UI element positioning method and related product

Abstract

The embodiment of the invention discloses a UI element positioning method and related products, and the method comprises the steps: carrying out the feature point matching of an obtained first local image and an obtained second local image, so as to screen feature point pairs meeting a preset matching precision requirement, and forming an effective feature matching set according to the feature point pairs, the to-be-processed first local image and the to-be-processed second local image are associated with the same target UI element; and calculating a spatial transformation matrix between the first local image and the second local image based on the effective feature matching set, so as to calculate a mapping calibration position of the target UI element in the second local image according to the spatial transformation matrix and the position of the target UI element in the first local image. According to the method, feature point matching is carried out on the two local images associated with the same target UI element, the effective feature matching set meeting the precision requirement is screened, and compared with traditional full-image template matching, local focusing is carried out, the matching range is narrowed, background interference is reduced, and matching pertinence is improved.

Description

Technical Field

[0001] This application relates to the field of user interface automation testing technology, and more specifically, to a method for locating UI elements and related products. Background Technology

[0002] In automated user interface (UI) testing, accurate positioning of UI elements is a core element to ensure smooth testing processes and reliable results, and is widely used in functional verification and compatibility testing of various software products. With the diversification of terminal device types, devices of different resolutions and sizes are emerging in large numbers, and the dynamic changes of UI interfaces are becoming increasingly frequent, placing higher demands on the adaptability and stability of UI element positioning technology.

[0003] In existing technologies, there is a UI element localization scheme based on image template matching. This scheme pre-creates a standard template image for each UI element to be located. During testing, the image to be detected is compared with the preset template image in a full-image traversal, and the position of the UI element is determined by calculating the similarity between the images. This scheme first requires manually collecting and creating a large number of template images and storing them in a template library; then, during localization, the corresponding template in the template library is called and compared pixel by pixel with the image to be detected; finally, a similarity threshold is used to determine whether a match is successful, thereby determining the position of the UI element.

[0004] However, this UI element localization scheme based on image template matching has significant technical flaws. It is extremely sensitive to subtle changes in UI elements (such as minor adjustments to color, brightness, and shape). If a UI element undergoes even a slight modification, the preset template image cannot accurately match, leading to localization failure. Furthermore, because a separate template needs to be created for each UI element, and the template image is strongly correlated with device resolution and size, a large number of templates need to be recreated for cross-device and cross-resolution scenarios. This not only increases the dependency cost of preset data but also severely reduces the efficiency and compatibility of automated testing, failing to meet the testing needs of diverse devices and dynamic UI interfaces. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a UI element positioning method and related products to at least alleviate these problems.

[0006] The technical solutions provided in this application are as follows: A UI element localization method based on feature matching includes: performing feature point matching on an acquired first local image and a second local image to filter feature point pairs that meet a preset matching accuracy requirement to form an effective feature matching set, wherein the first local image and the second local image to be processed are associated with the same target UI element; calculating a spatial transformation matrix between the first local image and the second local image based on the effective feature matching set, and calculating the mapping calibration position of the target UI element in the second local image according to the spatial transformation matrix and the position of the target UI element in the first local image.

[0007] Optionally, a first partial image is obtained by cropping a local area of ​​the first image according to a first preset ratio, using the position of the target UI element in the first image as a reference point.

[0008] Optionally, based on the cross-image mapping coefficient between the first image and the second image and the position of the target UI element in the first image, the mapping position of the target UI element in the second image is determined, wherein the cross-image mapping coefficient is the mapping coefficient of the spatial scale features between the first image and the second image; taking the mapping position of the target UI element in the second image as a reference point, a local area of ​​the second image is cropped according to a second preset ratio to obtain a second local image.

[0009] Optionally, the positional deviation between the mapping calibration position of the target UI element in the second local image and the mapping position of the target UI element in the second local image is calculated to obtain the mapping offset calibration parameter; based on the mapping offset calibration parameter and the mapping position of the target UI element in the second image, the mapping calibration position of the target UI element in the second image is determined.

[0010] Optionally, the mapping position of the target UI element in the second local image is obtained through the following steps: based on cropping the second image to obtain a second preset ratio of the second local image, the region boundary parameters of the second local image and the regional position reference standard of the second local image are determined; the position of the target UI element in the second image is converted into a relative position parameter based on the regional position reference standard of the second local image, thereby obtaining the mapping position of the target UI element in the second local image.

[0011] Optionally, taking the position of the target UI element in the first image as a reference point and cropping a local area of ​​the first image according to a first preset ratio to obtain a first local image includes: taking the position of the target UI element in the first image as a reference point and cropping the first image according to a first preset ratio to obtain an initial cropping area of ​​the first image; determining whether the edge of the initial cropping area of ​​the first image exceeds the size boundary threshold of the first image; if the edge of the initial cropping area of ​​the first image exceeds the size boundary threshold of the first image, then taking the position of the target UI element as a reference, performing a size correction on the initial cropping area of ​​the first image based on the size boundary constraint of the first image, so that each edge of the initial cropping area of ​​the first image does not exceed the size boundary threshold of the first image; if the edge of the initial cropping area of ​​the first image does not exceed the size boundary threshold of the first image, then taking the initial cropping area of ​​the first image as the first local image.

[0012] Optionally, using the mapping position of the target UI element in the second image as a reference point, a local area of ​​the second image is cropped according to a preset ratio to obtain a second local image. This includes: using the mapping position of the target UI element in the second image as a reference point, cropping the second image according to a second preset ratio to obtain an initial cropping area of ​​the second image; determining whether the edge of the initial cropping area of ​​the second image exceeds the size boundary threshold of the second image; if the edge of the initial cropping area of ​​the second image exceeds the size boundary threshold of the second image, then using the mapping position of the target UI element as a reference, performing size correction on the initial cropping area of ​​the second image based on the size boundary constraint of the second image, so that each edge of the initial cropping area of ​​the second image does not exceed the size boundary threshold of the second image; if the edge of the initial cropping area of ​​the second image does not exceed the size boundary threshold of the second image, then using the initial cropping area of ​​the second image as the second local image.

[0013] Optionally, determining the mapping position of the target UI element in the second image based on the cross-image mapping coefficient between the first image and the second image and the position of the target UI element in the first image includes: calculating the cross-image mapping coefficient between the first image and the second image based on the size parameters of the first image and the second image, wherein the cross-image mapping coefficient includes a horizontal mapping coefficient and a vertical mapping coefficient; performing mapping operations on the horizontal position component of the first image based on the horizontal mapping coefficient and on the vertical position component of the first image based on the vertical mapping coefficient, respectively, to determine the mapping position of the target UI element in the second image.

[0014] Optionally, feature point matching is performed on the acquired first and second local images to filter feature point pairs that meet preset matching accuracy requirements to form an effective feature matching set. This includes: performing feature enhancement preprocessing on the first and second local images to obtain preprocessed data for the first and second local images; extracting feature points from the preprocessed data for the first and second local images and obtaining feature point description information to form a first local image feature point description information set and a second local image feature point description information set; matching the first local image feature point description information set and the second local image feature point description information set according to preset criteria using a feature matcher to generate feature point pairs; and filtering the feature point pairs according to preset criteria to select an effective feature matching set that meets preset matching accuracy requirements.

[0015] Optionally, the feature point pairs are filtered according to preset criteria to select an effective feature matching set that meets preset matching accuracy requirements. This includes: determining a target screening ratio for the feature point pairs based on a preset mapping relationship between the size of the second local image and the screening ratio of the feature point pairs; and filtering the feature point pairs according to the target screening ratio to form an effective feature matching set that meets preset matching accuracy requirements.

[0016] Optionally, feature point extraction and feature point description information are performed on the first local image preprocessing data and the second local image preprocessing data to form a first local image feature point description information set and a second local image feature point description information set. This includes: performing feature point detection on the first local image preprocessing data and the second local image preprocessing data to extract feature point location information; synchronously triggering a feature point description information generation process based on the detected feature points to associate the detected feature points with the exclusive feature representation data of the detected feature points; and synchronously outputting the location information of the detected feature points and the exclusive feature representation data of the detected feature points to form a first local image feature point description information set and a second local image feature point description information set.

[0017] A UI element localization device based on feature matching includes: a feature point matching module, which performs feature point matching on an acquired first local image and a second local image to filter feature point pairs that meet a preset matching accuracy requirement to form an effective feature matching set, wherein the first local image and the second local image are associated with the same target UI element; and a mapping calibration position determination module for the target UI element in the second local image, which calculates a spatial transformation matrix between the first local image and the second local image based on the effective feature matching set, and calculates the mapping calibration position of the target UI element in the second local image according to the spatial transformation matrix and the position of the target UI element in the first local image.

[0018] An electronic device includes: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, causing the electronic device to perform a UI element location method based on feature matching as described in any one of the preceding claims.

[0019] A computer storage medium storing at least one instruction or at least one program, said at least one instruction or said at least one program being loaded and executed by a processor to implement a feature-matching-based UI element location method as described in any of the preceding claims.

[0020] A computer program product or computer program comprising computer instructions that, when executed by a processor, implement a UI element location method based on feature matching as described above.

[0021] The technical solution provided in this application has the following technical effects: This application first focuses on feature point matching between a first local image and a second local image associated with the same target UI element, filtering out an effective feature matching set that meets the accuracy requirements. Compared to traditional full-image template matching, the focusing characteristics of local images significantly narrow the matching range and reduce interference from irrelevant background information, making feature point matching more targeted. Simultaneously, by calculating the spatial transformation matrix through the effective feature matching set, the spatial mapping relationship between the two local images can be captured. Combined with the known position of the target UI element in the first local image, its mapping calibration position in the second local image is derived. This feature point-based spatial relationship modeling method avoids strong dependence on the overall appearance of the UI element (such as subtle changes in color, brightness, and shape). Attached Figure Description

[0022] Figure 1 A flowchart of a UI element positioning method provided in one embodiment of this application; Figure 2A block diagram of a UI element positioning device provided in one embodiment of this application; Figure 3 A structural block diagram of an electronic device provided in one embodiment of this application; Detailed Implementation like Figure 1 As shown in the figure, this application provides a UI element positioning method, including: Feature point matching is performed on the acquired first and second local images to filter feature point pairs that meet the preset matching accuracy requirements and form an effective feature matching set. The first and second local images to be processed are associated with the same target UI element. Based on the effective feature matching set, a spatial transformation matrix between the first and second local images is calculated. According to the spatial transformation matrix and the position of the target UI element in the first local image, the mapping calibration position of the target UI element in the second local image is calculated.

[0023] This application's solution extracts feature points from local images of target UI elements and constructs a spatial mapping relationship across resolutions. Unlike traditional positioning methods that rely on full-image matching or fixed parameters, this solution achieves accurate positioning of UI elements with low interference and high compatibility, reducing reliance on preset templates or coordinate data.

[0024] Specifically, in this application, for the implementation of feature point matching, the first step is to perform feature enhancement preprocessing on the first and second local images associated with the same target UI element. Both local images are converted to grayscale images, and the RGB three-channel pixel values ​​are fused using a weighted average method. The fusion weights are set according to the color characteristics of the UI elements in the learning machine (e.g., increasing the weight of the green channel for icon elements and distributing weights evenly for text elements), resulting in the first and second grayscale local images, eliminating the interference of color differences on feature extraction. Adaptive Gaussian filtering is then applied to the grayscale images, with the filter kernel size dynamically adjusted according to the estimated size of the target UI element in the local image (e.g., using a 3×3 kernel for smaller elements and a 5×5 kernel for larger elements), smoothing high-frequency noise to obtain the first and second denoised local images. Finally, contrast enhancement processing is performed on the denoised images, stretching the dynamic range of pixel grayscale values ​​to enhance the grayscale difference between the target UI element and the background, resulting in the first and second enhanced local images, providing a clearer image foundation for subsequent feature point extraction.

[0025] Specifically, the multi-scale feature point extraction algorithm designed in this application takes a first enhanced local image and a second enhanced local image as input. First, a multi-scale image pyramid is constructed, with each pyramid containing multiple scale layers (e.g., three to five scale layers). The image sizes of adjacent scale layers are scaled by a fixed ratio (e.g., a scaling ratio of 0.85). Each scale layer's image undergoes Gaussian blurring to reduce feature fluctuations caused by scale differences, resulting in a first multi-scale image set and a second multi-scale image set. In each scale layer's image, an improved corner detection operator is used to calculate the gray-level gradient magnitude and gradient direction of each pixel. The gradient magnitude is obtained through the difference operation of the pixel's neighborhood gray-level values, and the gradient direction is represented by a range of 0 to 360 degrees. Pixels with gradient magnitudes higher than a dynamic threshold (which is a fixed multiple of the average gradient magnitude of the current scale layer) are selected as candidate feature points, resulting in a first multi-scale candidate feature point set and a second multi-scale candidate feature point set. Non-maximum suppression is applied to the candidate feature points of each scale layer within a preset neighborhood range (e.g., 3×3). Within the pixel neighborhood, candidate feature points with the largest gradient magnitude are retained, and redundant adjacent points are removed to obtain the first multi-scale core feature point set and the second multi-scale core feature point set. These core feature points correspond to the stable structural positions such as the edges and corners of the target UI elements.

[0026] Specifically, in this application, feature descriptors are generated for the first and second multi-scale core feature point sets using a representation method based on local neighborhood gradient distribution. For each core feature point, a fixed-size neighborhood window (e.g., a 16×16 pixel window) is defined centered on the feature point, and the neighborhood window is divided into multiple uniform sub-regions (e.g., 4×4 sub-regions). The gradient direction and gradient magnitude of all pixels in each sub-region relative to the core feature point are calculated. The gradient direction of each sub-region is quantized (e.g., quantized into 8 direction intervals), and the sum of the gradient magnitudes in each direction interval is calculated to form a sub-region gradient statistical vector. The gradient statistical vectors of all sub-regions are concatenated in row-major order to obtain a fixed-length feature descriptor (e.g., a 256-dimensional vector). The value of each dimension corresponds to the cumulative magnitude of a specific gradient direction in a specific sub-region, ultimately forming the first and second feature descriptor subsets. Each feature descriptor subset is associated with a corresponding multi-scale core feature point set.

[0027] Specifically, the hierarchical feature matching mechanism designed in this application takes a first feature descriptor subset and a second feature descriptor subset as input. First, a brute-force matcher is used to calculate the Hamming distance between each feature descriptor in the first feature descriptor subset and all feature descriptors in the second feature descriptor subset. The Hamming distance measures the degree of difference between two binary descriptors; the smaller the distance value, the higher the matching degree of the feature points. The Hamming distances corresponding to each feature point are sorted in ascending order, and the two candidate matching feature points with the highest similarity are selected. A distance ratio threshold is set (e.g., between 0.7 and 0.8). If the ratio of the Hamming distance between the first and second candidate matching feature points is less than this threshold, the matching pair is retained as a preliminary matching feature point pair, and mismatched feature point pairs are eliminated, resulting in a preliminary matching feature point pair set. Spatial consistency is checked on the preliminary matching feature point pair set by calculating the horizontal and vertical coordinate offsets of all matching feature point pairs and statistically analyzing the offsets. The mean and standard deviation of the offsets are used to eliminate matching pairs whose offsets exceed the mean plus or minus twice the standard deviation, resulting in a spatially consistent matching feature point pair set. The filtering ratio is dynamically adjusted according to the size characteristics of the second local image. The size of the second local image is divided into three levels: small, medium, and large based on pixel area (e.g., small size is an area smaller than a preset small area threshold, medium size is an area between the small and medium area thresholds, and large size is an area larger than the medium area threshold). Small-sized images correspond to a higher filtering ratio, and large-sized images correspond to a lower filtering ratio (e.g., retaining the top 20% for small-sized images, the top 15% for medium-sized images, and the top 10% for large-sized images). Feature point pairs with the smallest Hamming distance are selected from the spatially consistent matching feature point pair set according to the filtering ratio. At the same time, a minimum threshold for the number of matching point pairs is set (e.g., 4). If the number of feature point pairs after filtering is lower than this threshold, the filtering ratio is gradually increased until the quantity requirement is met, ultimately forming an effective feature matching set.

[0028] Specifically, in this application, the calculation of the spatial transformation matrix is ​​based on the effective feature matching set. The spatial transformation matrix adopts a 3×3 homography matrix to describe the spatial mapping relationship of feature points between the first local image and the second local image. The rows and columns of the homography matrix correspond to the coordinate transformation dimensions. The first row corresponds to the scaling, rotation, and translation parameters of the horizontal coordinate, the second row corresponds to the scaling, rotation, and translation parameters of the vertical coordinate, and the third row is used for coordinate normalization. The element at each intersection position in the matrix is ​​the weight of a specific transformation parameter. For example, the element in the first row and first column represents the scaling factor of the horizontal coordinate, the element in the first row and third column represents the translation amount of the horizontal coordinate, the element in the second row and second column represents the scaling factor of the vertical coordinate, the element in the second row and third column represents the translation amount of the vertical coordinate, and the element in the third row and third column is fixed at 1 to achieve coordinate normalization. By matching the coordinates of feature points in the effective feature matching set, each feature point pair provides the coordinates (x1, y1) in the first local image and the corresponding coordinates (x2, y2) in the second local image. A system of linear equations is constructed, and the system of equations is solved by the least squares method to obtain the values ​​of each element of the homography matrix. This matrix can accurately reflect the spatial transformation law of feature points in the two local images.

[0029] Specifically, in this application, the mapping calibration position of the target UI element in the second local image is calculated based on the homography matrix and the local reference coordinates of the target UI element in the first local image. The local reference coordinates of the target UI element in the first local image are determined based on the pixel coordinate system of the first local image, with the origin at the upper left corner of the first local image, the x-axis pointing horizontally to the right, and the y-axis pointing vertically downward. These coordinates are obtained by converting the known coordinates of the target UI element in the first image (reference image) with the cropping parameters of the first local image. Substituting these local reference coordinates into the homography matrix, the initial mapping coordinates of the target UI element in the second local image are obtained through matrix multiplication. The average deviation between the initial mapping coordinates and the mapping coordinates of all feature points in the effective feature matching set is calculated. The average deviation is obtained by averaging the mapping deviations of all feature point pairs in the horizontal and vertical directions. This average deviation is used as the coordinate calibration value to compensate for the initial mapping coordinates, thereby obtaining the mapping calibration position of the target UI element in the second local image. This position can accurately reflect the actual distribution of the target UI element in the second local image, providing a precise basis for the subsequent global positioning of the target UI element in the second image (image to be located).

[0030] Optionally, a first partial image is obtained by cropping a local area of ​​the first image according to a first preset ratio, using the position of the target UI element in the first image as a reference point.

[0031] Specifically, regarding the determination of the reference coordinates of the target UI element, the initial coordinates (x_A, y_A) of the target UI element in the first image (such as the reference interface image of the learning machine) are first obtained. These coordinates are the center point coordinates of the target UI element, determined based on the pixel coordinate system of the first image, with the origin at the upper left corner of the first image. x_A represents the horizontal position, and y_A represents the vertical position. Their values ​​come from the preset reference element position library in the learning machine UI automated testing system. This position library is generated by element recognition and coordinate calibration of the reference interface to ensure that (x_A, y_A) can accurately correspond to the center position of the target UI element.

[0032] Specifically, in this application, for the dynamic configuration of the first preset ratio, a designed adaptive ratio algorithm is used to determine the first preset ratio based on the type of the target UI element and the resolution characteristics of the first image. The target UI element types include buttons, icons, text boxes, etc., and different types of elements have different size characteristics (e.g., icon elements are smaller, button elements are medium-sized). Combined with the resolution of the first image (e.g., elements in high-resolution images have a lower relative size proportion), the width and height ratios of the first preset ratio are set. Both the width and height ratios are within a preset reasonable range (e.g., between 15% and 25%), ensuring that the captured local area can completely contain the target UI element without introducing too much irrelevant background due to an excessively large range. For example, for icon-type target UI elements of a learning machine, if the first image resolution is high, the width and height of the first preset ratio are both set to 15% of the corresponding dimension of the first image; if it is a button-type element, the ratio is set to 20%.

[0033] Specifically, regarding the generation of the initial cropping region, the boundary coordinates of the initial cropping region are calculated based on the center point coordinates (x_A, y_A) of the target UI element and a first preset ratio. The initial cropping width is calculated according to the first preset width ratio, which is equal to the product of the width of the first image and its width ratio. The initial cropping height is calculated according to the first preset height ratio, which is equal to the product of the height of the first image and its height ratio. Using (x_A, y_A) as the center, the initial cropping width is extended to the left and right by half to obtain the left and right x-axis coordinates of the initial cropping region. The initial cropping height is extended upwards and downwards by half to obtain the upper and lower y-axis coordinates of the initial cropping region, ultimately forming the four boundary coordinates of the initial cropping region, resulting in the first initial local image.

[0034] Optionally, based on the cross-image mapping coefficient between the first image and the second image and the position of the target UI element in the first image, the mapping position of the target UI element in the second image is determined, wherein the cross-image mapping coefficient is the mapping coefficient of the spatial scale features between the first image and the second image; taking the mapping position of the target UI element in the second image as a reference point, a local area of ​​the second image is cropped according to a second preset ratio to obtain a second local image.

[0035] Specifically, for the calculation of cross-image mapping coefficients, the size parameters of the first image (e.g., the reference interface image of the learning machine) and the second image (e.g., the interface image to be located of the learning machine) are first obtained. The size parameters of the first image include the width w_A and the height h_A, where w_A is the total number of horizontal pixels and h_A is the total number of vertical pixels, both extracted from the attribute information of the first image; the size parameters of the second image include the width w_B and the height h_B, obtained in the same way as the first image. Cross-image mapping coefficients are calculated based on the size parameters of two images. These coefficients include a horizontal mapping coefficient k_x and a vertical mapping coefficient k_y. The horizontal mapping coefficient k_x is the ratio of the width w_B of the second image to the width w_A of the first image, representing the scale ratio between the two images in the horizontal direction. The vertical mapping coefficient k_y is the ratio of the height h_B of the second image to the height h_A of the first image, representing the scale ratio in the vertical direction. This yields a cross-image scale mapping set (k_x, k_y), which accurately reflects the spatial scale correspondence between interfaces of different resolutions in the learning machine.

[0036] Specifically, in this application, for the determination of the mapping position of the target UI element in the second image, the cross-image scale mapping set (k_x, k_y) and the reference coordinates (x_A, y_A) of the target UI element in the first image are used as inputs. The reference coordinates (x_A, y_A) of the target UI element in the first image are its center point coordinates, determined based on the pixel coordinate system of the first image, with the origin at the upper left corner of the first image. x_A represents the horizontal position, and y_A represents the vertical position, and their values ​​are derived from the reference element position library of the learning machine UI automated testing system. The reference coordinates (x_A, y_A) are scale-converted with the cross-image mapping coefficients. The horizontal position is converted to x_B = x_A × k_x, and the vertical position is converted to y_B = y_A × k_y, resulting in the mapping coordinates (x_B, y_B) of the target UI element in the second image. These coordinates are the preliminary center position of the target UI element in the second image, realizing scale adaptation between the reference image and the image to be located.

[0037] Specifically, in this application, for the dynamic configuration of the second preset ratio, a designed ratio adaptation algorithm is used to determine the second preset ratio by combining the target UI element type and the resolution characteristics of the second image. The target UI element types include buttons, icons, and functional controls, etc. Different types of elements have different size characteristics (e.g., icon elements are smaller, button elements are medium-sized). Simultaneously, the resolution of the second image is taken into account (elements occupy a smaller relative proportion in high-resolution images), and the width and height ratios of the second preset ratio are set. Both the width and height ratios are within a preset reasonable range (e.g., between 15% and 25%), ensuring that the captured local area can completely contain the target UI element while avoiding the introduction of too much irrelevant background. For example, for functional button elements of a learning machine, if the second image resolution is high, the width and height of the second preset ratio are both set to 15% of the corresponding dimension of the second image; if it is an icon element, the ratio is set to 20%.

[0038] Optionally, the positional deviation between the mapping calibration position of the target UI element in the second local image and the mapping position of the target UI element in the second local image is calculated to obtain the mapping offset calibration parameter; based on the mapping offset calibration parameter and the mapping position of the target UI element in the second image, the mapping calibration position of the target UI element in the second image is determined.

[0039] Specifically, regarding the determination of the mapping position of the target UI element in the second local image, the region parameters and position reference standard of the second local image are first obtained. The region parameters of the second local image include the width w_B' and the height h_B' of the second local image, both extracted from the attribute information of the second local image. This second local image is cropped from the second image (the learning machine's interface image to be located) based on a second preset ratio. The position reference standard is based on the pixel coordinate system of the second local image, with the origin at the upper left corner of the second local image, the positive x-axis pointing to the right horizontally, and the positive y-axis pointing downwards vertically. Combining the mapping coordinates (x_B, y_B) of the target UI element in the second image, which are calculated by the cross-image mapping coefficient between the first image (the learning machine's baseline interface image) and the second image, x_B represents the horizontal mapping position, and y_B represents the vertical mapping position. These coordinates are converted into relative coordinates (x_B_rel, y_B_rel) relative to the origin at the upper left corner of the second local image, which is the mapping position of the target UI element in the second local image. This position accurately reflects the initial distribution of the target UI element in the local image.

[0040] Specifically, in this application, the derivation of the mapping calibration position of the target UI element in the second local image uses a homography matrix and the reference coordinates (x_A', y_A') of the target UI element in the first local image as input. The homography matrix is ​​a 3×3 matrix calculated based on an effective feature matching set, used to describe the spatial transformation relationship of feature points between the first and second local images. The first row of the matrix corresponds to the scaling, rotation, and translation parameters of the horizontal coordinates, the second row corresponds to the relevant parameters of the vertical coordinates, and the third row is used for coordinate normalization. (x_A', y_A') are the center point coordinates of the target UI element in the first local image, determined based on the pixel coordinate system of the first local image. Substituting (x_A', y_A') into the homography matrix for transformation operations yields the mapping calibration coordinates (x_B'_cal, y_B'_cal) of the target UI element in the second local image. These coordinates are the precise position corrected by the spatial mapping relationship of feature points, which is closer to the actual distribution than the initial mapping position.

[0041] Specifically, in this application, for the calculation of the positional deviation, a designed deviation quantization algorithm is used, taking the mapped position (x_B_rel, y_B_rel) and the mapped calibration position (x_B'_cal, y_B'_cal) of the target UI element in the second local image as input. The lateral deviation delta_x is calculated, which is the difference between the x-axis coordinate x_B'_cal and the x-axis coordinate x_B_rel of the mapped calibration position; the longitudinal deviation delta_y is calculated, which is the difference between the y-axis coordinate y_B'_cal and the y-axis coordinate y_B_rel of the mapped calibration position. The lateral deviation delta_x and the longitudinal deviation delta_y are combined to obtain the mapped offset calibration parameters (delta_x, delta_y). These parameters quantify the deviation between the initial mapped position and the precise calibration position, providing a core basis for the position correction of the target UI element in the second image.

[0042] Specifically, in this application, the determination of the mapping calibration position of the target UI element in the second image uses the mapping offset calibration parameters (delta_x, delta_y) and the mapping coordinates (x_B, y_B) of the target UI element in the second image as input. The horizontal offset delta_x is superimposed on the x-axis coordinate x_B of the mapping coordinates to obtain the corrected horizontal coordinate x_B_cal = x_B + delta_x; the vertical offset delta_y is superimposed on the y-axis coordinate y_B of the mapping coordinates to obtain the corrected vertical coordinate y_B_cal = y_B + delta_y. Combining the corrected horizontal and vertical coordinates, the mapping calibration position (x_B_cal, y_B_cal) of the target UI element in the second image is obtained. This position is the final result after combining cross-image scale mapping and local feature calibration, and can accurately reflect the actual position of the target UI element in the second image.

[0043] Optionally, the mapping position of the target UI element in the second local image is obtained through the following steps: based on cropping the second image to obtain a second preset ratio of the second local image, the region boundary parameters of the second local image and the regional position reference standard of the second local image are determined; the position of the target UI element in the second image is converted into a relative position parameter based on the regional position reference standard of the second local image, thereby obtaining the mapping position of the target UI element in the second local image.

[0044] Specifically, regarding the analysis of the second preset ratio and the determination of the region boundary parameters, the specific value of the second preset ratio is first obtained. This ratio is the width and height ratio used when cropping the second local image. It is set based on the type of UI elements of the learning machine (such as buttons and icons) and the resolution characteristics of the second image (the interface image of the learning machine to be located). The width and height ratios are both within a preset reasonable range (for example, between 15% and 25%). Combining the size parameters of the second image (width w_B and height h_B), the theoretical size of the second local image is calculated. The theoretical width is the product of w_B and the width ratio, and the theoretical height is the product of h_B and the height ratio. Then, the mapping coordinates (x_B, y_B) of the target UI element in the second image are combined. These coordinates are obtained by converting the cross-image mapping coefficient between the first image (the learning machine's baseline interface image) and the second image. x_B represents the horizontal mapping position, and y_B represents the vertical mapping position. With (x_B, y_B) as the center, the initial boundary coordinates of the second local image are obtained by extending half of the theoretical size to the left and right, and up and down in both directions. (left boundary x1, right boundary x2, top boundary y1, bottom boundary y2) are obtained. The initial boundary coordinates are validated for compliance. If they exceed the size range of the second image, they are adjusted according to the boundary correction logic (e.g., if the left boundary is less than 0, it is fixed to 0; if the right boundary is greater than w_B, it is fixed to w_B). Finally, the actual boundary parameters of the second local image (x1_final, x2_final, y1_final, y2_final) are obtained. These parameters accurately define the spatial range of the second local image in the second image.

[0045] Specifically, in this application, regarding the establishment of the position reference standard within the second local image region, a dedicated pixel coordinate system is constructed as the position reference standard based on the actual boundary parameters of the second local image. The origin of this coordinate system is set as the upper left corner vertex of the second local image, i.e., the corresponding point in the second image coordinate system with coordinates (x1_final, y1_final); the positive direction of the x' axis is to the right horizontally, and the positive direction of the y' axis is downward vertically, with the units of the coordinate axes consistent with the pixel units of the second image. Simultaneously, the mapping relationship between this coordinate system and the original coordinate system of the second image is recorded, including the origin offset (x_offset = x1_final, y_offset = y1_final) and the scale scaling factor (since the second local image is a partial crop of the second image, the scale scaling factor is always 1), forming the core parameters of the position reference standard and providing a benchmark for subsequent coordinate transformations.

[0046] Specifically, in this application, the transformation from the position of the target UI element in the second image to its local relative position uses the core parameters of the target UI element's mapped coordinates (x_B, y_B) and position reference standard in the second image as input. Subtracting the origin's horizontal offset x_offset from x_B yields the target UI element's horizontal relative coordinates x_B_rel = x_B - x_offset in the second local image coordinate system; subtracting the origin's vertical offset y_offset from y_B yields the vertical relative coordinates y_B_rel = y_B - y_offset. Combining the horizontal and vertical relative coordinates yields the target UI element's mapped position (x_B_rel, y_B_rel) in the second local image. This position accurately reflects the relative distribution of the target UI element in the second local image and forms a logical closed loop with the boundary parameters and position reference standard of the second local image, providing a precise local position reference for subsequent feature point matching and offset calculation.

[0047] Optionally, taking the position of the target UI element in the first image as a reference point and cropping a local area of ​​the first image according to a first preset ratio to obtain a first local image includes: taking the position of the target UI element in the first image as a reference point and cropping the first image according to a first preset ratio to obtain an initial cropping area of ​​the first image; determining whether the edge of the initial cropping area of ​​the first image exceeds the size boundary threshold of the first image; if the edge of the initial cropping area of ​​the first image exceeds the size boundary threshold of the first image, then taking the position of the target UI element as a reference, performing a size correction on the initial cropping area of ​​the first image based on the size boundary constraint of the first image, so that each edge of the initial cropping area of ​​the first image does not exceed the size boundary threshold of the first image; if the edge of the initial cropping area of ​​the first image does not exceed the size boundary threshold of the first image, then taking the initial cropping area of ​​the first image as the first local image.

[0048] Specifically, in this application, regarding the confirmation of the reference position of the target UI element in the first image, the attribute parameters of the first image (the reference interface image of the learning machine) and the initial coordinates (x_A, y_A) of the target UI element are first obtained. The attribute parameters of the first image include the width w_A and the height h_A of the first image, both extracted from the metadata of the first image. w_A is the total number of horizontal pixels, and h_A is the total number of vertical pixels. (x_A, y_A) are the center point coordinates of the target UI element, determined based on the pixel coordinate system of the first image. The origin is the upper left corner of the first image, the horizontal direction to the right is the positive x-axis, and the vertical direction downwards is the positive y-axis. These coordinates are derived from the reference element position library preset by the learning machine UI automated testing system to ensure accurate correspondence with the center position of the target UI element (such as function buttons and learning icons).

[0049] Specifically, in this application, for the scenario-based configuration of the first preset ratio, a designed adaptive ratio algorithm is used to determine the ratio parameters by combining the target UI element type and the resolution characteristics of the first image. The target UI element types are divided into icon types, button types, text box types, etc., and the size proportions of different types of elements differ (icon elements are smaller, button elements are medium-sized); the resolution characteristics of the first image are quantized by w_A and h_A, with elements having a lower relative proportion in high-resolution images. Based on these characteristics, the width and height ratios of the first preset ratio are both set within a preset reasonable range (e.g., between 15% and 25%). For example, for icon elements of a learning machine, if the first image resolution is high, both the width and height ratios are configured to 15%; for button elements, the ratio is configured to 20%, ensuring that the cropped area can completely contain the target UI element while excluding irrelevant backgrounds to the greatest extent possible.

[0050] Specifically, in this application, for the generation of the initial cropping region of the first image, the center point coordinates (x_A, y_A) of the target UI element and a first preset ratio are used as inputs. The initial cropping width is calculated based on the first preset width ratio, and the initial cropping width is equal to the product of w_A and the width ratio; the initial cropping height is calculated based on the first preset height ratio, and the initial cropping height is equal to the product of h_A and the height ratio; with (x_A, y_A) as the center, half of the initial cropping width is extended to the left and right directions respectively to obtain the left boundary x-axis coordinates (x_left_init) and right boundary x-axis coordinates (x_right_init); half of the initial cropping height is extended to the up and down directions respectively to obtain the upper boundary y-axis coordinates (y_top_init) and lower boundary y-axis coordinates (y_bottom_init), which are combined to form the four boundary coordinates of the initial cropping region of the first image, thus obtaining the first initial cropping region. This region initially locks the core distribution range of the target UI element.

[0051] Specifically, in this application, for the boundary verification step of the first initial cropping region, a designed boundary compliance detection logic is adopted, using the four boundary coordinates of the first initial cropping region and the size boundary threshold of the first image as the processing basis. The size boundary threshold of the first image includes a left boundary threshold (0), a right boundary threshold (w_A), a top boundary threshold (0), and a bottom boundary threshold (h_A), which correspond to the four edge positions of the first image, respectively; it is determined one by one whether x_left_init is less than the left boundary threshold, whether x_right_init is greater than the right boundary threshold, whether y_top_init is less than the top boundary threshold, and whether y_bottom_init is greater than the bottom boundary threshold; based on the judgment results, a boundary violation label is generated. If any violation exists, it is labeled as "violation"; otherwise, it is labeled as "compliance", thus obtaining the boundary verification result. This result accurately reflects whether the initial cropping region exceeds the actual range of the first image.

[0052] Specifically, in this application, the dynamic correction of the initially captured area of ​​the violation is based on the boundary verification results and the center point coordinates (x_A, y_A) of the target UI element, and targeted correction is performed according to the direction of the violation. If `x_left_init` is less than 0 (left boundary violation), the corrected left boundary is fixed at 0, and the area is expanded to the right centered at (x_A, y_A). The corrected right boundary coordinates are 2×x_A, ensuring the target UI element remains centered within the region, thus obtaining the left boundary correction clipping area. If `x_right_init` is greater than `w_A` (right boundary violation), the corrected right boundary is fixed at `w_A`, and the area is expanded to the left to obtain the corrected left boundary coordinates of 2×w_A - x_A, forming the right boundary correction clipping area. If `y_top_init` is less than 0 (top boundary violation), the corrected top boundary is fixed at 0, and the area is expanded downwards to obtain the corrected bottom boundary coordinates of 2×y_A, forming the top boundary correction clipping area. If `y_bottom_init` is greater than `h_A` (bottom boundary violation), the corrected bottom boundary is fixed at `h_A`, and the area is expanded upwards to obtain the corrected top boundary coordinates of 2×h_A - y_A, forming the bottom boundary correction clipping area. If multiple boundary violations exist, the corrected boundary is determined by the following order: "Left-Right-Top-". The order of "down" is corrected sequentially to finally obtain a corrected intercepted area with compliant boundaries.

[0053] Specifically, in this application, for the final generation of the first partial image, the output content is determined based on the boundary verification result. If the boundary verification result is "compliant," the initial cropped area is directly used as the first partial image; if the boundary verification result is "non-compliant," the corrected cropped area with compliant boundaries is used as the first partial image. This first partial image centers on the target UI element, fully preserving the core feature areas of the element (such as button icons and text outlines), while avoiding interference from subsequent feature extraction caused by excessive background information. Compared to traditional full-image processing methods, the computational load is reduced to a preset ratio range, and it adapts to the cropping needs of different types of UI elements in learning machines, providing high-quality image data support for subsequent feature point extraction and matching.

[0054] Optionally, using the mapping position of the target UI element in the second image as a reference point, a local area of ​​the second image is cropped according to a preset ratio to obtain a second local image. This includes: using the mapping position of the target UI element in the second image as a reference point, cropping the second image according to a second preset ratio to obtain an initial cropping area of ​​the second image; determining whether the edge of the initial cropping area of ​​the second image exceeds the size boundary threshold of the second image; if the edge of the initial cropping area of ​​the second image exceeds the size boundary threshold of the second image, then using the mapping position of the target UI element as a reference, performing size correction on the initial cropping area of ​​the second image based on the size boundary constraint of the second image, so that each edge of the initial cropping area of ​​the second image does not exceed the size boundary threshold of the second image; if the edge of the initial cropping area of ​​the second image does not exceed the size boundary threshold of the second image, then using the initial cropping area of ​​the second image as the second local image.

[0055] Specifically, in this application, regarding the confirmation of the mapping position of the target UI element in the second image, the size parameters of the second image (the learning machine's interface image to be located) and the mapping coordinates (x_B, y_B) of the target UI element are first obtained. The size parameters of the second image include the width w_B and the height h_B of the second image, both extracted from the metadata of the second image. w_B is the total number of horizontal pixels, and h_B is the total number of vertical pixels. (x_B, y_B) are the center point mapping coordinates of the target UI element, calculated based on the cross-image mapping coefficient between the first image (the learning machine's baseline interface image) and the second image. x_B represents the horizontal mapping position, and y_B represents the vertical mapping position. These coordinates accurately reflect the initial distribution center of the target UI element in the second image.

[0056] Specifically, in this application, for the scenario-based configuration of the second preset ratio, a designed adaptive ratio algorithm is used to determine the ratio parameters by combining the target UI element type and the resolution characteristics of the second image. The target UI element types are divided into icons, buttons, and functional controls, etc., and the size proportions of different types of elements differ (icon elements are smaller, button elements are medium-sized); the resolution characteristics of the second image are quantized by w_B and h_B, with elements having a lower relative proportion in high-resolution images. Based on these characteristics, the width and height ratios of the second preset ratio are both set within a preset reasonable range (e.g., between 15% and 25%). For example, for icon elements of a learning machine, if the second image resolution is high, both the width and height ratios are configured to 15%; for button elements, the ratio is configured to 20%, ensuring that the cropped area can completely contain the target UI element while minimizing irrelevant background interference.

[0057] Specifically, in this application, for the generation of the initial cropping region of the second image, the mapping coordinates (x_B, y_B) of the target UI element and a second preset ratio are used as inputs. The initial cropping width is calculated based on the second preset width ratio, which is equal to the product of w_B and the width ratio. The initial cropping height is calculated based on the second preset height ratio, which is equal to the product of h_B and the height ratio. Using (x_B, y_B) as the center, the initial cropping width is extended to the left and right by half, respectively, to obtain the left and right boundary x-axis coordinates (x_left_init) and right boundary x-axis coordinates (x_right_init). The initial cropping height is extended upwards and downwards by half, respectively, to obtain the upper and lower boundary y-axis coordinates (y_top_init) and lower boundary y-axis coordinates (y_bottom_init). These coordinates are combined to form the four boundary coordinates of the initial cropping region of the second image, thus obtaining the second initial cropping region. This region initially defines the core distribution range of the target UI element in the second image.

[0058] Specifically, in this application, for the boundary verification step of the second initial cropping region, a designed boundary compliance detection logic is adopted, using the four boundary coordinates of the second initial cropping region and the size boundary threshold of the second image as the processing basis. The size boundary threshold of the second image includes a left boundary threshold (0), a right boundary threshold (w_B), a top boundary threshold (0), and a bottom boundary threshold (h_B), which correspond to the four edge positions of the second image, respectively; it is determined one by one whether x_left_init is less than the left boundary threshold, whether x_right_init is greater than the right boundary threshold, whether y_top_init is less than the top boundary threshold, and whether y_bottom_init is greater than the bottom boundary threshold; based on the judgment results, a boundary violation label is generated. If any violation exists, it is labeled as "violation"; otherwise, it is labeled as "compliance", thus obtaining the boundary verification result. This result accurately reflects whether the initial cropping region exceeds the actual range of the second image.

[0059] Specifically, in this application, the dynamic correction of the initially captured area of ​​the violation is based on the boundary verification results and the mapping coordinates (x_B, y_B) of the target UI element, and targeted correction is performed according to the direction of the violation. If x_left_init is less than 0 (left boundary violation), the corrected left boundary is fixed at 0, and the area is expanded to the right centered at (x_B, y_B). The corrected right boundary coordinates are 2×x_B, ensuring the target UI element remains centered in the region, thus obtaining the left boundary correction clipping area. If x_right_init is greater than w_B (right boundary violation), the corrected right boundary is fixed at w_B, and the area is expanded to the left to obtain the corrected left boundary coordinates of 2×w_B - x_B, forming the right boundary correction clipping area. If y_top_init is less than 0 (top boundary violation), the corrected top boundary is fixed at 0, and the area is expanded downwards to obtain the corrected bottom boundary coordinates of 2×y_B, thus obtaining the top boundary correction clipping area. If y_bottom_init is greater than h_B (bottom boundary violation), the corrected bottom boundary is fixed at h_B, and the area is expanded upwards to obtain the corrected top boundary coordinates of 2×h_B - y_B, forming the bottom boundary correction clipping area. If multiple boundary violations exist, the corrected boundary is determined by the following order: left-right-top- The order of "down" is corrected sequentially to finally obtain a corrected intercepted area with compliant boundaries.

[0060] Specifically, in this application, the output content is determined based on the boundary verification result during the final generation of the second local image. If the boundary verification result is "compliant," the initial cropped area is directly used as the second local image; if the boundary verification result is "non-compliant," the corrected cropped area with compliant boundaries is used as the second local image. This second local image, centered on the mapped coordinates of the target UI element, fully preserves the core feature areas of the element (such as button icons and control outlines), while avoiding interference from excessive background information in subsequent feature extraction. Compared to traditional full-image processing methods, the computational load is reduced to a preset proportion, and it adapts to the positioning needs of different resolution interfaces on learning machines, providing high-quality image data support for subsequent feature point extraction and matching.

[0061] Optionally, determining the mapping position of the target UI element in the second image based on the cross-image mapping coefficient between the first image and the second image and the position of the target UI element in the first image includes: calculating the cross-image mapping coefficient between the first image and the second image based on the size parameters of the first image and the second image, wherein the cross-image mapping coefficient includes a horizontal mapping coefficient and a vertical mapping coefficient; performing mapping operations on the horizontal position component of the first image based on the horizontal mapping coefficient and on the vertical position component of the first image based on the vertical mapping coefficient, respectively, to determine the mapping position of the target UI element in the second image.

[0062] Specifically, in this application, regarding the accurate acquisition of the size parameters of the first and second images, the core size information of the two images is first extracted. The size parameters of the first image (the learning machine's baseline interface image) include the width w_A and height h_A, where w_A is the total number of horizontal pixels and h_A is the total number of vertical pixels, both obtained by parsing from the metadata of the first image, which includes basic attributes such as image resolution and pixel density. The size parameters of the second image (the learning machine's interface image to be located) include the width w_B and height h_B, obtained in the same way as the first image. Simultaneously, the acquired size parameters are validated, and outliers (such as zero values ​​or values ​​exceeding a reasonable resolution range) are removed to ensure that the size parameters accurately reflect the actual scale of the image, resulting in a compliant set of size parameters (w_A, h_A, w_B, h_B).

[0063] Specifically, in this application, for the calculation and calibration of cross-image mapping coefficients, a designed coefficient adaptive algorithm is adopted, using a compliant size parameter set as input. The horizontal mapping coefficient k_x is calculated, its value being the ratio of the second image width w_B to the first image width w_A, used to characterize the scale scaling relationship between the two images in the horizontal direction; the vertical mapping coefficient k_y is calculated, its value being the ratio of the second image height h_B to the first image height h_A, used to characterize the scale scaling relationship in the vertical direction. Considering the slight stretching or compression deformation that may exist in different interfaces of the learning machine, a coefficient calibration factor is introduced. The calibration factor is corrected based on the size ratio of fixed reference elements (such as interface borders and fixed icons) in the two images. For example, if the width ratio of the reference element in the first image to the width ratio in the second image is a fixed value, then k_x is fine-tuned using this ratio. Similarly, k_y is calibrated to obtain the calibrated horizontal mapping coefficient k_x_cal and vertical mapping coefficient k_y_cal, forming a cross-image mapping coefficient set (k_x_cal, k_y_cal). This coefficient set can more accurately reflect the actual scale correspondence between the two images.

[0064] Specifically, in this application, for the confirmation of the reference position of the target UI element in the first image, the reference coordinates (x_A, y_A) of the target UI element are obtained. These coordinates are the center point coordinates of the target UI element, determined based on the pixel coordinate system of the first image, with the origin at the upper left corner of the first image, the positive x-axis pointing to the right horizontally, and the positive y-axis pointing downwards vertically. The values ​​of (x_A, y_A) are derived from the reference element position library preset by the learning machine UI automated testing system. This position library is generated by identifying and calibrating the coordinates of elements on the reference interface to ensure accurate correspondence with the center position of the target UI element (such as function buttons or learning icons), thus obtaining the reference position parameters (x_A, y_A) of the target UI element.

[0065] Specifically, in this application, the deduction of the mapping position of the target UI element in the second image uses a cross-image mapping coefficient set (k_x_cal, k_y_cal) and a reference position parameter (x_A, y_A) as input. The horizontal component x_A of the reference coordinate is multiplied by the calibrated horizontal mapping coefficient k_x_cal to obtain the horizontal mapping coordinate x_B = x_A × k_x_cal of the target UI element in the second image; the vertical component y_A of the reference coordinate is multiplied by the calibrated vertical mapping coefficient k_y_cal to obtain the vertical mapping coordinate y_B = y_A × k_y_cal. Combining the horizontal and vertical mapping coordinates yields the mapping position (x_B, y_B) of the target UI element in the second image. This position accurately reflects the initial distribution center of the target UI element in the second image, providing a core reference for the subsequent cropping of the second local image.

[0066] Specifically, in this application, for the consistency verification step of the mapping position, a designed position verification algorithm is adopted, taking the mapping position (x_B, y_B) and the size parameters (w_B, h_B) of the second image as input. It verifies whether x_B is within the range of 0 to w_B and whether y_B is within the range of 0 to h_B. If both are within the range, the mapping position is confirmed to be valid; if they are outside the range, the calculation and calibration process of the cross-image mapping coefficients is re-examined, the coefficient error is corrected, and the mapping operation is re-executed to ensure that the mapping position of the target UI element is always within a reasonable range of the second image, avoiding subsequent cropping failures due to scale adaptation deviations.

[0067] Optionally, feature point matching is performed on the acquired first and second local images to filter feature point pairs that meet preset matching accuracy requirements to form an effective feature matching set. This includes: performing feature enhancement preprocessing on the first and second local images to obtain preprocessed data for the first and second local images; extracting feature points from the preprocessed data for the first and second local images and obtaining feature point description information to form a first local image feature point description information set and a second local image feature point description information set; matching the first local image feature point description information set and the second local image feature point description information set according to preset criteria using a feature matcher to generate feature point pairs; and filtering the feature point pairs according to preset criteria to select an effective feature matching set that meets preset matching accuracy requirements.

[0068] Specifically, in this application, for the feature enhancement preprocessing stage, a multi-level preprocessing algorithm is adopted, with the first local image and the second local image (both associated with the same target UI element of the learning machine, such as function buttons and learning icons) as input. First, the two local images are converted to grayscale images. A channel weight dynamic allocation strategy is used to fuse the RGB three-channel pixel values. The weights are adjusted according to the color characteristics of the learning machine UI elements (icon elements have a high proportion of green channel, and text elements have a balanced proportion of channels), resulting in a first grayscale base image and a second grayscale base image, eliminating the interference of color fluctuations on feature extraction. Adaptive noise reduction processing is performed on the grayscale base images. The size of the Gaussian filter kernel is dynamically adjusted according to the estimated size of the target UI element in the local image (3×3 kernel is used when the element size is small, and 5×5 kernel is used when the element size is large), smoothing high-frequency noise while preserving feature details, resulting in a first denoised image and a second denoised image. Edge enhancement processing is performed on the denoised image. By calculating the gradient magnitude of the pixel neighborhood and stretching the dynamic range, the grayscale difference between the target UI element and the background is enhanced, resulting in the first local image preprocessing data and the second local image preprocessing data, laying a clear foundation for subsequent feature point extraction.

[0069] Specifically, in this application, for the feature point extraction and description information acquisition stage, a designed multi-scale feature extraction and representation algorithm is adopted, with the first local image preprocessing data and the second local image preprocessing data as input. First, a multi-level scale pyramid is constructed, with each pyramid containing multiple scaling levels (e.g., three to five levels). The image sizes of adjacent levels are scaled by a fixed ratio (e.g., a scaling ratio of 0.85). Each level undergoes Gaussian blurring to reduce feature distortion caused by scale differences, resulting in a first multi-scale image set and a second multi-scale image set. In the images at each scale level, the gradient direction, gradient magnitude, and curvature value of each pixel are calculated. The gradient direction is represented by a range of 0 to 360 degrees, the gradient magnitude is obtained through the difference operation of the pixel's neighborhood grayscale values, and the curvature value reflects the degree of contour curvature at the pixel's location. Pixels with gradient magnitudes higher than a dynamic threshold (a fixed multiple of the average gradient magnitude of the current level) and curvature values ​​within a reasonable range (e.g., between 0.1 and 0.9) are selected, resulting in a first multi-scale candidate feature point set and a second multi-scale candidate feature point set. Non-maximum suppression is applied to the candidate feature points at each scale level within a preset neighborhood range (e.g., 3×3). Within a pixel neighborhood, candidate feature points with the largest gradient magnitude are retained, and redundant adjacent points are removed to obtain the first core feature point set and the second core feature point set. For each core feature point, a fixed-size neighborhood window (e.g., a 16×16 pixel window) is defined centered on the feature point. The neighborhood window is divided into multiple uniform sub-regions (e.g., 4×4 sub-regions). The gradient direction and gradient magnitude of pixels in each sub-region are calculated. The gradient direction is quantized (e.g., quantized into 8 direction intervals). The sum of the gradient magnitudes in each direction interval is calculated to form a sub-region gradient statistical vector. The gradient statistical vectors of all sub-regions are concatenated in row-major order to obtain a fixed-length feature descriptor (e.g., a 256-dimensional vector). Finally, the first local image feature point description information set and the second local image feature point description information set are formed. Each description information set is associated with the location information of the core feature points and their dedicated feature representation data.

[0070] Specifically, in this application, for the feature point pair generation stage, a designed dual-standard matching algorithm is adopted, using the first local image feature point description information set and the second local image feature point description information set as input. A brute-force matcher is selected as the feature matcher. First, the Hamming distance between each feature descriptor in the first feature descriptor subset and all feature descriptors in the second feature descriptor subset is calculated. The Hamming distance measures the degree of difference between two binary descriptors; the smaller the distance value, the higher the matching degree of the feature points. Simultaneously, a normalized correlation coefficient is calculated to characterize the similarity between two feature descriptors; the closer the coefficient value is to 1, the higher the matching degree. A Hamming distance threshold (e.g., a fixed value within a preset reasonable range) and a correlation coefficient threshold (e.g., between 0.7 and 0.8) are set, and feature point pairs with a Hamming distance less than the distance threshold and a correlation coefficient greater than the correlation coefficient threshold are selected to obtain an initial feature point pair set. Duplicate matching is then performed on the initial feature point pair set to eliminate cases where the same feature point corresponds to multiple matching objects, resulting in the final feature point pairs, ensuring the uniqueness and reliability of the matching results.

[0071] Specifically, in this application, a dynamic filtering algorithm is used for the filtering of the effective feature matching set, with the feature point pairs and the size parameters of the second local image as input. First, the size parameters (width w_B' and height h_B') of the second local image are obtained. The pixel area is calculated based on these parameters, and the second local image is divided into three levels: small, medium, and large (e.g., small size is an area smaller than a preset small area threshold, medium size is an area between the small and medium area thresholds, and large size is an area larger than the medium area threshold). A corresponding target filtering ratio is set based on the image size level, with a higher filtering ratio for small images and a lower filtering ratio for large images (e.g., retaining the top 20% for small images, the top 15% for medium images, and the top 10% for large images). Feature point pairs with the smallest Hamming distance and the highest correlation coefficient are selected from the feature point pairs according to the filtering ratio. Simultaneously, a minimum matching point number threshold is set (e.g., 4). If the number of filtered feature point pairs is lower than this threshold, the filtering ratio is gradually increased until the quantity requirement is met, ultimately forming an effective feature matching set that meets the preset matching accuracy requirements. This set ensures matching accuracy while controlling computational load, adapting to the feature matching needs of different resolution interfaces on the learning machine.

[0072] Optionally, the feature point pairs are filtered according to preset criteria to select an effective feature matching set that meets preset matching accuracy requirements. This includes: determining a target screening ratio for the feature point pairs based on a preset mapping relationship between the size of the second local image and the screening ratio of the feature point pairs; and filtering the feature point pairs according to the target screening ratio to form an effective feature matching set that meets preset matching accuracy requirements.

[0073] Specifically, in this application, regarding the precise grading of the second local image size, the core size parameters of the second local image are first extracted. The second local image is a cropped image of the same target UI element (such as a function button or learning icon) associated with the learning machine. Its size parameters include the width w_B' and height h_B' of the second local image, both obtained from the metadata of the second local image. w_B' is the total number of horizontal pixels, and h_B' is the total number of vertical pixels. The pixel area S = w_B' × h_B' of the second local image is calculated. Based on the pixel area, the size is divided into three levels: small, medium, and large. The grading threshold is set based on the size distribution characteristics of common UI elements in the learning machine (for example, small size is S less than the preset small area threshold, medium size is S between the small and medium area thresholds, and large size is S greater than the medium area threshold). This yields the size level identifier of the second local image, which provides a basis for the dynamic matching of the selection ratio.

[0074] Specifically, in this application, for the construction of the preset mapping relationship and the determination of the target screening ratio, a designed ratio mapping algorithm is adopted, using the second local image size level identifier as input. The preset mapping relationship is a correspondence constructed based on the distribution law of feature points of UI elements in the learning machine, that is, different size levels of the second local image correspond to different feature point pair screening ratios. Small-sized images have fewer feature points in total, so a higher screening ratio is corresponding to ensure sufficient effective matching points; large-sized images have more feature points in total, and excessive feature points will increase the computational load and easily introduce interference, so a lower screening ratio is corresponding to them (for example, small size corresponds to the top 20% screening ratio, medium size corresponds to the top 15%, and large size corresponds to the top 10%). According to the second local image size level identifier, the corresponding screening ratio is matched from the preset mapping relationship to obtain the target screening ratio, which can balance matching accuracy and computational efficiency.

[0075] Specifically, in this application, the sorting and preliminary screening of feature point pairs takes the feature point pairs and the target screening ratio as inputs. Feature point pairs are generated by matching the feature point description information sets of the first and second local images using a feature matcher (such as a brute-force matcher). Each feature point pair contains a matching degree metric (such as Hamming distance). All feature point pairs are sorted in ascending order according to the matching degree metric (a smaller metric indicates a higher matching degree). Feature point pairs that represent the top 20% of the target screening ratio are then selected, resulting in a preliminary set of screened feature point pairs. For example, if the target screening ratio is the top 20%, then the top 20% of the sorted feature point pairs are retained, and the remaining low-matching feature point pairs are removed, achieving preliminary simplification of the feature point pairs.

[0076] Specifically, in this application, for the final verification and generation of the effective feature matching set, a designed matching quality verification algorithm is adopted, taking the initial screening of feature point pairs and the minimum matching point number threshold as inputs. The minimum matching point number threshold is set based on the homography matrix calculation requirements to ensure that there are enough feature point pairs to describe the spatial transformation relationship between two local images (e.g., the threshold is set to 4). The number of the initial screening feature point pairs is counted. If the number is greater than or equal to the minimum matching point number threshold, it is directly used as the effective feature matching set; if the number is less than the minimum matching point number threshold, the screening ratio is gradually increased according to a preset gradient (e.g., increased by 5% each time), and the sorting and truncation operations are re-executed until the number of screened feature point pairs meets the threshold requirement, thus obtaining the final effective feature matching set. This effective feature matching set ensures both matching accuracy and the reliability of subsequent spatial transformation matrix calculations, adapting to the positioning requirements of UI elements of different sizes in the learning machine.

[0077] Optionally, feature point extraction and feature point description information are performed on the first local image preprocessing data and the second local image preprocessing data to form a first local image feature point description information set and a second local image feature point description information set. This includes: performing feature point detection on the first local image preprocessing data and the second local image preprocessing data to extract feature point location information; synchronously triggering a feature point description information generation process based on the detected feature points to associate the detected feature points with the exclusive feature representation data of the detected feature points; and synchronously outputting the location information of the detected feature points and the exclusive feature representation data of the detected feature points to form a first local image feature point description information set and a second local image feature point description information set.

[0078] Specifically, in this application, for the feature point detection stage, a designed multi-scale corner enhancement algorithm is adopted, with the first local image preprocessing data and the second local image preprocessing data as input. Both preprocessing data are local images of the same target UI element (such as function buttons and learning icons) in the learning machine, and have been processed by grayscale conversion, noise reduction, and edge enhancement. A multi-level scale pyramid is constructed, with each pyramid containing multiple scaling levels (e.g., three to five levels). The image size of adjacent levels is scaled by a fixed ratio (e.g., a scaling ratio of 0.85). Each level undergoes Gaussian blurring to reduce feature distortion caused by scale differences, resulting in a first multi-scale dataset and a second multi-scale dataset. Within each scale level, the gradient direction, gradient magnitude, and curvature value of each pixel are calculated. The gradient direction is represented by a range of 0 to 360 degrees, the gradient magnitude is obtained through the difference operation of the pixel's neighborhood grayscale values, and the curvature value reflects the degree of contour curvature at the pixel's location. A dynamic threshold is set (this threshold is a fixed multiple of the average gradient magnitude of the current level), and pixels with gradient magnitudes higher than the threshold and curvature values ​​within a reasonable range (e.g., between 0.1 and 0.9) are selected, resulting in a first multi-scale candidate feature point set and a second multi-scale candidate feature point set. Non-maximum suppression is applied to the candidate feature points at each scale level within a preset neighborhood range (e.g., 3×3). Within a pixel neighborhood, candidate feature points with the largest gradient magnitude are retained, and redundant adjacent points are removed to obtain the first core feature point set and the second core feature point set. The coordinate information of the core feature points in the corresponding local image is extracted, that is, the feature point position information. The coordinates are determined based on the pixel coordinate system of the local image, with the origin at the upper left corner of the local image, the horizontal direction to the right being the positive x-axis, and the vertical direction downwards being the positive y-axis, forming the first feature point position information set and the second feature point position information set.

[0079] Specifically, in this application, the synchronous triggering step of the feature point description information generation process uses a first feature point location information set and a second feature point location information set as input. A unique feature point identifier is assigned to each core feature point, which is associated one-to-one with the feature point location information to ensure the traceability of the feature points. Based on the feature point identifier, a dedicated feature representation data generation process is triggered. For each core feature point, a fixed-size neighborhood window (e.g., a 16×16 pixel window) is defined centered on the feature point, and the neighborhood window is divided into multiple uniform sub-regions (e.g., 4×4 sub-regions). The gradient direction and gradient magnitude of all pixels within each sub-region relative to the core feature point are calculated. The gradient direction of each sub-region is quantized (e.g., quantized into 8 direction intervals). The sum of the gradient magnitudes and the variance of the gradient direction within each direction interval are calculated to form a sub-region gradient statistical vector. The gradient statistical vectors of all sub-regions are concatenated in row-major order to obtain a fixed-length feature descriptor (e.g., 256). Each dimension is a vector, and the value of each dimension corresponds to the sum of magnitudes or variance of directions of a specific gradient direction in a specific sub-region. This is the exclusive feature representation data of that feature point. The feature point identifier, feature point location information and exclusive feature representation data are associated and bound to form the first feature point association dataset and the second feature point association dataset, ensuring that the position of each feature point corresponds one-to-one with the feature representation.

[0080] Specifically, in this application, the synchronous output of the feature point description information set takes the first feature point association dataset and the second feature point association dataset as input. The association datasets are standardized in format, and the data storage structure is unified. Feature point location information (x-coordinate, y-coordinate) and dedicated feature representation data (feature descriptors) are arranged in a fixed field order. The standardized association data is validated, and data with abnormal feature descriptor dimensions or location information exceeding the local image range are removed to ensure the reliability of the output data. The validated standardized association data are integrated and encapsulated to form the first local image feature point description information set and the second local image feature point description information set. Each description information set contains multiple feature point entries, and each entry is associated with a unique feature point identifier, corresponding feature point location information, and dedicated feature representation data, providing complete and accurate feature data support for subsequent feature point matching and adapting to the feature matching needs of different types of UI elements in the learning machine.

[0081] like Figure 2Furthermore, this application embodiment also provides a UI element localization device based on feature matching, comprising: a feature point matching module, which performs feature point matching on the acquired first local image and second local image to filter feature point pairs that meet preset matching accuracy requirements to form an effective feature matching set, wherein the first local image and the second local image to be processed are associated with the same target UI element; and a mapping calibration position determination module for the target UI element in the second local image, which calculates a spatial transformation matrix between the first local image and the second local image based on the effective feature matching set, and calculates the mapping calibration position of the target UI element in the second local image according to the spatial transformation matrix and the position of the target UI element in the first local image.

[0082] Specifically, in this application, regarding the internal implementation of the feature point matching module, this module receives a first local image and a second local image of the same target UI element (such as a function button or learning icon) from the associated learning machine. It incorporates a multi-level preprocessing unit, a multi-scale feature extraction unit, a dual-standard matching unit, and a dynamic filtering unit. The multi-level preprocessing unit first converts the two local images into grayscale images, then uses a channel weight dynamic allocation strategy to fuse the RGB three-channel pixel values, adjusting the weights according to the color characteristics of the learning machine's UI element to obtain a grayscale base image. Next, it performs adaptive Gaussian filtering for noise reduction, adjusting the filter kernel size based on the estimated element size to obtain a denoised image. Finally, it performs gradient magnitude stretching to achieve edge enhancement, outputting the first and second local image preprocessing data. The multi-scale feature extraction unit uses the preprocessed data as input to construct a multi-level scale pyramid. At each scale level, it calculates the pixel gradient direction, magnitude, and curvature value, filters and suppresses redundant candidate feature points, generates a core feature point set, and then generates feature descriptors through neighborhood gradient statistics, forming the first and second local image feature point description information sets. The dual-standard matching unit calls a brute-force matcher to calculate the Hamming distance and normalized correlation coefficient of the feature descriptors, filters initial feature point pairs that meet the dual thresholds, and removes duplicate matches to obtain the final feature point pairs. The dynamic filtering unit divides the size levels according to the pixel area of ​​the second local image, matches the corresponding target filtering ratio, extracts high-matching feature point pairs according to the ratio, and combines the minimum matching point number threshold for verification to finally output the effective feature matching set.

[0083] Specifically, this application describes the internal implementation of a module for determining the mapping calibration position of a target UI element in the second local image. This module receives an effective feature matching set and the reference coordinates (x_A', y_A') of the target UI element in the first local image. It includes a built-in spatial transformation matrix calculation unit and a mapping calibration position deduction unit. The spatial transformation matrix calculation unit takes the effective feature matching set as input and extracts the coordinates (x1, y1) and (x2, y2) of each feature point pair in the first and second local images. x1 and y1 are based on the pixel coordinate system of the first local image (origin is the upper left corner, the horizontal direction to the right is the positive x-axis, and the vertical direction downward is the positive y-axis). x2 and y2 are based on the pixel coordinate system of the second local image (defined in the same way as the first local image). A system of linear equations is constructed using these coordinates, and a 3×3 homography matrix is ​​solved using the least squares method. The first row of the matrix corresponds to the scaling, rotation, and translation parameters of the horizontal coordinates, the second row corresponds to the relevant parameters of the vertical coordinates, and the third row is used for coordinate normalization. The matrix element values ​​accurately reflect the spatial transformation rules of the two local images, and the spatial transformation matrix H is output. The mapping calibration position deduction unit substitutes the reference coordinates (x_A', y_A') of the target UI element in the first local image into the spatial transformation matrix H, and obtains the initial mapping calibration coordinates (x_B'_cal, y_B'_cal) of the target UI element in the second local image through matrix multiplication. Then, it calculates the deviation compensation value between the initial mapping calibration coordinates and the mapping coordinates of the feature point pairs in the effective feature matching set. The horizontal deviation compensation value is the mean of all feature point pairs (x2 - x1 × horizontal scaling factor - horizontal translation amount), and the vertical deviation compensation value is the mean of all feature point pairs (y2 - y1 × vertical scaling factor - vertical translation amount). The deviation compensation value is superimposed on the initial mapping calibration coordinates to obtain the mapping calibration position (x_B'', y_B'') of the target UI element in the second local image. This position accurately reflects the actual distribution of the target UI element in the second local image.

[0084] Specifically, in this application, regarding the collaborative working mechanism between the two modules, the effective feature matching set output by the feature point matching module is directly used as the core input data for the mapping calibration position determination module of the target UI element in the second local image, without the need for additional data conversion or format adaptation, ensuring the efficiency of data flow. The reference coordinates (x_A', y_A') of the target UI element in the first local image are provided by the reference element position library of the learning machine UI automated testing system, and are passed to the mapping calibration position determination module through the preset data interface between the modules, participating in the spatial transformation matrix calculation and position inference in collaboration with the effective feature matching set. The internal algorithm parameters of both modules are preset based on scene data such as the UI element size distribution and resolution characteristics of the learning machine, and support dynamic adjustment according to the actual test scenario, ensuring that the modules can adapt to the positioning requirements of different types of UI elements and interfaces with different resolutions when working collaboratively, and have higher scene adaptability and positioning reliability than traditional module architectures.

[0085] like Figure 3 Furthermore, this application embodiment also provides an electronic device, the electronic device comprising: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, causing the electronic device to execute a UI element positioning method based on feature matching as described in any of the preceding claims.

[0086] The electronic devices include, but are not limited to, laptops, smartphones, tablets, smartwatches, etc.

[0087] This application also provides a computer storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement a UI element positioning method based on feature matching as described in any of the preceding claims.

[0088] This application also provides a computer program product or computer program, which includes computer instructions that, when executed by a processor, implement a UI element location method based on feature matching as described in any of the preceding claims.

[0089] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for locating UI elements, characterized in that, include: Feature point matching is performed on the acquired first local image and second local image to filter feature point pairs that meet the preset matching accuracy requirements, thereby forming an effective feature matching set. The first local image and second local image to be processed are associated with the same target UI element. Based on the effective feature matching set, a spatial transformation matrix is ​​calculated between the first local image and the second local image, so as to calculate the mapping calibration position of the target UI element in the second local image according to the spatial transformation matrix and the position of the target UI element in the first local image.

2. The UI element positioning method according to claim 1, characterized in that, Also includes: Using the position of the target UI element in the first image as a reference point, a local area of ​​the first image is cropped according to a first preset ratio to obtain a first local image.

3. The UI element positioning method according to claim 1, characterized in that, Also includes: Based on the cross-image mapping coefficient between the first image and the second image and the position of the target UI element in the first image, the mapping position of the target UI element in the second image is determined. The cross-image mapping coefficient is the mapping coefficient of the spatial scale features between the first image and the second image. Using the mapping position of the target UI element in the second image as a reference point, a local area of ​​the second image is cropped according to a second preset ratio to obtain a second local image.

4. The UI element positioning method according to claim 3, characterized in that, Also includes: The positional deviation between the target UI element's mapping calibration position in the second local image and its mapping position in the second local image is calculated to obtain the mapping offset calibration parameter; based on the mapping offset calibration parameter and the target UI element's mapping position in the second image, the mapping calibration position of the target UI element in the second image is determined.

5. A UI element positioning method according to claim 4, characterized in that, The mapping position of the target UI element in the second local image is obtained through the following steps: Based on cropping the second image to obtain a second preset ratio for the second local image, the region boundary parameters of the second local image and the position reference standard within the region of the second local image are determined; The position of the target UI element in the second image is converted into a relative position parameter based on the position reference standard within the region of the second local image, thereby obtaining the mapped position of the target UI element in the second local image.

6. A UI element positioning method according to claim 2, characterized in that, Using the position of the target UI element in the first image as a reference point, a first partial image is obtained by cropping a local area of ​​the first image according to a first preset ratio, including: Using the position of the target UI element in the first image as a reference point, the first image is cropped according to a first preset ratio to obtain the initial cropping area of ​​the first image; Determine whether the edge of the initial cropped region of the first image exceeds the size boundary threshold of the first image; If the edge of the initial cropping region of the first image exceeds the size boundary threshold of the first image, then based on the position of the target UI element, a size correction based on the size boundary constraint of the first image is performed on the initial cropping region of the first image so that none of the edges of the initial cropping region of the first image exceed the size boundary threshold of the first image. If the edge of the initial cropped region of the first image does not exceed the size boundary threshold of the first image, then the initial cropped region of the first image is taken as the first local image.

7. A UI element positioning method according to claim 2, wherein a local area of ​​the second image is cropped according to a preset ratio using the mapping position of the target UI element in the second image as a reference point to obtain a second local image, comprising: Using the mapping position of the target UI element in the second image as a reference point, the second image is cropped according to a second preset ratio to obtain the initial cropping area of ​​the second image; Determine whether the edge of the initial cropped region of the second image exceeds the size boundary threshold of the second image; If the edge of the initial cropping region of the second image exceeds the size boundary threshold of the second image, then based on the mapping position of the target UI element, a size correction based on the size boundary constraint of the second image is performed on the initial cropping region of the second image so that none of the edges of the initial cropping region of the second image exceed the size boundary threshold of the second image. If the edge of the initial cropped region of the second image does not exceed the size boundary threshold of the second image, then the initial cropped region of the second image is taken as the second local image.

8. The UI element positioning method according to claim 1, characterized in that, Based on the cross-image mapping coefficients between the first and second images and the position of the target UI element in the first image, the mapping position of the target UI element in the second image is determined, including: Based on the size parameters of the first image and the second image, a cross-image mapping coefficient is calculated between the first image and the second image. The cross-image mapping coefficient includes a horizontal mapping coefficient and a vertical mapping coefficient. Based on the horizontal mapping coefficient, the horizontal position component of the first image is mapped, and the vertical mapping coefficient is mapped to the vertical position component of the first image, respectively, to determine the mapping position of the target UI element in the second image.

9. A UI element positioning method according to claim 1, characterized in that, Feature point matching is performed on the acquired first and second local images to filter feature point pairs that meet the preset matching accuracy requirements to form an effective feature matching set, including: The first local image and the second local image are subjected to feature enhancement preprocessing to obtain the first local image preprocessing data and the second local image preprocessing data. Feature points are extracted from the first local image preprocessing data and the second local image preprocessing data, and feature point description information is obtained to form a first local image feature point description information set and a second local image feature point description information set. The feature matcher matches the first local image feature point description information set and the second local image feature point description information set according to a preset standard to generate feature point pairs. The feature point pairs are filtered according to preset criteria to select an effective feature matching set that meets the preset matching accuracy requirements.

10. A UI element positioning method according to claim 9, characterized in that, The feature point pairs are filtered according to preset criteria to select a valid feature matching set that meets preset matching accuracy requirements, including: Based on the preset mapping relationship between the size of the second local image and the screening ratio of the feature point pairs, the target screening ratio of the feature point pairs is determined. The feature point pairs are filtered according to the target filtering ratio to form an effective feature matching set that meets the preset matching accuracy requirements.

11. A UI element positioning method according to claim 9, characterized in that, It also includes extracting feature points from the first local image preprocessing data and the second local image preprocessing data and obtaining feature point description information to form a first local image feature point description information set and a second local image feature point description information set, including: Feature point detection is performed on the first local image preprocessing data and the second local image preprocessing data to extract feature point location information; Based on the detected feature points, a feature point description information generation process is synchronously triggered to associate the detected feature points with the exclusive feature representation data of the detected feature points; The location information of the detected feature points and the unique feature representation data of the detected feature points are synchronously output to form a first local image feature point description information set and a second local image feature point description information set.

12. A UI element positioning device, characterized in that, include: The feature point matching module performs feature point matching on the acquired first local image and second local image to filter feature point pairs that meet the preset matching accuracy requirements to form an effective feature matching set. The first local image and the second local image to be processed are associated with the same target UI element. The target UI element mapping calibration position determination module in the second local image calculates the spatial transformation matrix between the first local image and the second local image based on the effective feature matching set, so as to calculate the mapping calibration position of the target UI element in the second local image according to the spatial transformation matrix and the position of the target UI element in the first local image.

13. An electronic device, characterized in that, The electronic device includes: a memory for storing executable program code; and a processor for calling and running the executable program code from the memory, causing the electronic device to perform a UI element positioning method as described in any one of claims 1 to 11.

14. A computer storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement a UI element positioning method as described in any one of claims 1-11.

15. A computer program product or computer program, characterized in that, The computer program product or computer program includes computer instructions that, when executed by a processor, implement a UI element positioning method as described in any one of claims 1-11.

Images