Method for rapid detection of dynamic areas in mobile application pages

Abstract

A dynamic area detection method based on computer vision, comprising: first, opening the corresponding software, traversing the App page and obtaining screenshots; second, after preprocessing the adjacent two screenshots, extracting, screening and matching the feature points to obtain the single mapping matrix of the two pictures. According to the single mapping matrix, it is judged whether the two screenshots have offset caused by page sliding. If there is no offset, the dynamic area is directly detected according to the maximum range of successful matching of the feature points. If there is offset, the position of the App page navigation bar is detected first, and then the detection range of the dynamic area is determined. After determining the detection range, the Euclidean distance is calculated by using the feature vectors of the corresponding feature points of the two pictures according to the fixed sampling interval. If it exceeds a certain threshold, it is determined as the detection point. Through the above steps, whether the current page of the App has a dynamic content area can be detected according to the adjacent two screenshots.

Description

Technical fields:

[0001] This invention relates to a dynamic region detection algorithm based on computer vision, belonging to the field of computer technology software. Background technology:

[0002] The number of mobile applications has exploded with the development of the mobile internet, and software design has become increasingly complex. As a result, the demand for automated testing, page structure decomposition, and human-computer interaction analysis of mobile applications has also increased. In the analysis of page content, dynamic areas such as carousels, pop-ups, and videos need to be identified and processed separately.

[0003] In actual page detection and traversal, page scrolling will inevitably occur. In this case, simply comparing the corresponding pixels in two adjacent screenshots will classify the entire page as dynamic content, thus rendering the method ineffective. Existing image dynamic region detection algorithms based on monograph transformation use some highly complex algorithms, but the situation for mobile application pages is relatively simple, and unnecessary computational resources and time should not be wasted on redundant calculations. Summary of the Invention:

[0004] To address the above issues, this invention proposes a fast method for detecting dynamic regions in mobile application pages. Compared to methods based on comparing pixels at the same location, this method can adapt to situations where two screenshots are scrolled, offering greater versatility. Furthermore, it can calculate the transformations between the two images, such as the pixel distance during upward / downward scrolling. This value can also be provided to downstream tasks. Compared to existing dynamic region detection algorithms, this method significantly reduces the complexity of the dynamic region localization part and improves detection speed.

[0005] The specific steps of a fast detection method for dynamic areas in mobile application pages are as follows:

[0006] S1: Open the corresponding App, iterate through the App's pages and take screenshots;

[0007] S2: For the two adjacent screenshots obtained in S1, extract the image feature points of each, and select the point pairs with high matching degree to calculate the single mapping matrix between the two images.

[0008] S3: Based on the single mapping matrix obtained in S2, determine whether there is page scrolling between the two images. If so, detect the position of the navigation bar corresponding to the scrolling direction to determine the dynamic region detection range of the two images. If there is no scrolling, directly use the maximum range of matching feature points as the detection region.

[0009] S4: Within the detection range determined in S3, two images are sampled at fixed intervals, and SIFT features are extracted from the sampled points. The image of the current frame is transformed using the monomorphism matrix obtained in S2. Then, the Euclidean distance is calculated for the features of the corresponding sampled points in the two images. If the distance is less than the set threshold, the two points are judged to be similar, that is, the point is a non-dynamic region. If the distance is greater than the set threshold, the point is judged to be a dynamic region. Finally, a dot matrix composed of sampled points is obtained.

[0010] S5: Since the point matrix obtained by S4 is sparse on the original image, which is not conducive to the localization of the region outline, it is necessary to construct a binary image that is scaled down according to the distribution of the point matrix so that the sampling points are closely spaced in this binary image.

[0011] S6: In the binary image obtained in S5, use OpenCV functions to locate the contours and find the minimum bounding rectangle corresponding to the contours. Filter out contours whose area is greater than the set threshold. Then, restore the position of the minimum bounding rectangle of the contour to the original image according to the previously reduced ratio to obtain the detection results on the current page.

[0012] Specifically, in step S2, the specific algorithm for image feature point extraction and matching is the Scale-Invariant Feature Transform (SIFT) matching algorithm. This algorithm is a computer vision algorithm used to detect and describe local features in images. It finds extreme points in the scale space (the LoG pyramid approximated by the DoG subtraction of Gaussian images) and extracts their position, scale, and rotation invariants.

[0013] Specifically, in step S2, the specific scheme for calculating the distance between image feature points is as follows: calculate the Euclidean distance between the feature point vectors.

[0014] Specifically, in step S2, the method for filtering feature points is as follows: find the two points that best match each feature point in another image. If the nearest neighbor distance divided by the second nearest neighbor distance is less than a certain threshold, then the feature points are filtered; otherwise, the pair of feature points are deleted.

[0015] Specifically, in step S2, the monomorphism matrix is ​​a third-order matrix:

[0016]

[0017] X1 = H*X2, using four point pairs and eight equations to find the monomorphism matrix H.

[0018] Specifically, in step S3, the method for determining whether the page is sliding is as follows: based on the monomorphism matrix H obtained in S2, the H matrix represents the rotation and translation transformations required to transform the first image into the second image. H[1][2] corresponds to the translation distance on the y-axis, so this value is directly used to determine whether sliding is present. If the absolute value of this value is greater than 1, it is considered that sliding exists; if it is greater than 0 and less than 1, it is considered that the two images are not sliding up and down.

[0019] Specifically, in step S3, without sliding, the detection range is determined as follows: the maximum range of the sieved feature points in the figure in S2 is directly used as the detection range.

[0020] Specifically, in step S3, when there is sliding, the sliding direction is determined as follows: According to the H matrix obtained from S2, if H[1][2]>0, the current frame slides downward compared to the previous frame; if H[1][2]<0, the current frame slides upward compared to the previous frame.

[0021] Specifically, in step S3, when there is sliding, the detection range is determined as follows: Based on the sliding direction obtained in S3, if the current frame is sliding downwards relative to the previous frame, the upper half of the two images is sampled at fixed intervals, and the SIFT features of the sampled points are extracted. After extraction, the Euclidean distance between corresponding feature points in the two images is calculated pairwise. If it is greater than a set threshold, it is judged as unmatched; if it is less than the set threshold, it is judged as a successful match. Then, each row of sampled points is traversed from bottom to top. If the number of successfully matched points in a certain row accounts for more than 60% of the total, then this row is considered to be the end position of the upper navigation bar. Then, the end position of the navigation bar is used as the upper limit of the detection range, and the lower limit is the area below the maximum range of filtered feature points in the image in S2. Finally, the detection range is confirmed. The processing method is similar when the current frame is sliding downwards relative to the previous frame, except that the lower limit is found at the lower navigation bar position, and the upper limit is the area above the maximum range of filtered feature points in S2.

[0022] Specifically, in step S4, the transformation of the image using the monochromatic matrix is ​​as follows:

[0023]

[0024] In the above formula, the function src(x,y) refers to the gray level at pixel (x,y) in the image. The transformed image is represented as Dst(x,y), where x and y are the horizontal and vertical coordinates of the pixel in the image, respectively.

[0025] Specifically, in step S4, the detection algorithm works as follows: within the detection range, the previous image is sampled at fixed intervals, and the coordinates of the sampling points in the subsequent image are obtained by multiplying the coordinates of the sampling points in the previous image by the monomorphism matrix. Then, SIFT features are extracted from the sampling points in both images, and the Euclidean distance is calculated for the features of the sampling points at corresponding positions. If the distance is less than a set threshold, the two points are judged to be similar, that is, the point is a non-dynamic region. If the distance is greater than the set threshold, the point is judged to be a dynamic region. Finally, a dot matrix composed of sampling points is obtained.

[0026] Specifically, in step S5, the method for constructing the binary image is as follows: the width of the binary image is the width of the image divided by the sampling interval; the height of the binary image is the height of the detection range divided by the sampling interval.

[0027] Specifically, in step S6, the functions called are: findContours() to find the contour; and boundingRect() to find the minimum bounding rectangle.

[0028] Specifically, in step S6, the method to restore the minimum bounding rectangle of the contour to the original image size is to multiply the coordinates of the upper left corner and the length and width of the minimum bounding rectangle by the sampling interval.

[0029] In summary, this invention provides a rapid detection method for dynamic areas in mobile application pages, which has the following advantages: (1) It can be applied to complex scenarios, such as those with pop-ups or page scrolling. (2) It has universality and is applicable to different device models. Attached image description:

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of the overall process for a rapid detection method of dynamic areas in mobile application pages provided by the present invention.

[0032] Figure 2 This is a flowchart illustrating the determination of the detection range in the overall flowchart of the rapid detection method for dynamic areas in mobile application pages provided by this invention.

[0033] Figure 3 This is an example of a sampling point matrix in the overall flowchart of the fast detection method for dynamic areas in mobile application pages provided by the present invention. Specific implementation methods

[0034] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0035] This example uses a specific app as an example, and the method includes the following specific steps:

[0036] S1: Open the corresponding App, iterate through the App's pages and take screenshots;

[0037] S2: For the two adjacent screenshots obtained in S1, extract the image feature points of each, and select the point pairs with high matching degree to calculate the single mapping matrix between the two images.

[0038] S3: Based on the single mapping matrix obtained in S2, determine whether there is page scrolling between the two images. If so, detect the position of the navigation bar corresponding to the scrolling direction to determine the dynamic region detection range of the two images. If there is no scrolling, directly use the maximum range of matching feature points as the detection region.

[0039] S4: Within the detection range determined in S3, two images are sampled at fixed intervals, and SIFT features are extracted from the sampled points. The image of the current frame is transformed using the monomorphism matrix obtained in S12. Then, the Euclidean distance is calculated for the features of the corresponding sampled points in the two images. If the distance is less than the set threshold, the two points are judged to be similar, that is, the point is a non-dynamic region. If the distance is greater than the set threshold, the point is judged to be a dynamic region. Finally, a dot matrix composed of sampled points is obtained.

[0040] S5: Since the point matrix obtained by S4 is sparse on the original image, which is not conducive to the localization of the region outline, it is necessary to construct a binary image that is scaled down according to the distribution of the point matrix so that the sampling points are closely spaced in this binary image.

[0041] S6: In the binary image obtained in S5, use OpenCV functions to locate the contours and find the minimum bounding rectangle corresponding to the contours. Filter out contours whose area is greater than a set threshold. Then, restore the position of the minimum bounding rectangle of the contour to the original image according to the previously reduced ratio to obtain the detection results on the current page.

Claims

1. A method for rapid detection of dynamic areas in mobile application pages, comprising the following steps: S1: Open the corresponding App, iterate through the App's pages and take screenshots; S2: For the two adjacent screenshots obtained in S1, extract the image feature points of each, and select the point pairs with high matching degree to calculate the homograph matrix between the two images. S3: Based on the monomorphism matrix obtained in S2, determine whether there is page scrolling between the two images. If so, detect the position of the navigation bar corresponding to the scrolling direction to determine the dynamic region detection range of the two images. If there is no scrolling, directly use the maximum range of matching feature points as the detection region. S4: Within the detection range determined in S3, two images are sampled at fixed intervals, and SIFT features are extracted from the sampled points. The image of the current frame is transformed using the monomorphism matrix obtained in S2. Then, the Euclidean distance is calculated for the features of the corresponding sampled points in the two images. If the distance is less than the set threshold, the two points are judged to be similar, that is, the point is a non-dynamic region. If the distance is greater than the set threshold, the point is judged to be a dynamic region. Finally, a dot matrix composed of sampled points is obtained. S5: Since the point matrix obtained by S4 is sparse on the original image, which is not conducive to the localization of the region outline, it is necessary to construct a binary image that is scaled down according to the distribution of the point matrix so that the sampling points are closely spaced in this binary image. S6: In the binary image obtained in S5, use OpenCV functions to locate the contours and find the minimum bounding rectangle corresponding to the contours. Filter out contours whose area is greater than a set threshold. Then, restore the position of the minimum bounding rectangle of the contour to the original image according to the previously reduced ratio to obtain the detection results on the current page.

2. The method for rapid detection of dynamic regions in a mobile application page according to claim 1, characterized in that: In step S2, the specific algorithm for image feature point extraction and matching is as follows: Scale-invariant feature transformation matching algorithm; In step S2, the specific scheme for calculating the distance between image feature points is as follows: calculate the Euclidean distance between the feature point vectors; In step S2, the specific method for filtering feature points is to find the two points that best match each feature point in another image. If the nearest neighbor distance divided by the second nearest neighbor distance is less than a certain threshold, then the feature points are filtered; otherwise, the pair of feature points are deleted. In step S2, the monomorphism matrix is ​​a third-order matrix: H= X1=H X2 uses four point pairs and eight equations to find the monomorphism matrix H.

3. The method for rapid detection of dynamic regions in a mobile application page according to claim 1, characterized in that: In step S3, the method for determining whether the page is sliding is as follows: based on the monochromatic matrix H obtained in S2, the H matrix represents the rotation and translation transformations required to transform the first image into the second image; while H[1][2] corresponds to the translation distance on the y-axis, so this value is used directly to determine whether sliding is present. If the absolute value of this value is greater than 1, it is considered that sliding exists; if it is greater than 0 and less than 1, it is considered that the two images are not sliding up and down.

4. The method for rapid detection of dynamic regions in a mobile application page according to claim 1, characterized in that: In step S3, without sliding, the detection range is determined as follows: the maximum range of the sieved feature points in the figure in S2 is directly used as the detection range; In step S3, when there is sliding, the sliding direction is determined as follows: According to the H matrix obtained from S2, if H[1][2]>0, the current frame slides downward compared to the previous frame; if H[1][2]<0, the current frame slides upward compared to the previous frame. In the case of sliding, the detection range is determined as follows: Based on the sliding direction obtained in S3, if the current frame is sliding downwards relative to the previous frame, the upper half of the two images are sampled at a fixed interval, and the SIFT features of the sampling points are extracted. After extraction, the Euclidean distance is calculated pairwise for corresponding feature points in the two images. If the distance is greater than a set threshold, it is considered a non-match; otherwise, it is considered a successful match. Then, each row of sampling points is traversed from bottom to top. If the number of successfully matched points in a row exceeds 60% of the total, that row is considered the end position of the top navigation bar. The end position of the navigation bar is then used as the upper limit of the detection range, and the area below the maximum range of filtered feature points in image S2 is used as the lower limit. Finally, the detection range is confirmed. The handling method is similar when the current frame is sliding downwards relative to the previous frame, except that the lower limit is found at the bottom navigation bar position, and the upper limit is found at the area above the maximum range of filtered feature points in image S2.

5. The method for rapid detection of dynamic regions in a mobile application page according to claim 1, characterized in that: In step S4, the specific content of transforming the image using the monochromatic matrix is ​​as follows: In the above formula, the function src(x,y) refers to the gray level at pixel (x,y) in the image. The transformed image is represented as Dst(x,y), where x and y are the horizontal and vertical coordinates of the pixel in the image, respectively. In step S4, the specific content of the detection algorithm is as follows: within the detection range, the previous image is sampled at fixed intervals, and the coordinates of the sampling points in the subsequent image are obtained by multiplying the coordinates of the sampling points in the previous image by the monomorphism matrix. Then, SIFT features are extracted from the sampling points in the two images, and the Euclidean distance is calculated for the features of the sampling points at corresponding positions. If the distance is less than a set threshold, the two points are judged to be similar, that is, the point is a non-dynamic region. If the distance is greater than the set threshold, the point is judged to be a dynamic region. Finally, a dot matrix composed of sampling points is obtained.

6. The method for rapid detection of dynamic regions in a mobile application page according to claim 1, characterized in that: In step S5, the specific method for constructing the binary image is as follows: The width of a binary image is the image width divided by the sampling interval; The height of the binary graph is the height of the detection range divided by the sampling interval.

7. The method for rapid detection of dynamic regions in a mobile application page according to claim 1, characterized in that: In step S6, the function called is: Find the contour using `findContours()`; find the minimum bounding rectangle using `boundingRect()`. In step S6, the method to restore the minimum bounding rectangle of the contour to the original image size is to multiply the coordinates of the top left corner and the length and width of the minimum bounding rectangle by the sampling interval.

Images