Image positioning method and electronic equipment

Abstract

The invention provides an image positioning method and electronic equipment, and belongs to the technical field of image positioning, and the image positioning method comprises the steps: carrying out the division operation of a to-be-positioned pin image according to a current division scale, and obtaining a sub-positioning image; when the matching areas of all the sub-positioning images in the grass image are empty, reducing the current division scale to obtain a small division scale, and taking the small division scale as the current division scale to execute division operation; and when at least one non-empty region exists in the matching region of the sub-positioning image, positioning the region of the to-be-positioned needle image in the grass cluster image based on the sub-positioning image corresponding to the non-empty region. According to the method and the device, the needle image can be accurately positioned in the grass cluster image on the basis of not depending on manually intercepting a large number of needle images.

Description

Technical Field

[0001] The present disclosure relates to the field of image positioning technology, and in particular to an image positioning method and electronic equipment. Background Art

[0002] Image recognition-based user interface automated testing methods (such as OpenCV and Sikuli) rely on manually capturing images of target areas as pin images. The core process involves manually capturing image elements (such as buttons) in the user interface before testing begins, using them as pin images. Image matching algorithms (such as matchTemplate) are then used to locate the corresponding area in the user interface.

[0003] However, this method requires manual capture and management of a large number of needle images, which is prone to human errors; repeated screenshots are required during version iterations, which leads to low maintenance efficiency. The matching results are also easily affected by external factors such as background changes, resolution differences, and graphics card driver rendering, resulting in poor test stability. Summary of the Invention

[0004] The present disclosure provides an image positioning method and electronic device, which can accurately position a needle image in a bush image without relying on manually intercepting a large number of needle images.

[0005] The technical solution of the present disclosure is achieved as follows: In a first aspect, the present disclosure provides an image positioning method, which includes: dividing a pin image to be positioned according to a current division scale to obtain a sub-positioning image; when the matching areas of all sub-positioning images in a bush image are empty, reducing the current division scale to obtain a small division scale, and using the small division scale as the current division scale to perform the division operation; when there is at least one non-empty area in the matching area of the sub-positioning image, locating the area of the pin image to be positioned in the bush image based on the sub-positioning image corresponding to the non-empty area.

[0006] In a second aspect, the present disclosure provides an image positioning device, comprising: a division module, an update module, and a positioning module; the division module is configured to perform a division operation on a pin image to be positioned according to a current division scale to obtain a sub-positioning image; the update module is configured to, when the matching areas of all sub-positioning images in a bush image are empty, reduce the current division scale to obtain a small division scale, and use the small division scale as the current division scale to perform the division operation; the positioning module is configured to, when there is at least one non-empty area in the matching area of the sub-positioning image, locate the area of the pin image to be positioned in the bush image based on the sub-positioning image corresponding to the non-empty area.

[0007] In a third aspect, the present disclosure provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image positioning method as described in the first aspect.

[0008] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a program or instruction, which, when executed by a processor, implements the steps of the image positioning method as described in the first aspect.

[0009] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product includes a computer program or instructions. When the computer program product runs on a processor, the processor executes the computer program or instructions to implement the steps of the image positioning method as described in the first aspect.

[0010] In a sixth aspect, the present disclosure provides a chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the image positioning method as described in the first aspect.

[0011] The present disclosure provides an image positioning method. Pin images included with installed applications or games often have transparent or gradually transparent borders. These areas can easily become confused with the background during the matching process, resulting in a high failure rate when positioning the pin image in grass images. This solution utilizes a variable granularity partitioning mechanism to divide the complete pin image into multiple sub-positioning images. Even for grass images with continuously changing backgrounds, each sub-positioning image mitigates the impact of transparent edges, thereby improving image positioning accuracy. Furthermore, by directly using the pin image included with the installation (for applications or games), the efficiency issues associated with manual screenshots, the storage resource consumption caused by storing large numbers of screenshots, and the resulting version management issues associated with manual screenshots are avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of the first grass image with a gradient background provided by the present disclosure.

[0013] Figure 2 A schematic diagram of the first needle image under the installation path provided by the present disclosure.

[0014] Figure 3 A schematic diagram of a second grass image with a gradient background provided by the present disclosure.

[0015] Figure 4 A schematic diagram of a third type of grass image with a gradient background provided by the present disclosure.

[0016] Figure 5 A schematic diagram of a manually captured second needle image provided by the present disclosure.

[0017] Figure 6 A schematic diagram of a manually captured third needle image provided for the present disclosure.

[0018] Figure 7 Schematic diagram of the manually captured fourth needle image provided for the present disclosure.

[0019] Figure 8 A flowchart of an image positioning method provided by the present disclosure is provided.

[0020] Figure 9 This is a schematic diagram of a sub-positioning image obtained by dividing the needle image to be positioned into regular grids provided by the present disclosure.

[0021] Figure 10 A schematic flow chart of another image positioning method provided by the present disclosure.

[0022] Figure 11 A flowchart of another image positioning method provided by the present disclosure is provided.

[0023] Figure 12 A schematic diagram of a candidate needle image to be located provided by the present disclosure.

[0024] Figure 13 A schematic diagram of the intersection-over-union ratio of two regions provided in the present disclosure.

[0025] Figure 14 A schematic diagram with a fully transparent background provided for this disclosure.

[0026] Figure 15 This is a schematic diagram after replacing the fully transparent background provided by the present disclosure.

[0027] Figure 16 A schematic diagram of a fully transparent and semi-transparent background provided for the present disclosure.

[0028] Figure 17 A schematic diagram of replacing the fully transparent and semi-transparent backgrounds provided by the present disclosure.

[0029] Figure 18 This is a structural block diagram of an image positioning device provided by the present disclosure.

[0030] Figure 19 A schematic diagram of the hardware structure of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in this disclosure to clearly describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this disclosure.

[0032] Image localization is one of the core tasks in the field of computer vision. It refers to determining the spatial position information of specific image elements (objects, regions, or feature points) in a given image. Its core goal is to output the precise geometric position of image elements in the image coordinate system, usually represented in the form of coordinate points (such as center points), bounding boxes, etc.

[0033] For example, in the automated testing of user interfaces based on image recognition, the user interface image (the interface to be tested) includes at least one operable (responds after being triggered) image element. Each operable image element corresponds to a template image. During the automated testing process, it will be given which template image is operated in each step. It is necessary to locate the operable image element corresponding to the template image in the user interface image, and automatically simulate the operation of the image element in the user interface. Since the operation of searching for the corresponding operable image element in the user interface based on the template image is similar to the process of finding a needle in the grass, the user interface image is recorded as a grass image, and the operable template image is recorded as a needle image. Figure 1 As shown, the grass image 10 includes multiple operable image elements (each independent image element in the dotted box is an operable image element). Figure 2 As shown, Figure 1 The operable image element 101 corresponds to the needle image.

[0034] In user interface automation testing based on image recognition, the application or game under test generally provides pin images corresponding to each operable image element on the user interface under its installation path. However, most of these pin images contain transparent channels. This is because the original author, for artistic design reasons, made the edges of these pin images fully transparent, or made the transition from fully transparent to fully opaque.

[0035] The reason why computer image recognition interfaces (such as OpenCV or Sikuli) cannot directly use the needle image saved in the installation path is that when the program or game is running, the edge of the needle image will show the underlying background color due to transparency. At this time, the needle image with transparent edges cannot be matched with the operable image elements containing the underlying background color, resulting in the needle image through the transparent edges being unable to find the expected position in the grass image.

[0036] like Figure 2As shown in FIG, the first needle image corresponding to the operable image element 101 saved in the installation path, the two spaced border lines indicate that the edge of the first needle image is transparent. The grass image 10 mostly contains the background, such as Figure 3 In the grass image 10 shown, the background of the grass image 10 is included below the operable image element 101. Since the edges of the first needle image and the operable image element 101 are different, the matching degree between the two does not meet the requirements, making it impossible to find the corresponding operable image element 101 in the grass image based on the first needle image.

[0037] Therefore, based on computer image recognition interfaces (such as OpenCV or Sikuli), it is necessary to manually capture the operable image elements during the program or game runtime through a rectangular frame as a pin image. However, manual screenshots also have disadvantages. First, manual screenshots require capturing and managing a large number of images, especially for some game interfaces, which have a large number of operable image elements and a huge workload for manual screenshots. Second, for interfaces with changing background colors, manually captured operable image elements are also prone to positioning failures.

[0038] like Figure 1 、 Figure 3 and Figure 4 10 are different grass images with background gradient, where the dot fillings of different densities represent different backgrounds. Figure 1 In the grass image 10 shown, the area where the operable image element 101 is located is captured as the second needle image, as shown in FIG. Figure 5 As shown, the outer rectangular box is the manually intercepted boundary; Figure 3 In the grass image 10 shown, the area where the operable image element 101 is located is captured as the third needle image, as shown in FIG. Figure 6 shown; in Figure 4 In the grass image 10 shown, the area where the operable image element 101 is located is captured as the fourth needle image, as shown in FIG. Figure 7 As shown. Since manual capture will capture the background and the background with transparent edges, if only one needle image is manually captured in the grass image with a gradient background, such as the third needle image, then if the grass image 10 is in Figure 1 or Figure 3 If the image is in the same state as the grass, positioning fails because no matching area can be found (due to different edges). The solution is to capture the corresponding actionable image element in each frame of the grass image as the needle image. However, this method requires capturing a large number of needle images, which is difficult to operate and requires a large amount of space to store the captured needle images.

[0039] Based on the above problems, the present disclosure aims to provide an image positioning method that does not rely on manual interception of needle images and achieves effective image positioning. The image positioning method provided by the present disclosure is described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0040] like Figure 8 As shown, the present disclosure provides an image positioning method, which may include the following steps 801 to 804.

[0041] In step 801, the image of the pin to be positioned is divided according to the current division scale to obtain sub-positioning images.

[0042] The pin image to be located refers to the operable image element (such as a button or icon) that needs to be located in the grass image. The sub-positioning image refers to the local image block obtained by dividing the pin image to be located according to the current division scale (such as a 2×2 grid). For example, a 100×100 button image can be divided into four 50×50 sub-positioning images (top left, top right, bottom left, and bottom right).

[0043] The division scale refers to the spatial granularity of regional segmentation of the pin image to be positioned in the image positioning task, which directly affects the number and size of sub-positioning images. The granularity of dividing the pin image to be positioned determines the number and size of sub-positioning images.

[0044] In some embodiments, the current division scale is the current grid size. That is, the change of the division scale is to change the size of the grid. According to the current grid size, the image of the needle to be positioned is evenly divided into rectangular sub-images of the same size according to a fixed number of rows and columns, such as 2×2, 3×3, etc. Figure 2 The first needle image shown in is determined as the needle image to be positioned, as shown in Figure 9 As shown, 4 sub-positioning images 901 are obtained by dividing the image according to a 2×2 grid, and 9 sub-positioning images 902 are obtained by dividing the image according to a 3×3 grid. This division method is simple and improves the effectiveness of positioning after division.

[0045] The current division scale may also be to divide into rectangles of different sizes or irregular shapes. For example, the shape and size of the divided sub-positioning images may be dynamically adjusted according to the target features (such as edge, texture, transparency, etc.) of the pin image to be positioned.

[0046] Since a division scale that is too small will lead to large noise interference, in order to avoid this situation, the lower limit of the division scale can be limited, such as the maximum number of divisions = min (width of the image of the needle to be positioned, height of the image of the needle to be positioned) / 4. The size of the sub-positioning image after division according to the minimum division scale corresponding to this maximum number of divisions is 4 pixels × 4 pixels. The specific lower limit of the division scale can be determined based on actual conditions and is not limited in this disclosure.

[0047] In step 802 , it is determined whether the matching areas of all sub-positioning images in the grass image are empty.

[0048] If yes, execute step 803; otherwise, execute step 804.

[0049] In step 803 , the current division scale is reduced to obtain a smaller division scale, and the smaller division scale is used as the current division scale, and the process returns to step 801 .

[0050] The grass image is a user interface image (such as a screenshot or a game interface) that needs to be searched. For example, a 1920×1080 screenshot of a main interface of a game is a grass image.

[0051] The matching area refers to a local area in the grass image found by the image matching algorithm, whose similarity with the sub-positioning image is greater than the similarity threshold. The matching area includes the marked coordinates of the local area in the grass image (such as the upper left corner coordinates) and the size of the sub-positioning image.

[0052] The size of the sub-positioning image is used as the size of the sliding window. The sliding window is gradually slid in the grass image according to a certain step size (such as one pixel). The similarity between the image in each sliding window and the sub-positioning image is calculated until the entire grass image is traversed. If the similarity between the image in all sliding windows and the sub-positioning image is less than the similarity threshold, the matching area is determined to be empty, otherwise the matching area is determined to be non-empty.

[0053] If the similarity calculated between sub-image A (50×50) and the image within each sliding window of the grass image is only 0.6 (below the threshold of 0.8), then the matching area of sub-image A is determined to be empty. The similarity threshold is determined based on the required matching accuracy. For example, in OpenCV (the underlying image recognition API of Sikuli is also based on OpenCV), the default similarity threshold for the grass image recognition API is 0.8, but the similarity threshold can be smaller or larger, such as 0.7 or 0.9.

[0054] It should be noted that the image features contained in any sub-positioning image obtained by each division are smaller than the image features contained in any sub-positioning image obtained by the previous division.

[0055] In step 804 , when there is at least one non-empty area in the matching areas of all sub-locating images, the area where the needle image is to be located is located in the bush image based on the sub-locating image corresponding to the non-empty area.

[0056] The area of the pin image to be located refers to the area of the operable image element located in the grass image and corresponding to the pin image to be located.

[0057] When at least one non-empty region exists in the matching regions of all sub-positioning images, the region of the pin image to be located is located in the bush image based on the sub-positioning image corresponding to the non-empty region. Specifically, if at least one non-empty region exists in the multiple sub-positioning images obtained by the Nth (greater than 0) division, the region of the pin image to be located is determined in the bush image using the sub-positioning images corresponding to all non-empty regions obtained by the Nth division. For example, a region growing method is used, starting from the sub-positioning image with a non-empty matching region, to gradually merge adjacent sub-positioning image sub-images or expand the region boundaries until the region of the pin image to be located is covered. For example, after matching to sub-positioning image C (the top of the button), sub-positioning image D (the right side of the button) is found to the right, and the complete button region is restored based on this expansion. Alternatively, significant feature points (such as edges, corners, and textures) are extracted from the sub-positioning image corresponding to the non-empty matching region, and the corresponding feature point clusters are matched in the bush image to directly locate the corresponding operable image element. For example, 20 ORB feature points are extracted from sub-positioning image A (the left half of the button). Similar feature point clusters are found in the bush image, and the complete button position is determined through coordinate fitting.

[0058] If you directly use the needle image stored with the program installation to search for the operable image element corresponding to the needle image in the grass image, the needle image has a transparent edge, and the transparent edge of the operable image element corresponding to the needle image will show through the background, which reduces the similarity between the needle image and the operable image element. Ultimately, it is impossible to locate the area where the operable image element corresponding to the needle image is located in the grass image. In the present disclosure, the needle image is divided into multiple sub-positioning images. The transparent edge of each sub-positioning image is reduced relative to the needle image. Figure 2 and Figure 9 , Figure 2 The first needle image in includes four transparent edges that affect the matching accuracy. Figure 9 In the figure, each sub-positioning image after being divided into a 2×2 grid includes only two transparent edges that affect the matching accuracy. The sub-positioning images at the four corners after being divided into a 3×3 grid include only two transparent edges that affect the matching accuracy. The sub-positioning image at the center does not include a transparent edge that affects the matching accuracy. The remaining sub-positioning images include only one transparent edge that affects the matching accuracy. Therefore, the similarity between the sub-positioning image and the corresponding operable image element will be increased, that is, the difference between the needle image and the corresponding operable image element caused by the transparent edge is decomposed into multiple sub-positioning images, so that the similarity between each sub-positioning image and the corresponding operable image element is artificially increased.

[0059] In summary, the pin images that come with installed applications or games often have transparent or gradually transparent boundaries. These areas can easily become confused with the background environment during the matching process, resulting in a high failure rate for positioning based on the pin image in grass images. This solution uses a variable granularity partitioning mechanism to divide the complete pin image into multiple sub-positioning images. Even for grass images with continuously changing backgrounds, each sub-positioning image mitigates the impact of transparent edges, thereby improving image positioning accuracy. Furthermore, by directly using the pin image during installation (for applications or games), it avoids the efficiency issues associated with manual screenshots, the consumption of storage resources caused by storing large numbers of screenshots, and the resulting issue of manual screenshot version management (each version corresponds to a batch of manual screenshots).

[0060] In some embodiments, combined Figure 8 ,like Figure 10 As shown, the image positioning method further includes: the following step 805, and the above step 804 can be specifically implemented through the following step 804a.

[0061] In step 805 , when the matching score of the zoomed image corresponding to each sub-positioning image in the grass image is greater than a preset threshold, it is determined that the matching area of the sub-positioning image is not empty.

[0062] The match score indicates the similarity between the zoomed image and the area within the sliding window of the grass image. For example, using the Normalized Cross Correlation (NCC) algorithm, the score ranges from 0 to 1, with 1 indicating a perfect match. This disclosure does not limit the specific algorithm used to determine the match score; any algorithm that can quantify the degree of image matching is acceptable.

[0063] The preset threshold refers to a critical value parameter used to determine the validity of a match. For example, if the preset threshold is set to 0.8, a match is considered valid only when the NCC score of a region is greater than or equal to 0.8. The preset threshold can also be other values besides 0.8, which is not limited in this disclosure.

[0064] Each sub-positioning image is reduced or enlarged to obtain a scaled image. If the scaled image can find an area in the grass image with a similarity greater than a similarity threshold, and the matching score between the scaled image and the grass image is greater than a preset threshold, it is determined that the matching area of the sub-positioning image is not empty.

[0065] In some embodiments, the scaled image includes scaled sub-images; for each sub-positioning image, scaling is performed within a preset scaling range and at a scaling step size to obtain multiple scaled sub-images at different scaling ratios; for each scaled sub-image, a matching score of the scaled sub-image in the grass image is determined; and when any matching score exceeds a preset threshold, the matching region of the sub-positioning image is determined to be non-empty. Specifically, after all scaled sub-images are generated within the preset scaling range and at the scaling step size, matching is performed one by one.

[0066] The preset scaling range is an image scaling boundary parameter set to accommodate image elements within the grass image. For example, if the range is set between 80% and 120%, the sub-positioned image is adjusted between 0.8 and 1.2 times its original size. This preset scaling range is an adaptive adjustment mechanism designed to address the dynamic size changes of image elements in the display environment. The preset scaling range must adhere to a dual constraint principle: first, the upper limit of the scaling must ensure that the size of the enlarged sub-image does not exceed the display area of the grass image to avoid invalid matching due to overflow; second, a mapping relationship between scaling ratio and pixel density can be established by referring to standard resolution specifications (such as 1080P and 4K). Especially for cross-platform deployment scenarios, dynamic adjustment must be combined with the screen parameter distribution characteristics of the target device group.

[0067] The scaling step size refers to the granularity of each adjustment within the scaling range. For example, if the scaling step size is 5%, scaling versions starting at 80% will be generated at 85%, 90%, and so on, all the way to 120%. A smaller scaling step size results in higher potential matching accuracy but also requires more computation. The specific scaling step size is determined based on the matching accuracy and computational requirements and is not limited in this disclosure.

[0068] A scaled subimage is a set of derivative images generated by scaling a sub-localization image. For example, for a 50×50 sub-localization image, scaled subimages of different sizes, such as 52.5×52.5 and 55×55, are generated with a 5% scaling step. For each scaled subimage, a sliding window mechanism is used to traverse the grass image, and the matching score between each scaled subimage and the current window area is calculated.

[0069] In other embodiments, the zoomed image includes a current zoomed sub-image; for each sub-positioning image, a zoom operation is performed according to the current zoom ratio within a preset zoom ratio range to obtain the current zoomed sub-image; when the matching score of the current zoomed sub-image in the grass image is less than or equal to a preset threshold, the zoom operation is performed using the current zoom ratio increased by a preset step size as the current zoom ratio; when the matching score of the current zoomed sub-image in the grass image is greater than the preset threshold, the matching region of the sub-positioning image is determined to be non-empty; or when the current zoom ratio is not within the preset zoom ratio range, the matching region of the sub-positioning image is determined to be empty. Specifically, zoomed sub-images are generated stepwise within the preset zoom ratio range according to the zoom step size, and matching is performed once for each zoomed sub-image until a zoomed sub-image with a matching score greater than the preset threshold is found.

[0070] In step 804a, when there is at least one non-empty area in the matching areas of all sub-positioning images, the area of the needle image to be positioned is positioned in the bush image using the zoomed sub-image corresponding to the sub-positioning image with the non-empty matching area.

[0071] If multiple zoomed images corresponding to a sub-locating image have matching scores with the bush image that are greater than a preset threshold, one approach is to locate the area of the needle image to be located in the bush image based on any zoomed image with a matching score greater than the preset threshold. Another approach is to locate the area of the needle image to be located in the bush image based on the zoomed image with the highest matching score.

[0072] Since the number of scaled images is large, the matching process has a large amount of computational complexity. In order to reduce the computational complexity, the sub-positioning image can be first converted into a grayscale sub-positioning image, and then the grayscale sub-positioning image can be scaled to obtain a grayscale scaled image. The grass image can be converted into a grayscale grass image, and the matching score of the grayscale scaled image in the grayscale grass image can be determined.

[0073] Grayscale conversion uses the following formula: Grayscale value of each pixel = 0.299·R + 0.587·G + 0.114·B, where R, G, and B represent the red, green, and blue values of each pixel, respectively. This formula aligns with the following human perception characteristics: the human eye is most sensitive to green (weighted approximately 59%); second most sensitive to red (approximately 30%); and least sensitive to blue (approximately 11%).

[0074] Because sub-localization images are typically full-color images containing three channels (R, G, and B), OpenCV's image matching algorithm can only compare one channel at a time. Single-channel grayscale image matching reduces the number of comparisons compared to three-channel color image calculations, making the calculations faster. Furthermore, the color distribution in color images is very sensitive to changes in illumination, while grayscale images can, to a certain extent, ignore color noise, making them more suitable for stable and fast matching. Compared to the 3N operations required for three-channel parallel matching (N is the total number of pixels), grayscale conversion reduces the number of operations to N. Furthermore, the illumination invariance of grayscale space effectively suppresses the color temperature offset noise generated by the color channels when ambient light changes.

[0075] Due to the geometric deformation of image elements caused by differences in pixel density of display devices or dynamic scaling of the interface, in the embodiment of the present disclosure, multiple scaled sub-images covering a preset scaling range are generated through continuous proportional transformation, thereby solving the problem of size distortion under different resolutions or rendering environments, and ensuring the matching reliability through threshold judgment, and determining the scaled sub-image at the scaling scale that best matches the candidate area of the grass image.

[0076] In some embodiments, there are multiple non-empty regions; Figure 8 ,like Figure 11 As shown, the above step 804 can be implemented through the following steps 804b and 804c.

[0077] In step 804b, a corresponding candidate needle image to be positioned is determined based on each sub-positioning image in the plurality of sub-positioning images corresponding to the plurality of non-empty matching regions.

[0078] Candidate pin images for positioning refer to a set of complete pin image hypotheses reconstructed based on the spatial mapping relationship of the sub-positioning images in a non-empty matching region. If the matching region corresponding to the sub-positioning image is non-empty, the matching region includes the positioning coordinates of the sub-positioning image in the grass image. These coordinates are typically the coordinates of the upper-left corner of the sub-positioning image in the grass image, but may also be the coordinates of other feature points in the grass image, though this disclosure does not limit these coordinates.

[0079] Based on each of the multiple sub-positioning images corresponding to the multiple non-empty matching regions, a corresponding candidate pin image to be positioned is determined. This can be done by using the positioning coordinates of the determined sub-positioning image in the brush image as the geometric center of the pin image to be positioned. Based on the size of each sub-positioning image and the division scale within the pin image to be positioned, the size of the candidate pin image to be positioned is determined, thereby determining the candidate pin image to be positioned. This method is computationally simple, but the resulting candidate pin image may lack accuracy.

[0080] In some embodiments, based on each sub-positioning image in the multiple sub-positioning images corresponding to the multiple non-empty matching areas, the corresponding candidate needle image to be positioned is determined. Specifically, the positioning coordinates of the candidate needle image to be positioned are determined based on the row and column numbers and positioning coordinates of each sub-positioning image in the needle image to be positioned; the size of the candidate needle image to be positioned is determined based on the size of each sub-positioning image and the division scale in the needle image to be positioned.

[0081] For example, the image of the needle to be positioned is processed as follows Figure 9 The 2×2 grid shown on the left side of the figure is divided into 4 sub-positioning images. The row and column numbers of the sub-positioning image in the upper left corner are (0, 0), the row and column numbers of the sub-positioning image in the upper right corner are (0, 1), the row and column numbers of the sub-positioning image in the lower left corner are (1, 0), and the row and column numbers of the sub-positioning image in the lower right corner are (1, 1). If the sub-positioning image in the upper right corner with the row and column numbers (0, 1) is determined to have a non-empty matching area, the coordinates of the upper left corner of the matched sub-positioning image in the grass image are (x, y), as shown in the following example: Figure 12 The coordinates of the upper left corner of the candidate pin image are (xi×w, yj×h), then according to the coordinates (x, y) of the point p1 and the row and column numbers (0, 1) of the sub-positioning image, the following is determined: Figure 12 The coordinates of point p2 in the image are (x-0×w, y-1×h), where w and h represent the length and width of the sub-positioning image, respectively. Since the sub-positioning image is divided into a 2×2 grid, the size of the candidate pin image to be positioned is determined to be 2w×2h, that is, the coordinates of the upper left corner of the candidate pin image to be positioned are p2, and the size is 2w×2h, corresponding to Figure 9 The dotted box in the middle is the final candidate needle image to be positioned.

[0082] It should be noted that, if the sub-positioning image is scaled, the candidate needle image to be positioned is determined by using the size and division scale of the scaled image corresponding to the sub-positioning image having a non-empty matching area.

[0083] In step 804c, based on the plurality of candidate needle images to be located, the region of the needle image to be located is located in the bush image.

[0084] In some embodiments, based on multiple candidate pin images to be located, the area of the pin image to be located is located in the grass image. Specifically, the intersection over union (IoU) between any two of the multiple candidate pin images to be located is determined; and based on any one of the two candidate pin images to be located with the largest IoU, the area of the pin image to be located is located in the grass image. Since all candidate pairs only need to be traversed once and the candidate corresponding to the largest IoU (Intersection over Union, IoU) is recorded, the implementation is simple and relies only on the IoU sorting results. However, if the IoU between two incorrect candidates is the largest (for example, a false match caused by repeated background texture), the system will mistakenly select one of them. For example, if there are multiple similar icons in the grass image, incorrect candidates A and B have a high IoU due to similar local features, but both deviate from the true location.

[0085] IoU is a geometric similarity index used to measure the spatial overlap of two candidate pin images to be located. The calculation formula is: IoU = area of (A∩B) / area of (A∪B), such as Figure 13 As shown in the figure, A∩B is the area corresponding to the intersection of rectangular area A and rectangular area B, which is represented by the diagonal line filling area in the figure. A∪B is the sum of the area of the non-overlapping part and the area of the overlapping part of rectangular area A and rectangular area B, that is, the sum of the area of the blank area in rectangular area A and rectangular area B in the figure and the area of the diagonal line filling area.

[0086] In other embodiments, based on multiple candidate pin images to be located, the area of the pin image to be located is located in the grass image, specifically by determining the intersection-and-union ratio between any two of the multiple candidate pin images to be located; the number of times the intersection-and-union ratio of each candidate pin image to be located exceeds a preset threshold is determined as a confidence value; and based on the candidate pin image to be located with the highest confidence value, the area of the pin image to be located is located in the grass image.

[0087] The confidence value is a statistical indicator used to quantify the spatial consistency of candidate pin images. It is accumulated by accumulating the frequency of IoU exceeding a threshold. For example, if there are three candidate pin images to be located (Image 1, Image 2, and Image 3), and the IoU of Image 1 and Image 2 exceeds a preset threshold (such as 0.5, 0.6, or another value), the confidence value of Image 1 and Image 2 is increased by 1. If the IoU of Image 2 and Image 3 exceeds the preset threshold, the confidence value of Image 2 and Image 3 is increased by 1. If the IoU of Image 1 and Image 3 exceeds the preset threshold, the confidence value of Image 1 and Image 3 remains unchanged. This determines that Image 2 has the highest confidence value, and based on Image 2, the region of the pin image to be located is located in the grass image.

[0088] It should be noted that when multiple candidate pin images have the same confidence level, the pin image region is located in the brush image based on the candidate pin image with the higher matching score for its corresponding sub-positioning image or the scaled sub-image corresponding to the sub-positioning image. Alternatively, the pin image region is located in the brush image based on the candidate pin image with the higher weighted average of its intersection-over-union ratio.

[0089] When the intersection-over-union ratios of all candidate pin images to be located are smaller than the matching threshold, it is determined that the pin image to be located does not exist in the bush image.

[0090] In this way, the confidence mechanism selects hypotheses supported by the majority of candidates, and this confidence voting mechanism suppresses outlier noise interference. If the true target candidate C has an IoU exceeding the threshold with 7 of the 10 candidates, while the false candidate A has a high IoU with only 1 candidate, then C has a higher confidence.

[0091] In some embodiments, the image positioning method also includes: removing the background of the initial needle image when the background of the initial needle image is completely transparent; converting the initial needle image after background removal into a grayscale needle image; determining the main grayscale in the grayscale needle image, and the full-color value corresponding to the complementary value of the main grayscale; adjusting the background of the initial needle image to the full-color value to obtain the needle image to be positioned.

[0092] The initial needle image refers to the original needle image without preprocessing, usually stored in a format with a transparency channel (such as PNG). The alpha channel of the image is used to indicate whether the background of the image is transparent. 255 means opaque, and 0 means fully transparent. If the alpha channel value of the background area is marked as 0, it means fully transparent. Figure 14 As shown, the white area outside the wrench icon indicates full transparency.

[0093] The dominant grayscale refers to the grayscale value that appears most frequently in the grayscale needle image, reflecting the typical brightness characteristics of the grayscale needle image. This can be determined by statistically analyzing the grayscale histogram and taking the grayscale value corresponding to the frequency peak. If multiple peaks exist, the grayscale value corresponding to the largest connected area is selected. For example, if the main body of the icon is dark gray (grayscale value = 60) and occupies 70% of the image area, the dominant grayscale = 60.

[0094] The complementary value refers to the grayscale value that forms the maximum visual contrast with the main grayscale. For example, if the main grayscale is 60, the complementary value is 255-60=195.

[0095] Convert the complementary value to the corresponding R, G, B full color value. Specifically, the complementary value can be directly used as the R value, G value, and B value. For example, the full color value converted from the complementary value 195 is (195, 195, 195). Alternatively, the full color value corresponding to the complementary value can be determined by looking up the pseudo color mapping table (the correspondence between grayscale value and full color value). Use this full color value to replace the fully transparent background area, such as Figure 15 As shown, it is the image of the pin to be positioned obtained after replacement. The direction of the oblique lines filling the periphery of the wrench icon is different from the oblique lines filling the interior of the wrench icon, indicating that the sum of the grayscale values of the two is 255.

[0096] In the disclosed embodiment, the fully transparent background is replaced with a high-contrast full-color value to eliminate the matching interference caused by the fusion of transparent pixels with the background of the grass image. The transparent area of the original needle image is no longer affected by the underlying environmental color during the matching process, avoiding feature confusion caused by the background showing through, and significantly improving the recognition stability of the target subject's outline. The complementary color-filled background calculated based on the main grayscale forms a color difference with the target subject in the RGB space that is maximized (such as a dark gray subject and a light gray background). This forced contrast optimization enables the image matching algorithm (such as normalized cross correlation) to accurately capture the target edge features and reduce the probability of mismatching due to color similarity. In addition, the high-contrast boundary between the complementary color background and the target subject is equivalent to adding artificial edge enhancement to the original needle image during the matching process. This implicit sharpening operation enhances the template matching algorithm's ability to detect blurred or low-resolution targets, especially in scenes where the target size is small or there is motion blur.

[0097] In some embodiments, the image positioning method further includes: when there is a semi-transparent area with transparency less than a transparency threshold in the background of the initial needle image, adjusting the transparency value of the semi-transparent area to the maximum to obtain the needle image to be positioned.

[0098] like Figure 16 As shown, the outer white blank area indicates full transparency, and the dot-filled area indicates semi-transparency. Adjust the transparency value of the semi-transparent area to the maximum (255) to make it completely opaque. Figure 17As shown, the dense dot-filled area in the middle represents opacity, and the grayscale represented by the outer grid-filled area and the sum of the grayscale and the main grayscale is 255.

[0099] In the present disclosure, by forcibly increasing the alpha value of the semi-transparent area (transparency below the threshold) to the maximum value (255), the pixel blending effect caused by the semi-transparent gradient at the edge of the target is completely eliminated. The gradient transparent area (such as the feathered edge) caused by the artistic design in the original needle image is converted into a clear fully opaque boundary, avoiding the color superposition interference between the target outline and the grass image background during the matching process, and significantly improving the accuracy of edge feature recognition. The semi-transparent area usually contains mixed pixels of the target and the background (such as a light color coverage with transparency = 30%). After adjusting to full opacity, the RGB value of the mixed pixel is restored to the original color of the target body, so that the originally weakened local features of the target (such as texture details) are fully preserved, enhancing the effective signal input of the matching algorithm.

[0100] The image positioning method provided by the present disclosure effectively solves the interference of transparent edge features on matching accuracy through a dynamic multi-scale division mechanism. By dividing the needle image to be positioned into multi-level sub-positioning images, the sub-positioning images in the core area are completely stripped of the influence of the original transparent boundaries. Combined with the high-contrast replacement strategy of the background transparent area, the feature confusion problem caused by background fusion is significantly reduced. Grayscale conversion and multi-scale scaling technology work together to improve computational efficiency while adapting to the size deformation requirements of different resolution devices and dynamic rendering environments.

[0101] Furthermore, the confidence voting mechanism employed in this disclosure optimizes spatial consistency judgment. By integrating intersection-over-union analysis with candidate regions, it effectively mitigates the risk of mismatches caused by repeated textures. The inverse mapping model from local features to global positioning, combined with a scaling adaptive mechanism, enables robust adaptation to changes in interface element size during cross-platform deployment, eliminating the tedious manual maintenance of multiple template versions.

[0102] This technology, specifically designed for processing semi-transparent edges, enhances edge recognition in low-contrast scenes by enhancing the color characteristics of the target subject. While preserving the artistic design characteristics of the original needle image, this method, through intelligent background reconstruction and feature enhancement, provides a stable and reliable positioning benchmark for automated testing, significantly reducing maintenance costs during version iterations.

[0103] Figure 18 This is a structural block diagram of an image positioning device 180 shown in the present disclosure, such as Figure 18As shown, it includes: a division module 181, an updating module 182, and a positioning module 183; the division module 181 is configured to divide the pin image to be positioned according to the current division scale to obtain a sub-positioning image; the updating module 182 is configured to reduce the current division scale to obtain a small division scale when the matching areas of all sub-positioning images in the grass image are empty, and use the small division scale as the current division scale to perform the division operation; the positioning module 183 is configured to locate the area of the pin image to be positioned in the grass image based on the sub-positioning image corresponding to the non-empty area when there is at least one non-empty area in the matching area of the sub-positioning image.

[0104] In some embodiments, the current division scale is the current grid size.

[0105] In some embodiments, there are multiple non-empty areas; the positioning module 183 is configured to determine the corresponding candidate pin image to be positioned based on each sub-positioning image in the multiple sub-positioning images corresponding to the multiple non-empty matching areas; and locate the area of the pin image to be positioned in the grass image based on the multiple candidate pin images to be positioned.

[0106] In some embodiments, the image positioning device 180 further includes: a determination module; a determination module configured to determine the positioning coordinates of the candidate needle image to be positioned based on the row and column numbers and positioning coordinates of each sub-positioning image in the needle image to be positioned; and determine the size of the candidate needle image to be positioned based on the size of each sub-positioning image and the division scale in the needle image to be positioned.

[0107] In some embodiments, the determination module is configured to determine the intersection-and-union ratio between any two of a plurality of candidate pin images to be located; determine the number of times the intersection-and-union ratio of each candidate pin image to be located exceeds a preset threshold as a confidence value; and locate the area of the pin image to be located in the grass image based on the candidate pin image to be located with the highest confidence value.

[0108] In some embodiments, the image positioning device 180 further includes: a determination module; the determination module is configured to determine that the matching area of the sub-positioning image is not empty when the matching score of the zoomed image corresponding to each sub-positioning image in the grass image is greater than a preset threshold.

[0109] In some embodiments, the image positioning device 180 further includes: a scaling module; the scaling module is configured to perform scaling processing on each sub-positioning image according to a scaling step within a preset scaling ratio range to obtain multiple scaled sub-images at different scaling ratios; the determination module is configured to determine, for each scaled sub-image, a matching score of the scaled sub-image in the grass image; when any matching score is greater than a preset threshold, it is determined that the matching area of the sub-positioning image is not empty.

[0110] In some embodiments, the zoom module is configured to perform a zoom operation on each sub-positioning image according to the current zoom ratio within a preset zoom ratio range to obtain a current zoomed sub-image; when the matching score of the current zoomed sub-image in the grass image is less than or equal to a preset threshold, the zoom operation is performed using the zoom ratio after the current zoom ratio is increased by a preset step size as the current zoom ratio; the determination module is configured to determine that the matching area of the sub-positioning image is not empty when the matching score of the current zoomed sub-image in the grass image is greater than a preset threshold, or to determine that the matching area of the sub-positioning image is empty when the current zoom ratio is not within the preset zoom ratio range.

[0111] In some embodiments, the positioning module 183 is configured to locate the area where the needle image is to be located in the grass image using the zoomed image corresponding to the sub-positioning image having the non-empty matching area.

[0112] In some embodiments, the image positioning device 180 also includes: a removal module, a conversion module, and an adjustment module; the removal module is configured to remove the background of the initial needle image when the background of the initial needle image is completely transparent; the conversion module is configured to convert the initial needle image after background removal into a grayscale needle image; the determination module is also configured to determine the main grayscale in the grayscale needle image, and the full-color value corresponding to the complementary value of the main grayscale; the adjustment module is configured to adjust the background of the initial needle image to a full-color value to obtain the needle image to be positioned.

[0113] In some embodiments, the adjustment module is further configured to adjust the transparency value of a semi-transparent area to a maximum when there is a semi-transparent area with a transparency less than a transparency threshold in the background of the initial needle image, so as to obtain the needle image to be positioned.

[0114] It should be noted that each module of the above-mentioned image positioning device 180 can implement the image positioning method provided by the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0115] Please refer to Figure 19 , which shows a schematic diagram of the hardware structure of an electronic device provided by an exemplary embodiment of the present disclosure. In some examples, the electronic device can be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The electronic device has a communication function and can access a wired network or a wireless network. The electronic device can generally refer to one of a plurality of terminals. Those skilled in the art will appreciate that the number of the above-mentioned terminals can be more or less. It can be understood that the electronic device undertakes the calculation and processing work of the technical solution of the present disclosure, and the present disclosure does not limit this.

[0116] like Figure 19As shown, the electronic device in the present disclosure may include one or more of the following components: a processor 910 and a memory 920 .

[0117] Optionally, the processor 910 utilizes various interfaces and circuits to connect various components within the electronic device. It executes instructions, programs, code sets, or instruction sets stored in the memory 920, as well as accesses data stored in the memory 920, to perform various functions of the electronic device and process data. Optionally, the processor 910 can be implemented in at least one hardware form: a digital signal processing (DSP), a field programmable gate array (FPGA), or a programmable logic array (PLA). The processor 910 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), and a baseband chip. The CPU primarily handles the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the touchscreen display; the NPU implements artificial intelligence (AI) functions; and the baseband chip handles wireless communications. It is understandable that the above-mentioned baseband chip may not be integrated into the processor 910, but may be implemented by a separate chip.

[0118] The memory 920 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 920 includes a non-transitory computer-readable storage medium. The memory 920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), and instructions for implementing each of the above method embodiments. The data storage area may store data generated based on the use of the electronic device.

[0119] In addition, those skilled in the art will understand that the structures of the electronic devices shown in the above figures do not limit the electronic devices. The electronic devices may include more or fewer components than shown, or may combine certain components or arrange the components differently. For example, the electronic devices may also include a display screen, a camera assembly, a microphone, a speaker, a radio frequency circuit, an input unit, sensors (such as an accelerometer, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, and other components, which will not be described in detail here.

[0120] The present disclosure also provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction is configured to be executed by a processor to implement the image positioning method described in the above embodiments.

[0121] The present disclosure also provides a computer program product, which includes computer instructions stored in a computer-readable storage medium; a processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes to implement the image positioning method described in each of the above embodiments.

[0122] An embodiment of the present disclosure further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image positioning method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0123] It should be understood that the chip mentioned in the embodiments of the present disclosure can also be called a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0124] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, servers and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the portion that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, ROM, RAM, a magnetic disk, or an optical disk.

[0128] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in this disclosure can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0129] It should be noted that the technical solutions described in this disclosure can be combined arbitrarily without conflict.

[0130] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present disclosure, and they should all be covered by the protection scope of the present disclosure.

Claims

1. An image positioning method, characterized in that: The image positioning method comprises: The positioning pin image is divided according to the current division scale to obtain a sub-positioning image; When the matching areas of all the sub-positioning images in the grass image are empty, reducing the current division scale to obtain a small division scale, and using the small division scale as the current division scale to perform the division operation; When there is at least one non-empty area in the matching area of the sub-positioning image, the area of the needle image to be positioned is positioned in the bush image based on the sub-positioning image corresponding to the non-empty area.

2. The image positioning method according to claim 1, characterized in that: There are multiple non-empty areas, and the current division scale is the current grid size; The step of locating the area of the needle image to be located in the bush image based on the sub-locating image corresponding to the non-empty area includes: Determining a corresponding candidate needle image to be positioned according to each sub-positioning image in the plurality of sub-positioning images corresponding to the plurality of non-empty matching areas; Based on a plurality of candidate needle images to be located, a region of the needle image to be located is located in the bush image.

3. The image positioning method according to claim 2, characterized in that: The step of determining a corresponding candidate needle image to be positioned according to each of the plurality of sub-positioning images corresponding to the plurality of non-empty matching areas includes: Determine the positioning coordinates of the candidate pin image to be positioned according to the row and column numbers and positioning coordinates of each sub-positioning image in the pin image to be positioned; The size of the candidate image for the needle to be located is determined according to the size of each sub-positioning image and the division scale in the image for the needle to be located.

4. The image positioning method according to claim 2, characterized in that: The step of locating the region of the pin image to be located in the bush image based on the plurality of candidate pin images to be located comprises: determining an intersection-over-union ratio between any two of a plurality of candidate needle images to be located; The number of times the intersection-over-union ratio of each candidate pin image to be located exceeds a preset threshold is determined as a confidence value; According to the candidate needle image to be located with the highest confidence value, the region of the needle image to be located is located in the bush image.

5. The image positioning method according to claim 1, wherein: The image positioning method further includes: When the matching score of the zoomed image corresponding to each sub-positioning image in the grass image is greater than a preset threshold, it is determined that the matching area of the sub-positioning image is not empty.

6. The image positioning method according to claim 5, characterized in that: When the matching score of the zoomed image corresponding to each sub-positioning image in the bush image is greater than a preset threshold, determining that the matching area of the sub-positioning image is not empty includes: For each sub-positioning image, scaling is performed according to a scaling step within a preset scaling range to obtain multiple scaled sub-images at different scaling ratios; For each scaled sub-image, determining a matching score of the scaled sub-image in the grass image; When any of the matching scores is greater than a preset threshold, it is determined that the matching area of the sub-positioning image is not empty.

7. The image positioning method according to claim 5, characterized in that: The step of locating the area of the needle image to be located in the bush image based on the sub-locating image corresponding to the non-empty area includes: The region of the needle image to be located is located in the grass image by using the zoomed image corresponding to the sub-location image having the non-empty matching region.

8. The image positioning method according to any one of claims 1 to 7, characterized in that: The image positioning method further includes: When the background of the initial needle image is completely transparent, removing the background of the initial needle image; Convert the initial needle image after background removal into a grayscale needle image; Determining a main grayscale in the grayscale needle image and a full color value corresponding to a complementary value of the main grayscale; The background of the initial needle image is adjusted to the full color value to obtain the needle image to be located.

9. The image positioning method according to any one of claims 1 to 7, characterized in that: The image positioning method further includes: When a semi-transparent area with a transparency less than a transparency threshold exists in the background of the initial needle image, the transparency value of the semi-transparent area is adjusted to the maximum to obtain the needle image to be located.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image positioning method according to any one of claims 1 to 9.

Images