A UI icon detection method and related apparatus

Abstract

The application discloses a UI icon detection method and related device, the method comprises the following steps: obtaining a high-resolution image containing a UI icon; the high-resolution image is split to obtain a plurality of image blocks; wherein the size of each image block meets the input size of a target detection model; for each image block, the target detection model is input respectively to identify, and the icon detection result of the image block is obtained; wherein the icon detection result comprises at least one icon candidate box; the icon detection results of all image blocks are integrated and de-duplicated to obtain the UI icon detection result of the high-resolution image. When facing a high-resolution UI image, the positions of icons of different sizes can be accurately obtained.

Description

Technical Field

[0001] This application relates to a UI icon detection method and related apparatus, belonging to the field of software technology. Background Technology

[0002] In Robotic Process Automation (RPA) design, users can specify the UI (User Interface) elements to be clicked in the current step. During execution, the RPA program automatically searches for the specified elements on the current UI page based on the user's specifications. Typically, each UI element has a unique XPath path. The user specifies the corresponding XPath along with the element, and the execution process uses the XPath path to find the element. For elements that cannot be captured by the XPath path, RPA can use built-in Computer Vision (CV) image detection algorithms to obtain all elements of the UI interface.

[0003] Traditional CV image detection algorithms, such as Canny and Sobel edge detection, struggle to meet the high-precision detection requirements of UI images. Furthermore, traditional image detection algorithms primarily rely on pixel gradient changes to depict object contours and lack type recognition capabilities. Therefore, deep learning algorithms for image detection are more suitable for UI element detection. The main element types on a UI interface are text and icons. For text elements, OCR detection algorithms are sufficient and will not be discussed further here.

[0004] However, for icon elements, a suitable deep learning image detection algorithm and logical framework need to be customized according to the specific scenario. Summary of the Invention

[0005] Unlike text, icons vary considerably in size, with side lengths ranging from approximately 4 to 80 pixels. Most icons have a side length around 32 pixels, and object detection algorithms typically consider targets with a side length of 32 pixels to be small, making them difficult for general object detection algorithms to detect. Many icons have side lengths far less than 32 pixels, thus falling into the category of small targets and requiring enhancements to the detection algorithms to accommodate even smaller objects. Therefore, designing an image detection algorithm capable of accurately recognizing icons on high-resolution images is crucial for icon recognition technology.

[0006] In view of at least one of the above technical problems, this application provides a UI icon detection method and related apparatus, which can accurately obtain the position of icons of different sizes when faced with high-resolution UI images.

[0007] Technical solution: To solve the above-mentioned technical problems, the technical solution adopted in this application is as follows: Firstly, this application provides a UI icon detection method, the method comprising: Get a high-resolution image containing UI icons; The high-resolution image is segmented to obtain multiple image blocks; the size of each image block meets the input size of the target detection model. For each image patch, the target detection model is input for recognition to obtain the icon detection result of the image patch; wherein the icon detection result includes at least one icon candidate box; The icon detection results of all image blocks are integrated and deduplicated to obtain the UI icon detection results of the high-resolution image.

[0008] In some embodiments, segmenting the high-resolution image includes: The segmentation is performed along the width and height of the high-resolution image. If the input size of the target detection model is N*N, then the side length S of the segmentation window is set to N times A, and a stacking region is set between adjacent windows, the size of which is SN. If the width and / or height of the high-resolution image is less than the window side length S, or the side length of the image to which the last window is sliced ​​in the horizontal or vertical direction is less than S, then blank images are used to fill in the missing parts so that the side length of each window is S and the size of each image block is S*S.

[0009] In some embodiments, A is 1.1 to 1.4, preferably 1.25.

[0010] In some embodiments, the target detection model includes an improved backbone structure, an improved feature pyramid, and a prediction module.

[0011] In some embodiments, the method for constructing the improved backbone structure includes: (a) The convolutional layer used for feature extraction in the last stage of the backbone structure is converted into a deformable convolution; wherein the deformable convolution changes the receptive field according to the geometric features of the target by changing the offset position of the x-axis and y-axis of the receptive field, so as to better capture the actual geometric features of the target object.

[0012] (b) An attention module is added after the last stage of the backbone structure, wherein the attention module includes a spatial attention module and a channel attention module. The spatial attention module and the channel attention module are used to extract effective feature information from the spatial and channel dimensions. The residual structure in the last stage of the backbone structure is used to integrate the original features with the features extracted by the attention module.

[0013] In some embodiments, taking YOLO8 as an example, the convolutional layer used for feature extraction in the last stage of the backbone structure is converted into a deformable convolution, including: a stage of the backbone structure includes a downsampling layer and a C2F module; for the downsampling layer, the convolutional layer with a kernel of 3*3 and a step of 2 is replaced with a deformable convolution; for the C2F module, the convolutional layer with a kernel of 3*3 used for feature extraction is replaced with a deformable convolution, while retaining the point convolution with a kernel of 1*1.

[0014] In some embodiments, the improved feature pyramid construction method includes: the original pyramid integrates feature maps of different sizes from four stages of the backbone network, and the element pixel sizes of the four feature maps are 8*8, 16*16, 32*32, and 64*64, respectively. To detect extremely small targets, the second-stage feature map with a pixel size of 4*4 of the backbone structure is incorporated into the feature pyramid, and the structure of the feature pyramid is expanded from top to bottom. The third-stage feature map after being glued together in the original pyramid, after feature processing by the C2 module, will be used as the output of the bottom layer of the pyramid. The fourth, fifth, and sixth-stage feature maps will be output in the same form. This application removes the output of the original third-stage feature map, upsamples the third-stage feature map to make it the same size as the second-stage feature map, and then glues the newly introduced second-stage feature map together. At this time, the finest second-stage feature map is fused with all the coarse-grained features of the upper layer. The second-stage feature map processed by the C2 module is used as the output of the bottom layer of the feature pyramid. After expanding the structure from top to bottom, the pyramid structure is then supplemented from bottom to top: the second-stage feature map is convolved and then glued with the third-stage feature map, at which point the third-stage feature map incorporates more fine-grained features; the glued third-stage feature map is then processed by the C2 module and used as the output of the feature pyramid; sequentially upwards, the fourth to sixth-stage feature maps also incorporate the finest-grained second-stage feature maps; finally, the pyramid outputs five feature maps.

[0015] In some embodiments, the deduplication method uses the Intersection over Union (IOU) to deduplicate icons in the stacked area. Specifically, it includes: using the IOU to remove overlapping icon candidate boxes, and retaining icons with larger areas during deduplication to filter out truncated icons.

[0016] In some embodiments, after obtaining the icon detection results of the image block or the UI icon detection results of the high-resolution image, the method further includes: if the icon candidate box is offset, correcting the icon position, specifically including: Based on the offset icon candidate box, the width and height of the icon image are enlarged by B times the length and width of the icon candidate box; For the enlarged icon image, the position of the icon candidate box is recaptured and calibrated using a calibration algorithm; wherein the calibration algorithm includes edge detection, morphological transformation and contour description.

[0017] In some embodiments, B is 1.1 to 1.4, preferably 1.25.

[0018] In some embodiments, edge detection is used to delineate the outline of icons within an expanded area; Morphological transformations, such as closing and dilation, are used to enhance the connectivity of pixels in edge detection feature images. Contour description delineates the contour boundaries of the morphologically transformed image and defines the position of the new candidate boxes.

[0019] Secondly, this application provides a UI icon detection device, the device comprising: The acquisition module is used to: acquire high-resolution images containing UI icons; The segmentation module is used to segment the high-resolution image to obtain multiple image blocks; wherein the size of each image block meets the input size of the target detection model. The detection module is used to: input the target detection model into each image block for recognition, and obtain the icon detection result of the image block; wherein the icon detection result includes at least one icon candidate box; The integration module is used to integrate and deduplicate the icon detection results of all image blocks to obtain the UI icon detection results of the high-resolution image.

[0020] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0021] Fourthly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect.

[0022] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0023] Beneficial Effects: The UI icon detection method and related apparatus provided in this application have the following advantages: Multiple image blocks are obtained by segmenting a high-resolution image; the size of each image block meets the input size of the target detection model; for each image block, the target detection model is input for recognition to obtain the icon detection result of the image block; wherein the icon detection result includes at least one icon candidate box; the icon detection results of all image blocks are integrated and deduplicated to obtain the UI icon detection result of the high-resolution image. Furthermore, this application improves the structure of the target detection model, thereby enabling the detection of extremely small targets and improving detection capability and accuracy; in a further embodiment, this application also adds an icon position correction step to correct the position of icons with offset icon candidate boxes, further improving detection accuracy. Attached Figure Description

[0024] Figures 1 to 2 This is a schematic diagram illustrating an application scenario of a method according to an embodiment of this application; Figure 3 This is a schematic flowchart of a UI icon detection method according to an embodiment of this application; Figure 4 This is a schematic diagram of a UI icon detection device according to an embodiment of this application; Figure 5 This is a schematic diagram of an electronic device according to an embodiment of this application; Figure 6 This is a schematic diagram of deformable convolution replacement in a C2F module according to an embodiment of this application; Figure 7 This is a schematic diagram of an attention mechanism module according to an embodiment of this application; Figure 8 This is a schematic diagram of a miniature target feature pyramid provided according to an embodiment of this application; Figure 9 This is a schematic diagram of icon position calibration according to an embodiment of this application. Detailed Implementation

[0025] The present application will be further described below with reference to the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and should not be used to limit the scope of protection of the present application.

[0026] Unlike text, icons vary considerably in size, with side lengths ranging from approximately 4 to 80 pixels. Most icons have a side length around 32 pixels, and object detection algorithms typically consider targets with a side length of 32 pixels to be small, making them difficult for general object detection algorithms to detect. Many icons have side lengths far less than 32 pixels, thus falling into the category of small targets and requiring enhancements to the detection algorithms to accommodate even smaller objects. Therefore, designing an image detection algorithm capable of accurately recognizing icons on high-resolution images is crucial for icon recognition technology.

[0027] Based on this, this application proposes a UI icon detection method, which is divided into three parts to enhance the detection of icon elements: 1) image segmentation before entering the algorithm; 2) target detection algorithm enhanced for extremely small targets; 3) calibration of icon detection results.

[0028] like Figures 1 to 2 As shown, before providing a detailed description of the embodiments of this application, specific scenario examples are given to illustrate the scenarios in which the technical solution of this application can be used. In one possible scenario example, specifically as follows... Figure 1 As shown, a user performs a specific operation on client 110, which is equipped with a data acquisition device. A target detection model is deployed on the server. The data acquisition device is used to acquire high-resolution images containing UI icons on client 110. The acquired high-resolution images containing UI icons need to be transmitted to the server via the network. The server segments the high-resolution images to obtain multiple image blocks. The size of each image block meets the input size of the target detection model. For each image block, the target detection model is input for recognition to obtain the icon detection result of the image block. The icon detection result includes at least one icon candidate box. The icon detection results of all image blocks are integrated and deduplicated to obtain the UI icon detection result of the high-resolution image.

[0029] The target detection model can run on the server side. The method of this application can be configured in the target detection model on the server, and the client terminal and the server establish a network transmission connection to realize multi-terminal interaction.

[0030] The server can be an electronic device with a certain computing power. For example, a server can be a distributed system server, or a system with multiple processors, memory, network communication modules, etc., working together. A server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, a server can also be a new technological means capable of realizing the corresponding functions of the embodiments described in the specification. For example, it could be a new form of "server" based on quantum computing.

[0031] The client terminal can be an electronic device with network access capabilities. Specifically, for example, the terminal can be a desktop computer, tablet computer, laptop computer, smartphone, etc. Alternatively, the terminal can also be software that can run on the electronic device.

[0032] The network can be of any type and can use any of the various available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. One or more networks can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0033] Example 1

[0034] Firstly, such as Figure 3 As shown, this embodiment provides a UI icon detection method, including: Step S1: Obtain a high-resolution image containing UI icons; The image block containing the high-resolution image of the UI icon is extracted from the first interface; the first interface refers to the interface where the target UI element is located during the RPA process design phase. In this embodiment, the image block containing the high-resolution image of the UI icon can be an image block extracted from the first interface during the design phase. Specifically, during the design phase, when designing the RPA process, the user selects a baseline UI element. This selected baseline UI element serves as the basis for matching in subsequent tasks and is used to match the target UI element during the runtime phase. The first interface represents the interface where the UI elements involved in the RPA process are located. The involved UI elements are also the target UI elements. Specifically, the first interface can be the display interface of an application. For example, in an RPA process related to the financial field, the first interface can be the display interface of accounting software, or the display interface of a CRM system, etc. The first interface can also be the main interface of an operating system. The first interface can also include both the display interface of at least one application and the main interface of the operating system. For example, in an RPA process related to the financial field, the display interface of the accounting software is a scaled-down display interface, so the first interface also includes partial image information of the display interfaces of other applications or partial image information of the main interface of the operating system.

[0035] In this embodiment, unlike text, the high-resolution image containing UI icons exhibits a wide range in size, with side lengths varying from approximately 4 to 80. Most icons have a side length around 32, while object detection algorithms typically consider targets with a side length of 32 to be small, making them difficult for general object detection algorithms to capture. Many icons have side lengths far less than 32, therefore, icons fall into the category of targets far smaller than small targets, necessitating enhancements to the detection algorithm to adapt to the detection of even smaller targets.

[0036] Step S2: The high-resolution image is segmented to obtain multiple image blocks; wherein the size of each image block meets the input size of the target detection model; Object detection algorithms typically use square images as input, and these square images have fixed side lengths, such as 640*640 or 1280*1280. The algorithm's preprocessing will forcibly format the original UI image into a square UI image with standard side lengths. The computer screens that users normally use are generally 1920*1280 or higher, and forcibly formatting a higher resolution image will further degrade the results of the algorithm, which is already not good at detecting small objects.

[0037] In this embodiment, segmenting the high-resolution image may include: The segmentation is performed along the width and height of the high-resolution image. If the input size of the target detection model is N*N, then the side length S of the segmentation window is set to N times A, and a stacking region is set between adjacent windows, the size of which is SN. If the width and / or height of the high-resolution image is less than the window side length S, or the side length of the image to which the last window is sliced ​​in the horizontal or vertical direction is less than S, then blank images are used to fill in the missing parts so that the side length of each window is S and the size of each image block is S*S.

[0038] Furthermore, in some embodiments, A is taken as 1.1 to 1.4, preferably 1.25.

[0039] In this embodiment, to make the high-resolution image more compatible with the detection algorithm's capabilities, the high-resolution image is segmented according to a certain size ( Figure 2 The segmented image blocks are then fed into an object detection algorithm for identification, and the detection results of all image blocks are integrated.

[0040] The high-resolution image is segmented along its width and height. Assuming the fixed input size of the object detection algorithm is N*N, the side length S of the segmented window is set to 1.25 times N. This multiplier can be adjusted according to the actual resolution of the scene. It's worth noting that although the default side length is N, if the scaling ratio of the original image doesn't change significantly, it won't affect the detection results, and the choice of 1.25 times satisfies this condition. Furthermore, if the height or width of the original image is less than S, or if the side length of the image segmented by the last window in either the horizontal or vertical direction is much smaller than S, blank images are used to fill in the missing parts.

[0041] Window segmentation may truncate some elements at the window boundaries, so a certain stacking interval is set between each window, with the stack size exactly the difference between S and N. Icons in the stacked area are detected repeatedly; an element truncated in the first window can be detected completely in the second window. The detection algorithm sequentially detects the icon elements on each segmented window. To integrate the icon results from these segmented windows, the icons in the stacked area need to be deduplicated. The deduplication method here is IOU (Intersection of Union), which uses the intersection-union ratio to remove candidate boxes of overlapping icons, retaining icons with larger areas to filter out truncated icons.

[0042] Step S3: For each image block, input the target detection model for recognition to obtain the icon detection result of the image block; wherein the icon detection result includes at least one icon candidate box; The most common models in the field of object detection are the YOLO series, with YOLO5 and YOLO8 being the most widely used. Here, we take the YOLO8-P6 structure as an example. The main structure of mainstream convolutional neural network detection algorithms, such as YOLO, consists of three parts: a backbone structure, a feature pyramid, and a prediction module. The feature pyramid is designed to fuse multi-granularity features, enhancing the detection algorithm's ability to identify small objects. However, for the scenario addressed in this application, its performance is still insufficient. For icon detection, which involves extremely small objects, the algorithm is primarily improved by enhancing the backbone structure and the feature pyramid.

[0043] Therefore, in some embodiments, the target detection model described in this application includes an improved backbone structure, an improved feature pyramid, and a prediction module.

[0044] Smaller icons tend to have more irregular aspect ratios. "Anchor points" are a unique mechanism in detection networks, regressing candidate boxes based on preset aspect ratios and sizes. Regardless of whether the target detection algorithm uses "anchor points" or "non-anchor points," irregular aspect ratios increase the instability of candidate box regression; this instability is more pronounced with smaller icons. Deformable convolutions, by altering the x- and y-axis offsets of the receptive field, allow the receptive field to change according to the target's geometric features, enabling the detection algorithm to better capture the actual geometric features of the target object. To enhance the backbone structure's ability to capture the target's actual contour, this application converts the convolutional layer used for feature extraction in the last stage of the backbone into a deformable convolution. To reduce computational cost, replacing more convolutional layers is not considered.

[0045] In some embodiments, the method for constructing the improved backbone structure includes: Deformable convolution of the S31 backbone structure; The convolutional layer used for feature extraction in the last stage of the backbone structure is converted into a deformable convolution; wherein the deformable convolution changes the receptive field according to the geometric features of the target by changing the offset position of the receptive field along the x-axis and y-axis, so that the actual geometric features of the target object can be captured better.

[0046] In some embodiments, taking YOLO8 as an example, one stage of the backbone structure includes a downsampling layer and a C2F module; for the downsampling layer, the convolutional layer with a 3*3 kernel and a step size of 2 is replaced with a deformable convolution; the C2F module is a structured feature extraction module (such as...). Figure 6 As shown in the diagram, the point convolutions at the beginning and end are used to adjust the number of channels, while the middle part exists in the "base" structure module in "DenseNet". C2F first divides the feature map in half along the channel dimension into two blocks. One block goes into one or two parallel feature extraction modules, which are composed of stacked convolutions and have residual structures. The extracted feature blocks are then glued to the original feature blocks. For the C2F module, the 3*3 convolutional layers used for feature extraction are replaced with deformable convolutions, while retaining the 1*1 point convolutions.

[0047] Attention mechanisms of the S32 keel structure; The attention mechanism in convolutional neural networks is used to extract effective information from feature maps from different dimensions. The extracted feature maps are more conducive to the extraction or segmentation of effective targets, and it is a common enhancement method in convolutional neural networks. Adding an attention module after the last stage of the backbone, which has the highest feature density and the most channels, allows for the most effective extraction of effective information.

[0048] In this embodiment, as Figure 7 As shown, an attention module is added after the last stage of the backbone structure. The attention module includes a spatial attention module and a channel attention module. The spatial attention module and the channel attention module are used to extract effective feature information from the spatial and channel dimensions. The residual structure in the last stage of the backbone structure is used to integrate the original features with the features extracted by the attention module.

[0049] In some embodiments, the attention module can be added to more locations. However, due to computing power limitations, it is only used once in the backbone structure in this embodiment.

[0050] The feature pyramid of S33's smallest targets; The original P6 pyramid structure integrates feature maps of different sizes from four stages of the backbone network, representing four different granularities of image features. The element pixel sizes of these four feature maps are 8*8, 16*16, 32*32, and 64*64, respectively. The smallest granularity of 8*8 cannot cover all icons; some icons are only about 4*4 pixels in size. These icons can be considered extremely small targets, and the original pyramid's feature granularity cannot satisfy these targets. Considering that the original UI image may be larger than the model's fixed input size, the actual extremely small targets may even be smaller than 4*4 pixels. To detect extremely small targets, further fusion of finer-grained feature maps is needed. In this embodiment, as Figure 8 As shown, the improved feature pyramid construction method includes: the original pyramid integrates feature maps of different sizes from four stages of the backbone network, and the element pixel sizes corresponding to the four feature maps are 8*8, 16*16, 32*32, and 64*64, respectively. To detect extremely small targets, the second-stage feature map with a pixel size of 4*4 of the backbone structure is incorporated into the feature pyramid, and the structure of the feature pyramid is expanded from top to bottom. The third-stage feature map, after being glued together in the original pyramid and processed by the C2 module, will be the output of the bottom layer of the pyramid. The fourth, fifth, and sixth-stage feature maps will be output in the same form. In this application, the output of the original third-stage feature map is removed, and the third-stage feature map is upsampled to make it the same size as the second-stage feature map. Then, the newly introduced second-stage feature map is glued together. At this time, the finest second-stage feature map has fused all the coarse-grained features of the upper layer. The second-stage feature map processed by the C2 module is the output of the bottom layer of the feature pyramid. After expanding the structure from top to bottom, the pyramid structure is then supplemented from bottom to top: the second-stage feature map, after convolution processing, is joined with the third-stage feature map, at which point the third-stage feature map incorporates more fine-grained features; the joined third-stage feature map, after being processed by the C2 module, serves as the output of the feature pyramid; sequentially upwards, the fourth to sixth-stage feature maps also incorporate the most fine-grained second-stage feature map; finally, the pyramid outputs five feature maps, enhancing the algorithm's detection of extremely small targets, while also improving the regression of candidate boxes for larger icons due to the fusion of more fine-grained features.

[0051] Step S4: Integrate and deduplicate the icon detection results of all image blocks to obtain the UI icon detection results of the high-resolution image.

[0052] In some embodiments, the deduplication method uses the Intersection over Union (IOU) to deduplicate icons in the stacked area. Specifically, it includes: using the IOU to remove overlapping icon candidate boxes, and retaining icons with larger areas during deduplication to filter out truncated icons.

[0053] After detecting extremely small targets (S1-S4), the algorithm can accurately identify a large number of UI icons of different sizes. However, a problem remains: local changes in the UI image, such as the appearance of an open menu bar or a floating window, can cause regression bias in the candidate bounding boxes of icons in certain locations. For example, the candidate bounding boxes of icons near a floating window may differ by a few pixels in UI images with and without a floating window. These small differences in the candidate bounding boxes can negatively impact downstream tasks, such as inaccurate icon matching.

[0054] To address the above problems, in some embodiments, the method may further include: step S5, after obtaining the icon detection result of the image block or the UI icon detection result of the high-resolution image, further including: if the icon candidate box is offset, correcting the icon position, specifically including: Based on the offset icon candidate box, the width and height of the icon image are enlarged by B times the length and width of the icon candidate box; For the enlarged icon image, a calibration algorithm is used to recapture and calibrate the position of the icon candidate box; the calibration algorithm includes edge detection, morphological transformation, and contour description. Figure 9 As shown.

[0055] In some embodiments, B is 1.1 to 1.4, preferably 1.25.

[0056] In some embodiments, edge detection is used to delineate the outline of icons within an expanded region; morphological transformations, such as closing and dilation, are used to enhance the connectivity of pixels in the edge detection feature image; and contour description delineates the contour boundaries of the morphologically transformed image and characterizes the position of new candidate boxes.

[0057] The candidate boxes processed by the calibration algorithm can fit closely to the icons themselves, and the calibration algorithm has a shorter processing time and consumes less computing power.

[0058] Example 2

[0059] Secondly, based on Example 1, such as Figure 4 As shown, this application provides a UI icon detection device, the device comprising: The acquisition module is used to: acquire high-resolution images containing UI icons; The segmentation module is used to segment the high-resolution image to obtain multiple image blocks; wherein the size of each image block meets the input size of the target detection model. The detection module is used to: input the target detection model into each image block for recognition, and obtain the icon detection result of the image block; wherein the icon detection result includes at least one icon candidate box; The integration module is used to integrate and deduplicate the icon detection results of all image blocks to obtain the UI icon detection results of the high-resolution image.

[0060] In some embodiments, the system further includes: a correction module, configured to: correct the icon position if the icon candidate box is offset, specifically including: Based on the offset icon candidate box, the width and height of the icon image are enlarged by B times the length and width of the icon candidate box; For the enlarged icon image, the position of the icon candidate box is recaptured and calibrated using a calibration algorithm; wherein the calibration algorithm includes edge detection, morphological transformation and contour description.

[0061] In summary, the UI icon detection method and related apparatus provided in this application segment a high-resolution image to obtain multiple image blocks; the size of each image block meets the input size of the target detection model; for each image block, the target detection model is input for recognition to obtain the icon detection result of the image block; the icon detection result includes at least one icon candidate box; the icon detection results of all image blocks are integrated and deduplicated to obtain the UI icon detection result of the high-resolution image.

[0062] In addition, the structure of the target detection model is improved in this application, thereby enabling the detection of extremely small targets and improving detection capability and accuracy; In a further embodiment, this application adds an icon position correction step to correct the position of icons that have shifted in the icon candidate box, further improving detection accuracy. Example 3 Based on Embodiment 1, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0063] Example 3

[0064] Based on Example 1, such as Figure 5 As shown, this embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in Embodiment 1.

[0065] Example 4

[0066] Based on Embodiment 1, this embodiment proposes a hardware system that applies the UI icon detection method described in Embodiment 1. This method is applied on the server side.

[0067] A terminal can be an electronic device with network access capabilities. For example, a terminal can be a desktop computer, tablet computer, laptop computer, smartphone, digital assistant, shopping guide terminal, television set, etc.

[0068] A server can be an electronic device with a certain computing power. For example, a server can be a distributed system server, or a system with multiple processors, memory, network communication modules, etc., working together. A server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, a server can also be a new technological means capable of realizing the corresponding functions of the implementation method described in the manual. For example, it can be a new form of "server" based on quantum computing.

[0069] Example 5

[0070] Based on Embodiment 1, this embodiment also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform a UI icon detection method as described in Embodiment 1.

[0071] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the implementation methods of this specification, and are not intended to limit the scope of this application.

[0072] It is understood that in the various embodiments described in this specification, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments described in this specification.

[0073] It is understood that the various implementation methods described in this specification can be implemented individually or in combination, and the implementation methods in this specification are not limited in this respect.

[0074] Unless otherwise stated, all technical and scientific terms used in the embodiments of this specification have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this specification. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0075] It is understood that the processor in the embodiments of this specification can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this specification. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this specification can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0076] It is understood that the memory in the embodiments of this specification may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0077] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0079] In the several embodiments provided in this specification, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0081] In addition, the functional units in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0082] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this specification, in essence, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting UI icons, characterized in that, The method includes: Get a high-resolution image containing UI icons; The high-resolution image is segmented to obtain multiple image blocks; the size of each image block meets the input size of the target detection model. For each image patch, the target detection model is input for recognition to obtain the icon detection result of the image patch; wherein the icon detection result includes at least one icon candidate box; The icon detection results of all image blocks are integrated and deduplicated to obtain the UI icon detection results of the high-resolution image; The target detection model includes an improved backbone structure, an improved feature pyramid, and a prediction module. The improved feature pyramid construction method includes: the original pyramid integrates feature maps of different sizes from four stages of the backbone network, with the element pixel sizes corresponding to the four feature maps being 8*8, 16*16, 32*32, and 64*64 respectively; to detect extremely small targets, the second-stage feature map with a pixel size of 4*4 from the backbone structure is integrated into the feature pyramid, and the feature pyramid structure is expanded from top to bottom; the third-stage feature map, after being glued together in the original pyramid and processed by the C2 module, is used as the output of the bottom layer of the pyramid; the fourth, fifth, and sixth-stage feature maps are output in the same form; this application removes the original third-stage feature map. The output of the feature map is upsampled to the third-stage feature map, making it the same size as the second-stage feature map. Then, the newly introduced second-stage feature map is merged. At this point, the finest-grained second-stage feature map incorporates all the coarse-grained features from the upper layers. The second-stage feature map processed by the C2 module is used as the output of the bottom layer of the feature pyramid. After expanding the structure from top to bottom, the pyramid structure is then supplemented from bottom to top: the second-stage feature map is merged with the third-stage feature map after convolution. At this point, the third-stage feature map incorporates even finer-grained features. The merged third-stage feature map is then processed by the C2 module and used as the output of the feature pyramid. This process continues upwards, with the fourth to sixth-stage feature maps also incorporating the finest-grained second-stage feature maps. Finally, the pyramid outputs five feature maps.

2. The UI icon detection method according to claim 1, characterized in that, Segmenting the high-resolution image includes: The segmentation is performed along the width and height of the high-resolution image. If the input size of the target detection model is N*N, then the side length S of the segmentation window is set to N times A, and a stacking region is set between adjacent windows, the size of which is SN. If the width and / or height of the high-resolution image is less than the window side length S, or the side length of the image to which the last window is sliced ​​in the horizontal or vertical direction is less than S, then blank images are used to fill in the missing parts so that the side length of each window is S and the size of each image block is S*S.

3. The UI icon detection method according to claim 2, characterized in that, A is taken as 1.1 to 1.

4.

4. The UI icon detection method according to claim 2, characterized in that, A is set to 1.

25.

5. The UI icon detection method according to claim 1, characterized in that, The improved method for constructing the backbone structure includes: (a) The convolutional layer used for feature extraction in the last stage of the backbone structure is converted into a deformable convolution; wherein the deformable convolution changes the receptive field according to the geometric features of the target by changing the offset position of the x-axis and y-axis of the receptive field, so that the actual geometric features of the target object can be captured better. (b) An attention module is added after the last stage of the backbone structure, wherein the attention module includes a spatial attention module and a channel attention module. The spatial attention module and the channel attention module are used to extract effective feature information from the spatial and channel dimensions. The residual structure in the last stage of the backbone structure is used to integrate the original features with the features extracted by the attention module.

6. The UI icon detection method according to claim 5, characterized in that, The convolutional layers used for feature extraction in the last stage of the backbone structure are converted into deformable convolutions. Taking YOLO8 as an example, one stage of the backbone structure includes a downsampling layer and a C2F module. For the downsampling layer, the convolutional layer with a 3*3 kernel and a step size of 2 is replaced with a deformable convolution. For the C2F module, the convolutional layer with a 3*3 kernel used for feature extraction is replaced with a deformable convolution, while retaining the point convolution with a 1*1 kernel.

7. The UI icon detection method according to claim 1, characterized in that, After obtaining the icon detection results for the image patch or the UI icon detection results for the high-resolution image, the process also includes: if the icon candidate box is offset, correcting the icon position, specifically including: Based on the offset icon candidate box, the width and height of the icon image are enlarged by B times the length and width of the icon candidate box; For the enlarged icon image, the position of the icon candidate box is recaptured and calibrated using a calibration algorithm; wherein the calibration algorithm includes edge detection, morphological transformation and contour description.

8. A UI icon detection device, characterized in that, The device includes: The acquisition module is used to: acquire high-resolution images containing UI icons; The segmentation module is used to segment the high-resolution image to obtain multiple image blocks; wherein the size of each image block meets the input size of the target detection model. The detection module is used to: input the target detection model into each image block for recognition, and obtain the icon detection result of the image block; wherein the icon detection result includes at least one icon candidate box; The integration module is used to integrate and deduplicate the icon detection results of all image blocks to obtain the UI icon detection results of the high-resolution image. The target detection model includes an improved backbone structure, an improved feature pyramid, and a prediction module. The improved feature pyramid construction method includes: the original pyramid integrates feature maps of different sizes from four stages of the backbone network, with the element pixel sizes corresponding to the four feature maps being 8*8, 16*16, 32*32, and 64*64 respectively; to detect extremely small targets, the second-stage feature map with a pixel size of 4*4 from the backbone structure is integrated into the feature pyramid, and the feature pyramid structure is expanded from top to bottom; the third-stage feature map, after being glued together in the original pyramid and processed by the C2 module, is used as the output of the bottom layer of the pyramid; the fourth, fifth, and sixth-stage feature maps are output in the same form; this application removes the original third-stage feature map. The output of the feature map is upsampled to the third-stage feature map, making it the same size as the second-stage feature map. Then, the newly introduced second-stage feature map is merged. At this point, the finest-grained second-stage feature map incorporates all the coarse-grained features from the upper layers. The second-stage feature map processed by the C2 module is used as the output of the bottom layer of the feature pyramid. After expanding the structure from top to bottom, the pyramid structure is then supplemented from bottom to top: the second-stage feature map is merged with the third-stage feature map after convolution. At this point, the third-stage feature map incorporates even finer-grained features. The merged third-stage feature map is then processed by the C2 module and used as the output of the feature pyramid. This process continues upwards, with the fourth to sixth-stage feature maps also incorporating the finest-grained second-stage feature maps. Finally, the pyramid outputs five feature maps.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images