User interface similar element identification analysis method and system, storage medium and electronic equipment

Abstract

The invention provides a user interface similar element identification and analysis method and system, a storage medium and electronic equipment. The method comprises the following steps of obtaining a rectangular region where a to-be-identified similar element is located in a user interface image; preprocessing the user interface image to obtain a user interface grayscale image; for different adjustment parameters, performing gray adjustment on each pixel in the user interface gray image to obtain a gray adjustment image; constructing a similar element rectangular region group for each gray level adjustment image; for each similar element rectangular region group, screening the contained similar element rectangular regions to obtain an effective similar element rectangular region group; and calculating a score value of the effective similar element rectangular region group. According to the user interface similar element identification analysis method and system, the storage medium and the electronic equipment, automatic identification of the similar elements of the user interface can be realized only by the screenshot, and the accuracy is high.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and relates to a method, system, storage medium and electronic device for identifying and analyzing similar elements in a user interface. Background Art

[0002] Identifying similar elements in a user interface (UI) is a very important task in many fields, including UI design, automated testing, and user experience analysis. In the prior art, when dealing with repeated elements in a user interface, such as list items, it still relies heavily on labor-intensive operations, such as manually clicking on the interface one by one, which is extremely time-consuming and inefficient.

[0003] In addition, the similar element recognition algorithms of traditional RPA tools all need to obtain the UI element tree of the page, which not only requires the installation of specific applications, resulting in cumbersome operations, but also faces increasingly stringent mobile phone permission restrictions and technical barriers. In particular, some models have strict control and installation requirements for third-party applications. In addition, many applications have adopted anti-debugging and anti-automation strategies, resulting in the inability to obtain page elements within the application, completely blocking the path for automated operations through the UI element tree, and severely restricting the effectiveness and scope of application of traditional technologies. Summary of the invention

[0004] The present invention provides a method, system, storage medium and electronic device for identifying and analyzing similar elements of a user interface, which can realize automatic identification of similar elements of a user interface with high accuracy only by taking screenshots.

[0005] In a first aspect, the present invention provides a method for identifying and analyzing similar elements in a user interface, the method comprising the following steps: obtaining a rectangular area in a user interface image where similar elements to be identified are located; preprocessing the user interface image to obtain a grayscale image of the user interface; performing grayscale adjustment on each pixel in the grayscale image of the user interface for different adjustment parameters to obtain a grayscale adjusted image; for each grayscale adjusted image, calculating a pixel connected domain with a pixel value of 255 in the grayscale adjusted image, constructing a similar element rectangular area based on the pixel connected domain, and constructing the obtained similar element rectangular area into a similar element rectangular area group; for each similar element rectangular area group, screening the contained similar element rectangular areas to obtain a valid similar element rectangular area group; and calculating a score value of the valid similar element rectangular area group based on the rectangular area where the similar elements to be identified are located.

[0006] In an implementation of the first aspect, preprocessing the user interface image to obtain a user interface grayscale image includes the following steps:

[0007] Converting the user interface image into a grayscale image;

[0008] Extending a preset pixel size around the rectangular area of ​​the grayscale image to obtain a rectangular ring;

[0009] Clustering the grayscale values ​​of the pixels within the rectangular ring to divide them into three cluster sets, and taking the two cluster sets with the first and second largest number of pixels as target sets;

[0010] For the grayscale image, the grayscale values ​​of pixels whose grayscale values ​​belong to the target set are set to 0, and the grayscale values ​​of pixels that do not belong to the target set are set to 255, so as to obtain the user interface grayscale image.

[0011] In an implementation of the first aspect, grayscale adjustment is performed on each pixel in the user interface grayscale image, and obtaining the grayscale adjusted image includes the following steps:

[0012] When there is a pixel with a grayscale value of 255 in an area with a radius of k pixels around the pixel, the grayscale value of the pixel is set to 255, where k is an adjustment parameter;

[0013] When there is a pixel with a grayscale value of 0 in an area with a radius of k pixels around the pixel, the grayscale value of the pixel is set to 0.

[0014] In an implementation manner of the first aspect, for different adjustment parameters, the initial value of the adjustment parameter k is 5, and increases by 2 each time the adjustment is performed until the adjustment parameter is 151.

[0015] In an implementation of the first aspect, for each similar element rectangular region group, screening the contained similar element rectangular regions to obtain a valid similar element rectangular region group includes the following steps:

[0016] When the difference in left distance between the similar element rectangular area and the rectangular area where the to-be-identified similar element is located is greater than a first preset number of pixels, the similar element rectangular area is discarded;

[0017] When the width of the similar element rectangular area is less than a second preset number of pixels or the height of the similar element rectangular area is less than a second preset number of pixels, discard the similar element rectangular area;

[0018] When two similar element rectangular regions overlap, the similar element rectangular region having a smaller similarity to the rectangular region where the similar element to be identified is located is discarded;

[0019] The valid similar element rectangular area group is formed based on the remaining similar element rectangular areas.

[0020] In an implementation of the first aspect, based on the rectangular area where the similar elements to be identified are located, calculating the score value of the valid similar element rectangular area group includes the following steps:

[0021] For each valid similar element rectangular region group, the parameters of the valid similar element rectangular regions included therein are input into a neural network to obtain a similarity value between the valid similar element rectangular regions and the rectangular region where the similar element to be identified is located;

[0022] The sum of the similarity values ​​of all valid similar element rectangular regions is used as the score value of the valid similar element rectangular region group.

[0023] In an implementation of the first aspect, the parameters include one or more combinations of the square difference in area, square difference in length, square difference in width, square difference in left margin, square difference in right margin, right margin mean square error, structural similarity, and cosine similarity between the rectangular area of ​​the valid similar elements and the rectangular area where the similar elements to be identified are located.

[0024] In a second aspect, the present invention provides a user interface similar element recognition and analysis system, the system comprising a first acquisition module, a second acquisition module, an adjustment module, a calculation module, a screening module and a scoring module;

[0025] The first acquisition module is used to acquire a rectangular area where similar elements to be identified are located in the user interface image;

[0026] The second acquisition module is used to preprocess the user interface image to acquire a user interface grayscale image;

[0027] The adjustment module is used to perform grayscale adjustment on each pixel in the user interface grayscale image according to different adjustment parameters to obtain a grayscale adjusted image;

[0028] The calculation module is used to calculate, for each grayscale adjusted image, a pixel connected domain with a pixel value of 255 in the grayscale adjusted image, construct a similar element rectangular region based on the pixel connected domain, and construct the obtained similar element rectangular region into a similar element rectangular region group;

[0029] The screening module is used to screen the similar element rectangular areas contained in each similar element rectangular area group to obtain a valid similar element rectangular area group;

[0030] The scoring module is used to calculate the scoring value of the valid similar element rectangular area group based on the rectangular area where the similar elements to be identified are located.

[0031] In a third aspect, the present invention provides an electronic device, the electronic device comprising: a processor and a memory;

[0032] The memory is used to store computer programs;

[0033] The processor is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned user interface similar element recognition and analysis method.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned method for identifying and analyzing similar elements in a user interface.

[0035] As described above, the method, system, storage medium and electronic device for identifying and analyzing similar elements in a user interface according to the present invention have the following beneficial effects:

[0036] (1) Automatically identify similar elements of the user interface based on screenshots alone, breaking away from the limitations of traditional reliance on UI element tree parsing and without the need to install any applications on the target device;

[0037] (2) The accuracy of similar element recognition is high, and no complicated settings are required, and no knowledge of the user's device system is required;

[0038] (3) It is highly compatible. Almost all smartphones support screenshots and screen projection, and there are almost no restrictions on screenshots and screen projection. Therefore, it is easy to obtain screen images to locate similar elements, so that users can perform automated interactive operations and information recording later.

[0039] (4) When collecting evidence based on similar elements of user pages, it can greatly reduce the workload of evidence collectors and improve the versatility and compliance of the method; it not only greatly improves the efficiency and accuracy of evidence collection, but also provides strong technical support for ensuring the rapid locking and lossless collection of electronic evidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Shown is a schematic diagram of a scene of an electronic device in an embodiment of the present invention;

[0041] Figure 2 Shown is a flow chart of a method for identifying and analyzing similar elements in a user interface according to an embodiment of the present invention;

[0042] Figure 3 Shown is a flow chart of obtaining a grayscale image of a user interface in one embodiment of the present invention;

[0043] Figure 4 Shown is a schematic diagram of screening of rectangular regions of similar elements in one embodiment of the present invention;

[0044] Figure 5Shown is a schematic diagram of the structure of a user interface similar element recognition and analysis system in one embodiment of the present invention;

[0045] Figure 6 It is a schematic structural diagram of an electronic device of the present invention in one embodiment. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0048] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] The following embodiments of the present invention provide an instance segmentation method, which can be applied to Figure 1 The electronic device shown. The electronic device described in the present invention may include a mobile phone 11 with a wireless charging function, a tablet computer 12, a laptop computer 13, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present invention does not impose any restrictions on the specific type of the electronic device.

[0050] For example, the electronic device may be a station (STAION, ST) in a WLAN with a wireless charging function, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with a wireless charging function, a computing device or other processing device, a computer, a laptop computer, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system and a next generation communication system, such as a mobile terminal in a 5G network, a mobile terminal in a future evolved Public Land Mobile Network (PLMN), or a mobile terminal in a future evolved Non-terrestrial Network (NTN).

[0051] For example, the electronic device can communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobilecommunication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS can include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation Systems (SBAS).

[0052] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0053] like Figure 2 As shown, in one embodiment, the user interface similar element recognition and analysis method of the present invention includes steps S1 to S6.

[0054] Step S1: Obtain a rectangular area in a user interface image where similar elements to be identified are located.

[0055] Specifically, for the user interface of the smart terminal, the user interface image is obtained by screen projection, screenshot, etc. The user interface image is displayed to the user, and the user selects a similar element to be identified through a rectangular circle, thereby obtaining the rectangular area where the similar element to be identified is located based on the user's selection operation.

[0056] It should be noted that the smart terminal in the present invention refers to a general term for devices that have independent operating systems, can be installed by users themselves, such as software, games and other programs provided by third-party service providers, and can continuously expand the functions of the smart terminal through such programs, and can achieve wireless network access through mobile communication networks. The smart terminal includes smart phones, tablet computers, smart wearable devices, etc. The operating system used by the smart terminal can be an Android operating system or an iOS operating system. The present invention does not make specific restrictions.

[0057] Step S2: pre-process the user interface image to obtain a user interface grayscale image.

[0058] Specifically, Figure 3 As shown, preprocessing the user interface image to obtain a user interface grayscale image includes the following steps:

[0059] 21) Convert the user interface image into a grayscale image.

[0060] 22) Extending a preset pixel size around the rectangular area of ​​the grayscale image to obtain a rectangular ring.

[0061] Wherein, on the grayscale image, the rectangular area is extended outward by a preset pixel size, such as 10 pixels, to form a rectangular ring.

[0062] 23) Clustering the grayscale values ​​of pixels within the rectangular ring to divide them into three cluster sets, and taking the two cluster sets with the first and second largest number of pixels as target sets.

[0063] Among them, the pixel gray values within the rectangular ring are clustered based on the k-Means algorithm, and the pixel gray values within the rectangular ring are divided into three clustering combinations. At the same time, the center points and clustering radii of two clusters with the first and second numbers of pixels are obtained. That is, the two cluster centers are j1 and j2, and the two clustering radii are r1 and r2.

[0064] 24) For the grayscale image, set the gray value of the pixels whose gray values belong to the target set to 0, and set the gray values of the pixels that do not belong to the target set to 255 to obtain the user interface grayscale image.

[0065] Among them, in the grayscale image, if the gray value of a certain pixel belongs to the target set, the gray value of this pixel is set to 0; if the gray value of a certain pixel does not belong to the target set, the gray value of this pixel is set to 255. Set the pixel value of a certain pixel to v. When j1 - r1 < v < j1 + r1 or j2 - r2 < v < j2 + r2, it indicates that the gray value of this pixel belongs to the target set, then the gray value v of this pixel is set to 0; otherwise v is set to 255. After the above gray value setting, the user interface grayscale image can be obtained.

[0066] Step S3: For different adjustment parameters, perform gray level adjustment on each pixel in the user interface grayscale image to obtain a gray level adjusted image.

[0067] Specifically, set an adjustment parameter k, where k is a natural number. Preferably, the initial value of the adjustment parameter k is 5, and it is increased by 2 each time until the adjustment parameter is 151. That is, k takes values of 5, 7, 9... 151.

[0068] For each adjustment parameter k, traverse each pixel of the user interface grayscale image. When there is a pixel with a gray value of 255 in the area with a radius of k pixels around the pixel, set the gray value of the pixel to 255; when there is a pixel with a gray value of 0 in the area with a radius of k pixels around the pixel, set the gray value of the pixel to 0. Therefore, for each adjustment parameter, a gray level adjusted image can be obtained.

[0069] Step S4: For each gray level adjusted image, calculate the pixel connected domain with a pixel value of 255 in the gray level adjusted image, construct a similar element rectangular area based on the pixel connected domain, and construct the obtained similar element rectangular areas into a group of similar element rectangular areas.

[0070] Specifically, for each grayscale adjusted image, a pixel connected domain with a pixel value of 255 in the grayscale adjusted image is calculated, the top, bottom, left and right sides of the pixel connected domain are obtained, and a similar element rectangular region is constructed with the top, bottom, left and right sides as boundaries. All similar element rectangular regions in the grayscale adjusted image are constructed as the similar element rectangular region group.

[0071] Step S5: For each similar element rectangular region group, the included similar element rectangular regions are screened to obtain a valid similar element rectangular region group.

[0072] Specifically, the similar element rectangular regions in the similar element rectangular region group may not meet the conditions, so screening is required to discard similar element rectangular regions that do not meet the conditions. Figure 4 As shown, when the difference in the left distance between the similar element rectangular area and the rectangular area where the similar element to be identified is located is greater than a first preset number of pixels (such as 20 pixels), the similar element rectangular area is discarded. When the width of the similar element rectangular area is less than a second preset number of pixels (such as 5 pixels) or the height of the similar element rectangular area is less than a second preset number of pixels, the similar element rectangular area is discarded. When two similar element rectangular areas overlap, the similar element rectangular area with a smaller similarity to the rectangular area where the similar element to be identified is located is discarded, wherein the similarity can be structural similarity or cosine similarity. Finally, the valid similar element rectangular area group is formed based on the remaining similar element rectangular areas.

[0073] Step S6: Calculate the score value of the valid similar element rectangular area group based on the rectangular area where the similar elements to be identified are located.

[0074] Specifically, for each valid similar element rectangular area group, the parameters of the valid similar element rectangular area included are input into the neural network to obtain the similarity value between the valid similar element rectangular area and the rectangular area where the similar element to be identified is located; the sum of the similarity values ​​of all valid similar element rectangular areas is used as the score value of the valid similar element rectangular area group. Among them, the neural network shown uses a multi-layer perceptron. The parameters include the square difference of the area, square difference of the length, square difference of the width, square difference of the left margin, square difference of the right margin, mean square difference of the right margin, structural similarity, cosine similarity, etc. between the valid similar element rectangular area and the rectangular area where the similar element to be identified is located.

[0075] For each valid similar element rectangular area group, the score values ​​are arranged in descending order, and each valid similar element rectangular area group is sequentially displayed on the user interface image to be presented to the user. The user can select a valid similar element rectangular area group by himself to perform subsequent operations.

[0076] The protection scope of the user interface similar element recognition and analysis method described in the embodiment of the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0077] An embodiment of the present invention also provides a user interface similar element recognition and analysis system, which can implement the user interface similar element recognition and analysis method described in the present invention. However, the implementation device of the user interface similar element recognition and analysis system described in the present invention includes but is not limited to the structure of the user interface similar element recognition and analysis system listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0078] like Figure 5 As shown, in one embodiment, the user interface similar element recognition and analysis system of the present invention includes a first acquisition module 51 , a second acquisition module 52 , an adjustment module 53 , a calculation module 54 , a screening module 55 and a scoring module 56 .

[0079] The first acquisition module 51 is used to acquire a rectangular area where similar elements to be identified are located in the user interface image.

[0080] The second acquisition module 52 is connected to the first acquisition module 51 and is used to pre-process the user interface image to acquire a user interface grayscale image.

[0081] The adjustment module 53 is connected to the second acquisition module 52 and is used to perform grayscale adjustment on each pixel in the user interface grayscale image according to different adjustment parameters to obtain a grayscale adjusted image.

[0082] The calculation module 54 is connected to the adjustment module 53, and is used to calculate the pixel connected domain with a pixel value of 255 in the grayscale adjusted image for each grayscale adjusted image, construct a similar element rectangular area based on the pixel connected domain, and construct the obtained similar element rectangular area into a similar element rectangular area group.

[0083] The screening module 55 is connected to the calculation module 54 and is used for screening the similar element rectangular areas contained in each similar element rectangular area group to obtain a valid similar element rectangular area group.

[0084] The scoring module 56 is connected to the screening module 55 and is used to calculate the scoring value of the valid similar element rectangular area group based on the rectangular area where the similar elements to be identified are located.

[0085] Among them, the structures and principles of the first acquisition module 51, the second acquisition module 52, the adjustment module 53, the calculation module 54, the screening module 55 and the scoring module 56 correspond one by one to the steps in the above-mentioned user interface similar element identification and analysis method, so they are not repeated here.

[0086] In the several embodiments provided by the present invention, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0087] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0088] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0089] The embodiment of the present invention also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0090] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory.

[0091] The memory is used to store computer programs.

[0092] The memory includes: ROM, RAM, disk, USB flash drive, memory card or CD and other media that can store program codes.

[0093] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device executes the above-mentioned EEG signal acquisition method.

[0094] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0095] like Figure 6As shown, the electronic device of the present invention is in the form of a general computing device. The components of the electronic device may include but are not limited to: one or more processors or processing units 61, a memory 62, and a bus 63 connecting different system components (including the memory 62 and the processing unit 61).

[0096] Bus 63 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0097] Electronic devices typically include a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0098] The memory 62 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 621 and / or cache memory 622. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 623 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 63 via one or more data medium interfaces. The memory 62 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0099] A program / utility 624 having a set (at least one) of program modules 6241 may be stored, for example, in the memory 62, such program modules 6241 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 6241 generally perform the functions and / or methods of the embodiments described herein.

[0100] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, displays, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via input / output (I / O) interface 64. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 65. Figure 6 As shown, the network adapter 65 communicates with other modules of the electronic device via the bus 63. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0101] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for identifying and analyzing similar elements in a user interface, characterized in that: The method comprises the following steps: Obtain the rectangular area where the similar elements to be identified are located in the user interface image; Preprocessing the user interface image to obtain a user interface grayscale image; According to different adjustment parameters, grayscale adjustment is performed on each pixel in the grayscale image of the user interface to obtain a grayscale adjusted image; For each grayscale adjusted image, a pixel connected domain with a pixel value of 255 in the grayscale adjusted image is calculated, a similar element rectangular region is constructed based on the pixel connected domain, and the obtained similar element rectangular regions are constructed into a similar element rectangular region group; For each similar element rectangular region group, the contained similar element rectangular regions are screened to obtain a valid similar element rectangular region group; Based on the rectangular area where the similar elements to be identified are located, the score value of the valid similar element rectangular area group is calculated.

2. The method for identifying and analyzing similar elements in a user interface according to claim 1, characterized in that: Preprocessing the user interface image to obtain a user interface grayscale image includes the following steps: Converting the user interface image into a grayscale image; Extending a preset pixel size around the rectangular area of ​​the grayscale image to obtain a rectangular ring; Clustering the grayscale values ​​of the pixels within the rectangular ring to divide them into three cluster sets, and taking the two cluster sets with the first and second largest number of pixels as target sets; For the grayscale image, the grayscale values ​​of pixels whose grayscale values ​​belong to the target set are set to 0, and the grayscale values ​​of pixels that do not belong to the target set are set to 255, so as to obtain the user interface grayscale image.

3. The method for identifying and analyzing similar elements in a user interface according to claim 1, characterized in that: Performing grayscale adjustment on each pixel in the grayscale image of the user interface, and obtaining the grayscale adjusted image comprises the following steps: When there is a pixel with a grayscale value of 255 in an area with a radius of k pixels around the pixel, the grayscale value of the pixel is set to 255, where k is an adjustment parameter; When there is a pixel with a grayscale value of 0 in an area with a radius of k pixels around the pixel, the grayscale value of the pixel is set to 0.

4. The method for identifying and analyzing similar elements in a user interface according to claim 1, characterized in that: For different adjustment parameters, the initial value of the adjustment parameter k is 5, and increases by 2 each time the adjustment is made until the adjustment parameter is 151.

5. The method for identifying and analyzing similar elements in a user interface according to claim 1, characterized in that: For each similar element rectangular region group, the similar element rectangular regions contained therein are screened, and obtaining a valid similar element rectangular region group includes the following steps: When the difference in left distance between the similar element rectangular area and the rectangular area where the to-be-identified similar element is located is greater than a first preset number of pixels, the similar element rectangular area is discarded; When the width of the similar element rectangular area is less than a second preset number of pixels or the height of the similar element rectangular area is less than a second preset number of pixels, discard the similar element rectangular area; When two similar element rectangular regions overlap, the similar element rectangular region having a smaller similarity to the rectangular region where the similar element to be identified is located is discarded; The valid similar element rectangular area group is formed based on the remaining similar element rectangular areas.

6. The method for identifying and analyzing similar elements in a user interface according to claim 1, characterized in that: Based on the rectangular area where the similar elements to be identified are located, calculating the score value of the valid similar element rectangular area group comprises the following steps: For each valid similar element rectangular region group, the parameters of the valid similar element rectangular regions included in the group are input into a neural network to obtain a similarity value between the valid similar element rectangular region and the rectangular region where the similar element to be identified is located; The sum of the similarity values ​​of all valid similar element rectangular regions is used as the score value of the valid similar element rectangular region group.

7. The method for identifying and analyzing similar elements in a user interface according to claim 6, characterized in that: The parameters include one or more combinations of the square difference in area, square difference in length, square difference in width, square difference in left margin, square difference in right margin, right margin mean square difference, structural similarity, and cosine similarity between the rectangular area of ​​the valid similar element and the rectangular area where the similar element to be identified is located.

8. A user interface similar element recognition and analysis system, characterized by: The system includes a first acquisition module, a second acquisition module, an adjustment module, a calculation module, a screening module and a scoring module; The first acquisition module is used to acquire a rectangular area where similar elements to be identified are located in the user interface image; The second acquisition module is used to preprocess the user interface image to acquire a user interface grayscale image; The adjustment module is used to perform grayscale adjustment on each pixel in the user interface grayscale image according to different adjustment parameters to obtain a grayscale adjusted image; The calculation module is used for calculating, for each grayscale adjusted image, a pixel connected domain with a pixel value of 255 in the grayscale adjusted image, constructing a similar element rectangular region based on the pixel connected domain, and constructing the obtained similar element rectangular region into a similar element rectangular region group; The screening module is used to screen the similar element rectangular areas contained in each similar element rectangular area group to obtain a valid similar element rectangular area group; The scoring module is used to calculate the scoring value of the valid similar element rectangular area group based on the rectangular area where the similar elements to be identified are located.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for identifying and analyzing similar elements of a user interface according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the electronic device executes the user interface similar element recognition and analysis method according to any one of claims 1 to 7.

Images