UI semantic recognition method and device, equipment and storage medium

Abstract

The invention discloses a UI semantic recognition method and device, equipment and a storage medium, and belongs to the technical field of UI element recognition. The method comprises the steps of obtaining basic information of UI elements in a target UI image; processing the target UI image by adopting an edge detection algorithm to obtain a UI image with a region contour; constructing an image pyramid based on the UI image with the region contour; wherein the image pyramid comprises images of a plurality of layers, and the UI image with the regional contour is the image of the bottommost layer; and based on the image pyramid, endowing the basic information of the UI element with associated semantic information. The semantic hierarchical relationship of the UI elements in the UI interface is established, and effective UI interface semantic recognition is realized. Furthermore, by constructing the image pyramid, the technical problem of low region detection accuracy in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to a UI interface semantic recognition method, device, equipment and storage medium, belonging to the technical field of UI element recognition. Background Art

[0002] To achieve hyper-automation, we first need to solve the problem of UI element recognition, that is, we need to identify the elements on the UI interface and their relationships. Only by solving this problem can we realize the control of PRA robots based on natural language instructions to automatically complete automation tasks based on the UI interface, or other related tasks in hyper-automation scenarios.

[0003] Therefore, a UI interface semantic recognition method is needed that can recognize elements on the UI interface and their relationships. Summary of the invention

[0004] Purpose: In order to overcome the deficiencies in the prior art, the present invention provides a UI interface semantic recognition method, device, equipment and storage medium, which can realize the recognition of elements and their relationships on the UI interface.

[0005] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0006] In a first aspect, the present invention provides a method for UI interface semantic recognition, the method comprising:

[0007] Obtaining basic information of UI elements in the target UI image; wherein the basic information is information that can be directly extracted from the target UI image using a target detection algorithm;

[0008] For the target UI image, an edge detection algorithm is used to process the target UI image to obtain a UI image with a regional outline;

[0009] Based on the UI image with the region outline, an image pyramid is constructed; wherein the image pyramid includes multiple layers of images, and the UI image with the region outline is the image at the bottom layer;

[0010] Based on the image pyramid, associated semantic information is assigned to basic information of the UI element; wherein the associated semantic information is used to represent information on the association relationship between UI elements and between UI elements and regions.

[0011] In a second aspect, the present invention provides a UI interface semantic recognition device, the device comprising:

[0012] An acquisition module, which is used to acquire basic information of UI elements in a target UI image; wherein the basic information is information that can be directly extracted using a target detection algorithm;

[0013] A detection module, which is used to process the target UI image using an edge detection algorithm to obtain a UI image with a regional outline;

[0014] A construction module, which is used to construct an image pyramid based on the UI image with the region outline; wherein the image pyramid includes multiple layers of images, and the UI image with the region outline is the image at the bottom layer;

[0015] An assignment module is used to assign associated semantic information to basic information of UI elements based on the image pyramid; wherein the associated semantic information is used to represent information on the association relationship between UI elements and between UI elements and regions.

[0016] In a third aspect, the present invention provides a device, comprising:

[0017] Memory;

[0018] processor;

[0019] as well as

[0020] Computer programs;

[0021] The computer program is stored in the memory and is configured to be executed by the processor to implement the method described in the first aspect above.

[0022] In a fourth aspect, the present invention provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0023] Beneficial effects: The UI interface semantic recognition method, device, equipment and storage medium provided by the present invention have the following advantages:

[0024] 1. The present invention proposes a UI interface semantic recognition method that can obtain associated semantic information, establishes a semantic hierarchical relationship of UI elements in the UI interface, and realizes effective UI interface semantic recognition.

[0025] 2. Based on 1, the present invention solves the technical problem of low accuracy of region detection in the prior art by constructing an image pyramid. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a method for UI interface semantic recognition according to an embodiment of the present invention;

[0027] Figure 2 A block diagram of a UI interface recognition device provided according to an embodiment of the present invention;

[0028] Figure 3The present invention is a block diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0030] Among the related technologies, the purpose of interface semantic recognition is to use some methods, such as deep learning models, to identify UI elements, components and different UI elements in the screenshot, as well as the relationships between them, given an image of a UI interface.

[0031] Considering the existing UI design specifications, the elements in the UI interface do not exist in isolation, and there are some associations between UI elements. In other words, for UI elements, in addition to the information expressed by themselves, they also have other information. For example, a UI element with a rectangular shape and a prompt text of "Submit" can be considered as a submit button, in which the shape information of the UI element and the text information of the prompt text in the UI element are the information expressed by the UI element itself. It can be defined as basic information. For other information, it can be understood that in addition to the information expressed by the UI element's own shape or text, the area where the UI element is located in the UI image and the area level can also represent the semantic information of the UI element. For example, the above-mentioned submit button is in the form area, and the text information of other UI elements in the form area can only be saved after clicking the submit button; each form area has a UI element for representing the title information. In the form area, the remaining UI elements except the UI element representing the title information can be understood as the form attributes of the title information.

[0032] Therefore, in interface semantic recognition, it is very important to understand the relationship between UI elements. To understand this relationship, one feasible way is to obtain it through the hierarchical relationship between various regions in the UI image. The hierarchical relationship between regions can be understood as the relationship of inclusion and being included. Because, in UI design, UI elements are all in one region. To clarify the hierarchical relationship of regions in the UI interface, it is first necessary to accurately identify each region in the UI interface.

[0033] However, in some existing region detection schemes, it is impossible to accurately extract the regions in the UI interface due to interference factors such as watermarks, shadows, thin lines at the edge of the region, and too dense UI elements. Because it is impossible to accurately extract regions in UI images, especially large regions, it is impossible to effectively build a hierarchical relationship between regions, and then it is impossible to clarify the relationship between UI elements, making it impossible to achieve effective UI interface semantic recognition.

[0034] In this embodiment, a method for semantic recognition of a UI interface is proposed, in which an image pyramid carrying regional hierarchical information is constructed based on a UI image with regional contours; in the image pyramid, the UI image with regional contours is the bottom-level image, and the remaining images are obtained by multiple downsampling of the bottom-level image. This method is used to remove the above-mentioned interference factors, so that the regional contours in the UI interface can be accurately extracted, and then accurate regional hierarchical information can be obtained, and effective semantic recognition of the UI interface can be achieved based on the accurate regional hierarchical information.

[0035] Example 1

[0036] First, as Figure 1 As shown, this embodiment provides a UI interface semantic recognition method, including:

[0037] Step S100, obtaining basic information of UI elements in the target UI image; wherein the basic information is information that can be directly extracted from the target UI image using a target detection algorithm;

[0038] In this embodiment, the target UI image may be a UI image in a terminal or mobile terminal, a screenshot of any application, or a screenshot of a browser page. The basic information may include: category information and location information of UI elements, and text information.

[0039] Get the basic information of the UI elements in the target UI image, which may include:

[0040] Step S101: Detect UI elements of a target UI image using a target detection algorithm to obtain category information and location information of the UI elements; wherein the UI elements include icon UI elements and text UI elements.

[0041] In the present embodiment, the above-mentioned target algorithm can adopt the target detection algorithm of the YOLO series. For example, a pre-trained YOLOv5 model can be used to identify the target UI image, and the pre-trained YOLOv5 model can be trained using label data with text UI elements and icon UI elements. Text UI elements can be Chinese characters, or text elements of other languages, such as English. And text elements can be single characters or multiple characters. Icon UI elements represent graphics with a size less than a certain size limit, such as the size limit can be 64*64. Icon UI elements can be used to represent a certain function, state or concept. Icon UI elements can be used to form more complex UI elements such as buttons, menus and toolbars in UI images.

[0042] Step S102: for the text UI element, identify the text information of the UI element by using OCR.

[0043] In this embodiment, the above-mentioned OCR method is used to recognize the text information of the UI element. For example, a general OCR recognition model can be used, such as Google OCR engine, Baidu OCR engine, Baidu open source PPOCR and SenseTime MMOCR.

[0044] Step S200: The target UI image is processed by an edge detection algorithm to obtain a UI image with a region outline.

[0045] In this embodiment, considering a specific application scenario, the target UI image may be a UI image of a chat application main interface. The UI image of the chat application main interface may be defined as an original image. After the original image is converted into a corresponding grayscale image, the canny detection algorithm with a kernel of 5*5 may be used to perform boundary contour detection. Specifically, the step may include:

[0046] Step S201: convert the above original image into a corresponding grayscale image.

[0047] Step S202: Use the canny detection algorithm with a kernel of 5*5 to detect the region contour in the grayscale image.

[0048] In this implementation method, the canny detection algorithm with a kernel of 5*5 can better distinguish the boundary and background than the 3*3 in the prior art, and better reflect the regional contour than the binarization in the traditional method.

[0049] Step S203: Using a morphological transformation method based on a closing operation to enhance the continuity of the region contour in step S202.

[0050] In this embodiment, the area contour detected in step S202 may not be able to optimally express the area boundary information in the UI image. Therefore, some optimization operations are required. Therefore, in this embodiment, the edge detection result is enhanced by using the closing operation in the morphological transformation, that is, image expansion first, and then image erosion, so that the UI boundary contour is more continuous. The closing operation uses a 3*3 kernel, a cross kernel. The transformed contour information is more cohesive and the boundary sorting is stronger.

[0051] Step S300: construct an image pyramid based on the UI image with the region outline; wherein the image pyramid includes multiple layers of images, and the UI image with the region outline is the image at the bottom layer.

[0052] In this embodiment, the image pyramid may have multiple layers, such as three layers, five layers, etc. It should be noted that except for the image at the bottom layer, the images at the remaining layers are obtained by downsampling based on the image at the next layer. In some embodiments, other image processing methods may be used to make the image at the next layer lack some image information relative to the image at the previous layer, such as some image smoothing and denoising processing. However, in this embodiment, the downsampling operation is preferred.

[0053] In this implementation, step S300 may include:

[0054] Step S301: perform contour extraction on the bottom layer image to obtain a corresponding region contour; wherein the region contour is used to represent the position information of the region edge in the image.

[0055] In this embodiment, the above-mentioned bottom-level image can be an image after the morphological transformation method based on closing operation is executed in step S203, or it can be an image after further performing an image corrosion operation on the image output in step S203. The corrosion in the morphological transformation can be used to effectively remove noise points in the image. The corrosion operation can use a 1*1 kernel and iterate 3 times.

[0056] In this embodiment, the contour extraction operation is to extract the contours of each region in the bottom layer of the image, and convert them into boundary information through numerical calculation, such as calculating the maximum rectangular boundary of the contour. The boundary information is also the position information of the region contour in the image, which can be understood as a collection of multiple position coordinates. Specifically, the contour extraction can be performed using (cv2.findcontours) in the opencv library.

[0057] Step S302: performing a downsampling operation on the bottom layer image to obtain an upper layer image of the bottom layer image, and then performing contour extraction on the upper layer image to obtain a corresponding region contour.

[0058] Step S303, repeatedly performing the above-mentioned downsampling and contour extraction operations to obtain the region contours corresponding to the images of different layers in the image pyramid; wherein the image of the previous layer is obtained by performing a downsampling operation on the image of the next layer.

[0059] In this embodiment, the purpose of performing the downsampling operation is to reduce the resolution of the image to reduce the number of pixels in the image. As mentioned above, UI images often have a variety of interferences that make the regional detection inaccurate, especially the large area detection inaccurate. For example, the watermark in the image, which is often irregular, presents a font or symbol with a certain inclination, and will pass through multiple areas, resulting in multiple areas, especially large areas. In addition to the influence of the watermark, interference factors such as the size of the lines at the edge of the shadow and the edge of the area, such as too thin size, too dense UI elements, etc., will also affect the accurate recognition of the area in the UI image. This embodiment gradually filters out the interference factors that affect the detection of regional edges by constructing an image pyramid.

[0060] In this embodiment, after executing steps S301 to S303 for the above-mentioned image pyramid, a plurality of region outlines of different sizes will be obtained, among which some region outlines are exactly the same and overlap, and some small region outlines exist in the large region outline. Therefore, the image pyramid can be used to characterize the hierarchical relationship between regions. The hierarchical relationship of the regions can be an inclusion relationship. The inclusion relationship can be understood as the small region being in the large region, and the large region can be defined as the parent region, and the small region can be defined as the child region. If the small region also includes a smaller region, the smaller region can be defined as a sub-sub-region, and the sub-sub-region and the sub-region are in an inclusion relationship, and correspondingly, the sub-sub-region and the parent region are also in an inclusion relationship.

[0061] It should be noted that in this embodiment, the hierarchical relationship between the areas described is only the hierarchical relationship between the areas under the same window. In actual use, there are often overlapping windows in the same UI interface, resulting in overlapping areas. Therefore, some methods can be used to distinguish different windows, and then the method of this embodiment can be used to perform area division. The specific method to be used is not limited in this embodiment.

[0062] Therefore, in this embodiment, step S300 may further include:

[0063] Step S304: Determine the relative position relationship between the region contours based on the position information of the region contours.

[0064] In this embodiment, because the processing is performed on the same image, the image region has the characteristic of position invariance. Specifically, in the above-mentioned image pyramid, images at different levels may extract the contour of the same region. Therefore, two region contours with the same position information are the same region contour. For different region contours, the relative position relationship between them can be determined based on the position information.

[0065] Step S305: Determine the hierarchical relationship between the various regions in the UI image according to the relative positional relationship between the contours of the various regions; wherein the hierarchical relationship includes: a containment relationship.

[0066] In this embodiment, the relative positional relationship between the area contours is obtained, and the inclusion and being included relationship between the area contours can be determined.

[0067] In this embodiment, determining the hierarchical relationship between regions is an important basis for realizing semantic recognition of UI elements, and associated semantic information can be assigned based on the hierarchical relationship between regions.

[0068] Step S400: Assigning associated semantic information to basic information of UI elements based on the image pyramid; wherein the associated semantic information is used to represent information on the association relationship between UI elements and between UI elements and regions.

[0069] In this embodiment, the information characterizing the association between UI elements and between UI elements and regions may include region level information of the region where the UI element is located and information about the region where the UI element is located, such as text information of the region title.

[0070] As described above, the image pyramid can represent the hierarchical relationship between regions, so the basic information of the UI element can be given associated semantic information based on the image pyramid.

[0071] In this implementation, step S400 may include:

[0072] Step S401: Determine the area where the UI element is located based on the basic information of the UI element and the position information of the area outline.

[0073] In this implementation, the region where each UI element is located may be determined based on the position information in the basic information of the UI element and the position information of the region outline.

[0074] Step S402: extracting the region level information of the region where the UI element is located according to the level relationship between regions and the region where the UI element is located.

[0075] After the area where each UI element is located is determined, the area level information of the area where each UI element is located can be determined according to the level relationship between the areas determined in the above steps.

[0076] Specifically, taking a specific scenario example as an example, the target UI image obtained by the present embodiment as described above can be a UI image of the main interface of a certain chat application. The main interface of the UI image is divided into three areas from left to right, the leftmost area is the area about the function button, the middle is the user area about the user avatar; the rightmost area is the area where the chat information is displayed. Among them, there are multiple buttons in the area about the function button, such as text-type UI elements that display texts such as "message", "document", "workbench" and corresponding icon-type UI elements. Through the method of this embodiment, it can be determined that there is no hierarchical relationship of inclusion and inclusion in the three areas, because they can all be defined as parents, and considering the distinction, they can be defined as parent No. 1, parent No. 2, parent No. 3, etc. in turn. It can also be determined that the above-mentioned text-type UI elements that display texts such as "message", "document", "workbench" and corresponding icon-type UI elements are all set in the parent No. 1 area, because the regional hierarchical information of the area where the UI element is located can be extracted by this method.

[0077] Step S403: assign the area level information to the corresponding UI element.

[0078] In this embodiment, after extracting the regional hierarchical information, the corresponding UI element can be assigned. The assignment action can be understood as adding the regional hierarchical information to the basic information of the UI element. That is, through the method of this embodiment, the user can not only identify the semantic information of any UI meta-element itself, that is, the above-mentioned basic information, such as position and category, and the corresponding text information for text-type UI elements. The associated semantic information of any UI meta-element can also be obtained.

[0079] In summary, this implementation can clarify the hierarchical relationship of each region in the UI image and realize the semantic recognition of UI elements based on the hierarchical relationship of the regions. In the implementation of this method, by adopting the image pyramid method, the interference of region detection is eliminated, which greatly improves the accuracy of region detection.

[0080] In some implementations, considering the complexity of most UI image designs and the effect of extracting contours, the image pyramid may adopt a three-layer image pyramid, and the step of constructing the image pyramid based on the UI image with the regional contour may include:

[0081] The bottom layer image is the UI image with the area outline;

[0082] The bottom layer image is downsampled to obtain the middle layer image. Each pixel of the middle layer image corresponds to the weighted average of 5 pixels of the bottom layer image. The middle layer image is one quarter of the size of the bottom layer image.

[0083] The middle layer image is downsampled to obtain the top layer image. Each pixel of the top layer image corresponds to the weighted average of 5 pixels of the middle layer image. The top layer image is one quarter of the size of the middle layer image.

[0084] The bottom layer image, the middle layer image and the top layer image constitute an image pyramid.

[0085] Using a three-layer pyramid, the area contours in the UI image can be better extracted.

[0086] Example 2

[0087] In the second aspect, based on Example 1, Figure 2 As shown, this embodiment provides a UI interface semantic recognition device, including:

[0088] An acquisition module, which is used to acquire basic information of UI elements in a target UI image; wherein the basic information is information that can be directly extracted using a target detection algorithm;

[0089] A detection module, which is used to process the target UI image using an edge detection algorithm to obtain a UI image with a regional outline;

[0090] A construction module, which is used to construct an image pyramid based on the UI image with the region outline; wherein the image pyramid includes multiple layers of images, and the UI image with the region outline is the image at the bottom layer;

[0091] An assignment module is used to assign associated semantic information to basic information of UI elements based on the image pyramid; wherein the associated semantic information is used to represent information on the association relationship between UI elements and between UI elements and regions.

[0092] Example 3

[0093] In the third aspect, based on Example 1, Figure 3 As shown, this embodiment provides a device, including:

[0094] Memory;

[0095] processor;

[0096] as well as

[0097] Computer programs;

[0098] The computer program is stored in the memory and is configured to be executed by the processor to implement the method described in Example 1.

[0099] Example 4

[0100] In a fourth aspect, based on Embodiment 1, this embodiment provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the method described in Embodiment 1 is implemented.

[0101] Example 5

[0102] In a fifth aspect, based on Embodiment 1, this embodiment proposes a hardware system that applies a UI interface semantic recognition method in Embodiment 1. The method can be applied to both the terminal side and the server side.

[0103] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0104] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0105] Embodiment 6:

[0106] In a sixth aspect, based on Example 1, this embodiment also provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute a UI interface semantic recognition method in Example 1.

[0107] It should be understood that the specific examples herein are only intended to help those skilled in the art to better understand the embodiments of the present specification, rather than to limit the scope of the present invention.

[0108] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0109] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0110] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0111] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0112] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0115] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0117] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0118] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0119] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A UI interface semantic recognition method, characterized in that: The method comprises: Obtaining basic information of UI elements in the target UI image; wherein the basic information is information that can be directly extracted from the target UI image using a target detection algorithm; For the target UI image, an edge detection algorithm is used to process the target UI image to obtain a UI image with a regional outline; Based on the UI image with the region outline, an image pyramid is constructed; wherein the image pyramid includes multiple layers of images, and the UI image with the region outline is the image at the bottom layer; Based on the image pyramid, associated semantic information is assigned to basic information of the UI element; wherein the associated semantic information is information used to characterize the association relationship between UI elements and between UI elements and regions.

2. The UI interface semantic recognition method according to claim 1, characterized in that: The basic information includes: category information and location information of the UI element, and text information; obtaining the basic information of the UI element in the target UI image includes: Using a target detection algorithm to detect UI elements of a target UI image, and obtaining category information and location information of the UI elements; wherein the UI elements include icon-type UI elements and text-type UI elements; For the text-type UI elements, the text information of the UI elements is recognized by OCR.

3. The UI interface semantic recognition method according to claim 1, characterized in that: The step of constructing an image pyramid based on the UI image with the region outline comprises: Performing contour extraction on the bottom layer image to obtain a corresponding region contour; wherein the region contour is used to characterize the position information of the region edge in the image; A downsampling operation is performed on the bottom layer image to obtain an upper layer image of the bottom layer image, and then contour extraction is performed on the upper layer image to obtain a corresponding region contour; The above-mentioned downsampling and contour extraction operations are repeatedly performed to obtain the region contours corresponding to the images at different layers in the image pyramid; wherein the image at the previous layer is obtained by performing a downsampling operation on the image at the next layer.

4. The UI interface semantic recognition method according to claim 3, characterized in that: The image pyramid is a three-layer image pyramid. The steps of constructing the image pyramid based on the UI image with the region outline include: The bottom layer image is the UI image with the area outline; The bottom layer image is downsampled to obtain the middle layer image. Each pixel of the middle layer image corresponds to the weighted average of 5 pixels of the bottom layer image. The middle layer image is one quarter of the size of the bottom layer image. The middle layer image is downsampled to obtain the top layer image. Each pixel of the top layer image corresponds to the weighted average of 5 pixels of the middle layer image. The top layer image is one quarter of the size of the middle layer image. The bottom layer image, the middle layer image and the top layer image constitute an image pyramid.

5. The UI interface semantic recognition method according to claim 3, characterized in that: The image pyramid is used to represent the hierarchical relationship between regions. Based on the UI image with region outlines, the step of constructing the image pyramid further includes: Based on the position information of the area contours, determining the relative position relationship between the area contours; According to the relative position relationship between the contours of the various regions, the hierarchical relationship between the various regions in the UI image is determined; wherein the hierarchical relationship includes: a containment relationship.

6. The UI interface semantic recognition method according to claim 5, characterized in that: The associated semantic information includes: regional hierarchical information of the region where the UI element is located, wherein the regional hierarchical information is information representing the hierarchical relationship between the regions; The steps of assigning associated semantic information to the basic information of the UI element include: Determine the area where the UI element is located based on the basic information of the UI element and the position information of the area outline; Extracting regional hierarchical information of the region where the UI element is located according to the hierarchical relationship between the regions and the region where the UI element is located; The area level information is assigned to the corresponding UI element.

7. The UI interface semantic recognition method according to claim 1, characterized in that: The edge detection algorithm uses a canny-based edge detection algorithm, specifically including: After converting the target UI image into a corresponding grayscale image, the boundary contour is detected using a canny detection algorithm with a kernel of 5*5.

8. A UI interface semantic recognition device, characterized in that: The device comprises: An acquisition module, which is used to acquire basic information of UI elements in a target UI image; wherein the basic information is information that can be directly extracted using a target detection algorithm; A detection module, which is used to process the target UI image using an edge detection algorithm to obtain a UI image with a regional outline; A construction module, which is used to construct an image pyramid based on the UI image with the region outline; wherein the image pyramid includes multiple layers of images, and the UI image with the region outline is the image at the bottom layer; An assignment module is used to assign associated semantic information to basic information of UI elements based on the image pyramid; wherein the associated semantic information is used to represent information on the association relationship between UI elements and between UI elements and regions.

9. A device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images