Interface element semantic analysis method and device, equipment and medium

Abstract

The invention relates to the technical field of computer processing, and provides an interface element semantic analysis method and device, equipment and a medium, and the method comprises the steps: screening out a single element region and a combined element region on an interface of a business process; identifying element semantic information and element types of each element region in the single element region and the combined element region; and based on the element semantic information and the element type, determining an element region of the business process and element semantic information of the element region of the business process from a single element region and a combined element region on the interface. According to the method, multiple elements on the interface can be directly obtained in batches and reasonably planned in advance, multiple element areas are analyzed under the constraint of semantic information and element types, the element areas under the business process are obtained, and the efficiency and accuracy of RPA program development are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a method, device, equipment and medium for semantic analysis of interface elements. Background Art

[0002] RPA (Robotic Process Automation) is a type of automation software tool that can use and understand the company's existing applications through a user interface, automate routine rule-based operations, including mouse clicks, keyboard input, screen information acquisition, and page information capture, and simulate manual data processing, such as reading emails and systems, calculating, generating files and reports, and checking files.

[0003] For the development of RPA programs, it is necessary to capture the elements on the process interface. Interface elements are the basic units that constitute the user interface and are used to implement various components of human-computer interaction functions, such as buttons, input boxes, icons, etc. Each element on the process interface will be executed sequentially according to the needs of the process business. This execution process can be used as the automated execution process of the RPA program. To this end, it is necessary to accurately and quickly identify the element information on the interface that conforms to a certain business process.

[0004] The original method was "single picking", where the implementation personnel manually picked up the interface elements to be operated step by step according to the manual operation process. Even the same element of the same interface may need to be picked up multiple times, and when it needs to be modified, it is modified one by one, and the user needs to provide the interface semantics. This picking method makes the picking cumbersome, and also means that the RPA process can only be completed normally when the RPA developer develops according to the pre-designed process. In other cases, the RPA process will fail with a high probability, which greatly affects the development efficiency of the RPA robot. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method, device, equipment and medium for semantic analysis of interface elements.

[0006] The present invention provides a method for semantic analysis of interface elements, comprising: Filter out the single element area and the combined element area on the interface of the business process; Identify element semantic information and element type of each element area in the single element area and the combined element area; Based on the element semantic information and the element type, the element area of ​​the business process and the element semantic information of the element area of ​​the business process are determined from the single element area and the combined element area on the interface.

[0007] According to a method for semantic analysis of interface elements provided by the present invention, the method of determining the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type includes: Determine a target element region; When the element semantic information of the target element area is empty, calling a region association strategy based on the element type of the target element area; the region association strategy represents associating another element area on one side of the periphery of the target element area; Based on the area association strategy, an element area on one side of the target element area is determined. When the element semantic information of the determined element area is not empty, the determined element area and the target element area are merged into one element area as the element area of ​​the business process, and the non-empty element semantic information is used as the element semantic information of the element area of ​​the business process.

[0008] According to a method for semantic analysis of interface elements provided by the present invention, the method of determining the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type includes: Determine a target element region; When the element semantic information of the target element region is not empty, and the target element region is an element region in the existing combined element region, a first element region in the combined element region is screened out, and the element semantic information of the first element region has a preset relationship with the element semantic information of the target element region; Merging the first element region and the target element region into one element region as the element region of the business process, and determining element semantic information of the element region of the business process based on the preset relationship; When it is determined that two or more element regions remain in the existing combined element regions, the remaining two or more element regions are used as new combined element regions.

[0009] According to a method for semantic analysis of interface elements provided by the present invention, the method of determining the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type includes: Determine a target element region; When the element semantic information of the target element area is not empty, and the target element area is an element area in the existing combined element area, it is determined that the first element area does not exist in the combined element area, the target element area is used as the element area of ​​the business process, and the element semantic information of the target element area is used as the element semantic information of the element area of ​​the business process.

[0010] According to a method for semantic analysis of interface elements provided by the present invention, the method of determining the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type includes: Determine a target element region; When the element semantic information of the target element area is not empty and belongs to the single element area, the target element area is used as the element area of ​​the business process, and the element semantic information of the target element area is used as the element semantic information of the element area of ​​the business process.

[0011] According to a method for semantic analysis of interface elements provided by the present invention, the step of screening out single element areas and combined element areas on the interface of a business process includes: Divide the interface of the business process into multiple characteristic areas; When it is determined that two or more feature regions have overlapping regions, the two or more feature regions are merged to form one feature region as a single element region; When it is determined that a feature region has preset feature information, the feature region is widened in a preset direction. When the widened region overlaps with surrounding feature regions, the overlapping feature regions are used as element regions in the combined element region.

[0012] According to a method for semantic analysis of interface elements provided by the present invention, the method further includes: before screening the element area, performing rectangular trimming on each feature area to form a rectangular frame area.

[0013] The present invention also provides an interface element semantic analysis device, comprising: A screening module is used to screen out single element areas and combined element areas on the interface of a business process; An identification module, used to identify element semantic information and element type of each element region in the single element region and the combined element region; The analysis module is used to determine the element area of ​​the business process and the element semantic information of the element area of ​​the business process from multiple element areas on the interface based on the element semantic information and the element type.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above-mentioned methods for semantic analysis of interface elements is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned interface element semantic analysis methods.

[0016] The present invention also provides a computer program product, including a computer program, which implements any of the above-mentioned interface element semantic analysis methods when executed by a processor.

[0017] The present invention provides an interface element semantic analysis method, device, equipment and medium, which screens out single element areas and combined element areas on the interface of the business process, and identifies the element semantic information and element type of each element area, and then determines the element area and element semantics of the business process from the single element area and combined element area on the interface based on the element semantic information and element type. It can directly obtain multiple elements on the interface in batches and reasonably plan them in advance, and analyze multiple element areas under the constraints of semantic information and element types to obtain the element area under the business process, thereby improving the efficiency and accuracy of RPA program development. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of the interface element semantic analysis method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the elemental regional analysis provided by the present invention. Figure 1 .

[0021] Figure 3 This is a schematic diagram of the elemental regional analysis provided by the present invention. Figure 2 .

[0022] Figure 4 It is a structural schematic diagram of the interface element semantic analysis device provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figure 1-Figure 5 The interface element semantic analysis method, device, equipment and medium of the present invention are described.

[0026] Figure 1 A flow chart of a method for semantic analysis of interface elements provided by the present invention is shown in FIG. Figure 1 , the method comprises the following steps: Step 11: Filter out the single element area and the combined element area on the interface of the business process.

[0027] Step 12: Identify the element semantic information and element type of each element region in the single element region and the combined element region.

[0028] Step 13: Based on the element semantic information and the element type, determine the element area of ​​the business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface.

[0029] Regarding steps 11 to 13, it should be noted that RPA (Robotic Process Automation) is a type of automation software tool that can use and understand the company's existing applications through the user interface, and automate routine rule-based operations, including mouse clicks, keyboard input, screen information acquisition, and page information capture, simulating manual data processing, such as reading emails and systems, calculating, generating files and reports, and checking files.

[0030] For the development of RPA programs, it is necessary to capture the elements on the process interface. Interface elements are the basic units that constitute the user interface and are used to implement various components of human-computer interaction functions, such as buttons, input boxes, icons, etc. Each element on the process interface will be executed sequentially according to the needs of the process business. This execution process can be used as the automated execution process of the RPA program. To this end, it is necessary to accurately and quickly identify the element information on the interface that conforms to a certain business process.

[0031] In the present invention, for the interface of the business process, the area where the elements of the business process may be located is screened out on the interface according to the image processing method. Multiple element areas will be screened out on an interface, but in terms of the needs of the business process, there may be multiple elements under a certain business function, that is, multiple elements need to be displayed on a certain business function, so the multiple element areas screened out on the interface can be divided into combined element areas as much as possible, and each combined element area includes at least two element areas. Correspondingly, there are also separate element areas, which are regarded as single element areas. For example, in the business process of login, if "third-party login" appears, at this time, there will be multiple application software icons (such as icons of application a, application b, and application c). From the perspective of elements, the area occupied by the icons of these application software and the title of the third-party login can be regarded as a combined element area. Correspondingly, the input box for entering the account and password can be regarded as a single element area.

[0032] In the present invention, the necessary image features will be reflected in the element area. To this end, semantic analysis can be performed on these image features to obtain the element semantic information of each element area, as well as the element type. For example, an icon in a third-party login has its own characteristics, and its own semantic information can be identified based on the characteristics of the icon. If the icon indicates direct login to the current application software with the authorization of "software a". Then, the element semantic information of this icon can be the specific name of "software a". Accordingly, this icon is regarded as a clickable virtual button, and for this reason, the element type of this icon can be a "button" type. If there is an input box in the element area, then when no information is entered in the input box, its semantic information is empty. But its element type is an "input box" type.

[0033] To this end, for the feature information of the image contained in each element region, the corresponding element semantic information and element type can be identified.

[0034] In the present invention, there are many element areas screened out on the interface, and some element areas do not have their own semantics. There may also be several elements belonging to the same semantic type. For this reason, it is necessary to analyze the multiple element areas determined, and plan out the element areas that conform to the business process. The planned element areas have semantic information, and can also mark the specific position of the element areas on the interface, so that when the RPA program is developed for a certain business process in the future, it can be accurately operated on the interface when the program is executed. Therefore, in the present invention, it is necessary to combine the element semantic information and element type of each element area, perform semantic matching, area merging and other operations on each element area, and finally determine the element area belonging to the business process.

[0035] The interface element semantic analysis method provided by the present invention screens out single element areas and combined element areas on the interface of the business process, identifies the element semantic information and element type of each element area, and then determines the element area and element semantics of the business process from the single element area and combined element area on the interface based on the element semantic information and element type. It can directly acquire multiple elements on the interface in batches and reasonably plan them in advance, and analyze multiple element areas under the constraints of semantic information and element types to obtain the element area under the business process, thereby improving the efficiency and accuracy of RPA program development.

[0036] In a further method of the above method, the processing of determining the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type is mainly explained as follows: Identify a target element region.

[0037] When the element semantic information of the target element region is empty, the region association strategy is called based on the element type of the target element region; the region association strategy represents associating another element region on one side of the periphery of the target element region.

[0038] Based on the area association strategy, the element area on one side of the target element area is determined. When the element semantic information of the determined element area is not empty, the determined element area and the target element area are merged into one element area as the element area of ​​the business process, and the element semantic information of the element area of ​​the business process is determined based on the element type and non-empty element semantic information.

[0039] In this regard, it should be noted that in the present invention, multiple element factors are analyzed, and one of the element regions can be selected as the target element region to check whether the element semantic information of the target element region is empty. If it is empty, it indicates that it is difficult to parse text information from the element region. Such an element region may represent an input box, or an icon region such as a selection box, so that the element type can be identified by this type of element region.

[0040] In the present invention, the element semantic information of element areas of different element types is empty, and it is necessary to associate with other element areas to find appropriate element semantic information. For this purpose, the correspondence between element type and area association strategy is configured. After the element type of the target element area is determined, the area association strategy can be determined based on the correspondence. The area association strategy represents the association of another element area on one side of the periphery of the target element area. In fact, the target element area is based on a preset distance according to a direction of the periphery to see if there are other element areas within the preset distance. If so, this element area is used as the element area associated with the target element area. When the element semantic information of the element area is not empty, it means that the element area and the target element area belong to the same semantic range. At this time, the element area and the target element area are merged into one element area as the element area of ​​the business process, and then the element semantic information of the element area of ​​the business process is determined based on the element type and non-empty element semantic information.

[0041] join Figure 2 , if the target element area is an input box, its element semantic information is empty. The area association strategy corresponding to the element of the input box type is to associate another element area to the left. This strategy also configures other restrictions, such as the distance value to the left. Figure 2 There is an "Account" element on the left side of the target element area. The element semantic information of this element is not empty and is assigned the value of "Account". Therefore, the "Account" element area and the target element area are merged into a new element area as the element area of ​​the business process. The element semantics of this element area is "Account Input".

[0042] Furthermore, when the element semantic information of the target element area is not empty and belongs to a single element area, the target element area is used as the element area of ​​the business process. For example, if there are "register" and "login" elements on an interface, and the element semantic information of these two element areas is not empty and they belong to a single element area, these element areas are used as the element areas of the business process, and the element semantic information of the target element area is used as the element semantic information of the element area of ​​the business process.

[0043] Furthermore, a target element area is continued to be determined. When the element semantic information of the target element area is not empty, but the target element area belongs to an element area in the combined element area, it is also necessary to determine whether some element areas in the combined element area belong to the same element semantics, or whether they belong to elements under the same business function.

[0044] Next, the element semantic information in each element region of the combined element region is obtained, and then it is determined whether the element semantic information satisfies a preset relationship, and the elements satisfying the preset relationship are used as the first element region of the target element region.

[0045] Since the preset relationship belongs to the relationship between elements under the same business function, at this time, when the preset relationship is met, it is considered that these elements can be merged into one area, then the first element area and the target element area are merged into one element area as the element area of ​​the business process.

[0046] join Figure 3 , the target element area is the area where "third-party login" is located, and the other element areas in the combined element area are the areas where application a, application b, and application c are located. The element semantic information of the target element area is third-party login, and the element semantic information of the areas where application a, application b, and application c are located are the specific names of the applications. When the preset relationship is that when a third party logs in, the specific name of the application is paired. At this time, if the conditions are met, the various areas are merged into one element area, and the element semantics is "third-party login".

[0047] It should also be noted that when there is only one first element region in the existing combined element region, there will be remaining element regions. At this time, when it is determined that there are two or more element regions remaining in the existing combined element region, the remaining two or more element regions are used as new combined element regions. The new combined element region will continue to perform the screening analysis of the element region.

[0048] Furthermore, a target element area is determined. When the element semantic information of the target element area is not empty and the target element area is an element area in the existing combined element area, it is determined that the first element area does not exist in the combined element area, which is equivalent to that although the target element area is in the combined element area, it has no relationship with other element areas. At this time, the target element area is used as the element area of ​​the business process.

[0049] In addition, it should be noted that when the element semantic information of the target element area is empty and belongs to a single element area, the developer may be prompted to check the element area or directly delete the element area.

[0050] In a further method of the above method, the processing of filtering out the single element area and the combined element area on the interface of the business process is mainly explained, as follows: Multiple characteristic areas are divided on the interface of the business process.

[0051] When it is determined that two or more feature regions have overlapping regions, the two or more feature regions are merged to form one feature region as a single element region.

[0052] When it is determined that a feature region has preset feature information, the feature region is widened in a preset direction. When the widened region overlaps with surrounding feature regions, the overlapping feature regions are used as element regions in the combined element region.

[0053] In this regard, it should be noted that the image on the interface needs to be divided into a plurality of feature areas.

[0054] First, convert the color image of the screenshot into a grayscale image, use image corrosion and image expansion to achieve acidification, emphasize the local characteristics of the image, make the originally unclear feature image clear, expand the difference between the features of different objects in the image to achieve enhancement processing, use the grayscale threshold to binarize the image, generate multiple binary images, in each binary image, detect black and white areas as extreme areas, analyze the area changes of extreme areas under different thresholds, and find the area with the smallest area change within a certain threshold range. It should also be noted that before screening the element area, each feature area is rectangular trimmed to form a rectangular frame area. Realize the area detection of the image grayscale threshold change, and then use Canny edge detection to detect the rectangular position of the area where the interface element is located to generate the rectangular frame area.

[0055] These acquired feature regions can be merged according to the size of the overlapped range and then be treated as a single element region, such as the element region where the "account number" is located and the element region where the input box is located.

[0056] It should be noted that the rule of the size of the overlapped range can be understood as that when the size of the overlapped area and the size of the smallest area among the overlapped areas reach a certain value, it can be considered that the areas can overlap in one area. For overlapping to one area, the coordinates of the upper left corner and the lower right corner of the overall overlapped area can be used as the coordinates of the two diagonal corners of the rectangle, and then a rectangular area is constructed based on the coordinate values ​​to complete the merging of the overlapped areas.

[0057] When it is determined that there is preset feature information in a feature area, the preset feature information here indicates that there may be an associated feature area, so the feature area is widened in the preset direction. When the widened area overlaps with the surrounding feature areas, the overlapping multiple feature areas are used as element areas in the combined element area. For example, the target element area is the area where "third-party login" is located, and the feature information extracted from the target element area is the feature of "third copy login". At this time, the preset direction can be downward expansion. When the widened area overlaps with the surrounding feature areas (such as the area where application a, application b, and application c are located), the overlapping multiple feature areas (the area where third-party login, application a, application b, and application c are located) are used as element areas in the combined element area.

[0058] The interface element semantic analysis device provided by the present invention is described below. The interface element semantic analysis device described below and the interface element semantic analysis method described above can be referenced to each other.

[0059] Figure 4 A schematic diagram of a flow chart of an interface element semantic analysis device provided by the present invention is shown in FIG. Figure 4 The device comprises a screening module 41, an identification module 42 and an analysis module 43, wherein: A screening module is used to screen out single element areas and combined element areas on the interface of a business process; A recognition module, used to recognize element semantic information and element type of each element region in a single element region and a combined element region; The analysis module is used to determine the element area of ​​the business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type.

[0060] Since the principle of the device of the embodiment of the present invention is the same as that of the method of the above embodiment, a more detailed explanation is omitted here.

[0061] It should be noted that, in the embodiments of the present invention, relevant functional modules may be implemented by a hardware processor.

[0062] The interface element semantic analysis device provided by the present invention screens out single element areas and combined element areas on the interface of the business process, identifies the element semantic information and element type of each element area, and then determines the element area of ​​the business process from multiple element areas on the interface based on the element semantic information and element type. It can directly acquire multiple elements on the interface in batches and reasonably plan them in advance, and analyze multiple element areas under the constraints of semantic information and element types to obtain the element area under the business process, thereby improving the efficiency and accuracy of RPA program development.

[0063] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 51 (processor), a communication interface 52 (Communications Interface), a memory 53 (memory) and a communication bus 54, wherein the processor 51, the communication interface 52, and the memory 53 communicate with each other through the communication bus 54. The processor 51 may call the logic instructions in the memory 53 to execute the interface element semantic analysis method, which includes: screening out single element areas and combined element areas on the interface of the business process; identifying element semantic information and element types of each element area in the single element area and the combined element area; based on the element semantic information and the element type, determining the element area of ​​the business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface.

[0064] In addition, the logic instructions in the above-mentioned memory 53 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0065] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the interface element semantic analysis method provided by the above methods, which includes: screening out single element areas and combined element areas on the interface of the business process; identifying the element semantic information and element type of each element area in the single element area and the combined element area; based on the element semantic information and the element type, determining the element area of ​​the business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface.

[0066] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the interface element semantic analysis method provided by the above-mentioned methods, the method comprising: screening out single element areas and combined element areas on the interface of the business process; identifying the element semantic information and element type of each element area in the single element area and the combined element area; based on the element semantic information and the element type, determining the element area of ​​the business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface.

[0067] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for semantic analysis of interface elements, characterized in that: include: Filter out the single element area and the combined element area on the interface of the business process; Identify element semantic information and element type of each element area in the single element area and the combined element area; Based on the element semantic information and the element type, the element area of ​​the business process and the element semantic information of the element area of ​​the business process are determined from the single element area and the combined element area on the interface.

2. The interface element semantic analysis method according to claim 1, characterized in that: The determining, based on the element semantic information and the element type, the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface includes: Determine a target element region; When the element semantic information of the target element area is empty, calling a region association strategy based on the element type of the target element area; the region association strategy represents associating another element area on one side of the periphery of the target element area; Based on the area association strategy, an element area on one side of the target element area is determined; when the element semantic information of the determined element area is not empty, the determined element area and the target element area are merged into one element area as the element area of ​​the business process; and the element semantic information of the element area of ​​the business process is determined based on the element type and non-empty element semantic information.

3. The interface element semantic analysis method according to claim 1, characterized in that: The determining, based on the element semantic information and the element type, the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface includes: Determine a target element region; When the element semantic information of the target element region is not empty, and the target element region is an element region in the existing combined element region, a first element region in the combined element region is screened out, and the element semantic information of the first element region has a preset relationship with the element semantic information of the target element region; Merging the first element region and the target element region into one element region as the element region of the business process, and determining element semantic information of the element region of the business process based on the preset relationship; When it is determined that two or more element regions remain in the existing combined element regions, the remaining two or more element regions are used as new combined element regions.

4. The interface element semantic analysis method according to claim 3, characterized in that: The determining, based on the element semantic information and the element type, the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface includes: Determine a target element region; When the element semantic information of the target element area is not empty, and the target element area is an element area in the existing combined element area, it is determined that the first element area does not exist in the combined element area, the target element area is used as the element area of ​​the business process, and the element semantic information of the target element area is used as the element semantic information of the element area of ​​the business process.

5. The interface element semantic analysis method according to claim 2, characterized in that: The determining, based on the element semantic information and the element type, the element area of ​​the target business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface includes: Determine a target element region; When the element semantic information of the target element area is not empty and belongs to the single element area, the target element area is used as the element area of ​​the business process, and the element semantic information of the target element area is used as the element semantic information of the element area of ​​the business process.

6. The interface element semantic analysis method according to claim 1, characterized in that: The single element area and the combined element area on the interface for filtering out the business process include: Divide the interface of the business process into multiple characteristic areas; When it is determined that two or more feature regions have overlapping regions, the two or more feature regions are merged to form one feature region as a single element region; When it is determined that a feature region has preset feature information, the feature region is widened in a preset direction. When the widened region overlaps with surrounding feature regions, the overlapping feature regions are used as element regions in the combined element region.

7. The method for semantic analysis of interface elements according to claim 6, characterized in that: The method further comprises: before screening the element area, performing rectangular trimming on each feature area to form a rectangular frame area.

8. An interface element semantic analysis device, characterized in that: include: A screening module is used to screen out single element areas and combined element areas on the interface of a business process; An identification module, used to identify element semantic information and element type of each element region in the single element region and the combined element region; The analysis module is used to determine the element area of ​​the business process and the element semantic information of the element area of ​​the business process from the single element area and the combined element area on the interface based on the element semantic information and the element type.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the interface element semantic analysis method as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for semantic analysis of interface elements as described in any one of claims 1 to 7 is implemented.

Images