A method for labeling a graphical interface picture and an electronic device

Abstract

The application relates to the technical field of computer vision, in particular to a graphic interface picture labeling method and an electronic device, which are used to reduce labeling workload, thereby reducing labeling cost and improving graphic interface analysis efficiency. The method comprises the following steps: obtaining a to-be-labeled picture and at least two labeled pictures; matching the to-be-labeled picture with the at least two labeled pictures respectively to obtain at least two matching pictures; combining the at least two matching pictures to obtain a target matching picture; and obtaining a transcription labeling result of the to-be-labeled picture based on the target matching picture and labeling results of the at least two labeled pictures.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for annotating graphical interface images and an electronic device. Background Technology

[0002] A graphical user interface (GUI) is a computer user interface that uses graphics to display information. In some business scenarios, it is necessary to parse the images in the graphical interface to obtain information such as the image elements.

[0003] Currently, deep learning-based image recognition models are commonly used to parse graphical user interface images. However, image recognition models require training with a large number of labeled images, which incurs significant labeling costs.

[0004] In related technologies, considering that some graphical interface images may contain the same image elements, a template image is selected from these graphical interface images to reduce annotation costs, and the annotations in the template image are reused in other graphical interface images. Although this can reduce the annotation workload to some extent, the problem of duplicate annotation still exists. Summary of the Invention

[0005] This application provides a method and electronic device for annotating graphical interface images, which can avoid repetitive annotation as much as possible, thereby greatly reducing annotation costs and improving the parsing efficiency of graphical interfaces.

[0006] In a first aspect, embodiments of this application provide a method for annotating graphical interface images, comprising:

[0007] Obtain the image to be labeled and at least two already labeled images;

[0008] The image to be labeled is matched with each of the at least two labeled images to obtain at least two matching images;

[0009] The at least two matching images are combined to obtain the target matching image;

[0010] Based on the target matching image and the annotations in the at least two labeled images, the transcription annotation result of the image to be annotated is obtained.

[0011] In some possible embodiments, each matching image is marked with a matching region between the image to be labeled and the corresponding labeled image, the matching region being composed of multiple matching pixels;

[0012] The step of combining the at least two matching images to obtain the target matching image includes:

[0013] Mark the matching regions in the at least two matching images in the image to be labeled;

[0014] The labeled image to be labeled is used as the target matching image; wherein, the target matching image contains a labeled combined matching region, and the pixel value of each matching pixel in the combined matching region is the identifier of the corresponding labeled image.

[0015] In some possible embodiments, obtaining the transcription annotation result of the image to be annotated based on the target matching image and the annotations in the at least two annotated images includes:

[0016] For each of the at least two labeled images, perform the following operation: if the pixel value of each pixel in the target matching image that corresponds to any label in the labeled image is the identifier of the labeled image, then obtain the transcription label of the image to be labeled based on any label;

[0017] Based on all the transcription annotations obtained from the annotations in the at least two annotated images, the transcription annotation result of the image to be annotated is obtained.

[0018] In some possible embodiments, the method further includes:

[0019] For each copied annotation in the copied annotation result of the image to be annotated, the source of the copied annotation is recorded. The source of the copied annotation is an annotation in one of the at least two annotated images.

[0020] In some possible embodiments, obtaining the image to be labeled and at least two labeled images includes:

[0021] During the process of annotating multiple graphical interface images in sequence, after annotating at least two graphical interface images, each subsequent annotation will obtain the image to be annotated and at least two already annotated images.

[0022] In some possible embodiments, the method further includes:

[0023] The transcription and annotation results of the image to be annotated are displayed to perform the remaining annotations on the image to be annotated, thereby obtaining the remaining annotation results of the image to be annotated;

[0024] Based on the transcription and annotation results and the remaining annotation results, the annotation results of the image to be annotated are obtained.

[0025] In some possible embodiments, the method further includes:

[0026] In response to a user's trigger action regarding erroneous transcription annotations in the transcription annotation results, the source of the erroneous transcription annotations is displayed;

[0027] In response to a user's triggering action regarding the source of the copying, the target annotation of the labeled image corresponding to the source of the copying is displayed so that the target annotation can be modified.

[0028] In some possible embodiments, the method further includes:

[0029] Based on the copying source of all the copied annotations recorded, each annotation that has a copying relationship with the target annotation is identified, and each annotation is modified based on the modified target annotation.

[0030] In some possible embodiments, the method further includes:

[0031] For each remaining annotation in the remaining annotation results of the image to be annotated, perform the following operations:

[0032] If it is determined that the remaining annotations are completely contained by any already labeled image, then the remaining annotations are copied in reverse to the annotation results of any already labeled image.

[0033] In some possible embodiments, determining that the remaining annotations are completely contained by any already labeled image includes:

[0034] If, in the target matching image, the pixel value of each pixel corresponding to the remaining annotation is the identifier of any of the already labeled images, then it is determined that the remaining annotation is completely contained by any of the already labeled images.

[0035] In some possible embodiments, the method further includes:

[0036] The reverse-copy annotations in the annotation results of any of the annotated images are used as the source of the remaining annotations for the image to be annotated and recorded.

[0037] Secondly, embodiments of this application provide a graphical interface image annotation device, comprising:

[0038] The acquisition module is used to acquire the image to be labeled and at least two labeled images;

[0039] The matching module is used to match the image to be labeled with the at least two labeled images respectively to obtain at least two matching images;

[0040] The combination module is used to combine the at least two matching images to obtain a target matching image;

[0041] The first annotation module is used to obtain the transcription annotation result of the image to be annotated based on the target matching image and the annotations in the at least two annotated images.

[0042] In some possible embodiments, each matching image is marked with a matching region between the image to be labeled and the corresponding labeled image, the matching region being composed of multiple matching pixels;

[0043] The combined module is specifically used for:

[0044] Mark the matching regions in the at least two matching images in the image to be labeled;

[0045] The labeled image to be labeled is used as the target matching image; wherein, the target matching image contains a labeled combined matching region, and the pixel value of each matching pixel in the combined matching region is the identifier of the corresponding labeled image.

[0046] In some possible embodiments, the first annotation module is specifically used for:

[0047] For each of the at least two labeled images, perform the following operation: if the pixel value of each pixel in the target matching image that corresponds to any label in the labeled image is the identifier of the labeled image, then obtain the transcription label of the image to be labeled based on any label;

[0048] Based on all the transcription annotations obtained from the annotations in the at least two annotated images, the transcription annotation result of the image to be annotated is obtained.

[0049] In some possible embodiments, the device further includes:

[0050] The first recording module is used to record the source of copying for each copying annotation in the copying annotation result of the image to be annotated, wherein the source of copying is an annotation in one of the at least two annotated images.

[0051] In some possible embodiments, the acquisition module is specifically used for:

[0052] During the process of annotating multiple graphical interface images in sequence, after annotating at least two graphical interface images, each subsequent annotation will obtain the image to be annotated and at least two already annotated images.

[0053] In some possible embodiments, the device further includes:

[0054] The first display module is used to display the copying and annotation results of the image to be annotated, so as to perform the remaining annotations on the image to be annotated and obtain the remaining annotation results of the image to be annotated;

[0055] The second annotation module is used to obtain the annotation result of the image to be annotated based on the copying annotation result and the remaining annotation result.

[0056] In some possible embodiments, the device further includes:

[0057] The second display module is used to respond to the user's triggered operation on the erroneous copying annotations in the copying annotation results and display the copying source of the erroneous copying annotations;

[0058] The third display module is used to respond to the user's triggered operation on the source of the copying and to display the target annotation of the marked image corresponding to the source of the copying, so as to modify the target annotation.

[0059] In some possible embodiments, the device further includes:

[0060] The modification module is used to determine each annotation that has a copying relationship with the target annotation based on the copying source of all copied annotations in the record, and to modify each annotation based on the modified target annotation.

[0061] In some possible embodiments, the device further includes:

[0062] The third annotation module is used to perform the following operations for each remaining annotation in the remaining annotation results of the image to be annotated: if it is determined that the remaining annotation is completely contained by any annotated image, then the remaining annotation is copied in reverse to the annotation results of any annotated image.

[0063] In some possible embodiments, the third annotation module is further configured to:

[0064] If, in the target matching image, the pixel value of each pixel corresponding to the remaining annotation is the identifier of any of the already labeled images, then it is determined that the remaining annotation is completely contained by any of the already labeled images.

[0065] In some possible embodiments, the device further includes:

[0066] The second recording module is used to record the reverse copying annotations in the annotation results of any annotated image as the copying source of the remaining annotations of the image to be annotated.

[0067] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods in the first aspect.

[0068] Fourthly, embodiments of this application provide a computer storage medium storing a computer program that, when run on a computer, causes the computer to perform the steps of any of the methods in the first aspect.

[0069] Fifthly, embodiments of this application provide a computer program product, which includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any of the methods in the first aspect.

[0070] The solution in this application embodiment has at least the following beneficial effects:

[0071] This application considers that some graphical interface images may contain the same image elements. These image elements can be manually labeled once, and other graphical interface images containing the same image element can copy this label. In the solution of this application, when labeling the image to be labeled, the image to be labeled is matched with at least two labeled images to obtain at least two matching images. These matching images are combined to obtain a target matching image. Then, based on the target matching image and the labeling results of at least two labeled images, the labels in the labeled images can be copied into the copied labeling results of the image to be labeled. In this way, the labels in multiple labeled images can be reused, duplicate labeling can be avoided as much as possible, the labeling workload can be greatly reduced, thereby reducing labeling costs and improving the parsing efficiency of the graphical interface.

[0072] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1A flowchart illustrating another graphical interface image annotation method provided in this application embodiment;

[0075] Figure 2 A flowchart illustrating another graphical interface image annotation method provided in this application embodiment;

[0076] Figure 3 A flowchart illustrating another graphical interface image annotation method provided in this application embodiment;

[0077] Figure 4 This is a schematic diagram of a graphical interface image provided in an embodiment of this application;

[0078] Figure 5 A schematic diagram of another graphical interface image provided for an embodiment of this application;

[0079] Figure 6 A schematic diagram of another graphical interface image provided for an embodiment of this application;

[0080] Figure 7 A schematic diagram illustrating the annotation of a graphical interface image provided in an embodiment of this application;

[0081] Figure 8 A schematic diagram illustrating a matching image provided in an embodiment of this application;

[0082] Figure 9 A schematic diagram illustrating another matching image provided in an embodiment of this application;

[0083] Figure 10 A schematic diagram of a target matching image provided in an embodiment of this application;

[0084] Figure 11 A schematic diagram of the structure of a graphical interface image annotation device provided in an embodiment of this application;

[0085] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0087] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.

[0088] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0089] RPA (Robotic Process Automation) technology can simulate how employees operate computers using keyboards and mice in their daily work. It can replace humans in performing tasks such as logging into systems, operating software, reading and writing data, downloading files, and reading emails. By using automated robots as virtual labor, enterprises can free employees from repetitive, low-value tasks, allowing them to focus their energy on high-value-added work. This enables enterprises to reduce costs and increase efficiency while undergoing digital and intelligent transformation.

[0090] RPA (Robotic Process Automation) is a software robot that replaces manual tasks in business processes and interacts with computer front-end systems like a human. Therefore, RPA can be seen as a software-based program robot running on a personal PC or server, mimicking user actions on a computer to automate repetitive tasks such as retrieving emails, downloading attachments, logging into systems, and data processing and analysis—faster, more accurate, and more reliable. While both RPA and traditional physical robots address the speed and accuracy issues in human work through specific rules, traditional physical robots are hardware-software hybrids requiring specific hardware support and software to perform tasks. RPA robots, on the other hand, are purely software-based; once the appropriate software is installed, they can be deployed to any PC or server to complete the assigned tasks.

[0091] In other words, RPA is a method and related technologies that use "digital employees" to perform business operations in place of humans. Essentially, RPA uses software automation technology to simulate human operation of computer systems, software, web pages, and documents, acquiring business information, executing business actions, and ultimately achieving automated process processing, saving labor costs, and improving processing efficiency.

[0092] As can be seen from the description, the graphical interface image annotation method of this application can be applied to a certain process stage of RPA. For example, if a large number of graphical interface images need to be annotated in a certain process stage, the graphical interface image annotation method of this application can be used to complete the annotation of the graphical interface images so as to perform subsequent graphical interface parsing operations.

[0093] The design concept of the embodiments of this application is briefly introduced below:

[0094] In related technologies, considering that some graphical interface images may contain the same image elements, a template image is selected from these graphical interface images to reduce annotation costs, and the annotations in the template image are reused in other graphical interface images. Although this can reduce the annotation workload to some extent, the problem of duplicate annotation still exists. In view of this, embodiments of this application provide a method and electronic device for annotating graphical interface images, which can calculate matching images between the image to be annotated and multiple annotated images. Based on the target matching image formed by combining these matching images, the annotations from multiple annotated images can be copied into the image to be annotated, so as to avoid duplicate annotation as much as possible, greatly reduce the annotation workload, thereby reducing annotation costs and improving the parsing efficiency of graphical interfaces.

[0095] The preferred embodiments of this application will now be described with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features described herein can be combined with each other without conflict.

[0096] The annotation method for graphical interface images in this application embodiment can be executed independently by a server or a terminal device. Terminal devices include, but are not limited to, desktop computers, mobile phones, computers, smart home appliances, smart voice interaction devices, and in-vehicle terminals. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, cloud functions, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0097] Reference Figure 1 The diagram shows a flowchart of a graphical interface image annotation method provided in an embodiment of this application. The graphical interface image annotation method includes the following steps S101-S104:

[0098] Step S101: Obtain the image to be labeled and at least two labeled images.

[0099] Wherein, after a number of graphical interface images are input, these graphical interface images may be labeled in sequence. After at least two graphical interface images are labeled, when labeling each subsequent graphical interface image, the to-be-labeled image and at least two labeled images may be acquired. Assume that a total of n (n>2) graphical interface images are input, when labeling the i-th (2<i≤n) graphical interface image, the i-th graphical interface image is taken as the to-be-labeled image, and the 1st to (i-1)-th graphical interface images are taken as the labeled images.

[0100] It should be noted that the graphical user interface images may be images pre-stored in a database, or may be graphical interface images captured on site. The number of graphical user interface images may be a large number of images of the same application but in different graphical interface states directly input by the user, and the specific content of the graphical interfaces is not limited.

[0101] Step S102: matching the to-be-labeled image with the at least two labeled images to obtain at least two matched images.

[0102] Wherein, the to-be-labeled image is matched with each labeled image to determine a matching region between the to-be-labeled image and each labeled image. When matching the to-be-labeled image with a certain labeled image, matching pixels between them may be calculated to determine the matching region, and the matching region is marked out on the basis of the to-be-labeled image, so as to obtain a corresponding matched image.

[0103] Optionally, matching the to-be-labeled image with a certain labeled image specifically includes: overlapping the to-be-labeled image and the labeled image according to different overlapping placement methods, to obtain a candidate matching region in each overlapping placement method; acquiring an optimal matching region from all the candidate matching regions, and performing reduction processing on the optimal matching region to obtain the matching region of the to-be-labeled image and the labeled image.

[0104] Wherein, in consideration of the following situations that may exist when the graphical interface changes, such as scaling, stretching or clicking on the graphical interface:

[0105] (1) When the graphical user interface is scaled, stretched or clicked, the whole of the graphical user interface remains unchanged; if only a new window is generated, the disappearing part of some windows will be affected;

[0106] (2) When stretching or scaling, the resolution of the graphical elements may or may not change; but considering only the case where the resolution of the elements remains unchanged (in most cases), the changed elements will definitely maintain a certain alignment relationship with the elements before the change: left / center / right alignment.

[0107] Therefore, when matching the to-be-labeled image with a certain labeled image, the following operations may be specifically performed:

[0108] (1) Keep the image to be labeled unchanged and place the labeled images on top of each other.

[0109] The overlapping placement methods include:

[0110] Horizontally: Left / Center / Right alignment options

[0111] Vertical alignment: Top / Middle / Bottom

[0112] This results in 9 possible placement methods: top left alignment, center left alignment, bottom left alignment; top center alignment, center-center alignment, bottom center alignment; top right alignment, center right alignment, bottom right alignment.

[0113] (2) For each overlapping placement method, determine the candidate matching region and calculate the area of ​​the candidate matching region.

[0114] a. When placing images on top of each other, only consider the areas where both images have pixels (if the images are of different sizes, they will partially overlap).

[0115] b. For overlapping areas (where there are pixels), compare each pixel one by one. If they are completely identical, the pixel is considered to be a match. All matching pixels are obtained, and all matching pixels constitute the candidate matching area.

[0116] c. Calculate the number of matching pixels in the candidate matching region, that is, obtain the matching area.

[0117] (3) Select the candidate matching region with the largest matching area as the optimal matching region, that is, the matching region between the image to be labeled and the labeled image.

[0118] (4) Reduce the optimal matching region.

[0119] Specifically, for the optimal matching region, the matching region is reduced, and the non-matching region is expanded. The non-matching region refers to the area outside the matching region within the overlapping region. For example, an expansion operation (e.g., kernel 11, 11) is performed on the non-matching region to obtain the reduced optimal matching region.

[0120] This is because the optimal matching region consists of several pixels, which may have the following drawbacks in practical applications:

[0121] 1) Matching discrete pixels is likely to be coincidental: for example, two characters, even if they are not the same character, may have some identical pixels, but these pixels are definitely discrete, meaning that a large area (the entire character) of pixels will not be completely identical. Therefore, matching discrete pixels needs to be removed.

[0122] 2) The tightest pixels are somewhat abrupt: Because the mismatched areas are the tightest, for example, if two characters don't match, everything around them matches, so the mismatched area is the position of those two characters. However, this is too tight and not conducive to annotation. Therefore, the edges of the mismatched areas need to be expanded (by dilation) slightly to make the pixel boundaries more obvious.

[0123] In practical applications, when matching two images, the input is an labeled image and an image to be labeled, and the output is a matched image. The width and height of the matched image are the same as those of the image to be labeled. The number of channels is 1, and the value of each pixel is 0 or 1, where 0 means that the pixel does not match and 1 means that the pixel matches.

[0124] Step S103: Combine at least two matching images to obtain the target matching image.

[0125] Each matching image includes a matching region between the image to be labeled and the corresponding labeled image (or there may be no matching region). In the image to be labeled, the matching regions in at least two matching images are combined to obtain a combined matching region, which constitutes the target matching image mentioned above.

[0126] Step S104: Based on the target matching image and the annotation results of at least two labeled images, obtain the transcription annotation results of the image to be annotated.

[0127] In this step, based on the target matching image, the combined matching area between the image to be labeled and each labeled image can be determined. The annotations in each labeled image in the combined matching area can be copied into the copying annotation result of the image to be labeled.

[0128] In this embodiment, the matching region between the image to be labeled and at least two labeled images can be calculated to obtain at least two matching images. Based on the target matching image after combining these matching images, the image to be labeled can reuse the annotations in at least two labeled images to avoid repeated annotation as much as possible, greatly reduce the annotation workload, thereby reducing the annotation cost and improving the parsing efficiency of the graphical interface.

[0129] The following describes the possible implementation methods of step S103 above.

[0130] In some possible embodiments, each matching image is marked with a matching region between the image to be labeled and the corresponding labeled image, the matching region comprising multiple matching pixels. The step S103 above, which combines at least two matching images, may include:

[0131] Mark the matching regions in at least two matching images in the image to be labeled, and use the labeled image to be labeled as the target matching image; wherein, the target matching image contains the marked combined matching regions, and the pixel value of each matching pixel in the combined matching region is the identifier of the corresponding labeled image.

[0132] In this process, the width and height of each matching image are the same as those of the image to be labeled, and the number of channels is 1. The pixel value of each pixel is set to a first preset value or a second preset value. The first preset value represents that the pixel does not match, and the second preset value represents that the pixel matches, so as to distinguish the matching areas in the matching images. In the following embodiment, the first preset value is 0 and the second preset value is 1 as an example for explanation.

[0133] Furthermore, when marking the matching regions in each matching image in the image to be labeled, firstly, the pixel value of each pixel in the image to be labeled is set to 0 (representing that it does not match any labeled image). Then, for each matching image, each pixel in the matching image is traversed. If the pixel value of a certain pixel is 0 in the image to be labeled and 1 in the matching image, then the pixel value of that pixel in the image to be labeled is set to the identifier of the corresponding labeled image, indicating that the pixel matches the labeled image. Through the above process, the matching regions in each matching image can be marked in the image to be labeled, obtaining the marked combined matching regions, and the image to be labeled containing the combined matching regions is used as the target matching image. Specifically, A2, for each matching pixel in the combined matching region, the pixel value of the matching pixel is set to the identifier of the corresponding labeled image to obtain the target matching image. The identifier of each labeled image can be set as needed. For example, the identifier of each labeled image can be set according to the labeling order, that is, the identifier of the first labeled image is 1, the identifier of the second labeled image is 2, and so on.

[0134] For example, taking the identifier of the kth matching image as k, assuming the image to be labeled is the ith graphical interface image, then the 1st to i-1th graphical interface images are labeled images. For the kth matching image, each pixel in the matching image is traversed in turn. If a pixel is 0 in the image to be labeled (representing that it does not match a certain labeled image) and is 1 in the kth matching image (representing that it matches the kth labeled image), then the pixel value of that pixel in the image to be labeled is set to k.

[0135] Based on the above process, a target matching image is obtained. The width and height of the target matching image are the same as those of the image to be labeled. The number of channels is 1, and the pixel value is 0 or a positive integer k. 0 means that the pixel does not match, and k means that the pixel matches the kth labeled image.

[0136] In the above embodiments of this application, by setting each matching pixel in the combined matching region as the identifier of the corresponding labeled image, the target matching image is obtained, and the source of each matching region in the combined matching region can be determined so that the annotation can be copied from the corresponding labeled image later.

[0137] The following describes the possible implementation methods of step S104 above.

[0138] In some possible embodiments, step S104 above may include the following operations B1-B2:

[0139] A1. For each of the at least two labeled images, perform the following operation: If the pixel values ​​of each pixel in the target matching image that correspond to any label in the labeled image are all the identifiers of that labeled image, then obtain the transcription label of the image to be labeled based on that label.

[0140] The target matching image contains a combination matching region between the image to be labeled and each labeled image. This region contains the annotations in the labeled images. Once the annotations within the combination matching region are determined, these annotations can be directly copied into the image to be labeled to obtain the copied annotations of the image to be labeled.

[0141] Optionally, in step A1 above, the above operations can be performed on each labeled image in the order of their labeling.

[0142] For example, assuming the image to be labeled is the i-th graphical interface image, the first to i-1 graphical interface images (i.e. labeled images) are traversed. When the traversal reaches the k-th labeled image, for any label (position + category), if the pixel value of each pixel corresponding to any label in the target matching image is k (i.e., the label is a complete match), then the label is copied into the image to be labeled to obtain the copied label.

[0143] Optionally, to facilitate subsequent modifications to the transcription annotations, the transcription source of each transcription annotation in the transcription annotation results of the image to be annotated can be recorded. The transcription source is an annotation in an already annotated image.

[0144] For example, in an image to be annotated, the source of a certain transcription annotation is the qth annotation in the kth annotated image (each annotation in the annotated image has an annotation number).

[0145] A2. Based on all the transcription annotations obtained from the annotations in at least two annotated images, obtain the transcription annotation results for the image to be annotated.

[0146] In the above embodiments of this application, based on the target matching image, by determining the annotations in each of the labeled images corresponding to the combined matching area of ​​the target matching image, and copying the determined annotations to the copy annotations of the image to be labeled, the annotations in each of the labeled images can be reused, reducing the annotation workload.

[0147] Based on steps S101-S104 in the above embodiments, the transcription and annotation results of the image to be annotated can be obtained. Next, the remaining part of the image to be annotated (i.e., the mismatched area) can also be annotated.

[0148] In some possible embodiments, such as Figure 2 As shown, after steps S101-S104 above, the following steps S105-S106 can also be performed:

[0149] Step S105: Display the copied annotation results of the image to be annotated, so as to perform the remaining annotations on the image to be annotated and obtain the remaining annotation results of the image to be annotated.

[0150] Step S106: Based on the copying annotation results and the remaining annotation results, obtain the annotation results of the image to be annotated.

[0151] Among these features, displaying the copied annotation results of the image to be annotated allows users to add remaining annotations to unannotated parts, improving the user annotation experience. Furthermore, it allows users to visually review the copied annotations to determine their accuracy.

[0152] In some possible embodiments, when a user discovers an error in a transcription, in order to facilitate the correction of that transcription, such as... Figure 3 As shown, after performing step S105 above, the following steps S107-S108 can also be performed:

[0153] Step S107: In response to the user's triggering operation on the erroneous copying annotations in the copying annotation results, display the copying source of the erroneous copying annotations;

[0154] Step S108: In response to the user's triggered operation on the source of copying, display the target annotation of the marked image corresponding to the source of copying so as to modify the target annotation.

[0155] After displaying the copied annotations of the images to be annotated, if a user finds an error in a copied annotation, they can trigger the erroneous annotation. In response to this trigger, the source of the erroneous annotation will be displayed (e.g., the qth annotation in the kth annotated image). The user can trigger this source to access the target annotation in the corresponding annotated image and make corrections.

[0156] Optionally, after modifying the target annotations in the labeled images corresponding to the above-mentioned sources of copying, it is also possible to determine each annotation that has a copying relationship with the target annotation based on the sources of all the copied annotations recorded, and modify each annotation based on the modified target annotation.

[0157] Each time an image is annotated, the source of each annotated annotation in the annotation results is recorded. This allows the source of all recorded annotated annotations to be obtained, thus identifying the annotations that have a copying relationship with the target annotation (including annotations that use the target annotation as their source, and also annotations corresponding to the source of the target annotation), and then modifying each annotation.

[0158] In some possible embodiments, after performing the above step S105, for each remaining label in the obtained remaining labeling results, it can also be checked whether the remaining label is completely contained in a certain labeled image. If it is completely contained, it means that the remaining label was omitted in the labeled image. At this time, the remaining label can be copied back to the labeled image.

[0159] Therefore, for each remaining annotation in the remaining annotation results of the image to be annotated, the following operations B1-B2 can also be performed:

[0160] B1. Verify whether the remaining annotations are completely contained within any already annotated image;

[0161] Specifically, it verifies whether the pixel values ​​of each pixel corresponding to the remaining annotation in the target matching image are all identifiers of a certain labeled image. If so, it is determined that the remaining annotation is completely contained by the labeled image.

[0162] For example, for each remaining label, if the pixel value of each pixel corresponding to the remaining label in the target matching image is k (that is, the label is completely matched), then it is determined that the remaining label is completely contained by the kth labeled image.

[0163] B2. If it is determined that the remaining labels are completely contained within any labeled image, then the remaining labels are copied in reverse to the labeling results of any labeled image.

[0164] Note that the annotation is copied in reverse to the annotation result of the kth already annotated image, and is assumed to be the qth annotation.

[0165] Optionally, in order to maintain the unidirectionality of copying, the reverse copying annotations in the annotation results of any annotated image are used as the copying source of the remaining annotations of the image to be annotated and recorded.

[0166] For example, the qth annotation of the kth annotated image in the above example is used as the source for copying the remaining annotations and recorded.

[0167] In this embodiment, by maintaining the unidirectional nature of copying, the copying logic can be simplified. When modifying the source of an incorrect copying annotation, only subsequent copying annotations of the source need to be considered, and the copying relationships are not nested. This allows subsequent annotation modification processes to proceed in parallel, significantly improving the running speed. For example, q in image c... c The 'q' in image b is marked as copied. b The annotation is q in image b. b The annotation is copied from the q in image a. a Annotation.

[0168] The annotation process of the graphical interface image annotation system of this application embodiment will be described exemplarily below with specific examples.

[0169] For example, the annotation process of a graphical user interface image annotation system includes the following steps:

[0170] Step 1: The user uploads a large number (e.g., n) of graphical interface images (hereinafter referred to as images) of the same application in different states to this system, waits for labeling, and then proceeds to Step 2.

[0171] Step 2: Begin annotation. Let's assume the current image to be annotated is the i-th image; proceed to Step 3.

[0172] Step 3: Since the n images are labeled one by one, images 1 to i-1 have been labeled. At this point, calculate the matching images of images 1 to i-1 with the i-th image, and combine them to obtain the target matching image; proceed to step 4.

[0173] The method for calculating the matching image is the same as the matching method between the image to be labeled and the labeled image in the above embodiments of this application, and will not be repeated here.

[0174] The target matching image has the same width and height as the image to be labeled, has 1 channel, and a pixel value of 0 or a positive integer k. 0 represents a mismatch, and k represents the k-th image preceding the matching pixel (a number between 1 and i-1). Specific operations include:

[0175] For the k-th matching image, the width and height of the matching image are the same as the image to be labeled, the number of channels is 1, and the pixel value is 0 or 1, where 0 represents a mismatch and 1 represents a match. Iterate through each pixel in the k-th matching image. If a pixel's value is 0 in the image to be labeled (meaning it hasn't matched a labeled image yet) and 1 in the k-th matching image (meaning it matches the k-th labeled image), then set the pixel value of that pixel in the image to be labeled to k. Use the final image to be labeled as the target matching image.

[0176] Step 4: Based on the combined target matching image, copy the annotations from the first to i-1 images into the image to be annotated; proceed to step 5.

[0177] The transcription annotation result for the i-th image is empty by default. The process iterates through the images from 1 to i-1. For the k-th image, each annotation result (position + category) is:

[0178] If all pixels within the annotation in the target matching image are k (meaning the annotation is a complete match), then the annotation is copied into the i-th copied annotation result. Simultaneously, the system's backend records the copying relationship; for example, in the i-th image, the annotation is copied from the q-th pixel of the k-th image.

[0179] Step 5: The user continues to annotate the remaining images (location + category) on the i-th image to complete the annotation of the images to be annotated. If the annotation is not completed, go back to step 2 to annotate the (i+1)-th image. If the annotation is completed, the process ends.

[0180] In addition, if the user finds an error in a transcription, proceed to step 6; otherwise, proceed to step 7.

[0181] Step 6: If a user finds an error in a transcription annotation, the system can display the transcription source of that annotation (e.g., the qth annotation in the kth image). The user can click on this and access the annotation of the image corresponding to the transcription source, where they can make corrections. After the corrections are made, the system can automatically calculate all other annotations that have copied that annotation and correct them simultaneously.

[0182] Step 7: The user submits the remaining annotations for the i-th image. The system can verify whether the remaining annotations are completely contained in one of the images from the 1st to the (i-1)th images. If they are completely contained in one image, it means that the remaining annotations were missing in one image. In this case, the system can copy the remaining annotations back to that image.

[0183] Step 8: If an error is found in a transcription mark, the system will display the transcription source of that mark (the qth mark in the kth mark). The user can click on it to enter the image mark and make corrections. After the correction, the system will automatically calculate all places where the mark was transcribed and synchronize the correction.

[0184] In step 4, the system records the copying relationships in the background, allowing users to access the copying source. After the user makes a change, the system also uses the copying relationships to modify all annotations from that copying source.

[0185] Step 9: The user submits an annotation. The system will check whether the annotation is completely contained in a historical image. If it is completely contained, it means that the annotation was missed in the historical image. Therefore, the system will also copy the annotation back to the previous annotation.

[0186] For each remaining annotation (location + category) submitted by the user:

[0187] If, in the target matching image, all pixels within the remaining annotation have a pixel value of k (meaning the annotation is a complete match), then the remaining annotation is copied backwards into the annotation result of the k-th image, assuming it to be the q-th annotation. To maintain the unidirectionality of the copying, this remaining annotation in the i-th image is considered to be copied from the q-th annotation of the k-th image. Simultaneously, the system's backend records the copying relationship.

[0188] By maintaining the unidirectional nature of the copying process, the copying logic can be simplified. When modifying the annotations corresponding to the copying source in step 6, only the copying annotations in subsequent images need to be modified synchronously, and the copying relationships are not nested. In this way, the subsequent annotation modification process can be parallelized, which can significantly improve the running speed.

[0189] The following is combined Figures 4-10 The annotation process of the graphical interface images in the embodiments of this application will be described by way of example.

[0190] The annotation process for the graphical interface images in this application includes the following steps:

[0191] 1) The user inputs a batch of graphical interface images of the same application in different states to be labeled.

[0192] For example, the first image is as follows: Figure 4 As shown, the second image is as follows Figure 5 As shown, the third image is as follows Figure 6 As shown.

[0193] 2) Label the first image. At this time, there are no labeled images, so manually label the first image.

[0194] 3) Label the second image. At this point, the labeling results of the first image will be reused, and manual labeling will continue on this basis.

[0195] like Figure 7 As shown, the annotations are copied from the annotation results of the first image based on the matching region between the second image and the first image (see...). Figure 7 (The annotation box in the middle).

[0196] 4) Annotate the third image. At this point, the annotation results of the previous image will be reused, and manual annotation will continue on this basis.

[0197] like Figure 8 As shown, this is the matching image (including the matching region) between the third image and the first image. Figure 9 As shown, this is the matching image (including the matching area) between the third image and the second image.

[0198] Combine these two matching images to get Figure 10 As shown in the target matching image, it can be seen that after combining the two matching images, the matching area is greatly increased, with only a small area in the upper left corner remaining unmatched.

[0199] Based on the same inventive concept, this application also provides a graphical interface image annotation device. The principle of this device in solving the problem is similar to the method in the above embodiments. Therefore, the implementation of this device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0200] Reference Figure 11 As shown, this application embodiment provides a graphical interface image annotation device, including:

[0201] Module 111 is used to acquire the image to be labeled and at least two labeled images;

[0202] The matching module 112 is used to match the image to be labeled with at least two labeled images respectively to obtain at least two matching images;

[0203] The combination module 113 is used to combine at least two matching images to obtain a target matching image;

[0204] The first annotation module 114 is used to obtain the transcription annotation result of the image to be annotated based on the target matching image and the annotation results of at least two annotated images.

[0205] In this embodiment, the matching region between the image to be labeled and at least two labeled images can be calculated to obtain at least two matching images. Based on the target matching image after combining these matching images, the image to be labeled can reuse the annotations in at least two labeled images to avoid repeated annotation as much as possible, greatly reduce the annotation workload, thereby reducing the annotation cost and improving the parsing efficiency of the graphical interface.

[0206] In some possible embodiments, each matching image is marked with a matching region between the image to be labeled and the corresponding labeled image, the matching region consisting of multiple matching pixels;

[0207] The combination module 113 is specifically used for:

[0208] Mark the matching regions in the at least two matching images in the image to be labeled;

[0209] The labeled image to be labeled is used as the target matching image; wherein, the target matching image contains a labeled combined matching region, and the pixel value of each matching pixel in the combined matching region is the identifier of the corresponding labeled image.

[0210] In some possible embodiments, the first annotation module 114 is specifically used for:

[0211] For each of the at least two labeled images, perform the following operation: If in the target matching image, the pixel value of each pixel corresponding to any label in the labeled image is the identifier of that labeled image, then obtain the copying label of the image to be labeled based on that label;

[0212] Based on all the transcription annotations obtained from the annotations in at least two labeled images, obtain the transcription annotation results for the image to be annotated.

[0213] In some possible embodiments, the apparatus further includes:

[0214] The first recording module is used to record the source of each copied annotation in the copied annotation result of the image to be annotated. The source of the copied annotation is an annotation in one of at least two annotated images.

[0215] In some possible embodiments, the acquisition module 111 is specifically used for:

[0216] During the process of annotating multiple graphical interface images in sequence, after annotating at least two graphical interface images, each subsequent annotation will obtain the image to be annotated and at least two already annotated images.

[0217] In some possible embodiments, the apparatus further includes:

[0218] The first display module is used to display the copying and annotation results of the image to be annotated, so that the remaining annotations of the image to be annotated can be performed to obtain the remaining annotation results of the image to be annotated;

[0219] The second annotation module is used to obtain the annotation results of the image to be annotated based on the copied annotation results and the remaining annotation results.

[0220] In some possible embodiments, the apparatus further includes:

[0221] The second display module is used to respond to the user's triggered operation on the erroneous copying annotations in the copying annotation results and display the copying source of the erroneous copying annotations;

[0222] The third display module is used to respond to the user's triggered operation on the source of the copy and display the target annotation of the marked image corresponding to the source of the copy, so that the target annotation can be modified.

[0223] In some possible embodiments, the apparatus further includes:

[0224] The modification module is used to identify each annotation that has a copying relationship with the target annotation based on the copying source of all copied annotations in the record, and to modify each annotation based on the modified target annotation.

[0225] In some possible embodiments, the apparatus further includes:

[0226] The third annotation module is used to perform the following operations for each remaining annotation in the remaining annotation results of the image to be annotated: if it is determined that the remaining annotation is completely contained by any already annotated image, then the remaining annotation is copied in reverse to the annotation results of any already annotated image.

[0227] In some possible embodiments, the third annotation module is also used for:

[0228] If, in the target matching image, the pixel values ​​of each pixel corresponding to the remaining annotation are all identifiers of any already labeled image, then it is determined that the remaining annotation is completely contained by any already labeled image.

[0229] In some possible embodiments, the apparatus further includes:

[0230] The second recording module is used to record the reverse copying annotations in the annotation results of any annotated image as the source for copying the remaining annotations of the image to be annotated.

[0231] Based on the same inventive concept, this application also provides an electronic device. The principle of this electronic device in solving the problem is similar to the method of the above embodiments. Therefore, the implementation of this electronic device can refer to the implementation of the above method, and the repeated parts will not be described again.

[0232] See Figure 12 As shown, the electronic device may include a processor 122 and a memory 121. The memory 121 provides the processor 122 with program instructions and data stored in the memory 121. In this embodiment of the disclosure, the memory 121 may be used to store the program for generating tables in this embodiment of the disclosure.

[0233] Processor 122 executes the graphical interface image annotation method in any of the above method embodiments by calling program instructions stored in memory 121, for example... Figure 1 or Figure 6 The illustrated embodiment provides a method for annotating graphical interface images.

[0234] This disclosure does not limit the specific connection medium between the memory 121 and the processor 122 described above. This disclosure describes... Figure 12 The memory 121 and the processor 122 are connected via a bus 123, and the bus 123 is in Figure 12 The connections between other components are shown in thick lines only and are not intended to be limiting. Buses 1, 2, and 3 can be divided into address buses, data buses, control buses, etc. For ease of illustration, Figure 12 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0235] The memory may include read-only memory (ROM) and random access memory (RAM), and may also include non-volatile memory (NVM), such as at least two disk drives. Optionally, the memory may also be at least two storage devices located remotely from the aforementioned processor.

[0236] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0237] This disclosure also provides a computer storage medium storing a computer program. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the graphical interface image annotation method in any of the above method embodiments.

[0238] In specific implementation, computer storage media can include: Universal Serial Bus Flash Drive (USB), portable hard drive, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, and other storage media that can store program code.

[0239] Based on the same inventive concept as the above-described method embodiments, this application provides a computer program product comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps of any of the above-described graphical interface image annotation methods.

[0240] Computer program products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0241] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0242] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0243] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0244] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0245] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for annotating graphical interface images, characterized in that, include: Obtain the image to be labeled and at least two already labeled images; The image to be labeled is matched with the at least two labeled images to obtain at least two matched images; each matched image is marked with a matching region between the image to be labeled and the corresponding labeled image, the matching region being composed of multiple matching pixels; the width and height of each matched image are the same as the image to be labeled, the number of channels is 1, and the value of each pixel is 0 or 1, where 0 represents that the pixel does not match and 1 represents that the pixel matches; Mark the matching regions in the at least two matching images in the image to be labeled; The labeled image to be labeled is used as the target matching image; wherein, the target matching image contains a labeled combined matching region, and the pixel value of each matching pixel in the combined matching region is the identifier of the corresponding labeled image; Based on the target matching image and the annotations in the at least two annotated images, the transcription annotation result of the image to be annotated is obtained; For each copying annotation in the copying annotation result of the image to be annotated, the copying source of the copying annotation is recorded. The copying source is an annotation in one of the at least two annotated images, and the copying relationship between different copying annotations remains unidirectional. In response to a user's trigger action regarding erroneous transcription annotations in the transcription annotation results, the source of the erroneous transcription annotations is displayed; In response to a user's trigger operation for the source of copying, the target annotation of the marked image corresponding to the source of copying is displayed so as to modify the target annotation; based on the source of copying of all recorded copying annotations, each annotation that has a copying relationship with the target annotation is determined, and each annotation is modified based on the modified target annotation.

2. The method according to claim 1, characterized in that, The step of obtaining the transcription annotation result of the image to be annotated based on the target matching image and the annotations in the at least two annotated images includes: For each of the at least two labeled images, perform the following operation: if the pixel value of each pixel in the target matching image that corresponds to any label in the labeled image is the identifier of the labeled image, then obtain the transcription label of the image to be labeled based on any label; Based on all the transcription annotations obtained from the annotations in the at least two annotated images, the transcription annotation result of the image to be annotated is obtained.

3. The method according to claim 1, characterized in that, The process of obtaining the image to be labeled and at least two labeled images includes: During the process of annotating multiple graphical interface images in sequence, after annotating at least two graphical interface images, each subsequent annotation will obtain the image to be annotated and at least two already annotated images.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The transcription and annotation results of the image to be annotated are displayed to perform the remaining annotations on the image to be annotated, thereby obtaining the remaining annotation results of the image to be annotated; Based on the transcription and annotation results and the remaining annotation results, the annotation results of the image to be annotated are obtained.

5. The method according to claim 4, characterized in that, The method further includes: For each remaining annotation in the remaining annotation results of the image to be annotated, perform the following operations: If it is determined that the remaining annotations are completely contained by any already labeled image, then the remaining annotations are copied in reverse to the annotation results of any already labeled image.

6. The method according to claim 5, characterized in that, Determining that the remaining annotations are completely contained within any already labeled image includes: If, in the target matching image, the pixel value of each pixel corresponding to the remaining annotation is the identifier of any of the already labeled images, then it is determined that the remaining annotation is completely contained by any of the already labeled images.

7. The method according to claim 5, characterized in that, The method further includes: The reverse-copy annotations in the annotation results of any of the annotated images are used as the source of the remaining annotations for the image to be annotated and recorded.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 7.

Images