Method, device and electronic equipment for interface image labeling

Abstract

Embodiments of the present disclosure provide a method, device and electronic equipment for interface image labeling. In the method, a set of interface images to be labeled is aggregated to obtain at least one sub-image set, the similarity between the interface images in each sub-image set being greater than or equal to a similarity threshold; one interface image in each sub-image set is selected as a reference interface image; a user's labeling instruction for a to-be-labeled object in each reference interface image is received to obtain a labeling result; and the labeling result is used to label the remaining interface images in the at least one sub-image set. A part of the interface images can be labeled in a manual labeling manner, and then the labeled interface images are used to label the unlabeled interface images, thereby improving the labeling accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of image annotation technology, and in particular to a method, apparatus and electronic device for interface image annotation. Background Technology

[0002] The user interface (UI) of game applications includes text, UI controls, character images, etc. In automated testing scenarios for game applications, testing tools can use machine learning models to identify UI elements and automatically trigger UI controls to complete functional testing of the game application's interface. The machine learning model is trained based on annotations of UI controls within the game application's UI; therefore, the annotation of UI controls is particularly important.

[0003] Currently, computer vision technology can be used to identify UI controls on interface images for annotation. However, the accuracy of annotation in this method depends on the amount of training data of the model in computer vision technology, and the accuracy of UI control annotation cannot be guaranteed. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and electronic device for annotating interface images, which can improve the accuracy of interface image annotation.

[0005] In a first aspect, embodiments of this disclosure provide a method for annotating interface images, comprising: performing aggregation processing on a set of interface images to be annotated to obtain at least one sub-image set, wherein the similarity between interface images in each sub-image set is greater than or equal to a similarity threshold; selecting an interface image as a reference interface image in each sub-image set; receiving a user's annotation instruction for an object to be annotated in each reference interface image to obtain an annotation result; and using the annotation result to annotate the remaining interface images in the at least one sub-image set.

[0006] Secondly, embodiments of this disclosure provide an apparatus for annotating interface images, comprising:

[0007] The processing module is used to aggregate the set of interface images to be labeled to obtain at least one sub-image set, wherein the similarity between the interface images in each sub-image set is greater than or equal to a similarity threshold; and selects one interface image in each sub-image set as a reference interface image.

[0008] The transceiver module is used to receive annotation instructions from the user for each object to be annotated in the reference interface image and obtain the annotation results.

[0009] The processing module is further configured to use the annotation results to annotate the remaining interface images in the at least one sub-image set.

[0010] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the processor to perform the interface image annotation method as described in the first aspect above.

[0011] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the interface image annotation method described in the first aspect above.

[0012] Fifthly, embodiments of this disclosure provide a computer program product, including computer instructions that, when executed by a processor, implement the interface image annotation method described in the first aspect above.

[0013] The interface image annotation method, apparatus, and electronic device provided in this embodiment can annotate a portion of the interface images manually, and then use the annotated interface images to annotate the unannotated interface images. By relying on the matching similarity relationship between interface images, the accuracy of annotation can be improved. Furthermore, manual annotation can ensure the accuracy of annotation, and using manually annotated interface images to annotate the unannotated interface images can further improve the accuracy of annotation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of an interface image;

[0016] Figure 2 This is a schematic diagram of two different game interface images;

[0017] Figure 3A A flowchart illustrating a method for annotating interface images provided in an embodiment of this disclosure;

[0018] Figure 3B Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0019] Figure 3C Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0020] Figure 4 This is a schematic diagram of a sampling method according to an embodiment of the present disclosure;

[0021] Figure 5A Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0022] Figure 5B Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0023] Figure 6 Another schematic diagram of sampling for an embodiment of this disclosure;

[0024] Figure 7 Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0025] Figure 8 Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0026] Figure 9 Another flowchart illustrating the interface image annotation method provided in this embodiment of the disclosure;

[0027] Figure 10 A schematic diagram of a process for constructing a first annotation dictionary provided in an embodiment of this disclosure;

[0028] Figure 11 Structural block diagram of the interface image annotation device provided in the embodiments of this disclosure;

[0029] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] Figure 1 This is a schematic diagram of an interface image. Figure 1 The following explanation uses a game interface image as an example. (Refer to...) Figure 1The game interface image may include multiple objects, such as text 10, operable controls 11, and game character images 12. Operable controls 11 can be understood as controls that can be manipulated by the user, such as controls that the user can click, swipe, or long-press. Operable controls 11 can be text, images, or links, etc. This embodiment does not limit the form of the operable controls 11.

[0032] In one embodiment, the operable control may be referred to as a user interface (UI) control.

[0033] In one embodiment, labeling each object in the game interface image can be understood as: labeling the area where the object is located with a corresponding label, such as... Figure 1 The area where the game character image 12 is located is labeled "game character image". Figure 1 The area where the operable control 11 is located is labeled as "property control", etc. The labels shown here are for illustrative purposes only and should not be used as a limitation on the form of the labels.

[0034] Existing methods for annotating objects in game interface images can include the following three aspects:

[0035] 1. Inserting stubs into the runtime code of the game application to annotate text, UI controls, and other objects in various game interface images. However, this method is unfriendly to non-game developers. Firstly, the stub insertion strategy requires modifying the game application's runtime code, which is difficult, and the runtime code of the game application is constantly iterated with each version, making changes to this strategy often costly. Secondly, not all users have permission to access the game's source code. In one embodiment, the user in this disclosure can be a staff member annotating objects in the interface images.

[0036] 2. Using computer vision technology, a pre-trained network model can be used to annotate objects on the interface image. However, the accuracy of the annotation depends on the amount of training data of the network model and cannot guarantee the accuracy of the interface annotation.

[0037] Figure 2 The image shown is a schematic diagram of two different game interface images within the same version. For example... Figure 2 a and Figure 2 As shown in b, although the color composition of game scenes is relatively complex, for the same version of the game, the shape of the same operable control in the game interface image tends to remain consistent at the same resolution. For example, as... Figure 2 a and Figure 2The character list on the left (Character 1, Character 2, Character 3, etc.) and the text display on the right (attributes, weapons, artifacts, etc.) in the game interface image are all located in the same area (because they are similar controls). Based on this, if we find... Figure 2 a and Figure 2 The matching relationship to the left of 'b' in the equation can then be... Figure 2 The labels of the operable controls already marked in 'a' will be moved to the corresponding... Figure 2 On the operable controls of b in the game, on the one hand, there is no need to embed stubs in the running code of the game application, which can reduce the difficulty and cost of annotation. On the other hand, the annotation accuracy does not depend on the amount of training data of the network model, but on the matching similarity relationship between interface images, which can improve the accuracy of annotation.

[0038] Thus, this disclosure provides a method for annotating interface images, which can annotate unannotated interface images based on the similarity between annotated and unannotated interface images. Furthermore, to further improve annotation accuracy, the annotated interface images in this disclosure are manually annotated, and to reduce the workload of manual annotation, the interface images can be de-sampling, as described in the following embodiments.

[0039] It should be understood that the embodiments of this disclosure use game interface images as an example for illustration, but the interface image annotation method provided in the embodiments of this disclosure can also be applied to the annotation scenarios of interface images of other types of applications.

[0040] In one embodiment, the executing entity for the interface image annotation method provided in this disclosure can be an electronic device, such as a server, computer, or other device with a display screen and computing capabilities. This disclosure does not depict the physical form of the electronic device; the following embodiments use an electronic device as an example to illustrate the execution of the interface image annotation method provided in this disclosure.

[0041] The method for annotating interface images provided in this disclosure will be described below with reference to specific embodiments. These embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0042] refer to Figure 3A , Figure 3A This is a schematic flowchart illustrating a method for annotating interface images according to an embodiment of the present disclosure. The method for annotating interface images includes:

[0043] S301A, perform aggregation processing on the set of interface images to be labeled to obtain at least one sub-image set, wherein the similarity between the interface images in each sub-image set is greater than or equal to the similarity threshold.

[0044] In one embodiment, the set of interface images to be labeled can be a set of interface images from a game application. In another embodiment, the set of interface images to be labeled can be derived from actual video screenshots of a game application. As in this embodiment of the present disclosure, the electronic device can perform frame extraction processing on the video (i.e., the actual video) to be labeled in advance to obtain the set of interface images to be labeled. For example, the electronic device can extract one frame of interface image from the video to be labeled every 1 second to obtain the set of interface images to be labeled.

[0045] In one embodiment, the electronic device may employ k-means clustering, deep hierarchical traversal, or other algorithms to aggregate the set of interface images to be labeled, thereby obtaining at least one sub-image set. The similarity between the interface images in each sub-image set is greater than or equal to a corresponding similarity threshold.

[0046] S302A: Select one interface image as the reference interface image in each sub-image set.

[0047] In one embodiment, the electronic device may use any interface image from each sub-image set as the reference interface image for each sub-image set. Alternatively, in one embodiment, the electronic device may use the interface image with the highest number of similarity values ​​greater than or equal to a similarity threshold in each sub-image set as the reference interface image for each sub-image set.

[0048] S303A receives annotation instructions from the user for the objects to be annotated in each reference interface image and obtains the annotation results.

[0049] In this embodiment of the disclosure, the user can manually annotate the objects to be annotated in each reference interface image. In one embodiment, when the set of images to be annotated is a game interface image, the objects to be annotated can be UI controls in the game interface image. It should be understood that the user can be understood as the staff member annotating the interface images.

[0050] In one embodiment, after selecting one interface image as a reference interface image in each sub-image set, the electronic device can display (or sequentially display) each reference interface image on the interface, allowing the user to manually annotate each reference interface image on the electronic device's interface. In this way, the electronic device can obtain the results of manual annotation.

[0051] S304A, using the annotation results, annotate the remaining interface images in at least one sub-image set.

[0052] In this embodiment of the application, after obtaining the manual annotation results of each reference interface image, the electronic device can annotate the remaining interface images in at least one sub-image set based on the manual annotation results.

[0053] In one embodiment, because the similarity between interface images in each sub-image set is greater than a similarity threshold, it indicates that the interface images in each sub-image set are relatively similar. In one embodiment, the similarity threshold of this application can be preset to be sufficiently large so that the interface images in each sub-image set are similar. Therefore, the electronic device can label UI controls that are in the same position as UI controls in the reference interface image with the same label. For example, if the reference interface image in sub-image set 1 is interface image 1, and the UI control at position A in interface image 1 is labeled as the "Start" control, then the UI control at position A in interface image 2 in sub-image set 1 can also be labeled as the "Start" control.

[0054] In one embodiment, the electronic device can use a control recognition model to annotate the remaining interface images based on the manual annotation results of each reference interface image. For example, if the reference interface image in sub-image set 1 is interface image 1, and a UI control in interface image 1 is labeled as a "start" control, then the electronic device can use the control recognition model to identify whether there exists a target control in interface image 2 of sub-image set 1 that has a similarity greater than a preset similarity to the start control in interface image 1. If it exists, the electronic device can label the target control in interface image 2 as a "start" control. If it does not exist, the electronic device can ignore the target control and manually annotate it.

[0055] In this embodiment, a portion of the interface images can be annotated manually. Then, the unannotated interface images can be annotated using the similarity between the annotated and unannotated interface images (or other methods). Annotation based on the matching similarity relationship between interface images can improve the accuracy of the annotation. Furthermore, manual annotation can ensure the accuracy of the annotation. Therefore, using manually annotated interface images to annotate unannotated interface images can improve the accuracy of the annotation.

[0056] In this embodiment, after the electronic device annotates the interface images in each sub-image set, the annotation results of all interface images can be obtained. In one embodiment, the electronic device can use the annotation results of all interface images to train a recognition model. In the automated testing scenario of game applications, the testing tool can use this recognition model to identify the UI and automatically trigger UI controls to complete the functional testing of the game application's interface. The recognition model is trained based on the annotation results of the set of interface images to be annotated.

[0057] In this embodiment, reference is made to Figure 3B Following S304A, the electronic device can also execute S305A:

[0058] S305A uses interface images from a set of labeled interface images as training samples to train a recognition model, which is then used to identify objects to be labeled in the interface images.

[0059] The object to be annotated can be a UI control in the interface of a game application.

[0060] After the electronic device uses the annotation results to annotate the remaining interface images in the at least one sub-image set, it can obtain the annotation result for each interface image, i.e., the annotated interface image. The electronic device can use the interface images in the annotated interface image set as training samples to train a recognition model. The recognition model is used to identify the objects to be annotated in the interface images. The process of training the recognition model based on training samples is not described in detail in this embodiment; however, the training process of machine learning models in the prior art can be referred to.

[0061] As above Figure 3A The example shown illustrates a single aggregation process on the set of interface images to be annotated, resulting in at least one aggregated sub-image set. However, even after this aggregation, the image data volume in each sub-image set remains substantial, necessitating a large amount of manually annotated reference interface images. To reduce the amount of manually annotated reference interface images and improve annotation accuracy and efficiency, one embodiment employs different similarity thresholds for aggregation on the set of interface images to be annotated, achieving multi-level deduplication.

[0062] refer to Figure 3C , Figure 3C Another flowchart illustrating the interface image annotation method provided in this disclosure embodiment. The interface image annotation method includes:

[0063] S301, the interface image set to be labeled is aggregated using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. The similarity between the interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold.

[0064] In S301A, at least one sub-image set may include: multiple sub-image sets corresponding to each similarity threshold.

[0065] Taking the k-means clustering algorithm as an example, the similarity threshold between interface images is used as a clustering parameter to aggregate the set of interface images to be labeled. In this embodiment, different similarity thresholds can be used to aggregate the set of interface images to be labeled, so as to perform multiple deduplication processes on the set of interface images to be labeled. For example, refer to Figure 4 Different similarity thresholds can be used, such as 0.95, 0.85, and 0.75. The electronic device can first use 0.75 as the clustering parameter to aggregate the set of interface images to be labeled, obtaining multiple sub-image sets corresponding to 0.75. In other words, the electronic device can aggregate interface images with a similarity greater than 0.75 to form a sub-image set, thus obtaining multiple sub-image sets corresponding to 0.75. Among these, the similarity between interface images in each sub-image set corresponding to 0.75 is greater than or equal to the corresponding similarity threshold.

[0066] In one embodiment, the similarity between interface images can be the overall similarity of the interface images. In another embodiment, the electronic device may pre-store a trained first neural network model, which is used to obtain the similarity between interface images. In this embodiment, the electronic device can input the set of interface images to be labeled into the first neural network model to obtain the similarity between the interface images, and then use 0.75 as a clustering parameter to aggregate the set of interface images to be labeled to obtain multiple sub-image sets corresponding to 0.75.

[0067] Similar to a similarity threshold of 0.75, electronic devices can also aggregate the interface image set to be labeled using a clustering parameter of 0.85 to obtain multiple sub-image sets corresponding to 0.85, and aggregate the interface image set to be labeled using a clustering parameter of 0.95 to obtain multiple sub-image sets corresponding to 0.95. Specifically, the similarity between interface images in each sub-image set corresponding to 0.85 is greater than or equal to the corresponding similarity threshold, and the similarity between interface images in each sub-image set corresponding to 0.95 is greater than or equal to the corresponding similarity threshold.

[0068] Taking the deep hierarchical traversal algorithm as an example, the electronic device can sequentially obtain the similarity between the first interface image in the interface image set to be labeled and all other interface images, and then aggregate the images whose similarity with the first interface image is greater than the similarity threshold. The electronic device can sequentially obtain the similarity between the second interface image in the interface image set to be labeled and all other interface images, and then aggregate the images whose similarity with the second interface image is greater than the similarity threshold. By doing so, the electronic device can obtain multiple sub-image sets corresponding to the similarity threshold.

[0069] Using this method, electronic devices can also aggregate the interface image sets to be labeled using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. For example, the electronic device can obtain multiple sub-image sets corresponding to 0.75, multiple sub-image sets corresponding to 0.85, and multiple sub-image sets corresponding to 0.95.

[0070] S302, select one interface image as the first reference interface image in each sub-image set corresponding to the first similarity threshold, where the first similarity threshold is the minimum value among different similarity thresholds.

[0071] The first similarity threshold is the minimum value among different similarity thresholds. For example, when different similarity thresholds include 0.75, 0.85 and 0.95, the first similarity threshold is 0.75.

[0072] In this embodiment of the present disclosure, the electronic device may select one interface image as a first reference interface image from each sub-image set corresponding to a first similarity threshold. In the following embodiments, each sub-image set corresponding to the first similarity threshold is referred to as a first sub-image set. In one embodiment, the electronic device may use any interface image from each first sub-image set as the first reference interface image for each first sub-image set.

[0073] Alternatively, in one embodiment, the electronic device may use the interface image with the highest number of similarities greater than or equal to a first similarity threshold in each first sub-image set as the first reference interface image for each first sub-image set. For example, when acquiring a first sub-image set, the electronic device may acquire the similarity between each interface image in the first sub-image set and other interface images, and the electronic device may use the interface image with the highest number of similarities greater than or equal to the first similarity threshold in the first sub-image set as the first reference interface image for that first sub-image set.

[0074] S303, receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and obtain the first annotation result of the first similarity threshold.

[0075] In this embodiment of the disclosure, the user can manually annotate the objects to be annotated in each first reference interface image. In one embodiment, when the set of images to be annotated is a game interface image, the objects to be annotated can be UI controls in the game interface image.

[0076] In one embodiment, after the electronic device selects an interface image as a first reference interface image in each sub-image set corresponding to the first similarity threshold, it can display (or sequentially display) each first reference interface image on the interface, so that the user can manually annotate each first reference interface image on the interface of the electronic device.

[0077] When a user annotates an object to be annotated in each first reference interface image, the electronic device can receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and thus obtain a first annotation result with a first similarity threshold. The first annotation result may include the labels of the objects to be annotated in each region of each first reference interface image.

[0078] The reason for selecting one interface image as the first reference interface image in each sub-image set corresponding to the first similarity threshold in this embodiment, instead of selecting reference interface images in each sub-image set corresponding to other similarity thresholds, is to reduce the number of manually annotated interface images. The first similarity threshold is the minimum among different similarity thresholds, resulting in the fewest sub-image sets corresponding to the first similarity threshold. Therefore, the number of reference images selected in each sub-image set is also the fewest, thus reducing the workload of manually annotating interface images.

[0079] S304. Using the first annotation result, the interface images in each sub-image set corresponding to other similarity thresholds are annotated.

[0080] Since the user has already labeled each first reference image, the electronic device can use the first labeling results to label the interface images in each sub-image set corresponding to other similarity thresholds. Figure 4 As shown, if each sub-image set corresponding to the first similarity threshold is taken as an m-level sub-image set, then the first reference image is included in the sub-image sets corresponding to other similarity thresholds. In one embodiment, if each sub-image set corresponding to other similarity thresholds is taken as an m-1 level sub-image set, then the m-1 level sub-image set may include the labeled first reference interface image, as well as other interface images in the unlabeled m-1 level sub-image sets.

[0081] In one embodiment, the electronic device can obtain the similarity between each first reference interface image and each other interface image in the unlabeled m-1 level sub-image set, use the labeling results of the first reference interface image to automatically label other interface images with similarity greater than or equal to a preset similarity, and manually label other interface images with similarity less than the preset similarity.

[0082] For example, with a preset similarity of 0.8, taking a first reference interface image as an example, the electronic device can obtain the similarity between the first reference interface image and each other interface image in the unlabeled m-1 level sub-image set. For other interface images with similarity less than the preset similarity, manual labeling is performed, which can be referred to the relevant description of manual labeling in S303. For other interface images with similarity greater than or equal to the preset similarity, the electronic device can label the corresponding area of ​​the other interface images as the corresponding area of ​​the first reference interface image, thereby completing the labeling of the other interface images.

[0083] In this embodiment, a portion of the interface images can be manually annotated. The similarity between the annotated and unannotated interface images is then used to annotate the unannotated images. This approach eliminates the need for instrumentation and stubbing in the game application's runtime code, reducing the difficulty and cost of annotation. Furthermore, annotation accuracy relies on the matching similarity relationships between interface images, rather than the amount of training data for the network model, thus improving accuracy. Manual annotation also ensures accuracy; using manually annotated interface images to annotate unannotated images further enhances accuracy. Additionally, to reduce the workload of manual annotation, the interface images can be downsampled to allow for manual annotation of the subset of images with the least amount of data.

[0084] exist Figure 3C Based on the illustrated embodiment, the following details the process of "aggregating the interface image sets to be annotated using different similarity thresholds" and "annotating the interface images in each sub-image set corresponding to other similarity thresholds using the first annotation result." (Refer to...) Figure 5A The interface image annotation method provided in this disclosure embodiment may include:

[0085] S501, in the set of interface images to be labeled, the interface images with a similarity threshold greater than or equal to the first similarity threshold are aggregated to obtain multiple first sub-image sets corresponding to the first similarity threshold.

[0086] The different similarity thresholds include: a first similarity threshold, a second similarity threshold, and a third similarity threshold. In one embodiment, the first similarity threshold, the second similarity threshold, and the third similarity threshold indicate that the electronic device can perform aggregation processing on the interface image set to be annotated using at least three similarity thresholds. In this embodiment, the first similarity threshold, the second similarity threshold, and the third similarity threshold are used as examples for illustration.

[0087] Wherein, the first similarity threshold is less than the second similarity threshold, and the second similarity threshold is less than the third similarity threshold. This disclosure does not provide a detailed description of the aggregation processing method; however, the aggregation processing method can be the k-means clustering algorithm, a deep hierarchical traversal algorithm, etc. The aggregation processing method can be referred to the relevant description in S301. The following explanation uses a first similarity threshold of 0.75, a second similarity threshold of 0.85, and a third similarity threshold of 0.95 as examples.

[0088] In this embodiment of the disclosure, the electronic device can aggregate interface images with a similarity threshold greater than or equal to a first similarity threshold from a set of interface images to be labeled, thereby obtaining multiple first sub-image sets corresponding to the first similarity threshold. For example... Figure 6 As shown, the first similarity threshold of 0.75 corresponds to the two first sub-image sets 1 and 2.

[0089] S502, in the multiple first sub-image sets corresponding to the first similarity threshold, the interface images with a similarity threshold greater than or equal to the second similarity threshold are aggregated to obtain multiple second sub-image sets corresponding to the second similarity threshold.

[0090] Because the similarity between interface images in each first sub-image set is greater than or equal to a first similarity threshold, and the similarity between interface images in different first sub-image sets is less than the first similarity threshold, the electronic device can aggregate interface images with a similarity threshold greater than or equal to a second similarity threshold from multiple first sub-image sets corresponding to the first similarity threshold to obtain multiple second sub-image sets corresponding to the second similarity threshold. It should be noted that the multiple sub-image sets corresponding to the second similarity threshold can be referred to as second sub-image sets, i.e., the second similarity threshold corresponds to multiple second sub-image sets.

[0091] In other words, the electronic device can further aggregate interface images with a similarity threshold greater than or equal to a second similarity threshold within each first sub-image set to obtain multiple second sub-image sets corresponding to the second similarity threshold. For example, refer to... Figure 6 As shown, the electronic device can aggregate interface images with a similarity threshold greater than or equal to 0.85 in the first sub-image set 1 to obtain the second sub-image sets 3 and 4. Similarly, the electronic device can aggregate interface images with a similarity threshold greater than or equal to 0.85 in the first sub-image set 2 to obtain the second sub-image sets 5 and 6.

[0092] S503, in the multiple second sub-image sets corresponding to the second similarity threshold, the interface images with similarity thresholds greater than or equal to the third similarity threshold are aggregated to obtain multiple third sub-image sets corresponding to the third similarity threshold.

[0093] Similarly to S502 above, since the similarity between interface images in each second sub-image set is greater than or equal to the second similarity threshold, and the similarity between interface images in different second sub-image sets is less than the second similarity threshold, the electronic device can aggregate interface images with a similarity threshold greater than or equal to the third similarity threshold in multiple second sub-image sets corresponding to the second similarity threshold to obtain multiple third sub-image sets corresponding to the third similarity threshold. It should be noted that the multiple sub-image sets corresponding to the third similarity threshold can be referred to as third sub-image sets, i.e., the third similarity threshold corresponds to multiple third sub-image sets.

[0094] In other words, the electronic device can further aggregate interface images with a similarity threshold greater than or equal to a third similarity threshold within each second sub-image set to obtain multiple third sub-image sets corresponding to the third similarity threshold. For example, refer to... Figure 6 As shown, the electronic device can aggregate interface images with a similarity threshold greater than or equal to 0.95 in the second sub-image set 3 to obtain the third sub-image sets 7 and 8. Similarly, the electronic device can also obtain the third sub-image sets 9, 10, 11, 12, 13 and 14.

[0095] S504, select one interface image as the first reference interface image in each sub-first image set corresponding to the first similarity threshold.

[0096] S504 can be referred to the description of S302, and will not be repeated here.

[0097] S505, select one interface image as the second reference interface image in each second sub-image set corresponding to the second similarity threshold, and select one interface image as the third reference interface image in each third sub-image set corresponding to the third similarity threshold.

[0098] It should be understood that there is no specific order between S505 and S504; they can be executed simultaneously.

[0099] In S505, selecting one interface image as the second reference interface image in each second sub-image set corresponding to the second similarity threshold, and selecting one interface image as the third reference interface image in each third sub-image set corresponding to the third similarity threshold, can refer to the processing procedure of "the electronic device selecting one interface image as the first reference interface image in each sub-first image set corresponding to the first similarity threshold", which can be referred to the description in S302.

[0100] In other words, in this embodiment of the present disclosure, the electronic device can select one interface image as a reference interface image in each sub-image set corresponding to each similarity threshold. For example, the reference interface image in each first sub-image set is called the first reference interface image, the reference interface image in each second sub-image set is called the second reference interface image, and the reference interface image in each third sub-image set is called the third reference interface image.

[0101] pass Figure 6 It can be seen that for each first sub-image set, each first reference interface image is contained in a second sub-image set, that is, the second sub-image set contains one first reference interface image. Similarly, for each second sub-image set, each second reference interface image is contained in a third sub-image set, that is, the third sub-image set contains one second reference interface image. In one embodiment, the multiple first reference interface images can be referred to as an n-level image set, the multiple second reference interface images as an n-1-level image set, and the multiple third reference interface images as an n-2-level image set. Here, n is an integer greater than 2.

[0102] S506, receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and obtain the first annotation result of the first similarity threshold.

[0103] S506 can be referred to in the description of S303, and will not be repeated here. In other words, electronic devices can perform manual annotation on an n-level image set.

[0104] S507, using the first annotation result, annotate the second reference interface image in each second sub-image set to obtain the second annotation result of the second similarity threshold.

[0105] In this embodiment of the disclosure, the electronic device may use the first annotation result, namely the annotation result of each first reference interface image, to annotate the second reference interface image in each second sub-image set.

[0106] In one embodiment, the electronic device can acquire the similarity between each second reference interface image and each first reference interface image, and then annotate the corresponding second reference interface image based on the annotation results of the first reference interface images whose similarity is greater than or equal to a preset similarity. For second reference interface images whose similarity is less than the preset similarity, a manual annotation method can be used.

[0107] The preset similarity in S507 can be called the first preset similarity. In other words, in the similarity between each second reference interface image and each first reference interface image, the electronic device can determine whether the number of second reference interface images corresponding to a similarity greater than or equal to the first preset similarity is 0. If so, it indicates that the similarity between the second reference interface image and all other second reference interface images is less than the first preset similarity, and the second reference image can be labeled using manual annotation. If not, it indicates that the number of second reference interface images corresponding to a similarity greater than or equal to the first preset similarity is greater than 0, indicating that there exists a first reference interface image and the second reference image with a similarity greater than the first preset similarity. In this case, the electronic device can use the annotation result of the first reference interface image to label the second reference interface image.

[0108] For example, if the similarity between a second reference interface image and a first reference interface image is greater than or equal to a preset similarity, then the labels of each labeled region in the first reference interface image can be labeled in the corresponding region of the second reference interface image. If the similarity between a second reference interface image and all first reference interface images is less than the preset similarity, then the second reference interface image can be labeled manually.

[0109] In one embodiment, to improve annotation accuracy, the electronic device can acquire the mixed similarity between each second reference interface image and each first reference interface image, and then annotate the second reference interface images based on the mixed similarity. The mixed similarity is a weighted similarity of overall similarity and local similarity. Specifically, the overall similarity is directly calculated to match the similarity between the two interface images, aiming to match images from the same scene, while the local similarity aims to match images with the same region of interest. In one embodiment, the region of interest can be the already annotated region of an already annotated image (such as the first reference image), that is, the region inside the rectangle of the ground truth.

[0110] In this embodiment, reference is made to Figure 5B S507 may include:

[0111] S5071, obtain the first similarity between each second reference interface image and each first reference interface image, where the first similarity is the overall similarity between the interface images.

[0112] The electronic device can obtain a first similarity between each second reference interface image and each first reference interface image. The first similarity is the overall similarity between each second reference interface image and each first reference interface image. For example, the electronic device can input each second reference interface image and each first reference interface image into a first neural network model to obtain the overall similarity between each second reference interface image and each first reference interface image.

[0113] S5072, in each second reference interface image, obtain the second similarity at the corresponding position of each first region in each first reference interface image.

[0114] The second similarity is the local similarity between each reference interface image and each first reference interface image.

[0115] In one embodiment, the electronic device can obtain a second similarity in each second reference interface image to the position corresponding to each first region in each first reference interface image. For example, regarding the second similarity between a second reference interface image and a first reference interface image, the electronic device can obtain the similarity between each labeled region (i.e., first region) in the first reference interface image and the corresponding region in the second reference interface image, and then average (or weighted average) the similarities between the regions to obtain the second similarity between the second reference interface image and the first reference interface image.

[0116] In one embodiment, the electronic device may store a second neural network model, which is used to obtain a second similarity between corresponding regions of an image. Thus, the electronic device can input each second reference interface image and each first reference interface image into the second neural network model to obtain a second similarity (i.e., local similarity) between each second reference interface image and each first reference interface image.

[0117] S5073, based on the first similarity and the second similarity, obtain the similarity between each second reference interface image and each first reference interface image.

[0118] In one embodiment, the similarity (i.e., mixed similarity) between each second reference interface image and each first reference interface image can be: the average of the first similarity and the second similarity between each second reference interface image and each first reference interface image, or a weighted value of the first similarity and the second similarity. In one embodiment, the weights of the first similarity and the second similarity can be the same or different.

[0119] S5074, in the similarity between each second reference image and each first reference interface image, determine whether the number of similarities greater than or equal to the first preset similarity is 0. If yes, proceed to S5076; otherwise, proceed to S5075.

[0120] For each second reference image, the electronic device can detect whether the number of similarities between the second reference image and all first reference interface images that is greater than or equal to a first preset similarity is 0. For example, for second reference interface image 1 and second reference interface image 2, the number of similarities between second reference interface image 1 and all first reference interface images that is greater than or equal to the first preset similarity is 10, and the number of similarities between second reference interface image 2 and all first reference interface images that is greater than or equal to the first preset similarity is 0.

[0121] S5075, using the annotation results of the first reference interface image of the target, annotate the second reference interface image of the target.

[0122] In this embodiment, a second reference interface image with a non-zero number of similarities is designated as the target second reference interface image, and a first reference interface image with a similarity greater than a first preset similarity is designated as the target first reference interface image. In this embodiment, the electronic device can use the annotation results of the target first reference interface images to annotate the target second reference interface images. For example, if the number of similarities greater than or equal to the first preset similarity for second reference interface image 1 is 10, then second reference interface image 1 can be called the target second reference interface image, and correspondingly, a first reference interface image with a similarity greater than or equal to the first preset similarity for second reference interface image 1 can be called the target first reference interface image.

[0123] In one embodiment, the electronic device may use the annotation results of the target first reference interface image (i.e., the first reference interface image whose similarity to the second reference interface image 1 is greater than or equal to the first preset similarity) corresponding to the target second reference interface image to annotate the target second reference interface image.

[0124] In one embodiment, the electronic device may annotate the target second reference interface image in response to the following: the number of times the target second reference interface image corresponds to a first preset similarity value is greater than a preset number; the first similarity value between the target second reference interface image and the target first reference interface image is greater than or equal to a second preset similarity value; and the second similarity value between the target second reference interface image and the target first reference interface image is greater than or equal to a third preset similarity value. In one embodiment, at least two of the first preset similarity value, the second preset similarity value, and the third preset similarity value may be the same or different.

[0125] The electronic device uses the annotation results of the target first reference interface image corresponding to the target second reference interface image to annotate the target second reference interface image. Specifically, this can be done by labeling the regions of the target first reference interface image onto the corresponding regions of the target second reference interface image. In one embodiment, if there is more than one target first reference interface image whose similarity to the target second reference interface image is greater than or equal to a first preset similarity, the electronic device can label the regions of the target first reference interface image with the highest similarity to the target second reference interface image onto the corresponding regions of the target second reference interface image.

[0126] In this embodiment, the second annotation result of the second similarity threshold includes: the position of each region of each second reference interface image, and the label.

[0127] In one embodiment, in response to a quantity greater than a preset quantity, and a first similarity between the target second reference interface image and any first reference interface image being less than a second preset similarity, and / or a second similarity between the target second reference interface image and any first reference interface image being less than a third preset similarity, the electronic device may manually annotate the target second reference interface image. The electronic device may annotate the target second reference interface image according to a third annotation instruction from the user for the objects to be annotated in the target second reference interface image. It should be understood that the target second reference interface image in this embodiment is a second reference interface image whose first similarity is less than the second preset similarity, and / or whose second similarity is less than the third preset similarity.

[0128] In one embodiment, the annotation result of the target second reference interface image can be referred to as the second annotation result.

[0129] The manual annotation method can be found in the relevant description in S303.

[0130] S5076, according to the user's second annotation instruction for the object to be annotated in each second reference interface image, annotate each second reference interface image.

[0131] For a second reference interface image, if the number of second reference interface images with a similarity greater than or equal to the first preset similarity among all first reference interface images is 0, such as second reference interface image 2, the second reference interface image 2 can be labeled manually to avoid the problem of inaccurate automatic labeling.

[0132] In this embodiment, the electronic device can annotate the second reference interface images corresponding to a number of objects with a similarity greater than or equal to the first preset similarity, according to the user's second annotation instruction for the objects to be annotated in each second reference interface image. The manual annotation method can be referred to the relevant description in S303.

[0133] In one embodiment, the annotation result for each second reference interface image can be referred to as the second annotation result.

[0134] S508, using the second annotation results, annotate each third reference interface image.

[0135] In this embodiment of the disclosure, after obtaining the second annotation result, the electronic device can annotate each third reference interface image to obtain a third annotation result. The specific annotation process can be referred to the relevant description in S507 of "using the first annotation result to annotate each second reference interface image".

[0136] In this embodiment of the disclosure, the multiple third reference interface images are n-2 level sub-image sets. Assuming that three similarity thresholds (i.e., the first similarity threshold, the second similarity threshold, and the third similarity threshold) are selected in this embodiment of the disclosure, the n-2 level sub-image set is the set of the last reference interface images. In this embodiment, the electronic device can annotate other interface images in each third sub-image set corresponding to the third similarity threshold according to the third annotation result.

[0137] The electronic device can acquire the mixed similarity between each third reference interface image and each other interface image, and annotate the other interface images in each third sub-image set according to the mixed similarity. This process can be referred to the relevant descriptions in S5071-S5075.

[0138] In this embodiment, the simplified process of semi-automatic annotation can be referred to Figure 7 , Figure 8 As shown. It should be understood that, Figure 8 The manual labeling is S5076, and the automatic labeling is S5075.

[0139] In this embodiment of the disclosure, the electronic device can perform multi-level deduplication on the interface image set to be labeled. The goal of multi-level deduplication is to divide the complete large dataset hierarchically, thereby reducing the number of labels in the lower-level representative dataset while improving the labeling quality of the semi-automatic labeling, thus reducing the cost of subsequent manual fine-tuning. By adjusting the similarity threshold, the number of interface images to be labeled can be successively reduced to only 10%, 1%, etc., of the original amount, thereby reducing the amount of subsequent similarity calculation and matching.

[0140] As above Figure 5A and Figure 5BIn the example shown, labels for each region of the target first reference interface image corresponding to the target second reference interface image are used to mark the corresponding region of the target second reference interface image. In one scenario, if the region to be labeled in the target second reference interface image is not included in the regions of the target first reference interface image, then the region to be labeled in the target second reference interface image cannot be labeled.

[0141] Therefore, in this embodiment of the present disclosure, a first annotation dictionary can be generated based on the annotation results of all the target first reference interface images corresponding to the target second reference interface image. The target second reference interface image can be annotated using the annotation results of all the target first reference interface images corresponding to the target second reference interface image (i.e., the first annotation dictionary). This allows for complete annotation of all areas to be annotated in the target second reference interface image, thereby improving the annotation accuracy and precision of the target second reference interface image.

[0142] Reference Figure 9 In the interface image annotation method provided in this embodiment, the above-mentioned S5075 can be replaced by S901-S903.

[0143] S901, the position of each first region in the target first reference interface image is used as the key value of the first annotation dictionary, and the first p-hash value and label of each first region in the target first reference interface image are used as the value value of the first annotation dictionary to construct the first annotation dictionary.

[0144] The first annotation result includes: the position and label of at least one first region in each first reference interface image. In one embodiment, the position of the first region can be referred to as the coordinate information of a rectangular box (region), such as the coordinates (x0, y0, x1, y1) of the four points of the box on the interface image. The label of the first region is called a label. In this embodiment, after obtaining the annotation result of each first reference interface image, the electronic device can obtain the p-hash value of each first region of each first reference interface image. It should be understood that the p-hash value is the p-hash value of the first region after the first reference interface image has undergone hash hashing. This embodiment does not describe the hash hashing process and the process of obtaining the p-hash value of the first region in detail, but can refer to the description in the prior art.

[0145] By traversing all target first reference interface images of the target second reference interface image, the coordinate information, label, and p-hash value of the rectangular box corresponding to each target first reference interface image can be obtained. In this embodiment of the disclosure, the electronic device can construct a first annotation dictionary based on the coordinate information, label, and p-hash value of each rectangular box of all target first reference interface images of the target second reference interface image. The first annotation dictionary is a set of the coordinate information, label, and p-hash value of each rectangular box of all target first reference interface images of the target second reference interface image.

[0146] For example, the first labeled dictionary can be {(x0, y0, x1, y1):(p-hash1, label1), (x2, y02, x3, y3):(p-hash2, label2)……(xn, yn, xn+1, yn+1):(p-hash1, labeln)}.

[0147] For example, the process of constructing the first annotation dictionary in S901 can be simplified to: Figure 10 As shown. It should be understood that, Figure 10 The image displayed is the target first reference interface image, and the boxes are the rectangular boxes representing multiple labeled areas (first areas) in the target first reference interface image.

[0148] In one embodiment, the electronic device can select the top N target first reference interface images whose similarity to the target second reference interface image is ranked first, and then construct a first annotation dictionary based on these top N target first reference interface images. In this embodiment, constructing the first annotation dictionary based on the top N target first reference interface images ensures the accuracy of the first annotation dictionary and avoids using all the target first reference interface images corresponding to the target second reference interface image, thus reducing computational load and improving annotation efficiency. Here, N is an integer greater than 1.

[0149] S902, traverse each first region in the first annotation dictionary, and obtain the third similarity between the second p-hash value at the position corresponding to each first region in the target second reference interface image and the first p-hash value at the position of each first region.

[0150] In one embodiment, the electronic device can traverse each first region in the first annotation dictionary and obtain a third similarity between the second p-hash value at the location corresponding to each first region in the target second reference interface image and the first p-hash value at the location of each first region. In one embodiment, to eliminate redundant calculations of each region, a non-maximum suppression method can be used to traverse each first region in the first annotation dictionary.

[0151] In other words, in this embodiment of the present disclosure, a third similarity can be obtained between each first region in the target second reference interface image and each first region in the first annotation dictionary. In this embodiment, the similarity of each first region is not limited to the comparison of two interface images, but rather uses the first region of the target second reference interface image and its corresponding target first reference interface images for similarity calculation. This avoids the problem that the regions of the target second reference interface image cannot be annotated because they do not contain the regions to be annotated in the target first reference interface image.

[0152] S903, in response to the third similarity being greater than the third preset similarity, the second region in the target second reference interface image corresponding to the position of each first region is labeled according to the label of each first region.

[0153] For each second region in the target second reference interface image corresponding to each first region, each second region has at least one third similarity. In this embodiment of the present disclosure, the electronic device can, in response to the third similarity being greater than the third preset similarity, label the second region in the target second reference interface image corresponding to the position of each first region according to the label of each first region.

[0154] In one embodiment, if the second region corresponds to a third similarity and the third similarity is greater than a third preset similarity, then the electronic device can label the first region corresponding to the third similarity in the second region of the target second reference interface image.

[0155] In one embodiment, when the second region corresponds to multiple third similarities, the electronic device can use the label of the first region corresponding to the largest third similarity as the label of the second region corresponding to each first region, that is, the label of the first region corresponding to the largest third similarity is marked in the second region of the target second reference interface image.

[0156] Using this method, electronic devices can annotate the second region to be annotated in the target second reference interface image.

[0157] In one embodiment, the result of annotating the second region in the target second reference interface image corresponding to the positions of each first region can be referred to as the second annotation result.

[0158] In this embodiment, instead of using the annotation results from the target first reference interface image that has the highest similarity to the target second reference interface image, the annotation results from all the target first reference interface images corresponding to the target second reference interface image are used to generate a first annotation dictionary to annotate the target second reference interface image. This allows for complete annotation of all unannotated areas of the target second reference interface image, thereby improving the accuracy and precision of the annotation of the target second reference interface image.

[0159] refer to Figure 11 , Figure 11 This is a structural block diagram of an interface image annotation device provided for embodiments of this disclosure. For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. (Refer to...) Figure 11 The interface image annotation device 1100 includes a processing module 1101 and a transceiver module 1102. Wherein,

[0160] The processing module 1101 is used to aggregate the set of interface images to be labeled to obtain at least one sub-image set, wherein the similarity between the interface images in each sub-image set is greater than or equal to a similarity threshold; and to select one interface image in each sub-image set as a reference interface image.

[0161] The transceiver module 1102 is used to receive the user's annotation instructions for the objects to be annotated in each reference interface image and obtain the annotation results.

[0162] The processing module 1101 is further configured to use the annotation results to annotate the remaining interface images in the at least one sub-image set.

[0163] In one possible implementation, the processing module 1101 is specifically used to aggregate the set of interface images to be labeled using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. The similarity between the interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold. In addition, an interface image is selected as a first reference interface image in each sub-image set corresponding to a first similarity threshold. The first similarity threshold is the minimum value among the different similarity thresholds.

[0164] The transceiver module 1102 is specifically used to receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and to obtain the first annotation result of the first similarity threshold.

[0165] The processing module 1101 is specifically used to annotate the interface images in each sub-image set corresponding to other similarity thresholds using the first annotation result.

[0166] In one possible implementation, the different similarity thresholds include: a first similarity threshold, a second similarity threshold, and a third similarity threshold.

[0167] The processing module 1101 is specifically configured to: aggregate interface images with a similarity threshold greater than or equal to the first similarity threshold in the set of interface images to be labeled, to obtain a plurality of first sub-image sets corresponding to the first similarity threshold; aggregate interface images with a similarity threshold greater than or equal to the second similarity threshold in the plurality of first sub-image sets corresponding to the first similarity threshold, to obtain a plurality of second sub-image sets corresponding to the second similarity threshold, wherein the first similarity threshold is less than the second similarity threshold; and aggregate interface images with a similarity threshold greater than or equal to the third similarity threshold in the plurality of second sub-image sets corresponding to the second similarity threshold, to obtain a plurality of third sub-image sets corresponding to the third similarity threshold, wherein the second similarity threshold is less than the third similarity threshold.

[0168] In one possible implementation, the processing module 1101 is specifically used to take any interface image in each of the first sub-image sets as the first reference interface image of each of the first sub-image sets; or, to take the interface image with the largest number of similarities greater than or equal to the first similarity threshold in each of the first sub-image sets as the first reference interface image of each of the first sub-image sets.

[0169] In one possible implementation, the processing module 1101 is further configured to select an interface image as a second reference interface image in each second sub-image set corresponding to the second similarity threshold, and to select an interface image as a third reference interface image in each third sub-image set corresponding to the third similarity threshold.

[0170] The processing module 1101 is specifically used to use the first annotation result to annotate the second reference interface image in each second sub-image set to obtain the second annotation result of the second similarity threshold; and to use the second annotation result to annotate each third reference interface image.

[0171] In one possible implementation, the processing module 1101 is further configured to obtain the similarity between each second reference interface image and each first reference interface image, and to determine whether the number of similarities between each second reference interface image and each first reference interface image is greater than or equal to a first preset similarity is 0.

[0172] The processing module 1101 is specifically used to annotate the second reference interface image in each second sub-image set in response to the quantity not being zero, using the first annotation result.

[0173] In one possible implementation, the first annotation result includes the location of at least one first region in each first reference interface image.

[0174] The processing module 1101 is specifically used to obtain a first similarity between each second reference interface image and each first reference interface image, wherein the first similarity is the overall similarity between the interface images, and to obtain a second similarity in each second reference interface image at the position corresponding to each first region in each first reference interface image; and to obtain the similarity between each second reference interface image and each first reference interface image based on the first similarity and the second similarity.

[0175] In one possible implementation, the processing module 1101 is specifically used to annotate the target second reference interface image using the annotation result of the target first reference interface image in response to the quantity being greater than the preset quantity, and the first similarity between the target second reference interface image and the target first reference interface image being greater than or equal to the second preset similarity, and the second similarity between the target second reference interface image and the target first reference interface image being greater than or equal to the third preset similarity.

[0176] In one possible implementation, the first annotation result includes: a label for at least one first region in each first reference interface image.

[0177] The processing module 1101 is specifically configured to: use the position of each first region in the target first reference interface image as the key value of the first annotation dictionary; use the first p-hash value and label of each first region in the target first reference interface image as the value value of the first annotation dictionary to construct the first annotation dictionary; traverse each first region in the first annotation dictionary to obtain the third similarity between the second p-hash value at the position corresponding to each first region in the target second reference interface image and the first p-hash value at the position of each first region; and, in response to the third similarity being greater than the third preset similarity, annotate the second region in the target second reference interface image corresponding to the position of each first region according to the label of each first region.

[0178] In one possible implementation, there are multiple third similarities.

[0179] The step of labeling the second region in the target second reference interface image corresponding to the position of each of the first regions according to the labels of each of the first regions includes:

[0180] The processing module 1101 is specifically used to take the label of each first region corresponding to the largest third similarity as the label of the second region corresponding to each first region.

[0181] In one possible implementation, the processing module 1101 is specifically used to traverse each first region in the first annotation dictionary using a non-maximum suppression method.

[0182] In one possible implementation, the processing module 1101 is specifically configured to, in response to the quantity being 0, annotate each second reference interface image according to the user's second annotation instruction for the object to be annotated in each second reference interface image; or, in response to the quantity being greater than the preset quantity, and the first similarity between the target second reference interface image and any first reference interface image being less than the second preset similarity, and / or the second similarity between the target second reference interface image and any first reference interface image being less than the third preset similarity, annotate the target second reference interface image according to the user's third annotation instruction for the object to be annotated in the target second reference interface image.

[0183] In one possible implementation, the processing module 1101 is further configured to use the interface images in the labeled interface image set as training samples to train a recognition model, which is used to identify the objects to be labeled in the interface images.

[0184] In one possible implementation, the processing module 1101 is further configured to perform frame extraction processing on the video to be labeled, thereby obtaining the set of interface images to be labeled.

[0185] In one possible implementation, the video is a video of a game application, and the object to be labeled is a user-operable control in the interface image.

[0186] The interface image annotation apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0187] To implement the above embodiments, this disclosure also provides an electronic device.

[0188] refer to Figure 12 It shows a structural schematic diagram of an electronic device 1200 suitable for implementing the embodiments of the present disclosure, which can be the server in the above embodiments. Figure 12The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0189] like Figure 12 As shown, the electronic device 1200 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. The RAM 1203 also stores various programs and data required for the operation of the electronic device 1200. The processing unit 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0190] Typically, the following devices can be connected to I / O interface 1205: input devices 1206 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1207 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1208 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1209. Communication device 1209 allows electronic device 1200 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 12 An electronic device 1200 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0191] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1209, or installed from storage device 1208, or installed from ROM 1202. When the computer program is executed by processing device 1201, it performs the functions defined in the methods of embodiments of this disclosure.

[0192] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0193] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0194] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0195] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0196] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0197] The modules described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a module does not necessarily limit the module itself; for example, the first acquisition module can also be described as "a module for acquiring at least two Internet Protocol addresses".

[0198] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0199] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0200] In a first aspect, according to one or more embodiments of the present disclosure, a method for annotating interface images is provided, comprising: performing aggregation processing on a set of interface images to be annotated to obtain at least one sub-image set, wherein the similarity between interface images in each sub-image set is greater than or equal to a similarity threshold; selecting an interface image as a reference interface image in each sub-image set; receiving a user's annotation instruction for an object to be annotated in each reference interface image to obtain an annotation result; and using the annotation result to annotate the remaining interface images in the at least one sub-image set.

[0201] According to one or more embodiments of this disclosure, the aggregation processing of the interface image set to be annotated includes: aggregating the interface image set to be annotated using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold, wherein the similarity between interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold.

[0202] The step of selecting an interface image as a reference interface image in each sub-image set includes: selecting an interface image as a first reference interface image in each sub-image set corresponding to a first similarity threshold, wherein the first similarity threshold is the minimum value among the different similarity thresholds.

[0203] The step of receiving the user's annotation instruction for the object to be annotated in each reference interface image and obtaining the annotation result includes: receiving the user's first annotation instruction for the object to be annotated in each first reference interface image and obtaining the first annotation result of the first similarity threshold.

[0204] The step of using the annotation results to annotate the remaining interface images in the at least one sub-image set includes: using the first annotation results to annotate the interface images in each sub-image set corresponding to other similarity thresholds.

[0205] The interface image set to be labeled is aggregated using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. The similarity between the interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold.

[0206] In each sub-image set corresponding to the first similarity threshold, one interface image is selected as the first reference interface image, and the first similarity threshold is the minimum value among the different similarity thresholds;

[0207] Receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and obtain the first annotation result of the first similarity threshold;

[0208] Using the first annotation result, the interface images in each sub-image set corresponding to other similarity thresholds are annotated.

[0209] According to one or more embodiments of this disclosure, the different similarity thresholds include: a first similarity threshold, a second similarity threshold, and a third similarity threshold.

[0210] The process of aggregating the interface image set to be labeled using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold includes: aggregating interface images with similarity thresholds greater than or equal to the first similarity threshold in the interface image set to be labeled to obtain multiple first sub-image sets corresponding to the first similarity threshold; aggregating interface images with similarity thresholds greater than or equal to the second similarity threshold in the multiple first sub-image sets corresponding to the first similarity threshold to obtain multiple second sub-image sets corresponding to the second similarity threshold, wherein the first similarity threshold is less than the second similarity threshold; and aggregating interface images with similarity thresholds greater than or equal to the third similarity threshold in the multiple second sub-image sets corresponding to the second similarity threshold to obtain multiple third sub-image sets corresponding to the third similarity threshold, wherein the second similarity threshold is less than the third similarity threshold.

[0211] According to one or more embodiments of this disclosure, selecting an interface image as a first reference interface image in each sub-image set corresponding to a first similarity threshold includes: using any interface image in each first sub-image set as the first reference interface image of each first sub-image set; or, using the interface image with the highest number of similarities greater than or equal to the first similarity threshold in each first sub-image set as the first reference interface image of each first sub-image set.

[0212] According to one or more embodiments of this disclosure, before annotating the interface images in each sub-image set corresponding to other similarity thresholds using the first annotation result, the method includes: selecting an interface image as a second reference interface image in each second sub-image set corresponding to the second similarity threshold, and selecting an interface image as a third reference interface image in each third sub-image set corresponding to the third similarity threshold.

[0213] The step of using the first annotation result to annotate the interface images in each sub-image set corresponding to other similarity thresholds includes: using the first annotation result to annotate the second reference interface image in each second sub-image set to obtain the second annotation result of the second similarity threshold; and using the second annotation result to annotate each third reference interface image.

[0214] According to one or more embodiments of this disclosure, before annotating the second reference interface image in each second sub-image set using the first annotation result, the method further includes: obtaining the similarity between each second reference interface image and each first reference interface image: in the similarity between each second reference interface image and each first reference interface image, determining whether the number of similarities greater than or equal to a first preset similarity is 0.

[0215] The step of using the first annotation result to annotate the second reference interface image in each second sub-image set includes: in response to the number being non-zero, using the first annotation result to annotate the second reference interface image in each second sub-image set.

[0216] According to one or more embodiments of this disclosure, the first annotation result includes: the position of at least one first region in each first reference interface image; obtaining the similarity between each second reference interface image and each first reference interface image includes: obtaining a first similarity between each second reference interface image and each first reference interface image, wherein the first similarity is the overall similarity between interface images; in each second reference interface image, obtaining a second similarity at the position corresponding to each first region in each first reference interface image; and obtaining the similarity between each second reference interface image and each first reference interface image based on the first similarity and the second similarity.

[0217] According to one or more embodiments of this disclosure, the step of annotating the second reference interface image in each second sub-image set using the first annotation result in response to the number being greater than a preset number includes: in response to the number being greater than the preset number, and the first similarity between the target second reference interface image and the target first reference interface image being greater than or equal to a second preset similarity, and the second similarity between the target second reference interface image and the target first reference interface image being greater than or equal to a third preset similarity, then the annotation result of the target first reference interface image is used to annotate the target second reference interface image.

[0218] According to one or more embodiments of this disclosure, the first annotation result includes: a label for at least one first region in each first reference interface image. The step of annotating the target second reference interface image using the first annotation result includes: using the position of each first region in the target first reference interface image as the key value of a first annotation dictionary, and using the first p-hash value and label of each first region in the target first reference interface image as the value value of the first annotation dictionary to construct the first annotation dictionary; traversing each first region in the first annotation dictionary, obtaining a third similarity between the second p-hash value at the position corresponding to each first region in the target second reference interface image and the first p-hash value at the position of each first region; and, in response to the third similarity being greater than the third preset similarity, annotating the second region in the target second reference interface image corresponding to the position of each first region according to the label of each first region.

[0219] According to one or more embodiments of this disclosure, there are multiple third similarities; the step of labeling the second region in the target second reference interface image corresponding to the position of each of the first regions according to the labels of each of the first regions includes: taking the label of each of the first regions corresponding to the largest third similarity as the label of the second region corresponding to each of the first regions.

[0220] According to one or more embodiments of this disclosure, traversing each first region in the first annotation dictionary includes: traversing each first region in the first annotation dictionary using a non-maximum suppression method.

[0221] According to one or more embodiments of this disclosure, the method further includes: in response to the quantity being 0, annotating each second reference interface image according to the user's second annotation instruction for the object to be annotated in each second reference interface image; or,

[0222] In response to the quantity being greater than the preset quantity, and the first similarity between the target second reference interface image and any first reference interface image being less than the second preset similarity, and / or the second similarity between the target second reference interface image and any first reference interface image being less than the third preset similarity, the target second reference interface image is annotated according to the user's third annotation instruction for the object to be annotated in the target second reference interface image.

[0223] According to one or more embodiments of this disclosure, an interface image from a set of labeled interface images is used as training samples to train a recognition model, which is used to identify objects to be labeled in the interface images.

[0224] According to one or more embodiments of this disclosure, before aggregating the set of interface images to be labeled using different similarity thresholds, the method further includes: performing frame extraction processing on the video to be labeled to obtain the set of interface images to be labeled.

[0225] According to one or more embodiments of this disclosure, the video is a video of a game application, and the object to be labeled is a user-operable control in the interface image.

[0226] Secondly, according to one or more embodiments of this disclosure, an apparatus for interface image annotation is provided, comprising: a processing module, configured to aggregate a set of interface images to be annotated to obtain at least one sub-image set, wherein the similarity between interface images in each sub-image set is greater than or equal to a similarity threshold; and to select one interface image in each sub-image set as a reference interface image. A transceiver module, configured to receive annotation instructions from a user for objects to be annotated in each reference interface image, and to obtain annotation results. The processing module is further configured to use the annotation results to annotate the remaining interface images in the at least one sub-image set.

[0227] The processing module is specifically used to aggregate the set of interface images to be labeled using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. The similarity between the interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold. In addition, an interface image is selected as a first reference interface image in each sub-image set corresponding to a first similarity threshold. The first similarity threshold is the minimum value among the different similarity thresholds.

[0228] The transceiver module is specifically used to receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and to obtain the first annotation result of the first similarity threshold.

[0229] The processing module is specifically used to annotate the interface images in each sub-image set corresponding to other similarity thresholds using the first annotation result.

[0230] According to one or more embodiments of this disclosure, the different similarity thresholds include: a first similarity threshold, a second similarity threshold, and a third similarity threshold.

[0231] The processing module is specifically configured to: aggregate interface images with a similarity threshold greater than or equal to the first similarity threshold in the set of interface images to be labeled, to obtain multiple first sub-image sets corresponding to the first similarity threshold; aggregate interface images with a similarity threshold greater than or equal to the second similarity threshold in the multiple first sub-image sets corresponding to the first similarity threshold, to obtain multiple second sub-image sets corresponding to the second similarity threshold, wherein the first similarity threshold is less than the second similarity threshold; and aggregate interface images with a similarity threshold greater than or equal to the third similarity threshold in the multiple second sub-image sets corresponding to the second similarity threshold, to obtain multiple third sub-image sets corresponding to the third similarity threshold, wherein the second similarity threshold is less than the third similarity threshold.

[0232] According to one or more embodiments of this disclosure, the processing module is specifically used to take any interface image in each of the first sub-image sets as the first reference interface image of each of the first sub-image sets; or, take the interface image with the largest number of similarities greater than or equal to the first similarity threshold in each of the first sub-image sets as the first reference interface image of each of the first sub-image sets.

[0233] According to one or more embodiments of this disclosure, the processing module is further configured to select an interface image as a second reference interface image in each second sub-image set corresponding to the second similarity threshold, and to select an interface image as a third reference interface image in each third sub-image set corresponding to the third similarity threshold.

[0234] The processing module is specifically used to annotate the second reference interface image in each second sub-image set using the first annotation result to obtain the second annotation result of the second similarity threshold; and to annotate each third reference interface image using the second annotation result.

[0235] According to one or more embodiments of this disclosure, the processing module is further configured to obtain the similarity between each second reference interface image and each first reference interface image, and to determine whether the number of similarities between each second reference interface image and each first reference interface image is greater than or equal to a first preset similarity is 0.

[0236] The processing module is specifically used to annotate the second reference interface image in each second sub-image set in response to the quantity being non-zero, using the first annotation result.

[0237] According to one or more embodiments of this disclosure, the first annotation result includes: the position of at least one first region in each first reference interface image.

[0238] The processing module is specifically used to obtain a first similarity between each second reference interface image and each first reference interface image, wherein the first similarity is the overall similarity between the interface images, and to obtain a second similarity in each second reference interface image at the position corresponding to each first region in each first reference interface image; and to obtain the similarity between each second reference interface image and each first reference interface image based on the first similarity and the second similarity.

[0239] According to one or more embodiments of this disclosure, the processing module is specifically configured to, in response to the quantity being greater than the preset quantity, and the first similarity between the target second reference interface image and the target first reference interface image being greater than or equal to the second preset similarity, and the second similarity between the target second reference interface image and the target first reference interface image being greater than or equal to the third preset similarity, use the annotation result of the target first reference interface image to annotate the target second reference interface image.

[0240] According to one or more embodiments of this disclosure, the first annotation result includes: a label for at least one first region in each first reference interface image.

[0241] The processing module is specifically configured to use the position of each first region in the target first reference interface image as the key value of the first annotation dictionary, and use the first p-hash value and label of each first region in the target first reference interface image as the value value of the first annotation dictionary to construct the first annotation dictionary; traverse each first region in the first annotation dictionary to obtain the third similarity between the second p-hash value at the position corresponding to each first region in the target second reference interface image and the first p-hash value at the position of each first region; and in response to the third similarity being greater than the third preset similarity, annotate the second region in the target second reference interface image corresponding to the position of each first region according to the label of each first region.

[0242] According to one or more embodiments of this disclosure, there are multiple third similarities.

[0243] The step of labeling the second region in the target second reference interface image corresponding to the position of each of the first regions according to the labels of each of the first regions includes:

[0244] The processing module is specifically used to take the label of each first region corresponding to the largest third similarity as the label of the second region corresponding to each first region.

[0245] According to one or more embodiments of this disclosure, the processing module is specifically used to traverse each first region in the first annotation dictionary using a non-maximum suppression method.

[0246] According to one or more embodiments of this disclosure, the processing module is specifically configured to, in response to the quantity being 0, annotate each second reference interface image according to the user's second annotation instruction for the object to be annotated in each second reference interface image; or, in response to the quantity being greater than the preset quantity, and the first similarity between the target second reference interface image and any first reference interface image being less than the second preset similarity, and / or the second similarity between the target second reference interface image and any first reference interface image being less than the third preset similarity, annotate the target second reference interface image according to the user's third annotation instruction for the object to be annotated in the target second reference interface image.

[0247] According to one or more embodiments of this disclosure, the processing module is further configured to use interface images from the labeled interface image set as training samples to train a recognition model, the recognition model being used to identify objects to be labeled in the interface images.

[0248] According to one or more embodiments of this disclosure, the processing module is further configured to perform frame extraction processing on the video to be labeled to obtain the set of interface images to be labeled.

[0249] According to one or more embodiments of this disclosure, the video is a video of a game application, and the object to be labeled is a user-operable control in the interface image.

[0250] Thirdly, according to one or more embodiments of this disclosure, an electronic device is provided, comprising: a processor and a memory; the memory storing computer execution instructions; the processor executing the computer execution instructions stored in the memory, causing the processor to perform the interface image annotation method as described in the first aspect above and various possible designs of the first aspect.

[0251] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the method for interface image annotation as described in the first aspect and various possible designs of the first aspect is implemented.

[0252] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement the interface image annotation method as described in the first aspect and various possible designs of the first aspect.

[0253] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0254] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0255] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for annotating interface images, characterized in that, include: The interface image set to be labeled is aggregated using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. The similarity between the interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold. The different similarity thresholds include: a first similarity threshold, a second similarity threshold, and a third similarity threshold; In each sub-image set corresponding to the first similarity threshold, one interface image is selected as the first reference interface image, and the first similarity threshold is the minimum value among the different similarity thresholds. Receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and obtain the first annotation result of the first similarity threshold; One interface image is selected as a second reference interface image in each second sub-image set corresponding to the second similarity threshold, and one interface image is selected as a third reference interface image in each third sub-image set corresponding to the third similarity threshold; the second similarity threshold is less than the third similarity threshold. Using the first annotation result, the second reference interface image in each second sub-image set is annotated to obtain the second annotation result of the second similarity threshold; The second annotation result is used to annotate each third reference interface image; Obtain the mixed similarity between each third reference interface image and each other interface image, and then label the other interface images in each third sub-image set based on the mixed similarity.

2. The method according to claim 1, characterized in that, The process involves aggregating the interface image sets to be labeled using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold, including: In the set of interface images to be labeled, interface images with a similarity threshold greater than or equal to the first similarity threshold are aggregated to obtain multiple first sub-image sets corresponding to the first similarity threshold. In the multiple first sub-image sets corresponding to the first similarity threshold, the interface images with similarity thresholds greater than or equal to the second similarity threshold are aggregated to obtain multiple second sub-image sets corresponding to the second similarity threshold, wherein the first similarity threshold is less than the second similarity threshold. In the multiple second sub-image sets corresponding to the second similarity threshold, the interface images with a similarity threshold greater than or equal to the third similarity threshold are aggregated to obtain multiple third sub-image sets corresponding to the third similarity threshold.

3. The method according to claim 2, characterized in that, The step of selecting one interface image as the first reference interface image in each sub-image set corresponding to the first similarity threshold includes: Use any one interface image from each of the first sub-image sets as the first reference interface image for each of the first sub-image sets; or... The interface image with the highest number of similarities greater than or equal to the first similarity threshold in each of the first sub-image sets is used as the first reference interface image for each of the first sub-image sets.

4. The method according to claim 1, characterized in that, Before using the first annotation result to annotate the second reference interface image in each second sub-image set, the method further includes: Obtain the similarity between each second reference interface image and each first reference interface image; In the similarity between each second reference interface image and each first reference interface image, determine whether the number of similarities greater than or equal to the first preset similarity is 0; The step of annotating the second reference interface image in each second sub-image set using the first annotation result includes: In response to the quantity being non-zero, the first annotation result is used to annotate the second reference interface image in each second sub-image set.

5. The method according to claim 4, characterized in that, The first annotation result includes: the location of at least one first region in each first reference interface image; obtaining the similarity between each second reference interface image and each first reference interface image includes: Obtain the first similarity between each second reference interface image and each first reference interface image, where the first similarity is the overall similarity between the interface images; In each second reference interface image, obtain the second similarity at the corresponding position of each first region in each first reference interface image; Based on the first similarity and the second similarity, the similarity between each second reference interface image and each first reference interface image is obtained.

6. The method according to claim 5, characterized in that, In response to the quantity being non-zero, the first annotation result is used to annotate the second reference interface image in each second sub-image set, including: In response to the fact that the number is greater than a preset number, and the first similarity between the target second reference interface image and the target first reference interface image is greater than or equal to a second preset similarity, and the second similarity between the target second reference interface image and the target first reference interface image is greater than or equal to a third preset similarity, the annotation result of the target first reference interface image is used to annotate the target second reference interface image.

7. The method according to claim 6, characterized in that, The first annotation result includes: a label for at least one first region in each first reference interface image. The step of annotating the target second reference interface image using the first annotation result includes: The first annotation dictionary is constructed by using the position of each first region in the target first reference interface image as the key value of the first annotation dictionary, and the first p-hash value and label of each first region in the target first reference interface image as the value value of the first annotation dictionary. Traverse each first region in the first annotation dictionary and obtain the third similarity between the second p-hash value at the location corresponding to each first region in the target second reference interface image and the first p-hash value at the location of each first region; In response to the third similarity being greater than the third preset similarity, the second region in the target second reference interface image corresponding to the position of each of the first regions is labeled according to the labels of each of the first regions.

8. The method according to claim 7, characterized in that, There are multiple third-degree similarities; The step of labeling the second region in the target second reference interface image corresponding to the position of each of the first regions according to the labels of each of the first regions includes: The labels of the first regions corresponding to the highest third similarity are used as the labels of the second regions corresponding to the first regions.

9. The method according to claim 8, characterized in that, The traversal of each first region in the first labeled dictionary includes: The nonmaximum suppression method is used to traverse each first region in the first annotation dictionary.

10. The method according to claim 6, characterized in that, The method further includes: In response to the quantity being 0, each second reference interface image is annotated according to the user's second annotation instruction for the object to be annotated in each second reference interface image; or, In response to the quantity being greater than the preset quantity, and the first similarity between the target second reference interface image and any first reference interface image being less than the second preset similarity, and / or the second similarity between the target second reference interface image and any first reference interface image being less than the third preset similarity, the target second reference interface image is annotated according to the user's third annotation instruction for the object to be annotated in the target second reference interface image.

11. The method according to any one of claims 1-10, characterized in that, After annotating each third reference interface image using the second annotation result, the process further includes: The interface images in the already labeled interface image set are used as training samples to train a recognition model, which is used to identify the objects to be labeled in the interface images.

12. The method according to any one of claims 1-10, characterized in that, Before aggregating the set of interface images to be labeled using different similarity thresholds, the process also includes: The video to be labeled is processed by frame extraction to obtain the set of interface images to be labeled.

13. The method according to claim 12, characterized in that, The video is a video of a game application, and the object to be labeled is a user-operable control in the interface image.

14. An apparatus for annotating interface images, characterized in that, include: Processing module, used for: The interface image set to be labeled is aggregated using different similarity thresholds to obtain multiple sub-image sets corresponding to each similarity threshold. The similarity between the interface images in each sub-image set corresponding to each similarity threshold is greater than or equal to the corresponding similarity threshold. The different similarity thresholds include: a first similarity threshold, a second similarity threshold, and a third similarity threshold; In each sub-image set corresponding to the first similarity threshold, one interface image is selected as the first reference interface image, and the first similarity threshold is the minimum value among the different similarity thresholds. The transceiver module is used to receive the user's first annotation instruction for the object to be annotated in each first reference interface image, and to obtain the first annotation result of the first similarity threshold. The processing module is further configured to select one interface image as a second reference interface image in each second sub-image set corresponding to the second similarity threshold, and select one interface image as a third reference interface image in each third sub-image set corresponding to the third similarity threshold, wherein the second similarity threshold is less than the third similarity threshold; use the first annotation result to annotate the second reference interface image in each second sub-image set to obtain the second annotation result of the second similarity threshold; use the second annotation result to annotate each third reference interface image; obtain the mixed similarity between each third reference interface image and each other interface image, so as to annotate the other interface images in each third sub-image set according to the mixed similarity.

15. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-13.

17. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the method of any one of claims 1-13.

Images