Method, device and electronic device for gathering frame annotation data

Abstract

The present application provides a method, device, and electronic device for aggregating frame annotation data, the method comprising: obtaining frame annotation data corresponding to a target image; wherein the target image includes at least one target object; the frame annotation data comprising: position information corresponding to multiple first frames obtained by aggregating frame annotations performed by multiple annotators on target objects in the target image; performing frame clustering based on the position information corresponding to the multiple first frames to determine frame clustering data corresponding to each target object; for each target object, determining a frame aggregation result corresponding to the target object based on the frame clustering data corresponding to the target object; the present application can perform multi-target frame clustering on the frame annotation data of an image containing multiple target objects, obtain frame clustering data for each target object, and then determine the position information of the aggregated frame for each target object, thereby improving the efficiency and accuracy of multi-target frame aggregation.

Description

Technical Field

[0001] The present application relates to the field of annotation technology, and in particular to a method, device and electronic device for aggregating frame annotation data. Background Art

[0002] The development and application of artificial intelligence technology relies on extensive manual annotation. For example, object recognition applications require a large number of bounding box annotations to provide the model with an accurate labeled dataset for supervised training. Relying solely on expert bounding boxes in this process is often costly and difficult to produce massive amounts of labeled data in a timely manner. In recent years, data crowdsourcing, as a mature annotation model, has provided a better solution. However, due to the uncertainty of annotator quality in the crowdsourcing model, redundancy has become a key factor. That is, for each target object, multiple annotators are required to provide multiple bounding box results, which are then aggregated to produce the final bounding box. In this context, previous methods typically considered scenarios where only one bounding box target was present in an image. There was no direct aggregation solution for the broader problem of multiple bounding boxes in a single image. Summary of the Invention

[0003] The purpose of this application is to provide a method, device and electronic device for aggregating frame annotation data, which can perform multi-target frame clustering on the frame annotation data of an image containing multiple target objects, obtain clustered frame data for each target object, and then determine the position information of the aggregated frame for each target object, thereby improving the efficiency and accuracy of frame aggregation of multi-target annotation data.

[0004] In a first aspect, an embodiment of the present application provides a method for aggregating frame annotation data, the method comprising: obtaining frame annotation data corresponding to a target image; wherein the target image includes at least one target object; the frame annotation data comprises: position information corresponding to multiple first frames respectively obtained after multiple annotators perform frame annotation on the target object in the target image; frame clustering is performed based on the position information corresponding to the multiple first frames, and frame clustering data corresponding to each target object is determined; the frame clustering data comprises: position information corresponding to multiple second frames; for each target object, a frame aggregation result corresponding to the target object is determined based on the frame clustering data corresponding to the target object; the frame aggregation result comprises the position information of the aggregated frame corresponding to the target object.

[0005] In a preferred embodiment of the present application, the above-mentioned step of clustering the pull boxes based on the position information corresponding to the multiple first pull boxes and determining the pull box clustering data corresponding to each target object includes: calculating the IoU index between each pair of pull boxes according to the position information corresponding to the multiple first pull boxes; generating a pull box undirected graph according to the size relationship between the IoU index between each pair of pull boxes and the preset index threshold; performing a maximum cluster search based on the pull box undirected graph to determine the multiple second pull boxes corresponding to each cluster, and obtaining the pull box clustering data corresponding to each target object.

[0006] In a preferred embodiment of the present application, the above-mentioned position information includes: the upper left coordinate and the lower right coordinate of the pull box; according to the position information corresponding to multiple first pull boxes, the step of calculating the IoU index between each pull box includes: for any two first pull boxes, according to the upper left coordinate and the lower right coordinate corresponding to the two first pull boxes, calculating the intersection area and the merged area of ​​the two first pull boxes; taking the ratio of the intersection area to the merged area as the IoU index corresponding to the two first pull boxes.

[0007] In a preferred embodiment of the present application, the above-mentioned step of generating a pull-box undirected graph based on the size relationship between the IoU index between the two pull-boxes and the preset index threshold includes: constructing a distance matrix based on the IoU index between the two pull-boxes; each element in the distance matrix is ​​used to characterize the IoU index between the two pull-boxes; generating an adjacency matrix based on the size relationship between each element in the distance matrix and the preset index threshold; and generating a pull-box undirected graph based on the adjacency matrix.

[0008] In a preferred embodiment of the present application, the elements in the above-mentioned adjacency matrix include 0 or 1; the above-mentioned step of generating the adjacency matrix according to the size relationship between each element in the distance matrix and the preset index threshold includes: for each element in the distance matrix, comparing the IoU index corresponding to the element with the preset index threshold; setting the value corresponding to the element exceeding the preset index threshold to 1, and setting the value corresponding to the element not exceeding the preset index threshold to 0, to obtain the adjacency matrix.

[0009] In a preferred embodiment of the present application, the above-mentioned step of determining the pull-box aggregation result corresponding to the target object based on the pull-box clustering data corresponding to the target object includes: obtaining the initial annotation quality parameters of the labelers corresponding to multiple second pull boxes respectively; performing aggregation evaluation based on the position information and initial annotation quality parameters corresponding to the multiple second pull boxes of the target object, and determining the pull-box aggregation result corresponding to the target object.

[0010] In a preferred embodiment of the present application, the above-mentioned step of performing convergence evaluation based on the position information corresponding to the multiple second pull boxes of the target object and the initial annotation quality parameters to determine the convergence result of the pull boxes corresponding to the target object includes: taking the initial annotation quality parameter as the current annotation quality parameter, and executing the following convergence evaluation step: based on the position information corresponding to the multiple second pull boxes and the current annotation quality parameter, determining the position information of the current converged pull box corresponding to the multiple second pull boxes; calculating the estimated annotation quality parameter of the annotator corresponding to each second pull box based on the position information of the current converged pull box and the position information corresponding to each second pull box; when the change amplitude of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds a preset amplitude threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the convergence evaluation step is continued until the change amplitude does not exceed the preset amplitude threshold, and the position information of the current converged pull box is determined as the convergence result of the pull box corresponding to the target object.

[0011] In a preferred embodiment of the present application, the step of determining the position information of the current converged pull box corresponding to the multiple second pull boxes based on the position information corresponding to the multiple second pull boxes and the current annotation quality parameter includes: calculating the position information of the current converged pull box corresponding to the multiple second pull boxes according to the following first specified formula:

[0012]

[0013] Wherein, Z represents the position information of the current convergence pull frame corresponding to the multiple second pull frames; s i Indicates the upper left and lower right coordinates of the i-th second pull box, α i represents the current annotation quality parameter corresponding to the i-th second pull box; m represents the number of second pull boxes.

[0014] In a preferred embodiment of the present application, the step of calculating the estimated annotation quality parameter of the annotator corresponding to each second pull-box based on the position information of the current converged pull-box and the position information corresponding to each second pull-box includes: calculating the estimated annotation quality parameter of the annotator corresponding to each second pull-box according to the following second specified formula:

[0015] α i =Z / s i ;

[0016] Among them, α i " represents the estimated annotation quality parameter of the annotator corresponding to the i-th second pull box.

[0017] In a preferred embodiment of the present application, the above method further includes: when the change amplitude does not exceed the preset amplitude threshold, simultaneously outputting the position information of the current convergence pull frame and the current annotation quality parameters of the annotators corresponding to the multiple second pull frames.

[0018] In a second aspect, an embodiment of the present application further provides a device for aggregating frame annotation data, the device comprising: a data acquisition module for acquiring frame annotation data corresponding to a target image; wherein the target image includes at least one target object; the frame annotation data comprises: position information corresponding to multiple first frames respectively obtained after multiple labelers perform frame annotation on the target object in the target image; a frame clustering module for performing frame clustering based on the position information corresponding to the multiple first frames, and determining the frame clustering data corresponding to each target object; the frame clustering data comprises: position information corresponding to multiple second frames; a frame aggregation module for determining, for each target object, a frame aggregation result corresponding to the target object based on the frame clustering data corresponding to the target object; the frame aggregation result comprises the position information of the aggregated frame corresponding to the target object.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method described in the first aspect above.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method described in the first aspect above.

[0021] In a method, device, and electronic device for aggregating frame annotation data provided by an embodiment of the present application, first, frame annotation data corresponding to a target image is obtained; wherein the target image includes at least one target object; the frame annotation data includes: position information corresponding to multiple first frames obtained after multiple annotators perform frame annotation on the target object in the target image; then, frame clustering is performed based on the position information corresponding to the multiple first frames to determine the frame clustering data corresponding to each target object; the frame clustering data includes: position information corresponding to multiple second frames; finally, for each target object, based on the frame clustering data corresponding to the target object, a frame aggregation result corresponding to the target object is determined; the frame aggregation result includes the position information of the aggregation frame corresponding to the target object. The embodiment of the present application can perform multi-target clustering on the frame annotation data of an image containing multiple target objects, obtain clustered frame data for each target object, and then determine the position information of the aggregation frame for each target object, thereby improving the efficiency and accuracy of frame aggregation of multi-target annotation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is an application scenario diagram of a method for aggregating frame-marked data provided by an embodiment of the present application;

[0024] Figure 2 A flowchart of a method for aggregating frame-marked data provided in an embodiment of the present application;

[0025] Figure 3 A flowchart of the clustering process in a method for aggregating frame-annotated data provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a frame-pull undirected graph provided in an embodiment of the present application;

[0027] Figure 5 This is a structural block diagram of a device for aggregating frame-marked data provided by an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will clearly and completely describe the technical solutions of this application in conjunction with the embodiments. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0030] Related technologies primarily focus on converging single-image, single-target methods. However, when a single image contains multiple targets, i.e., multiple target objects to be annotated, the aforementioned converging methods for single targets are no longer applicable because the rectangular boxes annotated by the annotator essentially represent different targets. Another approach is process decomposition, whereby only one target in a single image is targeted at a time, converging the boxes for that target before processing the next target. However, this approach is based on serial logic, which greatly reduces the high scalability of crowdsourcing and reduces timeliness.

[0031] Based on this, the embodiments of the present application provide a method, device and electronic device for aggregating frame annotation data, which can perform multi-target clustering on the frame annotation data of a picture containing multiple target objects, obtain clustered frame data for each target object, and then determine the location information of the aggregated frame for each target object, thereby improving the efficiency and accuracy of frame aggregation of multi-target annotation data.

[0032] To facilitate understanding of this embodiment, a method for aggregating frame annotation data disclosed in an embodiment of the present application is first introduced in detail.

[0033] Figure 1 An application scenario of a method for aggregating frame annotation data provided in an embodiment of the present application includes a picture of three target objects that have been annotated with frames. Each target object corresponds to a frame obtained by three annotators. The goal of this solution is to perform frame clustering and frame aggregation on the frame annotation data for a picture containing multiple target objects, and finally obtain a clustered frame corresponding to each target object, thereby improving the efficiency of frame aggregation.

[0034] See also Figure 2 The flowchart of the method for aggregating frame annotation data shown in FIG. 1 specifically includes the following steps:

[0035] Step S202: Obtaining frame annotation data corresponding to the target image; wherein the target image includes at least one target object; the frame annotation data includes: position information corresponding to multiple first frames obtained by summarizing frame annotations performed by multiple annotators on the target object in the target image.

[0036] For the target image, each annotator provides a set of annotation results, each of which contains one or more boxes. Combining the box data of all annotators, the box annotation data corresponding to the target image can be obtained, which is expressed as l = {s0, s1, ..., s n}, where s n Indicates the position information corresponding to the drag box n.

[0037] by Figure 1 Taking the target image shown as an example, the box annotation data includes the position information corresponding to the nine boxes obtained by three annotators after box annotation of the three target objects. The position information here usually refers to the upper left and lower right coordinates of the rectangular box.

[0038] Step S204 , performing pull-box clustering based on the position information corresponding to the plurality of first pull-boxes, and determining pull-box clustering data corresponding to each target object; the pull-box clustering data includes: position information corresponding to the plurality of second pull-boxes.

[0039] The process of box clustering can be analyzed and determined by calculating the IoU metric between each box, constructing an adjacency matrix, and using the maximum cluster search algorithm. Ultimately, the multiple boxes contained in each cluster can be considered as the box cluster data corresponding to a target object.

[0040] It should be noted that in this application, each frame in the target image is referred to as a first frame; after frame clustering, each frame in the frame clustering data corresponding to each target object in the target image is referred to as a second frame. In other words, the first frame refers to the frame before frame clustering, and the second frame refers to the frame after frame clustering.

[0041] Step S206 : for each target object, determine the pull-box aggregation result corresponding to the target object according to the pull-box clustering data corresponding to the target object; the pull-box aggregation result includes the position information of the aggregation pull-box corresponding to the target object.

[0042] Here we can introduce the frame aggregation at the annotator level, that is, to conduct a convergence evaluation based on the annotation quality parameters of the annotators to determine the clustered frame corresponding to each target object.

[0043] In a method for aggregating frame annotation data provided by an embodiment of the present application, first, frame annotation data corresponding to a target image is obtained; wherein the target image includes at least one target object; the frame annotation data includes: position information corresponding to multiple first frames obtained after multiple annotators perform frame annotation on the target object in the target image; then, frame clustering is performed based on the position information corresponding to the multiple first frames to determine the frame clustering data corresponding to each target object; the frame clustering data includes: position information corresponding to multiple second frames; finally, for each target object, based on the frame clustering data corresponding to the target object, a frame aggregation result corresponding to the target object is determined; the frame aggregation result includes the position information of the aggregation frame corresponding to the target object. The embodiment of the present application can perform multi-target clustering on the frame annotation data of an image containing multiple target objects, obtain clustered frame data for each target object, and then determine the position information of the aggregation frame for each target object, thereby improving the efficiency and accuracy of frame aggregation of multi-target annotation data.

[0044] An embodiment of the present application also provides a method for aggregating frame annotation data. This method is implemented on the basis of the previous embodiment. This embodiment focuses on describing the frame clustering process and the frame aggregation process.

[0045] See also Figure 3 As shown, the step of clustering the pull boxes based on the position information corresponding to the plurality of first pull boxes and determining the pull box cluster data corresponding to each target object includes:

[0046] Step S302: Calculate the IoU index between any two of the first pull frames based on the position information corresponding to each of the plurality of first pull frames.

[0047] The above-mentioned position information includes: the upper left coordinate and the lower right coordinate of the pull box; in a specific implementation, for any two first pull boxes, the intersection area and the combined area of ​​the two first pull boxes are calculated according to the upper left coordinate and the lower right coordinate corresponding to the two first pull boxes respectively; the ratio of the intersection area to the combined area is used as the IoU indicator corresponding to the two first pull boxes.

[0048] IoU (Intersection over Union) is used to describe the degree of overlap between two bounding boxes. The larger the overlap, the larger the IoU value.

[0049] Step S304: Generate a frame undirected graph based on the relationship between the IoU index between each frame and a preset index threshold.

[0050] The specific steps include:

[0051] (1) Construct a distance matrix based on the IoU index between two boxes; each element in the distance matrix is ​​used to represent the IoU index between two boxes.

[0052] Using the IoU indicator to represent the distance between two boxes, we can get the following distance matrix:

[0053]

[0054] Among them, δ i, Represents the IoU index between the pull box i and the pull box j.

[0055] (2) Generate an adjacency matrix based on the size relationship between each element in the distance matrix and the preset index threshold; the elements in the adjacency matrix include 0 or 1;

[0056] For each element in the distance matrix, the IoU index corresponding to the element is compared with the preset index threshold; the value corresponding to the element exceeding the preset index threshold is set to 1, and the value corresponding to the element not exceeding the preset index threshold is set to 0, and the adjacency matrix is ​​obtained.

[0057] For example, given a configurable parameter τ, for each specific distance δ in the distance matrix Δ i, , compared with τ. For example, for δ i,j , if δ i,j >=τ, then we get a corresponding variable a i, =1. If δ i,j <τ, then we get a corresponding variable a i, = 0. Finally, a new matrix can be obtained, which is expressed as:

[0058]

[0059] Where A represents the adjacency matrix.

[0060] (3) Generate a pull-box undirected graph based on the adjacency matrix.

[0061] The matrix A is considered as an adjacency matrix and expressed by graph structure alignment. In the resulting graph, each node is a pull box, and each edge indicates that the IoU distance between the two pull boxes corresponding to the edge exceeds a given threshold τ. The pull box undirected graph is a graph structure, such as Figure 4 As shown, n=6, i.e., 6 nodes, and whether the edge is connected is determined by A.

[0062] Step S306 , performing a maximum cluster search based on the pull-box undirected graph, determining a plurality of second pull boxes corresponding to each cluster, and obtaining pull box clustering data corresponding to each target object.

[0063] For the graph formed in the previous step, the maximum cluster search method can be directly used to obtain the cluster to which each pull box belongs, completing the clustering of the pull boxes. At this time, without loss of generality, any cluster obtained can be expressed as c = {s0, s1, ..., s m}, that is, the group includes the position information corresponding to m second pull boxes.

[0064] In a preferred embodiment of the present application, the step of determining the pull-box aggregation result corresponding to the target object based on the pull-box clustering data corresponding to the target object includes:

[0065] Obtain the initial annotation quality parameters of the annotators corresponding to the multiple second draw boxes. Perform a convergence evaluation based on the position information and initial annotation quality parameters corresponding to the multiple second draw boxes of the target object, and determine the convergence result of the draw boxes corresponding to the target object. The initial annotation quality parameter of each annotator can be set to 1.

[0066] The specific implementation is achieved through the following steps:

[0067] (1) Taking the initial annotation quality parameters as the current annotation quality parameters, perform the following convergence evaluation steps:

[0068] (2) Based on the position information corresponding to the plurality of second pull frames and the current annotation quality parameter, the position information of the current converged pull frame corresponding to the plurality of second pull frames is determined.

[0069] Specifically, the position information of the current converged pull-box corresponding to the plurality of second pull-boxes is calculated according to the following first specified formula:

[0070]

[0071] Wherein, Z represents the position information of the current convergence pull frame corresponding to the multiple second pull frames; s i Indicates the upper left and lower right coordinates of the i-th second pull box, α i represents the current annotation quality parameter corresponding to the i-th second pull box; m represents the number of second pull boxes.

[0072] (3) Calculate the estimated annotation quality parameters of the annotator corresponding to each second pull frame based on the position information of the current converged pull frame and the position information corresponding to each second pull frame.

[0073] Specifically, the estimated annotation quality parameter of the annotator corresponding to each second pull box is calculated according to the following second specified formula:

[0074] α i =Z / s i ;

[0075] Among them, α i " represents the estimated annotation quality parameter of the annotator corresponding to the i-th second pull box.

[0076] (4) When the change range of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds the preset range threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the convergence evaluation step is continued until the change range does not exceed the preset range threshold, and the position information of the current convergence pull frame is determined as the pull frame convergence result corresponding to the target object.

[0077] The estimated annotation quality parameter α obtained by the annotator in the previous step i "After that, calculate and estimate the annotation quality parameter α i " and the current annotation quality parameter α i The magnitude of the change, and an artificially set threshold τ a For comparison. If the change is greater than τ a , then use α i "As the new current annotation quality parameter, continue to perform the above step (2) until the change is less than τ a , then the above loop is exited and the position information of the current converged pull-box is determined as the pull-box convergence result corresponding to the target object. Finally, when the change amplitude does not exceed the preset amplitude threshold, the position information of the current converged pull-box and the current annotation quality parameters of the annotators corresponding to the multiple second pull-boxes can be output simultaneously.

[0078] The method for aggregating frame annotation data provided in the embodiments of this application uses inter-frame distance measurement and graph conversion to find the largest cluster in the graph structure, clustering each frame accordingly. Within each cluster, frame aggregation is then performed, taking into account the annotator's frame level, i.e., the annotation quality, to obtain the final aggregation result. This solution addresses the problem of multi-target, multi-frame aggregation in crowdsourced frames, converting a multi-target scene into multiple single-target scenes, and completing frame aggregation for each, thereby ultimately completing the overall aggregation task and improving aggregation efficiency.

[0079] Based on the above method embodiment, the present application embodiment also provides a device for aggregating frame annotation data, see Figure 5 As shown, the device includes:

[0080] The data acquisition module 52 is used to obtain the frame annotation data corresponding to the target image; wherein the target image includes at least one target object; the frame annotation data includes: the position information corresponding to multiple first frames obtained after multiple labelers perform frame annotation on the target object in the target image; the frame clustering module 54 is used to perform frame clustering based on the position information corresponding to the multiple first frames, and determine the frame clustering data corresponding to each target object; the frame clustering data includes: the position information corresponding to multiple second frames; the frame aggregation module 56 is used to determine the frame aggregation result corresponding to each target object according to the frame clustering data corresponding to the target object; the frame aggregation result includes the position information of the aggregation frame corresponding to the target object.

[0081] In a preferred embodiment of the present application, the above-mentioned pull-box clustering module 54 is used to calculate the IoU index between each pair of pull-boxes based on the position information corresponding to each of the multiple first pull-boxes; generate a pull-box undirected graph based on the size relationship between the IoU index between each pair of pull-boxes and the preset index threshold; perform a maximum cluster search based on the pull-box undirected graph, determine the multiple second pull-boxes corresponding to each cluster, and obtain the pull-box clustering data corresponding to each target object.

[0082] In a preferred embodiment of the present application, the above-mentioned position information includes: the upper left coordinate and the lower right coordinate of the pull box; the pull box clustering module 54 is used to calculate the intersection area and the merged area of ​​the two first pull boxes for any two first pull boxes according to the upper left coordinate and the lower right coordinate corresponding to the two first pull boxes respectively; and the ratio of the intersection area to the merged area is used as the IoU indicator corresponding to the two first pull boxes.

[0083] In a preferred embodiment of the present application, the above-mentioned pull-box clustering module 54 is used to: construct a distance matrix based on the IoU index between two pull-boxes; each element in the distance matrix is ​​used to characterize the IoU index between two pull-boxes; generate an adjacency matrix based on the size relationship between each element in the distance matrix and the preset index threshold; generate a pull-box undirected graph based on the adjacency matrix.

[0084] In a preferred embodiment of the present application, the elements in the above-mentioned adjacency matrix include 0 or 1; the above-mentioned frame clustering module 54 is used to: for each element in the distance matrix, compare the IoU index corresponding to the element with the preset index threshold; set the value corresponding to the element exceeding the preset index threshold to 1, and set the value corresponding to the element not exceeding the preset index threshold to 0, to obtain the adjacency matrix.

[0085] In a preferred embodiment of the present application, the above-mentioned pull-frame aggregation module 56 is used to: obtain the initial annotation quality parameters of the labelers corresponding to multiple second pull-frames respectively; perform aggregation evaluation based on the position information and initial annotation quality parameters corresponding to multiple second pull-frames of the target object, and determine the pull-frame aggregation result corresponding to the target object.

[0086] In a preferred embodiment of the present application, the above-mentioned pull-frame aggregation module 56 is used to use the initial annotation quality parameter as the current annotation quality parameter to perform the following aggregation evaluation steps: based on the position information and current annotation quality parameters corresponding to the multiple second pull frames, determine the position information of the current aggregation pull frame corresponding to the multiple second pull frames; calculate the estimated annotation quality parameter of the annotator corresponding to each second pull frame according to the position information of the current aggregation pull frame and the position information corresponding to each second pull frame; when the change amplitude of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds a preset amplitude threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the aggregation evaluation steps are continued until the change amplitude does not exceed the preset amplitude threshold, and the position information of the current aggregation pull frame is determined as the pull-frame aggregation result corresponding to the target object.

[0087] In a preferred embodiment of the present application, the above-mentioned pull-frame aggregation module 56 is used to calculate the position information of the current aggregated pull-frame corresponding to the plurality of second pull-frames according to the following first specified formula:

[0088]

[0089] Wherein, Z represents the position information of the current convergence pull frame corresponding to the multiple second pull frames; s i Indicates the upper left and lower right coordinates of the i-th second pull box, α i represents the current annotation quality parameter corresponding to the i-th second pull box; m represents the number of second pull boxes.

[0090] In a preferred embodiment of the present application, the above-mentioned frame aggregation module 56 is used to calculate the estimated annotation quality parameter of the annotator corresponding to each second frame according to the following second specified formula:

[0091] α i =Z / s i ;

[0092] Among them, α i " represents the estimated annotation quality parameter of the annotator corresponding to the i-th second pull box.

[0093] In a preferred embodiment of the present application, the pull-frame aggregation module 56 is used to simultaneously output the position information of the current aggregated pull-frame and the current annotation quality parameters of the annotators corresponding to the multiple second pull-frames when the change amplitude does not exceed the preset amplitude threshold.

[0094] The device provided in the embodiment of the present application has the same implementation principle and technical effects as those in the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the embodiment of the device, reference can be made to the corresponding content in the aforementioned method embodiment.

[0095] The present application also provides an electronic device, such as Figure 6 FIG. 1 is a schematic diagram of the structure of the electronic device, wherein the electronic device includes a processor 61 and a memory 60. The memory 60 stores computer-executable instructions that can be executed by the processor 61. The processor 61 executes the computer-executable instructions to implement the following steps:

[0096] Obtaining frame annotation data corresponding to a target image; wherein the target image includes at least one target object; the frame annotation data includes: position information corresponding to multiple first frames obtained by summarizing frame annotations performed by multiple annotators on the target object in the target image; performing frame clustering based on the position information corresponding to the multiple first frames to determine frame clustering data corresponding to each target object; the frame clustering data includes: position information corresponding to multiple second frames; for each target object, determining a frame aggregation result corresponding to the target object based on the frame clustering data corresponding to the target object; the frame aggregation result includes the position information of the aggregated frame corresponding to the target object.

[0097] In a preferred embodiment of the present application, the above-mentioned step of clustering the pull boxes based on the position information corresponding to the multiple first pull boxes and determining the pull box clustering data corresponding to each target object includes: calculating the IoU index between each pair of pull boxes according to the position information corresponding to the multiple first pull boxes; generating a pull box undirected graph according to the size relationship between the IoU index between each pair of pull boxes and the preset index threshold; performing a maximum cluster search based on the pull box undirected graph to determine the multiple second pull boxes corresponding to each cluster, and obtaining the pull box clustering data corresponding to each target object.

[0098] In a preferred embodiment of the present application, the above-mentioned position information includes: the upper left coordinate and the lower right coordinate of the pull box; according to the position information corresponding to multiple first pull boxes, the step of calculating the IoU index between each pull box includes: for any two first pull boxes, according to the upper left coordinate and the lower right coordinate corresponding to the two first pull boxes, calculating the intersection area and the merged area of ​​the two first pull boxes; taking the ratio of the intersection area to the merged area as the IoU index corresponding to the two first pull boxes.

[0099] In a preferred embodiment of the present application, the above-mentioned step of generating a pull-box undirected graph based on the size relationship between the IoU index between the two pull-boxes and the preset index threshold includes: constructing a distance matrix based on the IoU index between the two pull-boxes; each element in the distance matrix is ​​used to characterize the IoU index between the two pull-boxes; generating an adjacency matrix based on the size relationship between each element in the distance matrix and the preset index threshold; and generating a pull-box undirected graph based on the adjacency matrix.

[0100] In a preferred embodiment of the present application, the elements in the above-mentioned adjacency matrix include 0 or 1; the above-mentioned step of generating the adjacency matrix according to the size relationship between each element in the distance matrix and the preset index threshold includes: for each element in the distance matrix, comparing the IoU index corresponding to the element with the preset index threshold; setting the value corresponding to the element exceeding the preset index threshold to 1, and setting the value corresponding to the element not exceeding the preset index threshold to 0, to obtain the adjacency matrix.

[0101] In a preferred embodiment of the present application, the above-mentioned step of determining the pull-box aggregation result corresponding to the target object based on the pull-box clustering data corresponding to the target object includes: obtaining the initial annotation quality parameters of the labelers corresponding to multiple second pull boxes respectively; performing aggregation evaluation based on the position information and initial annotation quality parameters corresponding to the multiple second pull boxes of the target object, and determining the pull-box aggregation result corresponding to the target object.

[0102] In a preferred embodiment of the present application, the above-mentioned step of performing convergence evaluation based on the position information corresponding to the multiple second pull boxes of the target object and the initial annotation quality parameters to determine the convergence result of the pull boxes corresponding to the target object includes: taking the initial annotation quality parameter as the current annotation quality parameter, and executing the following convergence evaluation step: based on the position information corresponding to the multiple second pull boxes and the current annotation quality parameter, determining the position information of the current converged pull box corresponding to the multiple second pull boxes; calculating the estimated annotation quality parameter of the annotator corresponding to each second pull box based on the position information of the current converged pull box and the position information corresponding to each second pull box; when the change amplitude of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds a preset amplitude threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the convergence evaluation step is continued until the change amplitude does not exceed the preset amplitude threshold, and the position information of the current converged pull box is determined as the convergence result of the pull box corresponding to the target object.

[0103] In a preferred embodiment of the present application, the step of determining the position information of the current converged pull box corresponding to the multiple second pull boxes based on the position information corresponding to the multiple second pull boxes and the current annotation quality parameter includes: calculating the position information of the current converged pull box corresponding to the multiple second pull boxes according to the following first specified formula:

[0104]

[0105] Wherein, Z represents the position information of the current convergence pull frame corresponding to the multiple second pull frames; s i Indicates the upper left and lower right coordinates of the i-th second pull box, α i represents the current annotation quality parameter corresponding to the i-th second pull box; m represents the number of second pull boxes.

[0106] In a preferred embodiment of the present application, the step of calculating the estimated annotation quality parameter of the annotator corresponding to each second pull-box based on the position information of the current converged pull-box and the position information corresponding to each second pull-box includes: calculating the estimated annotation quality parameter of the annotator corresponding to each second pull-box according to the following second specified formula:

[0107] α i =Z / s i ;

[0108] Among them, α i " represents the estimated annotation quality parameter of the annotator corresponding to the i-th second pull box.

[0109] In a preferred embodiment of the present application, the above method further includes: when the change amplitude does not exceed the preset amplitude threshold, simultaneously outputting the position information of the current convergence pull frame and the current annotation quality parameters of the annotators corresponding to the multiple second pull frames.

[0110] exist Figure 6 In the illustrated embodiment, the electronic device further includes a bus 62 and a communication interface 63 , wherein the processor 61 , the communication interface 63 and the memory 60 are connected via the bus 62 .

[0111] Among them, the memory 60 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 62 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 62 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0112] The processor 61 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor 61 or by software instructions. The processor 61 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor 61 reads the information in the memory and completes the steps of the method of the above embodiment in combination with its hardware.

[0113] The present application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the following steps:

[0114] Obtaining frame annotation data corresponding to a target image; wherein the target image includes at least one target object; the frame annotation data includes: position information corresponding to multiple first frames obtained by summarizing frame annotations performed by multiple annotators on the target object in the target image; performing frame clustering based on the position information corresponding to the multiple first frames to determine frame clustering data corresponding to each target object; the frame clustering data includes: position information corresponding to multiple second frames; for each target object, determining a frame aggregation result corresponding to the target object based on the frame clustering data corresponding to the target object; the frame aggregation result includes the position information of the aggregated frame corresponding to the target object.

[0115] In a preferred embodiment of the present application, the above-mentioned step of clustering the pull boxes based on the position information corresponding to the multiple first pull boxes and determining the pull box clustering data corresponding to each target object includes: calculating the IoU index between each pair of pull boxes according to the position information corresponding to the multiple first pull boxes; generating a pull box undirected graph according to the size relationship between the IoU index between each pair of pull boxes and the preset index threshold; performing a maximum cluster search based on the pull box undirected graph to determine the multiple second pull boxes corresponding to each cluster, and obtaining the pull box clustering data corresponding to each target object.

[0116] In a preferred embodiment of the present application, the above-mentioned position information includes: the upper left coordinate and the lower right coordinate of the pull box; according to the position information corresponding to multiple first pull boxes, the step of calculating the IoU index between each pull box includes: for any two first pull boxes, according to the upper left coordinate and the lower right coordinate corresponding to the two first pull boxes, calculating the intersection area and the merged area of ​​the two first pull boxes; taking the ratio of the intersection area to the merged area as the IoU index corresponding to the two first pull boxes.

[0117] In a preferred embodiment of the present application, the above-mentioned step of generating a pull-box undirected graph based on the size relationship between the IoU index between the two pull-boxes and the preset index threshold includes: constructing a distance matrix based on the IoU index between the two pull-boxes; each element in the distance matrix is ​​used to characterize the IoU index between the two pull-boxes; generating an adjacency matrix based on the size relationship between each element in the distance matrix and the preset index threshold; and generating a pull-box undirected graph based on the adjacency matrix.

[0118] In a preferred embodiment of the present application, the elements in the above-mentioned adjacency matrix include 0 or 1; the above-mentioned step of generating the adjacency matrix according to the size relationship between each element in the distance matrix and the preset index threshold includes: for each element in the distance matrix, comparing the IoU index corresponding to the element with the preset index threshold; setting the value corresponding to the element exceeding the preset index threshold to 1, and setting the value corresponding to the element not exceeding the preset index threshold to 0, to obtain the adjacency matrix.

[0119] In a preferred embodiment of the present application, the above-mentioned step of determining the pull-box aggregation result corresponding to the target object based on the pull-box clustering data corresponding to the target object includes: obtaining the initial annotation quality parameters of the labelers corresponding to multiple second pull boxes respectively; performing aggregation evaluation based on the position information and initial annotation quality parameters corresponding to the multiple second pull boxes of the target object, and determining the pull-box aggregation result corresponding to the target object.

[0120] In a preferred embodiment of the present application, the above-mentioned step of performing convergence evaluation based on the position information corresponding to the multiple second pull boxes of the target object and the initial annotation quality parameters to determine the convergence result of the pull boxes corresponding to the target object includes: taking the initial annotation quality parameter as the current annotation quality parameter, and executing the following convergence evaluation step: based on the position information corresponding to the multiple second pull boxes and the current annotation quality parameter, determining the position information of the current converged pull box corresponding to the multiple second pull boxes; calculating the estimated annotation quality parameter of the annotator corresponding to each second pull box based on the position information of the current converged pull box and the position information corresponding to each second pull box; when the change amplitude of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds a preset amplitude threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the convergence evaluation step is continued until the change amplitude does not exceed the preset amplitude threshold, and the position information of the current converged pull box is determined as the convergence result of the pull box corresponding to the target object.

[0121] In a preferred embodiment of the present application, the step of determining the position information of the current converged pull box corresponding to the multiple second pull boxes based on the position information corresponding to the multiple second pull boxes and the current annotation quality parameter includes: calculating the position information of the current converged pull box corresponding to the multiple second pull boxes according to the following first specified formula:

[0122]

[0123] Wherein, Z represents the position information of the current convergence pull frame corresponding to the multiple second pull frames; s i Indicates the upper left and lower right coordinates of the i-th second pull box, α i represents the current annotation quality parameter corresponding to the i-th second pull box; m represents the number of second pull boxes.

[0124] In a preferred embodiment of the present application, the step of calculating the estimated annotation quality parameter of the annotator corresponding to each second pull-box based on the position information of the current converged pull-box and the position information corresponding to each second pull-box includes: calculating the estimated annotation quality parameter of the annotator corresponding to each second pull-box according to the following second specified formula:

[0125] α i =Z / s i ;

[0126] Among them, α i " represents the estimated annotation quality parameter of the annotator corresponding to the i-th second pull box.

[0127] In a preferred embodiment of the present application, the above method further includes: when the change amplitude does not exceed the preset amplitude threshold, simultaneously outputting the position information of the current convergence pull frame and the current annotation quality parameters of the annotators corresponding to the multiple second pull frames.

[0128] The computer program products of the methods, devices, and electronic devices provided in the embodiments of the present application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. For specific implementation, please refer to the method embodiments and will not be repeated here.

[0129] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0130] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0131] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of this application and simplify the description. They do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0132] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. A method for aggregating frame-marked data, characterized in that: The method comprises: Obtaining frame annotation data corresponding to a target image; wherein the target image includes at least one target object; the frame annotation data includes: position information corresponding to a plurality of first frames obtained by summarizing frame annotations performed by a plurality of annotators on the target object in the target image; Performing frame clustering based on the position information corresponding to the plurality of first pull frames, and determining the frame clustering data corresponding to each target object; the frame clustering data includes: the position information corresponding to the plurality of second pull frames; For each target object, determining a pull-box aggregation result corresponding to the target object according to the pull-box clustering data corresponding to the target object; the pull-box aggregation result includes position information of the aggregation pull-box corresponding to the target object; The step of determining the pull-box aggregation result corresponding to the target object according to the pull-box clustering data corresponding to the target object includes: Obtaining initial annotation quality parameters of the annotators corresponding to the plurality of second pull boxes; Using the initial annotation quality parameter as the current annotation quality parameter, perform the following convergence evaluation steps: Determining the position information of the current converged pull frame corresponding to the plurality of second pull frames based on the position information respectively corresponding to the plurality of second pull frames and the current annotation quality parameter; Calculating an estimated annotation quality parameter of the annotator corresponding to each second pull frame based on the position information of the current converged pull frame and the position information corresponding to each second pull frame; When the change amplitude of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds a preset amplitude threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the convergence evaluation step is continued until the change amplitude does not exceed the preset amplitude threshold, and the position information of the current convergence pull box is determined as the pull box convergence result corresponding to the target object.

2. The method according to claim 1, characterized in that The step of clustering the pull boxes based on the position information corresponding to the plurality of first pull boxes and determining the pull box cluster data corresponding to each target object includes: Calculate the IoU index between any two of the first pull frames based on the position information corresponding to each of the first pull frames. Generate an undirected frame graph based on the relationship between the IoU index between each frame and the preset index threshold; A maximum cluster search is performed based on the pull-box undirected graph to determine a plurality of second pull boxes corresponding to each cluster, and to obtain pull box clustering data corresponding to each target object.

3. The method according to claim 2, characterized in that The position information includes: the upper left coordinate and the lower right coordinate of the pull box; the step of calculating the IoU index between each of the pull boxes according to the position information corresponding to the plurality of first pull boxes includes: For any two first pull boxes, the intersection area and the combined area of ​​the two first pull boxes are calculated according to the upper left coordinates and lower right coordinates corresponding to the two first pull boxes respectively; and the ratio of the intersection area to the combined area is used as the IoU index corresponding to the two first pull boxes.

4. The method according to claim 2, characterized in that The steps of generating an undirected graph of pull boxes according to the relationship between the IoU index between two pull boxes and the preset index threshold include: Construct a distance matrix based on the IoU index between two boxes; each element in the distance matrix is ​​used to represent the IoU index between two boxes; Generate an adjacency matrix based on the size relationship between each element in the distance matrix and the preset indicator threshold; A pull-box undirected graph is generated according to the adjacency matrix.

5. The method according to claim 4, characterized in that The elements in the adjacency matrix include 0 or 1; and the step of generating the adjacency matrix according to the size relationship between each element in the distance matrix and a preset index threshold includes: For each element in the distance matrix, the IoU index corresponding to the element is compared with the preset index threshold; The values ​​corresponding to the elements exceeding the preset index threshold are set to 1, and the values ​​corresponding to the elements not exceeding the preset index threshold are set to 0, to obtain an adjacency matrix.

6. The method according to claim 1, characterized in that The step of determining the position information of the current converged pull box corresponding to the plurality of second pull boxes based on the position information respectively corresponding to the plurality of second pull boxes and the current annotation quality parameter includes: Calculate the position information of the current converged pull-box corresponding to the multiple second pull-boxes according to the following first specified formula: ； in, Indicates the position information of the current converged pull-box corresponding to the multiple second pull-boxes; Indicates the The upper left and lower right coordinates corresponding to the second pull box, Indicates the The current annotation quality parameter corresponding to the second pull-box, Indicates the number of the second pull box.

7. The method according to claim 6, characterized in that The step of calculating the estimated annotation quality parameter of the annotator corresponding to each second pull frame according to the position information of the current converged pull frame and the position information corresponding to each second pull frame includes: Calculate the estimated annotation quality parameter of each annotator corresponding to the second pull box according to the following second specified formula: ； in, Indicates the The estimated annotation quality parameters of the annotator corresponding to the second pull box.

8. The method according to claim 1, characterized in that The method further comprises: When the change amplitude does not exceed a preset amplitude threshold, the position information of the current convergence pull frame and the current annotation quality parameters of the annotators corresponding to the plurality of second pull frames are output simultaneously.

9. A device for gathering frame-marked data, characterized in that: The device comprises: A data acquisition module is configured to acquire frame annotation data corresponding to a target image; wherein the target image includes at least one target object; and the frame annotation data includes: position information corresponding to a plurality of first frames obtained by summarizing frame annotations performed by a plurality of annotators on the target object in the target image; A frame clustering module is configured to perform frame clustering based on the position information corresponding to the plurality of first frame pulls, and determine frame clustering data corresponding to each target object; the frame clustering data includes: position information corresponding to the plurality of second frame pulls; A pull-frame aggregation module is configured to determine, for each target object, a pull-frame aggregation result corresponding to the target object based on the pull-frame clustering data corresponding to the target object; the pull-frame aggregation result includes position information of the aggregation pull-frame corresponding to the target object; The step of determining the pull-box aggregation result corresponding to the target object according to the pull-box clustering data corresponding to the target object includes: Obtaining initial annotation quality parameters of the annotators corresponding to the plurality of second pull boxes; Using the initial annotation quality parameter as the current annotation quality parameter, perform the following convergence evaluation steps: Determining the position information of the current converged pull frame corresponding to the plurality of second pull frames based on the position information respectively corresponding to the plurality of second pull frames and the current annotation quality parameter; Calculating an estimated annotation quality parameter of the annotator corresponding to each second pull frame based on the position information of the current converged pull frame and the position information corresponding to each second pull frame; When the change amplitude of the estimated annotation quality parameter relative to the current annotation quality parameter exceeds a preset amplitude threshold, the estimated annotation quality parameter is updated to the current annotation quality parameter, and the convergence evaluation step is continued until the change amplitude does not exceed the preset amplitude threshold, and the position information of the current convergence pull box is determined as the pull box convergence result corresponding to the target object.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-executable instructions that can be executed by the processor, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 8.

Images