Method and device for reviewing annotated images

Abstract

The present invention discloses a method and device for reviewing annotated images, and relates to the field of computer technology. A specific implementation of the method includes: counting the number of annotation frames for multiple annotated images of the same image data to determine the correct number of frames that the image data should be annotated with; reviewing the number of annotation frames for multiple annotated images based on the correct number of frames; matching the position and annotation text of the annotation frame of each annotated image that has passed the review of the number of annotation frames with the reference frame; and determining the annotated image that has passed the review of the number of annotation frames and has successfully matched the position and annotation text of each annotation frame as the annotated image that has finally passed the review. This implementation does not rely on the subjective experience and attitude of the reviewer, avoids review errors such as misjudgment and omission, improves the accuracy of the annotated images after review, can quickly locate annotation errors during the review process, can effectively judge the spaces and pixel differences within the text line, and does not require the annotator to locate the parts that need to be modified after the data is rejected.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for reviewing annotated images. Background Art

[0002] Natural scene text annotation (OCR marking) uses computer technology to annotate text within images using boxes. This technology can be combined with ID card recognition, bank card recognition, and invoice recognition, and has applications in artificial intelligence systems, assisted driving systems, intelligent robotics, intelligent video surveillance, and intelligent transportation. Currently, natural scene text annotation relies on human reviewers to verify the accuracy of the annotation results and then return the verified annotated data. Existing manual review methods are prone to the following issues when it comes to text review: reviewers cannot focus for long periods of time to verify the accuracy of the text, which can lead to errors; when there are large amounts of text, although each line is numbered, omissions are likely to occur during review; the review of similar text can lead to misjudgments; and it is difficult to effectively determine the spacing within text lines during review. The review of box annotations is prone to the following issues: pixel differences cannot be effectively determined, relying on subjective judgment; when there are many boxes, incorrect or missing boxes cannot be quickly located; and when there are many boxes, the current review method requires human reviewers to identify the areas that need to be revised after the data is rejected (rejection means failure).

[0003] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0004] Relying on the subjective experience and attitude of the reviewers, it is easy to make review errors such as misjudgment and omission. Reviewers cannot quickly locate annotation errors during the review, and it is difficult to effectively judge the space and pixel differences within the text line. After the data is rejected, the annotators need to locate the areas that need to be modified by themselves. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a method and device for reviewing annotated images, which can be independent of the subjective experience and attitude of the reviewer, avoid review errors such as misjudgment and omission, improve the accuracy of the reviewed annotated images, quickly locate annotation errors during the review process, and effectively judge the spaces and pixel differences within text lines. The rejected data does not require the annotator to locate the areas that need to be modified, making modification convenient.

[0006] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for reviewing annotated images is provided.

[0007] A method for reviewing an annotated image comprises: counting the number of annotated frames for a plurality of annotated images of the same image data to determine the correct number of frames that the image data should be annotated with; reviewing the number of annotated frames for the plurality of annotated images based on the correct number of frames; matching the position and annotation text of the annotation frame of each annotated image that has passed the review with a reference frame of the image data generated according to a preset rule; and determining an annotated image that has passed the review with the number of annotated frames and in which the position and annotation text of each annotated frame have successfully matched as the annotated image that has finally passed the review.

[0008] Optionally, the step of counting the number of annotated frames for multiple annotated pictures of the same picture data to determine the correct number of frames that the picture data should be annotated includes: counting the number of annotated frames for multiple annotated pictures of the same picture data to obtain the number of annotated frames for each annotated picture; if the number of annotated frames for each annotated picture is consistent, then the number of annotated frames is used as the correct number of frames; if the number of annotated frames for each annotated picture is inconsistent, then the number of annotated frames corresponding to the largest number of annotated pictures is selected as the correct number of frames.

[0009] Optionally, the annotated picture whose number of annotated frames has passed the review is an annotated picture whose number of annotated frames is equal to the correct number of frames; the step of generating the reference frame of the picture data according to a preset rule includes: selecting two annotated pictures from the annotated pictures whose number of annotated frames has passed the review; matching each annotated frame of one of the selected annotated pictures with each annotated frame of the other annotated picture, and taking the intersection area of ​​the two annotated frames that are successfully matched each time as the reference frame of the picture data, and storing the annotated frames in the selected annotated picture that cannot be successfully matched with any annotated frame in a disputed frame list; if the number of reference frames obtained by the above matching is less than the correct number of frames, selecting another annotated picture from the annotated pictures whose number of annotated frames has passed the review, matching the annotated frames in the disputed frame list with the annotated frames of the latest selected annotated picture to continue determining the reference frame, and repeating the above process until the number of reference frames finally obtained is equal to the correct number of frames, then stopping the matching, thereby obtaining all the reference frames of the picture data.

[0010] Optionally, for a labeled picture or each labeled box in the disputed box list, matching is performed with each labeled box of another labeled picture in the following manner: taking the labeled picture or each labeled box in the disputed box list as the target labeled box, calculating the intersection-and-union (IoU) of the target labeled box with each labeled box of the other labeled picture, obtaining the maximum IoU of the target labeled box, and judging whether the maximum IoU is greater than or equal to an IoU threshold; if so, determining that the target labeled box is successfully matched with the labeled box of the other labeled picture with the largest IoU; otherwise, determining that the target labeled box cannot be successfully matched with any labeled box.

[0011] Optionally, the step of positionally matching the annotation frame of each annotated image whose number of annotation frames has been reviewed with the reference frame of the image data generated according to preset rules includes: for each annotation frame of each annotated image whose number of annotation frames has been reviewed, calculating the intersection-and-union ratio of the annotation frame with each reference frame of the image data; and determining whether the maximum intersection-and-union ratio of the annotation frame with the reference frame is greater than or equal to a preset threshold; if so, the position matching of the annotation frame is successful; otherwise, the position matching fails.

[0012] Optionally, the step of matching the annotation text of the annotation box of each annotated picture whose number of annotation boxes has been reviewed with the reference box includes: selecting target annotated pictures whose annotation box positions are successfully matched from the annotated pictures whose number of annotation boxes has been reviewed; counting the annotation texts of the annotation boxes at the same position of each target annotated picture, wherein, for each position, the annotation text that appears the most times at the position is selected as the correct annotation text for the position; for the annotation box at each position of each target annotated picture, comparing the annotation box with the correct annotation text at the position; if the comparison is consistent, the annotation text of the annotation box is successfully matched, otherwise the annotation text matching fails.

[0013] Optionally, when the annotated image meets one or more of the following conditions, it is determined that the final review has failed: the number of the annotated boxes has failed to be reviewed, the position matching of at least one of the annotated boxes has failed, and the annotated text matching of at least one of the annotated boxes has failed; the method also includes: adding review mark information to the annotated images that have passed the final review and those that have failed the final review, wherein in the review mark information of the annotated images that have failed the final review, different categories of modification objects are displayed with different review marks, and the categories of the modification objects include the position of the annotation box to be modified and the annotation text to be modified.

[0014] According to another aspect of an embodiment of the present invention, a device for reviewing annotated images is provided.

[0015] A device for reviewing annotated images comprises: a frame number counting module for counting the number of annotated frames of a plurality of annotated images of the same image data to determine the correct number of frames that the image data should be annotated with; a frame number review module for reviewing the number of annotated frames of the plurality of annotated images according to the correct number of frames; a matching module for matching the position and annotation text of the annotation frame of each annotated image that has passed the annotated frame number review with a reference frame of the image data generated according to preset rules; and an audit result output module for determining an annotated image that has passed the annotated frame number review and has successfully matched the position and annotation text of each annotated frame as an annotated image that has finally passed the annotated review.

[0016] Optionally, the frame number statistics module is also used to: count the number of annotated frames for multiple annotated pictures of the same image data to obtain the number of annotated frames of each annotated picture; if the number of annotated frames of each annotated picture is consistent, then the number of annotated frames is used as the correct number of frames; if the number of annotated frames of each annotated picture is inconsistent, then the number of annotated frames corresponding to the largest number of annotated pictures is selected as the correct number of frames.

[0017] Optionally, the annotated picture whose number of annotated frames has been reviewed and approved is an annotated picture whose number of annotated frames is equal to the correct number of frames; the device also includes a reference frame generation module, which is used to: select two annotated pictures from the annotated pictures whose number of annotated frames has been reviewed and approved; match each annotated frame of one of the selected annotated pictures with each annotated frame of the other annotated picture, use the intersection area of ​​the two annotated frames that are successfully matched each time as the reference frame of the picture data, and store the annotated frames in the selected annotated picture that cannot be successfully matched with any annotated frame in a disputed frame list; if the number of reference frames obtained by the above matching is less than the correct number of frames, select another annotated picture from the annotated pictures whose number of annotated frames has been reviewed and approved, match the annotated frames in the disputed frame list with the annotated frames of the most recently selected annotated picture to continue to determine the reference frame, repeat the above process until the number of reference frames finally obtained is equal to the correct number of frames, then stop the matching, and thus obtain all the reference frames of the picture data.

[0018] Optionally, the reference frame generation module includes a labeling frame matching submodule, which is used to match each labeling frame in an labeled picture or the disputed frame list with each labeling frame of another labeled picture in the following manner: taking each labeling frame in the labeled picture or the disputed frame list as the target labeling frame, calculating the intersection and union (IoU) of the target labeling frame with each labeling frame of the other labeled picture, obtaining the maximum IoU of the target labeling frame, and judging whether the maximum IoU is greater than or equal to an IoU threshold; if so, determining that the target labeling frame is successfully matched with the labeling frame of the other labeled picture with the largest IoU; otherwise, determining that the target labeling frame cannot be successfully matched with any labeling frame.

[0019] Optionally, the matching module includes a position matching submodule, which is used to: calculate the intersection-and-union ratio of each annotation frame of each annotated image that has passed the number of annotation frames and each reference frame of the image data; determine whether the maximum intersection-and-union ratio of the annotation frame and the reference frame is greater than or equal to a preset threshold value, if so, the position matching of the annotation frame is successful, otherwise the position matching fails.

[0020] Optionally, the matching module includes an annotation text matching submodule, which is used to: select the target annotation pictures whose annotation box positions are successfully matched from the annotation pictures that have passed the inspection; count the annotation texts of the annotation boxes at the same position of each target annotation picture, wherein, for each position, the annotation text that appears the most times at the position is selected as the correct annotation text for the position; for each position annotation box of each target annotation picture, compare the annotation box with the correct annotation text at the position; if the comparison is consistent, the annotation text of the annotation box is successfully matched, otherwise the annotation text matching fails.

[0021] Optionally, the audit result output module is also used to: determine that the final audit fails when the annotated image meets one or more of the following conditions: the number of annotated boxes fails to be audited, the position matching of at least one of the annotated boxes fails, and the annotated text matching of at least one of the annotated boxes fails; the audit result output module is also used to: add audit mark information to the annotated images that pass the final audit and fail the final audit, wherein in the audit mark information of the annotated images that fail the final audit, different categories of modification objects are displayed with different audit marks, and the categories of the modification objects include the annotation box positions to be modified and the annotation texts to be modified.

[0022] According to yet another aspect of the embodiments of the present invention, an electronic device is provided.

[0023] An electronic device includes: one or more processors; and a memory for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the annotated image review method provided in an embodiment of the present invention.

[0024] According to yet another aspect of an embodiment of the present invention, a computer-readable medium is provided.

[0025] A computer-readable medium stores a computer program, which, when executed by a processor, implements the annotated image review method provided by an embodiment of the present invention.

[0026] One embodiment of the above invention has the following advantages or beneficial effects: the number of annotated frames is counted for multiple annotated images of the same image data to determine the correct number of frames that the image data should be annotated with; the number of annotated frames is reviewed for multiple annotated images based on the correct number of frames; the annotated frame of each annotated image that has passed the annotated frame review is position-matched and the annotated text is matched with the reference frame of the image data generated according to the preset rules; the annotated image that has passed the annotated frame review and whose position and annotated text are successfully matched is determined as the annotated image that has finally passed the review. It can be independent of the subjective experience and attitude of the reviewer, avoid review errors such as misjudgment and omission, improve the accuracy of the annotated image after review, quickly locate the annotated errors during the review process, and effectively judge the spaces and pixel differences within the text line. By adding review mark information to the annotated image after review and displaying different categories of modification objects with different review marks, the rejected data does not need the annotator to locate the areas that need to be modified, making modification convenient.

[0027] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0029] Figure 1 This is a schematic diagram of the main steps of a method for reviewing annotated images according to one embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of the positional relationship between two rectangular frames according to an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of calculating the intersection-over-union ratio of two rectangular frames according to one embodiment of the present invention;

[0032] Figure 4 This is a diagram of the distribution of labeled image review tasks according to one embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram of the annotated image review process according to one embodiment of the present invention;

[0034] Figure 6 is a schematic diagram of a process for generating a reference frame for image data according to one embodiment of the present invention;

[0035] Figure 7 is a schematic diagram of a process for determining correctly annotated text according to one embodiment of the present invention;

[0036] Figure 8is a schematic diagram of the main modules of a device for reviewing annotated images according to an embodiment of the present invention;

[0037] Figure 9 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0038] Figure 10 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0040] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0041] Figure 1 1 is a schematic diagram of the main steps of a method for reviewing annotated images according to an embodiment of the present invention.

[0042] like Figure 1 As shown, a method for reviewing annotated images according to an embodiment of the present invention mainly includes the following steps S101 to S104.

[0043] Step S101: performing statistics on the number of annotated frames for multiple annotated pictures of the same picture data to determine the correct number of frames that the picture data should be annotated with.

[0044] Step S101 specifically includes: counting the number of annotation frames of multiple annotated pictures of the same picture data to obtain the number of annotation frames of each annotated picture; if the number of annotation frames of each annotated picture is consistent, then the number of annotation frames is used as the correct number of frames; if the number of annotation frames of each annotated picture is inconsistent, then selecting the number of annotation frames corresponding to the largest number of annotated pictures, and using the selected number of annotation frames as the correct number of frames.

[0045] The number of annotation frames corresponding to the largest number of annotated pictures can be selected as the correct number of frames by voting. For example, if 7 annotators annotate a piece of picture data at the same time, 7 annotated pictures can be obtained. Assume that the number of annotation frames for 4 of the annotated pictures is 2, and the number of annotation frames for 3 of the annotated pictures is 1. Each annotation frame number is regarded as one vote. The number of annotation frames 2 corresponds to 4 annotated pictures, which is the annotation frame number with the most votes. Therefore, the number of annotation frames 2 is regarded as the correct number of frames.

[0046] Step S102: reviewing the number of annotated frames for the plurality of annotated images based on the correct number of frames.

[0047] Specifically, it is to review whether the number of annotated frames of the annotated image is equal to the determined correct number of frames. If the number of annotated frames of the annotated image is equal to the correct number of frames, the number of annotated frames of the annotated image will pass the review; otherwise, if the number of annotated frames of the annotated image is not equal to the correct number of frames, the number of annotated frames of the annotated image will fail the review.

[0048] Step S103: performing position matching and annotation text matching on the annotation frame of each annotated picture that has passed the verification of the number of annotation frames, and the reference frame of the picture data generated according to the preset rules.

[0049] The reference frame is used to indicate the location where the image data should be correctly annotated.

[0050] The step of generating a reference frame for the image data according to a preset rule may include:

[0051] Select two annotated pictures from each annotated picture whose number of annotated frames has been reviewed and approved; match each annotated frame of one of the selected annotated pictures with each annotated frame of the other annotated picture, and use the intersection area of ​​the two successfully matched annotated frames each time as the reference frame of the picture data, and store the annotated frames in the selected annotated picture that cannot be successfully matched with any annotated frame in the disputed frame list; if the number of reference frames obtained through the above matching is less than the number of correct frames, select another annotated picture from each annotated picture whose number of annotated frames has been reviewed and approved, and match the annotated frames in the disputed frame list with each annotated frame of the latest selected annotated picture to continue to determine the reference frame, and repeat the above process until the final number of reference frames is equal to the number of correct frames, then stop matching, and thus obtain all the reference frames of the picture data.

[0052] The annotation box that cannot be successfully matched with any annotation box is a disputed box, which means that its correctness cannot be judged. It will then be stored in the disputed box list. The annotation boxes in the disputed box list can continue to be matched with the annotation boxes of the remaining other annotated images.

[0053] For example, suppose there are three annotated pictures whose number of annotation frames has passed the review: annotated pictures 1, 2, and 3. Each annotated picture includes three annotation frames, represented by A, B, and C. That is, the annotation frames of annotated pictures 1, 2, and 3 are: {A1, B1, C1}, {A2, B2, C2}, and {A3, B3, C3}, respectively. Then, two annotated pictures can be randomly selected, for example, annotated pictures 1 and 2 are selected, and each annotation frame of annotated picture 1 is matched with each annotation frame of annotated picture 2. Assuming that A1 and A2 are matched successfully, and B1 and B2 are matched successfully, then the intersection area of ​​A1 and A2, and the intersection area of ​​B1 and B2 are used as the two reference frames of the picture data, C1 and A2, and B2 respectively. , none of C2 is matched successfully. Similarly, C2 is not matched successfully with any of A1, B1, and C1. C1 and C2 are stored in the dispute frame list. Through the above matching, two reference frames are obtained. The number of reference frames is less than the number of correct frames (three). Then, a labeled image is randomly selected. In this example, there are only three labeled images, so labeled image 3 is selected. C1 and C2 in the dispute frame list are matched with A3, B3, and C3 of labeled image 3 respectively. Assuming that C2 and C3 are matched successfully, the intersection area of ​​C2 and C3 is used as the reference frame of the image data. At this time, the number of reference frames obtained is equal to the number of correct frames, which is three. Matching is stopped, and all the reference frames of the image data are obtained.

[0054] If all the annotation frames of each annotated image that has passed the annotation frame number review are matched, and the final number of reference frames is still less than the correct number of frames, these annotated images will be rejected, indicating that these annotated images have failed the final review, and the annotation image review process will end.

[0055] For each annotation frame in an annotated image or disputed frame list, it can be matched with each annotation frame in another annotated image in the following way:

[0056] Take each annotation frame in a labeled image or disputed frame list as the target annotation frame, calculate the intersection and union ratio of the target annotation frame with each annotation frame of the other labeled image, and obtain the maximum intersection and union ratio of the target annotation frame. The maximum intersection and union ratio of the target annotation frame is the maximum value of the intersection and union ratio of the target annotation frame with each annotation frame of the other labeled image. Determine whether the maximum intersection and union ratio is greater than or equal to the intersection and union ratio threshold. If so, it is determined that the target annotation frame is successfully matched with the annotation frame of the other labeled image with the largest intersection and union ratio. Otherwise, it is determined that the target annotation frame cannot be successfully matched with any annotation frame.

[0057] The intersection-over-union (IOU) of two rectangles is equal to the intersection of the two rectangles divided by the union of the two rectangles. Figure 2It is a schematic diagram of the positional relationship between two rectangular boxes according to an embodiment of the present invention. An IOU can be calculated for every two rectangular boxes. In the figure, box represents a rectangular box. In an embodiment of the present invention, both the reference box and the annotation box are rectangular boxes. When the maximum intersection-and-union (IoU) is greater than or equal to the IoU threshold, it indicates that the maximum IoU is legal. Otherwise, the maximum IoU is illegal, that is: when the maximum IoU of the target annotation box is legal, the target annotation box successfully matches the annotation box of another annotation picture with the largest IoU; when the maximum IoU of the target annotation box is illegal, the target annotation box cannot successfully match any annotation box.

[0058] Calculate the intersection of two rectangular boxes IOU as follows Figure 3 As shown in the figure, the horizontal line represents the division between the area intersection and the area union. When calculating the intersection-and-union ratio of two rectangular boxes, the coordinates of the two rectangular boxes can be analyzed, and the area intersection and area union of the two rectangular boxes can be calculated through coordinate calculation. The intersection-and-union ratio of the two rectangular boxes is obtained based on the ratio of the area intersection to the area union.

[0059] The intersection-in-union (IoU) threshold can be set according to needs. For example, it can be found through pre-testing that when the IoU threshold is set to a value in the range of (60%, 80%), the position matching accuracy of the annotation box is the best. Then the IoU threshold can be taken according to the above range, for example, set to 70%. When the maximum IoU of the target annotation box and the annotation box of another annotation image is greater than or equal to 70%, it means that the target annotation box and the corresponding annotation box of the other annotation image correspond to the same annotation object, and the match is successful. A successful match indicates that the position of the annotation box at this position is correct.

[0060] Combined with the above introduction, the annotated boxes in the annotated image that cannot be successfully matched with any annotated box are stored in the disputed box list, that is, the annotated boxes whose maximum intersection-over-union ratio is less than the intersection-over-union ratio threshold (maximum IOU is illegal) are stored in the disputed box list.

[0061] In another embodiment, an intermediate threshold value, such as 40%, can also be set. When 70%>maximum intersection-in-union ratio>40%, there may be some rotated images in the annotation process, and when the annotation box for the text on them intersects with the annotation box of other text, it is impossible to determine whether the two compared boxes are annotating the same text. In this case, such an annotation box can be considered a disputed box and stored in the disputed box list. An annotation box with a maximum intersection-in-union ratio of ≤40% is considered to be an incorrectly annotated box and is directly rejected. For example, if the maximum intersection-in-union ratio is 0, it can be determined that the two annotation boxes are not annotating the same text, and can be rejected.

[0062] In another embodiment, if there are only two annotated pictures with the correct number of annotated frames (the correct number of annotated frames means that the number of annotated frames is equal to the correct number of frames), if there are disputed frames in the two annotated pictures, it means that the position of the annotated frames in the two annotated pictures is incorrect, and the two annotated pictures are directly rejected.

[0063] After obtaining all the reference frames of the image data, the position matching of the annotation frame of each annotated image that has passed the number of annotation frames is performed with each reference frame. The specific steps include: for each annotation frame of each annotated image that has passed the number of annotation frames, calculating the intersection-over-union ratio of the annotation frame with each reference frame of the image data; judging whether the maximum intersection-over-union ratio of the annotation frame with the reference frame is greater than or equal to a preset threshold. If so, the position matching of the annotation frame is successful; otherwise, the position matching fails. It should be noted that in the process of generating the reference frame, if some annotated images are rejected, then this step only needs to perform the above-mentioned position matching on the annotation frames of the remaining annotated images that have passed the number of annotation frames.

[0064] The step of matching the annotation text of the annotation frame of each annotated image that has passed the review of the number of annotation frames with the reference frame specifically includes: selecting target annotated images whose annotation frames are successfully matched at all positions from the annotated images that have passed the review of the number of annotation frames; counting the annotation texts of the annotation frames at the same position of each target annotated image, wherein, for each position, the annotation text that appears the most times at the position can be selected by voting as the correct annotation text for the position; for each position annotation frame of each target annotated image, comparing the annotation frame with the correct annotation text at the position; if the comparison is consistent, the annotation text of the annotation frame is successfully matched, otherwise the annotation text matching fails.

[0065] Step S104: The annotated pictures whose number of annotation frames has passed the review and whose positions and annotation texts of the annotation frames have successfully matched are determined as the annotated pictures that have passed the final review.

[0066] When an annotated image meets one or more of the following conditions, it is determined to have failed the final review: the number of annotation boxes fails to be reviewed, the position matching of at least one annotation box fails, and the annotation text matching of at least one annotation box fails.

[0067] Review mark information can also be added to the annotated images that have passed or failed the final review. In the review mark information of the annotated images that have failed the final review, different review marks are displayed for different categories of modification objects. The categories of modification objects include the location of the annotation box to be modified and the annotation text to be modified. The review mark information can be review mark boxes of different colors, such as adding a green review mark box to the correct annotation location, a red review mark box to the location of the annotation box to be modified, and an orange review mark box to the annotation text to be modified, etc.

[0068] The annotated pictures of the embodiments of the present invention can be annotated pictures of identity cards, bank cards, invoices, etc., and the annotated pictures that have finally passed the review can be used accordingly as the data input end of the model for visually identifying and entering page information such as identity cards, bank cards, invoices, etc. Since the annotators only annotate manually, the annotation results cannot be completely correct due to subjective factors and fatigue factors, and the size of the annotation box can only be seen by the human eye. There are no quantitative indicators and the accuracy cannot be determined. The general practice is to first let the annotator mark, the reviewer conducts the review, and the model is trained on the reviewed data, and then compared with the vocabulary in the model to obtain the correct natural scene text information. If it is inconsistent with the model result, check the annotation result, return the wrong annotation result to the annotator for further modification, or directly reject the data for further modification when the data user uses it.

[0069] The embodiment of the present invention quantifies the position of the annotation box, obtains the annotation results by voting, and automatically reviews the annotated pictures. In this way, both the accuracy of the annotation box and the correctness of the content can be guaranteed. Moreover, the embodiment of the present invention matches the annotation box and the annotation text, so that the spaces and pixel differences within the text line can be effectively judged. In addition, the embodiment of the present invention can also be connected with the deep learning model, that is, the deep learning model is used to perform secondary annotation or review, modify the results of the annotation errors, and improve the accuracy of the annotation results through repeated iterations. For example, the annotated pictures of the embodiment of the present invention can be annotated pictures obtained by secondary annotation or review by a deep learning model (that is, secondary annotation or review based on manual annotation), and these annotated pictures are reviewed by the annotated picture review method of the embodiment of the present invention.

[0070] Figure 4 2 is a schematic diagram of distributing annotated image review tasks according to an embodiment of the present invention.

[0071] like Figure 4 As shown, one embodiment of the present invention can establish tasks for the review of annotated images. The review of all annotated images is a main task, which is further divided into multiple subtasks (or subtask packages). The same image data needs to be annotated more than twice, that is, the same image data has multiple annotated images. Each annotator can only annotate the same image data once.

[0072] For example, the main task corresponds to 10,000 images, and each subtask corresponds to 500 images. In theory, 10,000 / 500 = 20 subtasks. Since an image needs to be annotated multiple times, the "multiple times" is reflected in the subtask package, such as Figure 4As shown, the first "1" in subtask 1-1 indicates the first subtask package of 20 subtask packages, and the second "1" indicates that the image in this subtask is annotated for the first time). Similarly, subtask 1-2 indicates that the image in the first task package is annotated for the second time, and so on. For each image data in each subtask, the annotated image review process of the embodiment of the present invention is reviewed.

[0073] Figure 5 2 is a schematic diagram of the annotated image review process according to an embodiment of the present invention.

[0074] like Figure 5 As shown, the annotated image review process of one embodiment of the present invention includes the following steps S501 to S506.

[0075] Step S501: Receive multiple annotated pictures of picture data.

[0076] For a single annotated picture among the multiple annotated pictures, steps S502 to S505 are performed.

[0077] Step S502: Determine whether the number of annotation frames of the annotated image is correct. If correct, execute step S503; if incorrect, reject.

[0078] First, the correct number of frames that the image data should be annotated with is determined, and then whether the number of annotated frames of the annotated image is correct is determined based on the correct number of frames. For this step, please refer to the detailed introduction of step S101 and step S102.

[0079] Step S503: Generate a reference frame for the image data according to a preset rule.

[0080] For images with multiple annotated boxes, the accuracy of the annotation positions and text content cannot be determined solely based on the number of boxes. Therefore, a reference box is needed to determine this. The reference box is the intersection of two successfully matched annotation boxes and indicates where the image data should be correctly annotated. The reference box generation process has been detailed above and will not be repeated here.

[0081] Step S504: For each annotation frame of the annotated image, calculate the intersection over union (IOU) between the annotation frame and each reference frame of the image data.

[0082] The annotated images in this step refer to the annotated images that have not been rejected.

[0083] Step S505: Determine whether the maximum intersection-over-union ratios of the annotation boxes of the annotated image and the reference box are all legal. If so, execute step S506; otherwise, reject.

[0084] The maximum intersection-and-union (IoU) of a label box and a reference box is the maximum value of the IoUs of the label box and each reference box.

[0085] Determine whether the maximum IoU of the annotation box and the reference box is legal. Specifically, determine whether the maximum IoU is greater than or equal to a preset threshold. This threshold can be the same as the IoU threshold described above, or a different value. If the maximum IoU is greater than or equal to the preset threshold, the maximum IoU is legal; otherwise, it is illegal.

[0086] If the maximum intersection-and-union ratio of the annotation box and the reference box is legal, it means that the annotation box and the reference box with the largest intersection-and-union ratio correspond to the same position, that is, the annotation box is marked at a correct position.

[0087] If the maximum intersection-over-union ratio is illegal, it means that the annotation box is not in the same position as the corresponding position of any reference frame, indicating that the position of the annotation box is marked incorrectly. In this case, the annotation box can be fed back to the annotator with a red review mark box.

[0088] If the maximum intersection-over-union (IOU) between at least one annotation box and the reference box in an annotated image is illegal, the annotated image will be rejected. The rejected annotated image will carry the above-mentioned red review mark box, so that the annotator can modify the annotation box corresponding to the red review mark box without having to locate the part that needs to be modified by themselves.

[0089] Step S506: Collect statistics on the annotated texts in the annotated boxes at the same position of each annotated image that has not been rejected to determine whether the content of the annotated texts is correct, and reject or approve the annotated image based on the determination result.

[0090] The remaining annotated images refer to those images whose number of annotation boxes has been reviewed and approved, and whose maximum intersection-over-union ratio between the annotation box and the reference box is also legal.

[0091] Take these remaining annotated images as target annotated images. For each position, vote to select the annotation text that appears most times at that position as the correct annotation text for that position. For each position annotation box of each target annotated image, compare the annotation box with the correct annotation text for that position. If the comparison is consistent, the annotation text of the annotation box is successfully matched, otherwise the annotation text matching fails.

[0092] If all the annotation texts in a labeled image match successfully, the annotation text content of the labeled image is correct and the image is approved. If at least one annotation text fails to match successfully, it means that there are annotation boxes with incorrect annotation text in the labeled image. The incorrect annotation box can be marked with an orange review mark box and the labeled image will be rejected.

[0093] Figure 6FIG. 4 is a flow chart of generating a reference frame for image data according to an embodiment of the present invention.

[0094] like Figure 6 As shown, the process of generating a reference frame for image data includes:

[0095] Get multiple annotation results (i.e., annotated images) for a single image from the task pool;

[0096] Count the number of annotation boxes for each annotation result;

[0097] Take each marked box as a vote, and select the marked box with the most votes as the correct number of boxes;

[0098] The annotation results with the correct number of annotated boxes are stored in a correct box number result list, which includes the first annotated box list, the second annotated box list, and other annotated box lists. Each annotated box list corresponds to an annotation result with the correct number of annotated boxes. The other annotated box list represents multiple annotated box lists other than the first annotated box list and the second annotated box list.

[0099] First, calculate the maximum IOU between a single annotation box in the first annotation box list and the second annotation box list, where the maximum IOU is the maximum IOU between a single annotation box in the first annotation box list and each annotation box in the second annotation box list;

[0100] Determine whether the maximum IOU is legal;

[0101] If it is legal, the intersection area of ​​the two annotation boxes corresponding to the maximum IOU is used as the reference box of the image data, stored in the reference box list, and then it is determined whether the number of reference boxes in the reference box list is less than the number of correct boxes. If not, the process ends;

[0102] If it is illegal, the single annotation box in the first annotation box list is stored in the disputed box list. The disputed box list is used to store all the annotation boxes in the first and second annotation box lists whose maximum IOU is illegal (that is, they cannot be successfully matched with any annotation box in the other of the two annotation box lists). Subsequent annotation boxes with illegal maximum IOU will also be stored in the disputed box list;

[0103] If the number of reference boxes in the reference box list is less than the number of correct boxes, the maximum IOU is calculated for the single annotation box in the disputed box list and one of the annotation box lists in the other annotation box lists, and then the above step of determining whether the maximum IOU is legal is returned.

[0104] Through the above process, all reference frames of the image data can be obtained.

[0105] Figure 74 is a flow chart of determining correctly annotated text according to an embodiment of the present invention.

[0106] like Figure 7 As shown, the process of determining the correct annotated text according to one embodiment of the present invention includes:

[0107] Obtain a list of annotated images whose positions all match successfully (referred to as annotated image list X), and record the annotated images in the annotated image list as target annotated images;

[0108] Perform frame matching on each target annotated image to determine the annotated frame at the same position;

[0109] Count the annotated texts in the annotated boxes at the same position to obtain a list of annotated texts corresponding to the same position;

[0110] Each annotation text in the annotation text list corresponding to the same position is used as a vote to select the annotation text that appears the most times (ie, has the most votes) at the position as the correct annotation text for the position.

[0111] Figure 8 2 is a schematic diagram of main modules of a device for reviewing annotated images according to an embodiment of the present invention.

[0112] like Figure 8 As shown, an annotated image review device 800 according to an embodiment of the present invention mainly includes: a frame number counting module 801 , a frame number review module 802 , a matching module 803 , and a review result output module 804 .

[0113] The frame number counting module 801 is used to count the number of annotated frames for multiple annotated pictures of the same picture data to determine the correct number of frames that should be annotated for the picture data.

[0114] The frame number counting module 801 can be specifically used to: count the number of annotated frames for multiple annotated pictures of the same image data to obtain the number of annotated frames for each annotated picture; if the number of annotated frames for each annotated picture is the same, then the number of annotated frames is used as the correct number of frames; if the number of annotated frames for each annotated picture is inconsistent, then the number of annotated frames corresponding to the largest number of annotated pictures can be selected by voting, and the selected number of annotated frames is used as the correct number of frames.

[0115] The frame number review module 802 is used to review the number of annotated frames for multiple annotated images based on the correct number of frames.

[0116] Annotated images that pass the review for the number of annotation frames are those with the correct number of annotation frames. If the number of annotation frames is not equal to the correct number of frames, the review for the number of annotation frames will be rejected.

[0117] The matching module 803 is used to perform position matching and annotation text matching on the annotation frame of each annotated picture that has passed the annotation frame number review with the reference frame of the picture data generated according to the preset rules.

[0118] The matching module 803 includes a position matching submodule, which is used to: calculate the intersection-and-union ratio (IoU) of each annotation frame of each annotated image whose number of annotation frames has passed the review with each reference frame of the image data; determine whether the maximum IoU of the annotation frame and the reference frame is greater than or equal to a preset threshold. If so, the position matching of the annotation frame is successful; otherwise, the position matching fails.

[0119] The matching module also includes a label text matching submodule, which is used to: select the target label pictures whose label box positions are successfully matched from the label pictures that have passed the review of the number of label boxes; count the label texts of the label boxes at the same position of each target label picture, wherein, for each position, the label text that appears the most times at that position can be selected by voting as the correct label text for that position; for each label box at each position of each target label picture, compare the label box with the correct label text at that position; if the comparison is consistent, the label text of the label box is successfully matched, otherwise the label text matching fails.

[0120] The annotated image review device 800 may also include a reference frame generation module, which is used to: select two annotated images from the annotated images whose number of annotated frames has been reviewed; match each annotated frame of one of the selected annotated images with each annotated frame of the other annotated image, and use the intersection area of ​​the two annotated frames that are successfully matched each time as the reference frame of the image data, and store the annotated frames in the selected annotated image that cannot be successfully matched with any annotated frame in a disputed frame list; if the number of reference frames obtained through the above matching is less than the number of correct frames, then select another annotated image from the annotated images whose number of annotated frames has been reviewed, and match the annotated frames in the disputed frame list with the annotated frames of the most recently selected annotated image to continue determining the reference frame, and repeat the above process until the number of reference frames finally obtained is equal to the number of correct frames, then stop matching, and obtain all the reference frames of the image data.

[0121] The reference frame generation module may include a labeling frame matching submodule, which is used to match each labeling frame in a labeling picture or a disputed frame list with each labeling frame of another labeling picture in the following manner: taking each labeling frame in a labeling picture or a disputed frame list as a target labeling frame, calculating the intersection and union ratio of the target labeling frame with each labeling frame of another labeling picture, obtaining the maximum intersection and union ratio of the target labeling frame, and judging whether the maximum intersection and union ratio is greater than or equal to the intersection and union ratio threshold; if so, determining that the target labeling frame is successfully matched with the labeling frame of the other labeling picture with the largest intersection and union ratio; otherwise, determining that the target labeling frame cannot be successfully matched with any labeling frame.

[0122] The review result output module 804 is used to determine the annotated pictures that have passed the review in terms of the number of annotated boxes and the positions and annotated texts of each annotated box are successfully matched as the annotated pictures that have passed the final review.

[0123] The review result output module 804 is also used to determine that the annotated image fails the final review when it meets one or more of the following conditions: the number of annotation boxes fails to be reviewed, the position matching of at least one annotation box fails, and the annotation text matching of at least one annotation box fails.

[0124] The audit result output module 804 is also used to add audit mark information to the annotated pictures that have passed the final review and failed the final review. In the audit mark information of the annotated pictures that have failed the final review, different categories of modification objects are displayed with different audit marks. The categories of modification objects include the annotation box position to be modified and the annotation text to be modified.

[0125] In addition, the specific implementation content of the marked image review device in the embodiment of the present invention has been described in detail in the marked image review method above, so the repeated content will not be described again here.

[0126] Figure 9 An exemplary system architecture 900 is shown to which the method or apparatus for reviewing annotated images according to an embodiment of the present invention may be applied.

[0127] like Figure 9 As shown, system architecture 900 may include terminal devices 901, 902, 903, a network 904, and a server 905. Network 904 is used to provide a medium for communication links between terminal devices 901, 902, 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0128] Users can use terminal devices 901, 902, and 903 to interact with server 905 via network 904 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 901, 902, and 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0129] The terminal devices 901 , 902 , and 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0130] Server 905 may be a server that provides various services, such as a backend management server (for example only) that supports shopping websites browsed by users using terminal devices 901, 902, and 903. The backend management server may analyze and process received data such as product information query requests, and feed back processing results (for example, target push information and product information—for example only) to the terminal device.

[0131] It should be noted that the annotated image review method provided in the embodiment of the present invention is generally executed by the server 905 , and accordingly, the annotated image review device is generally set in the server 905 .

[0132] It should be understood that Figure 9 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0133] Reference below Figure 10 , which shows a structural diagram of a computer system 1000 suitable for implementing a terminal device or server of an embodiment of the present application. Figure 10 The terminal device or server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0134] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the system 1000 are also stored in the RAM 1003. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0135] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0136] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present application are executed.

[0137] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as follows: a processor including a frame count module, a frame count review module, a matching module, and an review result output module. The names of these modules do not, in some cases, limit the modules themselves. For example, the frame count module may also be described as a module for performing frame counts on multiple annotated images of the same image data to determine the correct number of frames that should be annotated for the image data.

[0140] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: counting the number of annotation frames of multiple annotated images of the same image data to determine the correct number of frames that the image data should be annotated with; reviewing the number of annotation frames of the multiple annotated images based on the correct number of frames; matching the position and annotation text of each annotated image that has passed the annotated frame number review with the reference frame of the image data generated according to preset rules; and determining the annotated image that has passed the annotated frame number review and whose position and annotation text of each annotated frame have successfully matched as the annotated image that has finally passed the annotated review.

[0141] According to the technical solution of the embodiment of the present invention, the number of annotation frames of multiple annotated pictures of the same image data is counted to determine the correct number of frames that the image data should be annotated with; the number of annotation frames of the multiple annotated pictures is reviewed based on the correct number of frames; the annotation frame of each annotated picture that has passed the review of the number of annotation frames is matched with the position and annotation text of the reference frame of the image data generated according to the preset rules; the annotated picture that has passed the review of the number of annotation frames and the position and annotation text of each annotation frame are successfully matched is determined as the annotated picture that has finally passed the review. It can be independent of the subjective experience and attitude of the reviewer, avoid audit errors such as misjudgment and omission, improve the accuracy of the annotated pictures after review, quickly locate annotation errors during the review process, and effectively judge the spaces and pixel differences within the text line. By adding review mark information to the annotated pictures after review, and displaying different categories of modification objects with different review marks, the rejected data does not need the annotator to locate the parts that need to be modified, which facilitates modification.

[0142] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for reviewing an annotated image, characterized in that: include: Counting the number of annotated frames of multiple annotated images obtained by annotating the same image data multiple times to determine the correct number of frames that should be annotated for the image data; Reviewing the number of annotated frames on the multiple annotated images according to the correct number of frames; Performing position matching and annotation text matching on the annotation frame of each annotated image whose number of annotation frames has been reviewed and approved, with the reference frame of the image data generated according to the preset rules; The annotated images whose number of annotation boxes has passed the review and whose positions and annotation texts of each annotation box have successfully matched are determined as the annotated images that have passed the final review; The step of generating a reference frame of the image data according to a preset rule includes: Select two annotated pictures from the annotated pictures that have passed the review of the number of annotated frames; Matching each annotation frame of one of the selected annotated images with each annotation frame of the other annotated image, using the intersection area of ​​the two successfully matched annotation frames as the reference frame of the image data, and storing the annotation frames in the selected annotated image that cannot be successfully matched with any annotation frame in a disputed frame list; If the number of reference frames obtained through the above matching is less than the number of correct frames, then another annotated picture is selected from the annotated pictures that have passed the review of the number of annotated frames, and the annotated frames in the disputed frame list are matched with the annotated frames of the latest selected annotated picture to continue to determine the reference frame. The above process is repeated until the final number of reference frames is equal to the number of correct frames, and then the matching is stopped to obtain all the reference frames of the picture data.

2. The method according to claim 1, characterized in that The step of counting the number of annotated frames of a plurality of annotated images obtained by annotating the same image data multiple times to determine the correct number of frames that should be annotated for the image data includes: Counting the number of annotation frames of multiple annotated images obtained by multiple annotations of the same image data to obtain the number of annotation frames of each annotated image; If the number of marked frames of each marked image is the same, then the number of marked frames is used as the correct number of frames; If the number of annotation frames of the annotated pictures is inconsistent, the number of annotation frames corresponding to the largest number of annotated pictures is selected as the correct number of frames.

3. The method according to claim 1, characterized in that The annotated picture whose number of annotated frames has passed the review is an annotated picture whose number of annotated frames is equal to the correct number of frames.

4. The method according to claim 1, wherein For each annotated frame in a labeled image or in the disputed frame list, matching is performed with each annotated frame in another labeled image in the following manner: Taking each labeled frame in the labeled image or the disputed frame list as the target labeled frame, calculating the intersection-and-union (IoU) of the target labeled frame with each labeled frame of the other labeled image, obtaining the maximum IoU of the target labeled frame, and judging whether the maximum IoU is greater than or equal to an IoU threshold. If so, determining that the target labeled frame successfully matches the labeled frame of the other labeled image with the maximum IoU; otherwise, determining that the target labeled frame cannot successfully match any labeled frame.

5. The method according to claim 1, wherein The step of positionally matching the annotation frame of each annotated image whose number of annotation frames has been reviewed with the reference frame of the image data generated according to a preset rule comprises: For each annotated frame of each annotated image that has passed the number of annotated frames review, calculating an intersection-over-union ratio between the annotated frame and each reference frame of the image data; It is determined whether the maximum intersection-over-union ratio of the annotation frame and the reference frame is greater than or equal to a preset threshold. If so, the position matching of the annotation frame is successful; otherwise, the position matching fails.

6. The method according to claim 5, characterized in that The step of matching the annotation text of each annotated image's annotation frame with the reference frame after the number of annotation frames has been reviewed includes: From the annotated images whose annotated frames have passed the review, select the target annotated images whose annotated frames have all been successfully matched; Collecting statistics on the annotated texts in the annotated boxes at the same position of each target annotated image, wherein for each position, the annotated text that appears the most times at that position is selected as the correct annotated text for that position; For each position annotation frame of each target annotated image, the annotation frame is compared with the correct annotation text of the position. If the comparison is consistent, the annotation text of the annotation frame is matched successfully, otherwise the annotation text matching fails.

7. The method according to claim 1, characterized in that When the annotated image meets one or more of the following conditions, it is determined that the final review fails: the number of the annotated frames fails to be reviewed, the position matching of at least one of the annotated frames fails, and the annotated text matching of at least one of the annotated frames fails; The method also includes: adding review mark information to the annotated pictures that have passed the final review and those that have failed the final review, wherein in the review mark information of the annotated pictures that have failed the final review, different categories of modification objects are displayed with different review marks, and the categories of the modification objects include the position of the annotation box to be modified and the annotation text to be modified.

8. A device for reviewing annotated images, characterized in that: include: A frame number counting module is used to count the number of annotated frames of multiple annotated images obtained by multiple annotating of the same image data, so as to determine the correct number of frames that should be annotated for the image data; A frame number review module, configured to review the number of annotated frames on the plurality of annotated images according to the correct number of frames; A matching module, configured to perform position matching and annotation text matching on the annotation frame of each annotated image whose number of annotation frames has been reviewed and approved, with a reference frame of the image data generated according to a preset rule; An audit result output module is used to determine the annotated pictures that have passed the audit in terms of the number of annotated boxes and the position and annotated text of each annotated box as the annotated pictures that have passed the final audit; The step of generating a reference frame of the image data according to a preset rule includes: Select two annotated pictures from the annotated pictures that have passed the review of the number of annotated frames; Matching each annotation frame of one of the selected annotated images with each annotation frame of the other annotated image, using the intersection area of ​​the two successfully matched annotation frames as the reference frame of the image data, and storing the annotation frames in the selected annotated image that cannot be successfully matched with any annotation frame in a disputed frame list; If the number of reference frames obtained through the above matching is less than the number of correct frames, then another annotated picture is selected from the annotated pictures that have passed the review of the number of annotated frames, and the annotated frames in the disputed frame list are matched with the annotated frames of the latest selected annotated picture to continue to determine the reference frame. The above process is repeated until the final number of reference frames is equal to the number of correct frames, and then the matching is stopped to obtain all the reference frames of the picture data.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images