Annotation verification method, annotation verification apparatus, and annotation verification program
The annotation verification method uses a regression model to predict and compare object positions, effectively identifying and correcting abnormal annotations in image sequences, enhancing the accuracy of machine learning models.
Patent Information
- Application Number
- JP2023081699
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-17
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2043-05-17
AI Technical Summary
The quality of annotations in image sequences is often compromised by human error or malicious intent, leading to abnormal annotations that degrade the performance of machine learning models, particularly when used for object detection, and verifying these annotations manually is labor-intensive.
An annotation verification method and device that utilize a regression model to predict the position of target objects in images based on reference information, comparing predicted positions with actual specified positions to verify the accuracy of annotations, thereby identifying abnormal annotations.
Facilitates easy and appropriate verification of annotations in image sequences, ensuring consistency and improving the performance of machine learning models by reducing the impact of abnormal annotations.
Smart Images

Figure 2024165470000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to techniques for verifying the results of annotation work on image sequences. [Background technology]
[0002] Machine learning models that use images as input to perform tasks such as object detection are trained using annotated images as training data. It is known that the performance of such machine learning models is highly dependent on the quality of annotations in the training data. For this reason, technologies to ensure the quality of annotations in training data have been developed.
[0003] For example, Patent Document 1 discloses a technology that aims to suppress a decrease in object detection accuracy when object detection training is performed using a dataset that includes images that have not been annotated as desired. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2022-043364 A Summary of the Invention [Problem to be solved by the invention]
[0005] The quality of annotation depends on the work of the annotator. Due to annotator mistakes or malicious annotators, annotation results may contain abnormal annotations.
[0006] In particular, in annotation of image sequences, abnormal annotations in some images can significantly reduce the quality of annotation. However, image sequences generally consist of many images, and manually verifying the results of annotation for each image in each image sequence requires a lot of effort.
[0007] One objective of the present disclosure is to provide a technology that enables easy and appropriate verification of the results of annotation work on each image included in an image sequence. [Means for solving the problem]
[0008] A first aspect of the present disclosure relates to an annotation verification method for verifying a result of annotation work on an image sequence, where annotation is work in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence.
[0009] An annotation verification method according to a first aspect is executed by a computer and includes obtaining a first work result which is a verified work result for a first image sequence included in the image sequence; obtaining first reference information related to a position of a target object area in each image included in the first image sequence from the first work result; calculating a predicted position of the target object area in a target image which is an image adjacent to the first image sequence based on reference information including the first reference information; obtaining an actual designated position which is the position of the target object area actually designated in the target image from the work result for the target image; and verifying the work result for the target image by comparing the target object area at the predicted position in the target image with the target object area at the actual designated position.
[0010] A second aspect of the present disclosure relates to an annotation verification device that verifies the results of annotation work on an image sequence, where annotation is work in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence.
[0011] An annotation verification device according to a second aspect includes one or more processors configured to execute the following processes: acquiring a first work result, which is a verified work result for a first image sequence included in the image sequence; acquiring first reference information related to a position of a target object area in each image included in the first image sequence from the first work result; calculating a predicted position of the target object area in a target image, which is an image adjacent to the first image sequence, based on reference information including the first reference information; acquiring an actual designated position, which is the position of the target object area actually designated in the target image, from the work result for the target image; and verifying the work result for the target image by comparing the target object area at the predicted position in the target image with the target object area at the actual designated position.
[0012] A third aspect of the present disclosure relates to an annotation verification program for causing a computer to execute a process of verifying a result of an annotation operation on an image sequence, where annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence.
[0013] An annotation verification program relating to a third aspect is configured to cause a computer to execute the following processes: a process of obtaining a first work result, which is a verified work result for a first image sequence included in the image sequence; a process of obtaining first reference information regarding the position of a target object area in each image included in the first image sequence from the first work result; a process of calculating a predicted position of the target object area in a target image, which is an image adjacent to the first image sequence, based on reference information including the first reference information; a process of obtaining an actual designated position, which is the position of the target object area actually designated in the target image, from the work result for the target image; and a process of verifying the work result for the target image by comparing the target object area at the predicted position in the target image with the target object area at the actual designated position. Effect of the Invention
[0014] According to the present disclosure, first reference information regarding the position of a target object region in each image included in a first image sequence is obtained. A predicted position of a target object region in a target image adjacent to the first image sequence is calculated based on reference information including the first reference information. Then, the result of the operation on the target image is verified by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position. This allows the result of the operation on each image included in the image sequence to be easily and appropriately verified. [Brief description of the drawings]
[0015] [Figure 1] FIG. 13 is a conceptual diagram for explaining the results of annotation work on an image sequence. [Diagram 2] FIG. 13 is a conceptual diagram illustrating an example of a work result including an abnormal annotation. [Diagram 3] FIG. 1 is a conceptual diagram for explaining an overview of an annotation verification method according to an embodiment of the present invention. [Figure 4] 1 is a conceptual diagram showing an example of a predicted position calculated in the annotation verification method according to the present embodiment. [Diagram 5]FIG. 1 is a diagram illustrating an example of a regression model configured by a machine learning model. [Figure 6] FIG. 1 is a conceptual diagram showing an example of two image sequences with different target objects. [Figure 7] FIG. 1 is a conceptual diagram illustrating an example of two image sequences in which the distance from the camera to the target object is different. [Figure 8] FIG. 2 is a conceptual diagram showing an example of two image sequences captured by a camera mounted on a moving object, the image sequences being different in speed from one another. [Figure 9] 11A to 11C are conceptual diagrams showing an example of a case where the result of the work on the target image is deemed to have no abnormality and a case where the result is deemed to have an abnormality in the annotation verification method according to the present embodiment. [Figure 10] 1 is a diagram illustrating an example of a configuration of an annotation verification apparatus according to an embodiment of the present invention. [Figure 11] 10 is a flowchart illustrating an example of processing executed by the annotation verification device according to the present embodiment. [Figure 12] FIG. 11 is a conceptual diagram for explaining an overview of an annotation verification method according to a second embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of the configuration of an annotation verification device according to a second embodiment. [Figure 14] 11 is a flowchart showing an example of processing executed by the annotation verification device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, an embodiment will be described with reference to the drawings.
[0017] 1. First embodiment 1-1. Overview The annotation verification method according to the present embodiment is implemented to verify the results of annotation work on an image sequence. Fig. 1 is a conceptual diagram for explaining the results 20 of annotation work on an image sequence 10.
[0018] The image sequence 10 is composed of a series of images having a predetermined order. The image sequence 10 is typically video data captured continuously by a camera. In this case, each image included in the image sequence 10 is a frame of the video data. Also in this case, each image included in the image sequence 10 has an order according to the time when it was captured. In the following description, the number of images included in the image sequence 10 is N, and the order of each image included in the image sequence 10 is represented by the numbers #1, #2, #3, . . . , #N.
[0019] The image sequence 10 may have been captured by a camera fixed at a predetermined position, such as a surveillance camera or a live camera, or may have been captured by a camera (hereinafter referred to as an "mounted camera") mounted on a moving body (vehicle, drone, etc.), such as an in-vehicle camera.
[0020] The image sequence 10 may further include data representing additional information about each image (hereinafter referred to as "additional data"). Examples of the additional data include depth information of each image, information on the capture time of each image, information on the capture location of each image, etc.
[0021] The annotation of the image sequence 10 is a task in which an annotator 1 specifies an area surrounding a target object (hereinafter, referred to as a "target object area") for each image included in the image sequence 10. However, in this embodiment, the annotator 1 does not have to be a human being. For example, the annotation of the image sequence 10 may be performed mechanically.
[0022] 1, a bounding box 21 surrounding a vehicle shown in an image is shown as an annotation result 20. That is, in FIG 1, the target object is a vehicle, and the target object region is the bounding box 21.
[0023] The target object is usually determined appropriately depending on the contents of the image sequence 10, the purpose of the annotation, and the like. For example, when annotation is performed for the purpose of creating training data for a machine learning model that performs person detection, the target object is assumed to be a person. In this embodiment, the form of the target object region is not limited to the bounding box 21. For example, the target object region may be a polygon or a segmentation. In the following, an annotation in which the target object region is the bounding box 21 will be described as an example.
[0024] The work result 20 is managed as data. The work result 20 can also be called "annotation data." The work result 20 includes at least position information of the bounding box 21 specified in each image. For example, the position information is the coordinate positions of the four corners (upper left corner, upper right corner, lower left corner, and lower right corner) and center of gravity of the bounding box 21 specified in each image. In addition, the work result 20 may include classification information of the target object (vehicle, person, airplane, etc.), attribute information of the annotator 1 (identification number, track record, etc.), etc.
[0025] The result of the operation 20 is added to the image sequence 10. The image sequence 10 with the result of the operation 20 added thereto is used as training data.
[0026] However, the work result 20 may contain abnormal annotations due to a work mistake by the annotator 1 or work by a malicious annotator 1. FIG. 2 is a conceptual diagram showing an example of the work result 20 containing abnormal annotations. In the work result 20 shown in FIG. 2, an abnormal annotation is made on the image #k+1. Using training data created by the work result 20 containing such abnormal annotations may cause a decrease in the performance of the machine learning model. Furthermore, when a system related to an autonomous vehicle or the like is realized by a machine learning model, there is a risk of decreasing the safety of the system.
[0027] In addition, in the annotation of the image sequence 10, the inclusion of abnormal annotations in the work result 20 locally impairs the consistency of annotation. For example, in the example shown in FIG. 2, three consecutive images #k, #k+1, and #k+2 continuously show a vehicle traveling on a road. Therefore, annotations for the three consecutive images #k, #k+1, and #k+2 are expected to be performed consistently to track the vehicle. However, in the example shown in FIG. 2, the position of the bounding box 21 in the image #k+1 is significantly off from the vehicle. Therefore, the work result 20 shown in FIG. 2 has an inconsistent annotation for the three consecutive images #k, #k+1, and #k+2.
[0028] When the image sequence 10 is used as training data, such localized inconsistencies in the annotations can also degrade the performance of a machine learning model.
[0029] The annotation verification method according to the present embodiment makes it possible to verify the presence or absence of abnormal annotations in the annotation work result 20 of the image sequence 10. Furthermore, the annotation verification method according to the present embodiment makes it possible to verify the local consistency of the annotations.
[0030] Hereinafter, an overview of the annotation verification method according to this embodiment will be described with reference to FIG.
[0031] In the annotation verification method according to this embodiment, the work result 20 for each image included in the image sequence 10 is verified sequentially according to a predetermined verification direction. In FIG. 3, the verification direction is ascending order. This is the direction in which images are verified from the past to the future when the image sequence 10 is video data. However, the verification direction can also be descending order. Alternatively, the verification direction can be a direction in which images are verified in ascending and descending order from the intermediate image to image #1 and image #N.
[0032] Hereinafter, an image to be verified as the result of the operation 20 will be referred to as a "target image." In Fig. 3, a case will be described in which the image #i is the target image. In other words, it is assumed that images up to #i-1 have already been verified.
[0033] In the annotation verification method according to this embodiment, first, a verified work result 22 (hereinafter referred to as a "first work result 22") for a partial image sequence 12 (hereinafter referred to as a "first image sequence 12") included in an image sequence 10 is obtained. In particular, the first image sequence 12 is an image sequence adjacent to a target image. The size of the first image sequence 12 may be determined in advance. In FIG. 3, the size of the first image sequence 12 is M. That is, in FIG. 3, the first image sequence 12 is M consecutive images from #iM to #i-1.
[0034] At least information on the position of the bounding box 21 in each image included in the first image sequence 12 (hereinafter referred to as "first reference information") is obtained from the first work result 22. For example, the first reference information is information on the coordinate positions of the four corners and the center of gravity of the bounding box 21 in each image included in the first image sequence 12.
[0035] Next, in the annotation verification method according to the present embodiment, the position of the bounding box 21 in the target image is predicted based on information including the first reference information (hereinafter simply referred to as "reference information"). In FIG. 3, the bounding box 21 at the predicted position (hereinafter referred to as "predicted position") is indicated by a dotted line. By calculating the predicted position based on at least the first reference information, it is possible to calculate a predicted position that is appropriate for sequential changes in the position of the bounding box 21 in each image included in the first image sequence 12. In particular, the predicted position calculated based on the reference information is expected to be a position that maintains the consistency of the annotation for the first image sequence 12.
[0036] Fig. 4 is a conceptual diagram showing an example of a predicted position calculated in the annotation verification method according to the present embodiment. In Fig. 4, the first image sequence 12 is three consecutive images #i-3, #i-2, and #i-1. Also in Fig. 4, bounding boxes 21a, 21b, and 21c are shown as an example of a first operation result 22 for the first image sequence 12. That is, the first reference information is information about the positions of the bounding boxes 21a, 21b, and 21c.
[0037] In FIG. 4, an example of a bounding box 21 (dotted line) of a predicted position is shown. The predicted position shown in FIG. 4 is an extrapolation of the positions of the bounding boxes 21a, 21b, and 21c. It can be seen that the bounding box 21 of the predicted position is a reasonable position with respect to the change in the positions of the bounding boxes 21a, 21b, and 21c. If the size of the first image sequence 12 is increased, it is expected that the predicted position can be calculated with higher accuracy.
[0038] The calculation of such a predicted position can be performed using a regression model with the reference information as an explanatory variable. In this case, the explanatory variables can be expressed by M variables corresponding to each image included in the first image sequence 12. For example, when the reference information is the first reference information, the explanatory variables can be expressed by the following M vectors wk (k=1, 2, . . . , M). Here, w1, w2, . . . , wM correspond to images #iM, #i-M+1, . . . , #i-1 in the first image sequence 12, respectively. Ptl, Ptr, Pbl, and Pbr, which are elements of the vector wk, are the coordinate positions of the upper left corner, upper right corner, lower left corner, and lower right corner of the bounding box 21 in the corresponding image, respectively. Also, CP is the coordinate position of the center of gravity of the bounding box 21 in the corresponding image.
[0039]
number
[0040] The regression model can be configured, for example, by a trained machine learning model. In this case, the machine learning model can employ a recurrent neural network (RNN) that inputs the above M vectors wk as time-series data. In this case, the vectors wk can also be called "feature vectors."
[0041] FIG. 5 is a diagram showing an example of the regression model 122 configured by a machine learning model. The regression model 122 shown in FIG. 5 is configured by a machine learning model that employs an RNN. That is, each vector wk is input to a corresponding layer, and each layer except for the layer #M at the final stage outputs a hidden state to the next layer. Each layer is configured by, for example, an LSTM (Long Short Term Memory). Then, the output y of the layer #M at the final stage becomes the output of the regression model 122. The output y is, for example, a vector whose elements are the coordinate positions of the four corners and the center of gravity of the bounding box 21 of the predicted position.
[0042] In the annotation verification method according to this embodiment, in order to improve the accuracy of the predicted position, the reference information may further include the information described below.
[0043] One of the pieces of reference information is information on the classification of the target object (hereinafter, referred to as "second reference information"). For example, the second reference information is information that specifies the classification of the target object by vehicle, person, airplane, etc. The second reference information can be acquired, for example, from the work result 20. In the image sequence 10, it is assumed that the tendency of the change in the position of the target object differs depending on the classification of the target object. FIG. 6 is a conceptual diagram showing an example of two image sequences 10 having different target objects. FIG. 6 (A) shows a case where the target object is a person. FIG. 6 (B) shows a case where the target object is a vehicle. As shown in FIG. 6 (A), when the target object is a person, the position of the person reflected in each image can change freely to some extent depending on the person's walking. In addition, it is assumed that the range in which the person is reflected in each image changes according to the person's movement. On the other hand, as shown in FIG. 6 (B), when the target object is a vehicle, the position and range of the vehicle reflected in each image tend to change linearly depending on the vehicle's running. In this way, it is assumed that the tendency of the change in the position of the target object reflected in each image differs depending on the classification of the target object. Therefore, by including the second reference information in the reference information, the tendency of the position change according to the classification of the target object can be taken into consideration in the calculation of the predicted position based on the reference information, which in turn can improve the accuracy of the predicted position.
[0044] Another piece of reference information is information on the distance from the camera to the target object in each image (hereinafter referred to as "third reference information"). For example, the third reference information is depth information of the target object in each image. The third reference information can be acquired, for example, from the position of the bounding box 21 specified in each image and the depth information of each image. The depth information of each image is given, for example, as additional data included in the image sequence 10. Alternatively, the depth information of each image may be calculated from the additional data included in the image sequence 10. In the image sequence 10, it is assumed that the degree of change in the position of the target object differs depending on the distance from the camera to the target object. FIG. 7 is a conceptual diagram showing an example of two image sequences 10 in which the distance from the camera to the target object (person) is different. (A) of FIG. 7 is a case in which the distance from the camera to the target object is small, that is, the target object is close. (B) of FIG. 7 is a case in which the distance from the camera to the target object is large, that is, the target object is far away. As shown in (A) of FIG. 7, when the target object is close, the change in the position of the target object shown in each image is large. On the other hand, as shown in (B) of FIG. 7, when the target object is far away, the change in the position of the target object shown in each image is small. In this way, it is expected that the degree of change in the position of the target object varies depending on the distance from the camera to the target object. Therefore, by including the third reference information in the reference information, the degree of change in the position depending on the distance from the camera to the target object can be taken into consideration in calculating the predicted position based on the reference information. In turn, the accuracy of the predicted position can be improved.
[0045] The other is information (hereinafter referred to as "fourth reference information") on the speed of the moving body when each image is captured in the image sequence 10 when the image sequence 10 is captured by a camera mounted on the moving body. For example, the fourth reference information is information on the speed of the vehicle when each image is captured by an on-board camera. The information on the speed of the vehicle is given, for example, as additional data included in the image sequence 10. It is assumed that the degree of change in the position of the target object in the image sequence 10 differs depending on the speed of the moving body mounting the camera. FIG. 8 is a conceptual diagram showing an example of two image sequences 10 with different speeds of the moving body mounting the camera. FIG. 8 (A) shows a case where the speed of the moving body mounting the camera is high. FIG. 8 (B) shows a case where the speed of the moving body mounting the camera is low. As shown in FIG. 8 (A), when the speed of the moving body is high, the change in the position of the target object (tree) in each image is large. On the other hand, as shown in FIG. 8 (B), when the speed of the moving body is low, the change in the position of the target object in each image is small. In this way, it is assumed that the degree of change in the position of the target object varies depending on the speed of the moving body on which the camera is mounted. Therefore, by including the fourth reference information in the reference information, the degree of change in the position depending on the speed of the moving body on which the camera is mounted can be taken into consideration in the calculation of the predicted position based on the reference information. As a result, the accuracy of the predicted position can be improved.
[0046] When the reference information includes the above information, the above information may be included in the elements of vector wk serving as explanatory variables when calculating a predicted position using regression model 122. For example, when the reference information further includes depth information (third reference information) of a target object shown in each image, vector wk may further include, as an element, the depth information of the target object shown in the corresponding image.
[0047] As described above, in the annotation verification method according to this embodiment, the predicted position of the bounding box 21 in the target image is calculated based on the reference information.
[0048] Furthermore, in the annotation verification method according to this embodiment, the position of the bounding box 21 that is actually specified in the target image (hereinafter, referred to as the “actual specified position”) is acquired from the operation result 20.
[0049] In the annotation verification method according to this embodiment, the work result 20 on the target image is verified by comparing the bounding box 21 of the predicted position in the target image with the bounding box 21 of the actual specified position. In other words, the work result 20 on the target image is verified depending on the degree to which the bounding box 21 of the predicted position in the target image matches the bounding box 21 of the actual specified position. In particular, it is verified whether or not there is an abnormality in the work result 20 on the target image.
[0050] Fig. 9 is a conceptual diagram showing an example of a case where the work result 20 for the target image is determined to have no abnormality and a case where the work result 20 for the target image is determined to have an abnormality in the annotation verification method according to this embodiment. As shown in Fig. 9, according to the annotation verification method according to this embodiment, when the bounding box 21 (dotted line) of the predicted position and the bounding box 21 (solid line) of the actual specified position almost match, it is determined that the work result 20 for the target image has no abnormality. On the other hand, when the bounding box 21 (dotted line) of the predicted position and the bounding box 21 (solid line) of the actual specified position deviate from each other, it is determined that the work result 20 for the target image has an abnormality.
[0051] As described above, according to the annotation verification method of this embodiment, it is possible to calculate a predicted position that is appropriate for sequential changes in the position of the bounding box 21 in each image included in the first image sequence 12. Therefore, by comparing the bounding box 21 of the predicted position with the bounding box 21 of the actual specified position, it is possible to properly verify whether or not there is an abnormality in the work result 20 for the target image. Furthermore, it is expected that the calculated predicted position will be a position that maintains the consistency of the annotation for the first image sequence 12. Therefore, it is possible to verify whether or not there is an abnormality in the work result 20 for the target image from the viewpoint of local consistency of the annotation as well.
[0052] The degree of overlap between the bounding box 21 of the predicted position and the bounding box 21 of the actual specified position can be used as an index for the degree of agreement between them. In this case, when the degree of overlap is smaller than a predetermined threshold, it is determined that there is an abnormality in the work result 20 for the target image. The degree of overlap is expressed, for example, by IoU (Intersection over Union).
[0053] By performing a threshold determination using the degree of overlap as an index in this manner, verification by comparison between the bounding box 21 of the predicted position and the bounding box 21 of the actually specified position can be easily realized.
[0054] When the verification of the work result 20 for the target image is completed, the above steps are repeated by shifting the target image along the verification direction. In the case shown in FIG. 3, when the verification of the image #i as the target image is completed, the image #i+1 becomes the next target image. At this time, the first image sequence 12 is M consecutive images from #i-M+1 to #i. However, the first image sequence 12 may be selected excluding images determined to have an abnormality in the work result 20. For example, when it is determined that the work result 20 for the image #i has an abnormality, the first image sequence 12 when the image #i+1 is the target image may be M consecutive images from #iM to #i-1. By selecting the first image sequence 12 in this manner, it is possible to suppress a decrease in the accuracy of the predicted position.
[0055] In this manner, in the annotation verification method according to the present embodiment, the work result 20 for each image included in the image sequence 10 is verified sequentially. The verification result by the annotation verification method according to the present embodiment is generated so as to provide a determination result as to whether or not there is an abnormality in the work result 20 for each image. For example, as shown in the table below, the verification result is data in which an abnormality determination flag that becomes TRUE when it is determined that there is an abnormality in the work result 20 is managed for each image. In the example of the verification result shown below, it is determined that there is an abnormality in the work result 20 for image #2. The verification result may be added as data to the work result 20 or the image sequence 10. Alternatively, the verification result may be provided to the user as data.
[0056] [Table 1]
[0057] As described above, according to the annotation verification method of this embodiment, first reference information is obtained regarding the position of the bounding box 21 in each image included in the first image sequence 12 adjacent to the target image. Furthermore, a predicted position of the bounding box 21 in the target image is calculated based on reference information including the first reference information. Then, the bounding box 21 at the predicted position in the target image is compared with the bounding box 21 at the actual specified position to verify the work result 20 on the target image. This makes it possible to properly verify the work result 20 on the target image.
[0058] Furthermore, by shifting the target image along a predetermined verification direction, the work result 20 for each image included in the image sequence 10 can be verified sequentially. Therefore, by executing the annotation verification method according to this embodiment by a computer, it is possible to easily verify the work result 20 for each image included in the image sequence 10.
[0059] Moreover, according to the annotation verification method of this embodiment, the first reference information can be acquired from the work result 20. Therefore, the annotation verification method of this embodiment can perform verification if at least the work result 20 is given. Moreover, according to the annotation verification method of this embodiment, the work result 20 can be verified without the need for comparison with the work result 20 of another annotator 1, etc.
[0060] According to the annotation verification method of this embodiment, it is assumed that the work result 20 of each image included in the first image sequence 12 has already been verified. Therefore, the work result 20 of each image included in the first image sequence 12 for the initial target image may have already been verified by other means. For example, in the case shown in FIG. 3, the work results 20 of M images from #1 to #M included in the first image sequence 12 for the image #M+1 that is the initial target image may have already been verified by other means. The verification by other means may be manual verification. Even in this case, according to the annotation verification method of this embodiment, it is possible to easily verify the work result 20 for each image included in the image sequence 10 by simply manually verifying the work result 20 for the images of the size of the first image sequence 12.
[0061] 1-2.Annotation verification device The annotation verification method according to the present embodiment is realized by processing executed by a computer. An annotation verification device for implementing the annotation verification method according to the present embodiment will be described below.
[0062] 10 is a block diagram showing an example of the configuration of an annotation verification device 100 according to this embodiment. The annotation verification device 100 is configured to be able to access an image database D10. For example, the annotation verification device 100 is connected to a server that stores the image database D10 via the Internet. Alternatively, the annotation verification device 100 can be configured to store the image database D10 in a storage device 120.
[0063] The image database D10 manages an image sequence 10 captured by the camera 200 or the on-board camera 310.
[0064] An image sequence 10 stored in an image database D10 includes image data 11, additional data 13, and a processing result 20.
[0065] The image data 11 is data representing each image included in the image sequence 10. The additional data 13 is data representing additional information relating to each image included in the image sequence 10.
[0066] The camera 200 may take various forms. Examples of the camera 200 include a video camera, a surveillance camera, a live camera, and the like. The image sequence 10 captured by the camera 200 is uploaded to the image database D10 as appropriate. The camera 200 may be configured to upload captured images to the image database D10 in sequence.
[0067] The on-board camera 310 is mounted on the moving body 300. Examples of the moving body 300 include a vehicle and a drone. The moving body 300 includes a sensor 320 that detects the state of the moving body 300 and the surrounding environment. Examples of the sensor 320 include a LIDAR (Light Detection And Ranging), an IMU (Inertial Measurement Unit), a speed sensor, a GPS receiver, and the like. Examples of information detected by the sensor 320 include the speed of the moving body 300, the position of the moving body 300, and the distance between the moving body 300 and surrounding objects. The moving body 300 uploads the image sequence 10 captured by the on-board camera 310 to the image database D10 as appropriate. Furthermore, the moving body 300 may be configured to upload the information detected by the sensor 320 to the image database D10 as additional data 13 of the image sequence 10. For example, the moving object 300 uploads the speed of the moving object 300 when each image included in the image sequence 10 was captured to the image database D10 as additional data 13.
[0068] The work terminal 400 is a device for annotating an image sequence 10 managed by an image database D10. The work terminal 400 reads out the image sequence 10 from the image database D10. Then, an annotator 1 operates the work terminal 400 to annotate the read out image sequence 10. The work terminal 400 uploads the annotation work result 20 to the image database D10. do.
[0069] The user interface 500 provides an interface for a user of the annotation verification device 100. For example, the user interface 500 is configured with input devices such as a keyboard, a mouse, a touch panel, etc., and output devices such as a display, a speaker, etc.
[0070] The annotation verification device 100 according to this embodiment reads an image sequence 10 managed by an image database D10, and executes a process of verifying the work result 20 for the read image sequence 10 by the annotation verification method described above. The image sequence 10 to be read is determined by a user via, for example, a user interface 500. Alternatively, the annotation verification device 100 may be configured to refer to the image database D10 and sequentially read the image sequences 10 to which the work result 20 has been added. The verification result by the annotation verification device 100 is, for example, transmitted to the image database D10 and added to the corresponding image sequence 10. Alternatively, the verification result is provided to the user via the user interface 500.
[0071] The annotation verification device 100 according to this embodiment is a computer including one or more processors 110 (hereinafter simply referred to as "processor 110") and one or more storage devices 120 (hereinafter simply referred to as "storage device 120"). The processor 110 executes various processes. The processor 110 may be configured, for example, as a central processing unit (CPU) including an arithmetic unit, a register, and the like. The storage device 120 is connected to the processor 110 and stores various information required for the processor 110 to execute the processes. The storage device 120 may be configured, for example, as a recording medium such as a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a solid state drive (SSD).
[0072] The storage device 120 stores a computer program 121 and a regression model 122 .
[0073] The computer program 121 is stored in a computer-readable recording medium. The computer program 121 includes a plurality of instructions configured to cause the processor 110 to execute various processes. The processor 110 operates according to the plurality of instructions, thereby realizing the execution of various processes by the processor 110.
[0074] 1-3. Processing The process executed by the annotation verification device 100, more specifically, the process executed by the processor 110, will be described below.
[0075] Fig. 11 is a flowchart showing an example of processing executed by the processor 110. The processing shown in Fig. 11 starts, for example, when a request to start execution is received and the target image sequence 10 is read.
[0076] In step S100, the processor 110 executes an initialization process. In the initialization process, the processor 110 acquires various information such as the additional data 13 and the work result 20, determines the initial target image, checks the verification direction, and so on.
[0077] Next, in step S110, the processor 110 selects a first image sequence 12 adjacent to the target image, and obtains a verified first operation result 22 for the first image sequence 12.
[0078] Next, in step S120, processor 110 acquires reference information. At least in step S120, processor 110 acquires first reference information from first work result 22 acquired in step S110. Further in step S120, processor 110 may acquire second reference information, third reference information, or fourth reference information from additional data 13 or first work result 22.
[0079] Next, in step S130, the processor 110 calculates a predicted position of the bounding box 21 in the target image based on the reference information obtained in step S120.
[0080] Next, in step S140, the processor 110 obtains the actual specified position of the bounding box 21 in the target image.
[0081] Next, in step S150, the processor 110 calculates the degree of overlap between the bounding box 21 of the predicted position and the bounding box 21 of the actual specified position.
[0082] Next, in step S160, the processor 110 determines whether the degree of overlap calculated in step S150 is smaller than a predetermined threshold value. The predetermined threshold value may be appropriately set depending on the environment to which the present embodiment is applied.
[0083] If the degree of overlap is smaller than the predetermined threshold (step S160; Yes), processor 110 determines that there is an abnormality in work result 20 for the target image (step S170). For example, processor 110 sets the abnormality determination flag corresponding to the target image to TRUE. Then, the process proceeds to step S180.
[0084] If the degree of overlap is equal to or greater than the predetermined threshold value (step S160; No), it is determined that there is no abnormality in the work result 20 for the target image, and the process proceeds to step S180.
[0085] In step S180, the processor 110 determines whether or not to end the verification. For example, the processor 110 determines to end the verification on the condition that the verification of the operation result 20 for all images included in the image sequence 10 has been completed.
[0086] When it is determined that the verification should be ended (Step S180; Yes), the process ends.
[0087] When it is determined that the verification should not be ended (step S180; No), processor 110 shifts the target image (step S190). After that, processor 110 repeats the process from step S110 again.
[0088] As described above, the functions of the annotation verification device 100 according to this embodiment are realized by the processor 110 executing the processes. The annotation verification method according to this embodiment is implemented by the processor 110 executing the processes in this manner. The annotation verification program according to this embodiment is realized by the computer program 121 that causes the processor 110 to execute the processes in this manner.
[0089] 2. Second embodiment The second embodiment will be described below. In the following description, parts that overlap with the above description will be omitted as appropriate.
[0090] 2-1. Overview In the annotation of an image sequence 10, multiple annotators 1 may annotate the same image sequence 10. In this case, the annotation work result 20 of the image sequence 10 includes the work results 20 of each of the multiple annotators 1. It is usually assumed that the work results 20 of each of the multiple annotators 1 differ from each other depending on the skill, tendency, etc. of each annotator 1.
[0091] The annotation verification method according to the second embodiment is applied to a case where a plurality of annotators 1 perform annotation on the same image sequence 10. Below, an overview of the annotation verification method according to the second embodiment will be described with reference to Fig. 12. Fig. 12 conceptually illustrates work results 20 of three annotators 1, #1, #2, and #3, on the same image sequence 10.
[0092] In the annotation verification method according to the second embodiment, the reliability of each annotation by a plurality of annotators 1 is managed. The reliability of each annotator 1 is a value that estimates the correctness of the annotation by each annotator 1. In FIG. 12, the reliability of annotators 1 #1, #2, and #3 is given as 90%, 80%, and 60%, respectively. That is, the annotation by annotator 1 #1 is estimated to be 90% correct, and the annotation by annotator 1 #3 is estimated to be 60% correct. Note that the expression of reliability is not limited to percentage. For example, reliability may be expressed by a decimal.
[0093] The reliability of each annotator 1 can be given by updating based on the verification results by the annotation verification method, as described later. In this case, the initial value of the reliability of each annotator 1 may be suitably given. Alternatively, the reliability of each annotator 1 may be given using the number of annotations performed, years of experience, etc. as indicators.
[0094] The annotation verification method according to the second embodiment sequentially verifies the work results 20 of each of the multiple annotators 1 for each image included in an image sequence 10 according to a predetermined verification direction. In Fig. 12, the verification direction is ascending order. In Fig. 12, a case will be described in which the image #i is the target image.
[0095] In the annotation verification method according to the second embodiment, first, a first work result 22 is obtained. In the second embodiment, the first work result 22 includes verified work results 20 of each of the multiple annotators 1 on the first image sequence 12.
[0096] Next, in the annotation verification method according to the second embodiment, the reliability of each of the multiple annotators 1 is acquired. Then, in the annotation verification method according to the second embodiment, first reference information is acquired from the first work result 22 and the reliability of each of the multiple annotators 1.
[0097] In the second embodiment, the first reference information is a weighted average position 23 of the positions of bounding boxes 21 designated by the multiple annotators 1 for each image included in the first image sequence 12. In particular, the weights used in calculating the weighted average position 23 are the reliability of each of the multiple annotators 1.
[0098] In FIG. 12, the weighted average position 23 is indicated by a vector wk (k=1, 2, . . . , M). For example, w1 is a vector whose elements are the weighted average of the coordinate positions of the four corners and the center of gravity of the bounding box 21 designated by the annotators 1 of #1, #2, and #3, respectively, for the image of #iM. For example, the vector wk (k=1, 2, . . . , M) can be expressed by the following formula. Here, α1, α2, and α3 are the reliability of the annotators 1 of #1, #2, and #3, respectively. Also, v1k, v2k, and v3k are vectors whose elements are the coordinate positions of the four corners and the center of gravity of the bounding box 21 designated by the annotators of #1, #2, and #3, respectively, for the corresponding image.
[0099]
number
[0100] Next, in the annotation verification method according to the second embodiment, a predicted position of the bounding box 21 in the target image is calculated based on reference information including the first reference information. The calculation of the predicted position may be the same as that in the first embodiment. That is, the predicted position may be calculated using a regression model 122 that uses reference information including the weighted average position 23 as the first reference information as an explanatory variable. The reference information may also include the second reference information, the third reference information, or the fourth reference information.
[0101] Next, in the annotation verification method according to the second embodiment, from the work results 20 of each of the multiple annotators 1, the actual specified position of the bounding box 21 in the target image is acquired for each of the multiple annotators 1.
[0102] In the annotation verification method according to the second embodiment, the bounding box 21 of the predicted position in the target image is compared with the bounding box 21 of the actual specified position for each of the multiple annotators 1, thereby verifying the work result 20 for the target image. The bounding box 21 of the predicted position is common to the verification of each of the multiple annotators 1. The method of comparison in the verification of each of the multiple annotators 1 may be the same as that of the first embodiment. In particular, the comparison in the verification of each of the multiple annotators 1 may be performed using the degree of overlap between the bounding box 21 of the predicted position in the target image and the bounding box 21 of the actual specified position as an index.
[0103] When the verification of the work result 20 for the target image is completed for each of the multiple annotators 1, the target image is shifted along the verification direction and the above steps are repeated.
[0104] In this manner, in the annotation verification method according to the second embodiment, the work results 20 for each image included in the image sequence 10 are verified sequentially for each of the multiple annotators 1.
[0105] In the annotation verification method according to the second embodiment, the reliability of each of the multiple annotators 1 is updated based on the verification result. Typically, the reliability of an annotator 1 whose verification result determines that the work result 20 has an abnormality is decreased. Furthermore, the reliability of an annotator 1 whose verification result determines that the work result 20 has no abnormality (the work result 20 is normal) may be increased. The reliability may be updated each time the verification of the work result 20 for the target image is completed, or when the verification of the work result 20 for all images is completed.
[0106] When updating the reliability of the annotator 1, the change in the reliability may be adjusted according to the difficulty of the annotation. The difficulty of the annotation is a value representing the difficulty of the annotator 1 to correctly perform the annotation.
[0107] The difficulty of annotation can be determined based on the size of the target object. For example, if the target object is small, such as a pebble, the difficulty of annotation is set high.
[0108] The degree of difficulty of annotation can be determined based on the degree of freedom of the target object's movement. For example, if the target object is a flying object such as a bird or an airplane, the degree of difficulty of annotation is set high.
[0109] The difficulty of annotation can be determined by the ease of distinguishing the target object in the image. For example, when the classification of the target object is confusing with other classifications that are not the target object (e.g., vans and SUVs, sidewalks and bicycle paths, etc.), the difficulty of annotation is set high.
[0110] The difficulty of annotation can be determined based on the appearance of the image. For example, the lower the brightness or contrast of an image, the higher the difficulty of annotation is set.
[0111] The degree of difficulty of annotation can be an index of the degree of difference in the work results 20 between the multiple annotators 1. In this case, Krippendorff's alpha can be used as the degree of difference in the work results 20 between the multiple annotators 1. For example, the larger the alpha coefficient is, that is, the smaller the difference in the work results 20 between the multiple annotators 1, the higher the difficulty of annotation is set.
[0112] The degree of difficulty of annotation may be calculated by a trained machine learning model that uses an image as an input. In this case, the machine learning model is configured by, for example, a Convolutional Neural Network (CNN).
[0113] The degree of difficulty of annotation may be set for the image sequence 10, or may be set individually for each image included in the image sequence 10.
[0114] The amount of change in reliability depending on the difficulty of annotation is adjusted, for example, as shown in the following table. However, the amount of change in reliability may be adjusted in more stages or continuously depending on the difficulty of annotation. By adjusting the amount of change in reliability depending on the difficulty of annotation in this way, the reliability of each of the multiple annotators 1 can be managed more accurately.
[0115] [Table 2]
[0116] As described above, according to the annotation verification method of the second embodiment, the predicted position of the bounding box 21 in the target image is calculated based on the reference information including the weighted average position 23 as the first reference information. Then, for each of the multiple annotators 1, the bounding box 21 of the predicted position in the target image is compared with the bounding box 21 of the actual specified position, thereby verifying the work result 20 on the target image. In addition, by shifting the target image along a predetermined verification direction, the work result 20 on each image included in the image sequence 10 is sequentially verified for each of the multiple annotators 1. This makes it possible to verify the work result 20 while maintaining the efficiency of verification when multiple annotators 1 are annotating the same image sequence 10. In particular, since the weighted average position 23 is calculated using the reliability of each of the multiple annotators 1 as a weight, the work result 20 of each of the multiple annotators 1 can be verified taking into account the skill of each annotator 1.
[0117] Furthermore, according to the annotation verification method according to the second embodiment, the reliability of each of the multiple annotators 1 is updated based on the verification result. This makes it possible to dynamically manage the reliability of each of the multiple annotators 1.
[0118] 2-2.Annotation verification device The annotation verification device 100 for implementing the annotation verification method according to the second embodiment will be described below.
[0119] 13 is a block diagram showing an example of the configuration of an annotation verification device 100 according to the second embodiment. In the annotation verification device 100 according to the second embodiment, reliability information 123 is stored in a storage device 120, unlike the first embodiment.
[0120] The reliability information 123 manages the reliability of each annotator 1. For example, the reliability information 123 is data in which identification information of each annotator 1 is associated with the reliability of each annotator 1. The reliability information 123 is updated and managed by processing executed by the processor 110.
[0121] 2-3. Processing Hereinafter, the processing executed by the annotation verification device 100 according to the second embodiment, more specifically, the processing executed by the processor 110 will be described.
[0122] The process executed by the processor 110 according to the second embodiment may be the same as the process shown in Fig. 11. However, the process relating to steps S140 to S170 is executed for each of the multiple annotators 1. In addition, in step S120, the processor 110 according to the second embodiment executes at least the following process shown in Fig. 14.
[0123] In step S210, the processor 110 acquires the reliability of each of the multiple annotators 1. For example, the processor 110 acquires the reliability of each annotator 1 by referring to the reliability information 123 using the identification information of each annotator 1.
[0124] Next, in step S220, the processor 110 calculates a weighted average position 23 as first reference information based on the first work result 22 acquired in step S110 and the reliability of each of the multiple annotators 1 acquired in step S210. [Explanation of symbols]
[0125] 1 Annotator 10 Image Sequences 11 Image data 12 First Image Sequence 13 Additional Data 20 Work Results 21 Bounding Box 22 First Work Result 23 Weighted average position 100 Annotation Verification Device 110 Processor 120 Storage device 121 Computer Programs 122 Regression Models 123 Reliability Information D10 Image Database
Claims
1. An annotation verification method for verifying an annotation work result for an image sequence, comprising: The annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence; The annotation verification method is executed by a computer, obtaining a first operation result, the first operation result being a verified operation result for a first image sequence included in the image sequences; obtaining first reference information relating to a position of the object region in each image of the first image sequence from the first operation result; calculating a predicted position of the target object region in a target image, the target image being an image adjacent to the first image sequence, based on reference information including the first reference information; acquiring an actual designated position, which is a position of the target object region actually designated in the target image, from the operation result for the target image; verifying the operation result for the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position; Includes Annotation validation methods.
2. 2. The method of claim 1, further comprising: Verifying the operation result on the target image includes: calculating an overlapping degree of the target object region at the predicted position and the target object region at the actual specified position; determining that there is an abnormality in the result of the operation on the target image when the degree of overlap is smaller than a predetermined threshold value; Includes Annotation validation methods.
3. 2. The method of claim 1, further comprising: obtaining second reference information related to a classification of the target object; The reference information further includes the second reference information. Annotation validation methods.
4. 2. The method of claim 1, further comprising: the image sequence is captured by a camera; The annotation verification method further comprises obtaining third reference information related to a distance from the camera to the target object in each image in the first image sequence; The reference information further includes the third reference information. Annotation validation methods.
5. 2. The method of claim 1, further comprising: the image sequence is captured by a camera mounted on a moving object; The annotation verification method further includes obtaining fourth reference information on a speed of the moving object when each image in the first image sequence was captured; The reference information further includes the fourth reference information. Annotation validation methods.
6. 6. The annotation verification method according to claim 1, further comprising: the annotation is performed by a plurality of annotators; Obtaining the first reference information includes: obtaining a confidence level for the annotation of each of the plurality of annotators; calculating a weighted average position of positions of the target object regions designated by each of the plurality of annotators for each image included in the first image sequence, using the confidence as a weight; Including, The first reference information is the weighted average position for each image in the first image sequence. Annotation validation methods.
7. 7. A method for verifying annotations according to claim 6, comprising: and updating the reliability based on a result of verifying the operation result for the target image. Annotation validation methods.
8. 8. A method for verifying annotations according to claim 7, comprising: Updating the confidence level includes: obtaining a difficulty level of the annotation; adjusting an amount of change in the reliability at the time of updating according to the degree of difficulty; Includes Annotation validation methods.
9. An annotation verification device for verifying an annotation work result for an image sequence, comprising: One or more processors; The annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence; The one or more processors: obtaining a first operation result, the first operation result being a verified operation result for a first image sequence included in the image sequences; obtaining first reference information relating to a position of the object region in each image of the first image sequence from the first operation result; calculating a predicted position of the target object region in a target image, the target image being an image adjacent to the first image sequence, based on reference information including the first reference information; A process of acquiring an actual designated position, which is a position of the target object region actually designated in the target image, from the operation result for the target image; verifying the result of the operation on the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position; It is configured to run Annotation verification device.
10. An annotation verification program that causes a computer to execute a process for verifying an annotation work result for an image sequence, comprising: The annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence; The annotation verification program includes: obtaining a first operation result, the first operation result being a verified operation result for a first image sequence included in the image sequences; obtaining first reference information relating to a position of the object region in each image of the first image sequence from the first operation result; calculating a predicted position of the target object region in a target image, the target image being an image adjacent to the first image sequence, based on reference information including the first reference information; A process of acquiring an actual designated position, which is a position of the target object region actually designated in the target image, from the operation result for the target image; verifying the result of the operation on the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position; The computer is configured to execute An annotation verification program comprising:
Citation Information
Patent Citations
Discrimination device and machine learning method
JP2020030692A
Image analysis device, image analysis method, and image analysis program
JP2020181255A
Training data generation system, estimation system, training data generation method, estimation method, and program
JP2022029257A
Information processing device and information processing method
JP2022068012A
Information processing device, information processing method, and information processing program
JP2023022262A
Cited By
Information processing system, its control method, and program
JP2026115992A
