Annotation verification method, annotation verification device, and annotation verification program

Abstract

To provide an annotation verification method for enabling easy and appropriate verification of an annotation work result for each image included in an image sequence.SOLUTION: The annotation verification method is executed by a computer, and includes: acquiring a first work result for a first image sequence included in an image sequence; acquiring first reference information regarding a position of a target object region in each image included in the first image sequence from the first work result; calculating a predicted position of the target object region in a target image on the basis of reference information including the first reference information; acquiring an actually specified position of the target object region in the target image from the work result for the target image; and verifying the work result for the target image by comparing the target object region at the actually specified position and the target object region at the predicted position in the target image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to techniques for verifying annotation work results for image sequences. [Background technology]

[0002] Machine learning models that perform tasks such as object detection using images as input are trained using annotated images as training data. It is known that the performance of such machine learning models is highly dependent on the quality of the annotations in the training data. For this reason, technologies to ensure the quality of annotations in the training data have been developed.

[0003] For example, Patent Document 1 discloses a technology that aims to suppress a decrease in object detection accuracy when object detection training is performed using a dataset that includes images that have not been annotated as desired. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-043364 Summary of the Invention [Problem to be solved by the invention]

[0005] The quality of annotation depends on the work of the annotator. Due to annotator mistakes or the work of malicious annotators, the annotation work results may contain abnormal annotations.

[0006] In particular, when annotating image sequences, abnormal annotations in some images can significantly reduce the quality of the annotation. However, image sequences generally consist of many images, and manually verifying the results of the work on each image in each image sequence requires a great deal of effort.

[0007] One object of the present disclosure is to provide a technology that enables easy and appropriate verification of the results of annotations on each image included in an image sequence. [Means for solving the problem]

[0008] A first aspect of the present disclosure relates to an annotation verification method for verifying the results of annotation work on an image sequence, where annotation is work in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence.

[0009] An annotation verification method according to a first aspect is executed by a computer and includes: obtaining a first work result, which is a verified work result for a first image sequence included in an image sequence; obtaining first reference information regarding the position of a target object region in each image included in the first image sequence from the first work result; calculating a predicted position of the target object region in a target image, which is an image adjacent to the first image sequence, based on reference information including the first reference information; obtaining an actual designated position, which is the position of the target object region actually designated in the target image, from the work result for the target image; and verifying the work result for the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual designated position.

[0010] A second aspect of the present disclosure relates to an annotation verification device that verifies the results of annotation work on an image sequence, where annotation is work in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence.

[0011] An annotation verification device according to a second aspect includes one or more processors configured to execute the following processes: a process of acquiring a first work result, which is a verified work result for a first image sequence included in an image sequence; a process of acquiring first reference information relating to the position of a target object area in each image included in the first image sequence from the first work result; a process of calculating a predicted position of the target object area in a target image, which is an image adjacent to the first image sequence, based on reference information including the first reference information; a process of acquiring an actual designated position, which is the position of the target object area actually designated in the target image, from the work result for the target image; and a process of verifying the work result for the target image by comparing the target object area at the predicted position in the target image with the target object area at the actual designated position.

[0012] A third aspect of the present disclosure relates to an annotation verification program that causes a computer to execute a process for verifying the results of annotation work on an image sequence, where annotation is work in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence.

[0013] An annotation verification program according to a third aspect is configured to cause a computer to execute the following processes: a process of acquiring a first work result, which is a verified work result for a first image sequence included in the image sequence; a process of acquiring first reference information relating to the position of a target object area in each image included in the first image sequence from the first work result; a process of calculating a predicted position of the target object area in a target image, which is an image adjacent to the first image sequence, based on reference information including the first reference information; a process of acquiring an actual designated position, which is the position of the target object area actually designated in the target image, from the work result for the target image; and a process of verifying the work result for the target image by comparing the target object area at the predicted position in the target image with the target object area at the actual designated position. [Effects of the Invention]

[0014] According to the present disclosure, first reference information relating to the position of a target object region in each image included in a first image sequence is acquired. Furthermore, a predicted position of the target object region in a target image adjacent to the first image sequence is calculated based on reference information including the first reference information. Then, the target object region at the predicted position in the target image is compared with the target object region at the actual specified position to verify the work result on the target image. This allows the work result on each image included in the image sequence to be easily and appropriately verified. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 10 is a conceptual diagram for explaining the results of annotation work on an image sequence. [Figure 2] FIG. 10 is a conceptual diagram illustrating an example of a work result including an abnormal annotation. [Figure 3] FIG. 1 is a conceptual diagram for explaining an overview of an annotation verification method according to an embodiment of the present invention. [Figure 4] FIG. 10 is a conceptual diagram showing an example of a predicted position calculated in the annotation verification method according to the present embodiment. [Figure 5]FIG. 1 is a diagram illustrating an example of a regression model configured using a machine learning model. [Figure 6] FIG. 1 is a conceptual diagram illustrating an example of two image sequences with different target objects. [Figure 7] FIG. 1 is a conceptual diagram illustrating an example of two image sequences in which the distance from the camera to the target object is different. [Figure 8] FIG. 1 is a conceptual diagram showing an example of two image sequences of a moving object at different speeds when the image sequences are captured by a camera mounted on the moving object. [Figure 9] 10A and 10B are conceptual diagrams showing an example of a case where there is no abnormality in the work result on the target image and a case where there is an abnormality in the annotation verification method according to the present embodiment. [Figure 10] FIG. 1 is a diagram illustrating an example of the configuration of an annotation verification apparatus according to an embodiment of the present invention. [Figure 11] 10 is a flowchart illustrating an example of processing executed by the annotation verification device according to the present embodiment. [Figure 12] FIG. 10 is a conceptual diagram for explaining an overview of an annotation verification method according to a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of an annotation verification device according to a second embodiment. [Figure 14] 10 is a flowchart showing an example of processing executed by the annotation verification device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, an embodiment will be described with reference to the drawings.

[0017] 1. First embodiment Overview The annotation verification method according to this embodiment is implemented to verify the results of annotation work on an image sequence. Fig. 1 is a conceptual diagram for explaining the results 20 of annotation work on an image sequence 10.

[0018] An image sequence 10 is composed of a series of images in a predetermined order. The image sequence 10 is typically video data captured continuously by a camera. In this case, each image included in the image sequence 10 is a frame of the video data. In addition, in this case, each image included in the image sequence 10 is ordered according to the time at which it was captured. In the following description, the number of images included in the image sequence 10 is represented as N, and the order of each image included in the image sequence 10 is represented by the numbers #1, #2, #3, . . . , #N.

[0019] The image sequence 10 may be captured by a camera fixed at a predetermined location, such as a surveillance camera or live camera, or may be captured by a camera (hereinafter referred to as an "mounted camera") mounted on a moving object (vehicle, drone, etc.), such as an in-vehicle camera.

[0020] The image sequence 10 may further include data representing additional information about each image (hereinafter referred to as "additional data"). Examples of additional data include depth information for each image, information about the capture time of each image, information about the capture location of each image, etc.

[0021] Annotation of the image sequence 10 is a task in which an annotator 1 specifies an area surrounding a target object (hereinafter referred to as a "target object area") for each image included in the image sequence 10. However, in this embodiment, the annotator 1 does not have to be a human. For example, annotation of the image sequence 10 may be performed mechanically.

[0022] 1 shows a bounding box 21 surrounding a vehicle shown in an image as an annotation result 20. That is, in FIG. 1, the target object is a vehicle, and the target object region is the bounding box 21.

[0023] The target object is usually determined appropriately depending on the content of the image sequence 10, the purpose of the annotation, etc. For example, when annotation is performed for the purpose of creating training data for a machine learning model that performs person detection, the target object is expected to be a person. Furthermore, in this embodiment, the form of the target object region is not limited to the bounding box 21. For example, the target object region may be a polygon or a segmentation. In the following, an annotation in which the target object region is the bounding box 21 will be described as an example.

[0024] The work result 20 is managed as data. The work result 20 can also be called "annotation data." The work result 20 includes at least the position information of the bounding box 21 specified in each image. For example, the position information is the coordinate positions of the four corners (upper left corner, upper right corner, lower left corner, and lower right corner) and center of gravity of the bounding box 21 specified in each image. In addition, the work result 20 may include classification information of the target object (vehicle, person, airplane, etc.), attribute information of the annotator 1 (identification number, track record, etc.), etc.

[0025] The work result 20 is added to the image sequence 10. The image sequence 10 to which the work result 20 has been added is used as training data.

[0026] However, the work result 20 may contain abnormal annotations due to a work error by the annotator 1 or the work of a malicious annotator 1. FIG. 2 is a conceptual diagram showing an example of the work result 20 containing an abnormal annotation. In the work result 20 shown in FIG. 2, an abnormal annotation has been made on the #k+1 image. Using training data created from the work result 20 containing such an abnormal annotation can cause a decrease in the performance of the machine learning model. Furthermore, when a system related to an autonomous vehicle or the like is realized using a machine learning model, there is a risk that the safety of the system may be reduced.

[0027] Furthermore, when annotating an image sequence 10, the inclusion of abnormal annotations in the result 20 locally disrupts annotation consistency. For example, in the example shown in FIG. 2, three consecutive images #k, #k+1, and #k+2 continuously capture a vehicle traveling on a road. Therefore, annotations for the three consecutive images #k, #k+1, and #k+2 are expected to be performed consistently to track the vehicle. However, in the example shown in FIG. 2, the position of the bounding box 21 in image #k+1 is significantly off-center from the vehicle. Therefore, the result 20 shown in FIG. 2 has inconsistent annotations for the three consecutive images #k, #k+1, and #k+2.

[0028] When the image sequence 10 is used as training data, such local inconsistencies in annotations can also degrade the performance of a machine learning model.

[0029] The annotation verification method according to this embodiment makes it possible to verify whether or not there are any abnormal annotations in the annotation work result 20 for the image sequence 10. Furthermore, the annotation verification method according to this embodiment makes it possible to verify the local consistency of the annotations.

[0030] Hereinafter, an outline of the annotation verification method according to this embodiment will be described with reference to FIG.

[0031] In the annotation verification method according to this embodiment, the work result 20 for each image included in the image sequence 10 is verified sequentially in a predetermined verification direction. In FIG. 3, the verification direction is ascending order. When the image sequence 10 is video data, this is the direction in which images are verified from the past to the future. However, the verification direction can also be descending order. Alternatively, the verification direction can be a direction in which images are verified in ascending and descending order from the middle image to image #1 and image #N.

[0032] Hereinafter, the image to be verified for the work result 20 will be referred to as the "target image." In Fig. 3, a case will be described in which the image #i is the target image. In other words, it is assumed that verification has been completed up to the image #i-1.

[0033] In the annotation verification method according to this embodiment, first, a verified work result 22 (hereinafter referred to as the "first work result 22") for a partial image sequence 12 (hereinafter referred to as the "first image sequence 12") included in the image sequence 10 is obtained. In particular, the first image sequence 12 is an image sequence adjacent to the target image. The size of the first image sequence 12 may be predetermined. In FIG. 3, the size of the first image sequence 12 is M. That is, in FIG. 3, the first image sequence 12 is M consecutive images from #iM to #i-1.

[0034] At least information (hereinafter referred to as "first reference information") regarding the position of the bounding box 21 in each image included in the first image sequence 12 is obtained from the first work result 22. For example, the first reference information is information regarding the coordinate positions of the four corners and the center of gravity of the bounding box 21 in each image included in the first image sequence 12.

[0035] Next, in the annotation verification method according to this embodiment, the position of the bounding box 21 in the target image is predicted based on information including the first reference information (hereinafter simply referred to as "reference information"). In FIG. 3, the bounding box 21 at the predicted position (hereinafter referred to as "predicted position") is indicated by a dotted line. By calculating the predicted position based on at least the first reference information, it is possible to calculate a predicted position that is appropriate for sequential changes in the position of the bounding box 21 in each image included in the first image sequence 12. In particular, the predicted position calculated based on the reference information is expected to be a position that maintains consistency of the annotation for the first image sequence 12.

[0036] 4 is a conceptual diagram showing an example of a predicted position calculated in the annotation verification method according to this embodiment. In FIG. 4, the first image sequence 12 is three consecutive images #i-3, #i-2, and #i-1. In addition, FIG. 4 shows bounding boxes 21a, 21b, and 21c as an example of a first operation result 22 for the first image sequence 12. In other words, the first reference information is information about the positions of the bounding boxes 21a, 21b, and 21c.

[0037] FIG. 4 shows an example of a bounding box 21 (dotted line) of a predicted position. The predicted position shown in FIG. 4 is an extrapolation of the positions of bounding boxes 21a, 21b, and 21c. It can be seen that the bounding box 21 of the predicted position is a reasonable position relative to the changes in the positions of bounding boxes 21a, 21b, and 21c. If the size of the first image sequence 12 is increased, it is expected that the predicted position can be calculated with higher accuracy.

[0038] Such a predicted position can be calculated using a regression model that uses the reference information as explanatory variables. In this case, the explanatory variables can be represented by M variables corresponding to each image included in the first image sequence 12. For example, if the reference information is first reference information, the explanatory variables can be represented by the following M vectors wk (k=1, 2, . . . , M). Here, w1, w2, . . . , wM correspond to images #iM, #i-M+1, . . . , #i-1 in the first image sequence 12, respectively. Ptl, Ptr, Pbl, and Pbr, which are elements of the vector wk, are the coordinate positions of the upper left corner, upper right corner, lower left corner, and lower right corner of the bounding box 21 in the corresponding image, respectively. CP is the coordinate position of the center of gravity of the bounding box 21 in the corresponding image.

[0039]

number

[0040] The regression model can be configured, for example, by a trained machine learning model. In this case, the machine learning model can employ a recurrent neural network (RNN) that inputs the above M vectors wk as time-series data. In this case, the vectors wk can also be called "feature vectors."

[0041] FIG. 5 is a diagram showing an example of a regression model 122 configured as a machine learning model. The regression model 122 shown in FIG. 5 is configured as a machine learning model that employs an RNN. That is, each vector wk is input to a corresponding layer, and each layer except for the final layer #M outputs a hidden state to the next layer. Each layer is configured, for example, with an LSTM (Long Short Term Memory). The output y of the final layer #M becomes the output of the regression model 122. The output y is, for example, a vector whose elements are the coordinate positions of the four corners and the center of gravity of a bounding box 21 of the predicted position.

[0042] In the annotation verification method according to this embodiment, the reference information may further include the information described below in order to improve the accuracy of the predicted position.

[0043] One type of reference information is information regarding the classification of the target object (hereinafter referred to as "second reference information"). For example, the second reference information is information specifying the classification of the target object, such as a vehicle, a person, or an airplane. The second reference information can be acquired, for example, from the work result 20. In the image sequence 10, the tendency of the change in the position of the target object is expected to differ depending on the classification of the target object. FIG. 6 is a conceptual diagram showing an example of two image sequences 10 containing different target objects. (A) of FIG. 6 shows a case where the target object is a person. (B) of FIG. 6 shows a case where the target object is a vehicle. As shown in (A) of FIG. 6, when the target object is a person, the position of the person in each image can change freely to some extent depending on the person's walking. It is also expected that the range in which the person is captured in each image will change in accordance with the person's movement. On the other hand, as shown in (B) of FIG. 6, when the target object is a vehicle, the position and range of the vehicle captured in each image tend to change linearly as the vehicle moves. In this way, it is expected that the tendency of the change in the position of the target object in each image will differ depending on the classification of the target object. Therefore, by including the second reference information in the reference information, it is possible to take into account the tendency of position change according to the classification of the target object in calculating the predicted position based on the reference information, thereby improving the accuracy of the predicted position.

[0044] Another type of reference information is information regarding the distance from the camera to the target object in each image (hereinafter referred to as "third reference information"). For example, the third reference information is depth information of the target object in each image. The third reference information can be obtained, for example, from the position of the bounding box 21 specified in each image and the depth information of each image. The depth information of each image is provided, for example, as additional data included in the image sequence 10. Alternatively, the depth information of each image may be calculated from the additional data included in the image sequence 10. It is assumed that the degree of change in the position of the target object in the image sequence 10 varies depending on the distance from the camera to the target object. FIG. 7 is a conceptual diagram showing an example of two image sequences 10 in which the distance from the camera to the target object (person) is different. (A) in FIG. 7 shows a case where the distance from the camera to the target object is small, i.e., the target object is close. (B) in FIG. 7 shows a case where the distance from the camera to the target object is large, i.e., the target object is far away. As shown in (A) of FIG. 7, when the target object is close, the change in the position of the target object shown in each image becomes large. On the other hand, as shown in (B) of FIG. 7, when the target object is far away, the change in the position of the target object shown in each image becomes small. In this way, it is expected that the degree of change in the position of the target object will differ depending on the distance from the camera to the target object. Therefore, by including the third reference information in the reference information, the degree of change in position depending on the distance from the camera to the target object can be taken into consideration when calculating the predicted position based on the reference information. This in turn can improve the accuracy of the predicted position.

[0045] The other is information (hereinafter referred to as "fourth reference information") regarding the speed of the moving object when each image was captured in the image sequence 10 when the image sequence 10 was captured by a camera mounted on the moving object. For example, the fourth reference information is information regarding the speed of the vehicle when each image was captured by an on-board camera. The vehicle speed information is provided, for example, as additional data included in the image sequence 10. It is assumed that the degree of change in the position of the target object in the image sequence 10 varies depending on the speed of the moving object carrying the camera. FIG. 8 is a conceptual diagram showing an example of two image sequences 10 with different speeds of the moving object carrying the camera. (A) of FIG. 8 shows a case where the speed of the moving object carrying the camera is high. (B) of FIG. 8 shows a case where the speed of the moving object carrying the camera is low. As shown in (A) of FIG. 8, when the speed of the moving object is high, the change in the position of the target object (tree) in each image is large. On the other hand, as shown in (B) of FIG. 8, when the speed of the moving object is low, the change in the position of the target object in each image is small. As described above, it is expected that the degree of change in the position of the target object will differ depending on the speed of the moving object carrying the camera. Therefore, by including the fourth reference information in the reference information, the degree of change in the position depending on the speed of the moving object carrying the camera can be taken into consideration when calculating the predicted position based on the reference information. As a result, the accuracy of the predicted position can be improved.

[0046] When the reference information includes the above information, the above information may be included in the elements of the vector wk, which serves as an explanatory variable, when calculating a predicted position using the regression model 122. For example, when the reference information further includes depth information of a target object shown in each image (third reference information), the vector wk may further include, as an element, the depth information of the target object shown in the corresponding image.

[0047] As described above, in the annotation verification method according to this embodiment, the predicted position of the bounding box 21 in the target image is calculated based on the reference information.

[0048] Furthermore, in the annotation verification method according to this embodiment, the position of the bounding box 21 that is actually specified in the target image (hereinafter referred to as the "actual specified position") is acquired from the work result 20.

[0049] In the annotation verification method according to this embodiment, the work result 20 on the target image is verified by comparing the bounding box 21 of the predicted position in the target image with the bounding box 21 of the actual specified position. In other words, the work result 20 on the target image is verified based on the degree to which the bounding box 21 of the predicted position in the target image matches the bounding box 21 of the actual specified position. In particular, it is verified whether or not there is an abnormality in the work result 20 on the target image.

[0050] 9 is a conceptual diagram showing an example of when the work result 20 for the target image is determined to have no abnormality and when it is determined to have an abnormality in the annotation verification method according to this embodiment. As shown in FIG. 9, according to the annotation verification method according to this embodiment, when the bounding box 21 (dotted line) of the predicted position and the bounding box 21 (solid line) of the actually specified position almost match, it is determined that the work result 20 for the target image has no abnormality. On the other hand, when the bounding box 21 (dotted line) of the predicted position and the bounding box 21 (solid line) of the actually specified position deviate from each other, it is determined that the work result 20 for the target image has an abnormality.

[0051] As described above, the annotation verification method according to this embodiment makes it possible to calculate a predicted position that is appropriate for sequential changes in the position of the bounding box 21 in each image included in the first image sequence 12. Therefore, by comparing the bounding box 21 at the predicted position with the bounding box 21 at the actual specified position, it is possible to appropriately verify whether or not there is an abnormality in the work result 20 for the target image. Furthermore, it is expected that the calculated predicted position will be a position that maintains the consistency of the annotation for the first image sequence 12. Therefore, it is possible to verify whether or not there is an abnormality in the work result 20 for the target image from the perspective of local consistency of the annotation as well.

[0052] The degree of overlap between the bounding box 21 of the predicted position and the bounding box 21 of the actual specified position can be used as an indicator of how closely the bounding box 21 of the predicted position and the bounding box 21 of the actual specified position match. In this case, if the degree of overlap is smaller than a predetermined threshold, it is determined that there is an abnormality in the work result 20 for the target image. The degree of overlap is expressed, for example, by IoU (Intersection over Union).

[0053] By performing threshold determination using the degree of overlap as an index in this way, verification by comparing the bounding box 21 of the predicted position with the bounding box 21 of the actually specified position can be easily realized.

[0054] When verification of the work result 20 for the target image is completed, the target image is shifted along the verification direction and the above steps are repeated. In the example shown in FIG. 3, when verification of image #i as the target image is completed, image #i+1 becomes the next target image. In this case, the first image sequence 12 is M consecutive images from #i-M+1 to #i. However, the first image sequence 12 may be selected excluding images determined to have an abnormality in the work result 20. For example, if it is determined that the work result 20 for image #i has an abnormality, the first image sequence 12 when image #i+1 is the target image may be M consecutive images from #iM to #i-1. By selecting the first image sequence 12 in this manner, it is possible to suppress a decrease in the accuracy of the predicted position.

[0055] In this way, in the annotation verification method according to this embodiment, the work result 20 for each image included in the image sequence 10 is sequentially verified. The verification result by the annotation verification method according to this embodiment is generated so as to provide a determination result as to whether or not there is an abnormality in the work result 20 for each image. For example, as shown in the table below, the verification result is data in which an abnormality determination flag that becomes TRUE when it is determined that there is an abnormality in the work result 20 is managed for each image. In the example of the verification result shown below, it is determined that there is an abnormality in the work result 20 for image #2. The verification result may be added as data to the work result 20 or the image sequence 10. Alternatively, the verification result may be provided to the user as data.

[0056] [Table 1]

[0057] As described above, according to the annotation verification method of this embodiment, first reference information is acquired regarding the position of the bounding box 21 in each image included in the first image sequence 12 adjacent to the target image. Furthermore, a predicted position of the bounding box 21 in the target image is calculated based on reference information including the first reference information. Then, the bounding box 21 at the predicted position in the target image is compared with the bounding box 21 at the actual specified position, thereby verifying the work result 20 on the target image. This makes it possible to appropriately verify the work result 20 on the target image.

[0058] Furthermore, by shifting the target image along a predetermined verification direction, it is possible to sequentially verify the work result 20 for each image included in the image sequence 10. Therefore, by executing the annotation verification method according to this embodiment on a computer, it is possible to easily verify the work result 20 for each image included in the image sequence 10.

[0059] Furthermore, according to the annotation verification method of this embodiment, the first reference information can be acquired from the work result 20. Therefore, the annotation verification method of this embodiment can perform verification if at least the work result 20 is given. Furthermore, according to the annotation verification method of this embodiment, the work result 20 can be verified without the need to compare it with work results 20 by other annotators 1, etc.

[0060] According to the annotation verification method of this embodiment, it is assumed that the work results 20 of each image included in the first image sequence 12 have already been verified. Therefore, the work results 20 of each image included in the first image sequence 12 for the initial target image may have already been verified by other means. For example, in the case shown in FIG. 3 , the work results 20 of M images #1 to #M included in the first image sequence 12 for image #M+1, which is the initial target image, may have already been verified by other means. The verification by other means may be manual verification. Even in this case, according to the annotation verification method of this embodiment, it is possible to easily verify the work results 20 for each image included in the image sequence 10 simply by manually verifying the work results 20 for images corresponding to the size of the first image sequence 12.

[0061] 1-2.Annotation verification device The annotation verification method according to this embodiment is realized by processing executed by a computer. An annotation verification device for implementing the annotation verification method according to this embodiment will be described below.

[0062] 10 is a block diagram showing an example of the configuration of an annotation verification device 100 according to this embodiment. The annotation verification device 100 is configured to be able to access an image database D10. For example, the annotation verification device 100 is connected to a server that stores the image database D10 via the Internet. Alternatively, the annotation verification device 100 can be configured to store the image database D10 in a storage device 120.

[0063] The image database D10 manages image sequences 10 captured by the camera 200 or the on-board camera 310.

[0064] An image sequence 10 stored in an image database D10 includes image data 11, additional data 13, and a work result 20.

[0065] The image data 11 is data representing each image included in the image sequence 10. The additional data 13 is data representing additional information relating to each image included in the image sequence 10.

[0066] The camera 200 may take various forms. Examples of the camera 200 include a video camera, a surveillance camera, and a live camera. The image sequence 10 captured by the camera 200 is uploaded to the image database D10 as appropriate. The camera 200 may be configured to upload captured images to the image database D10 sequentially.

[0067] The onboard camera 310 is mounted on a moving object 300. Examples of the moving object 300 include a vehicle and a drone. The moving object 300 includes a sensor 320 that detects the state of the moving object 300 and the surrounding environment. Examples of the sensor 320 include a light detection and ranging (LIDAR), an inertial measurement unit (IMU), a speed sensor, and a GPS receiver. Examples of information detected by the sensor 320 include the speed of the moving object 300, the position of the moving object 300, and the distance between the moving object 300 and surrounding objects. The moving object 300 appropriately uploads an image sequence 10 captured by the onboard camera 310 to an image database D10. Furthermore, the moving object 300 may be configured to upload the information detected by the sensor 320 to the image database D10 as additional data 13 of the image sequence 10. For example, the moving body 300 uploads the velocity of the moving body 300 when each image included in the image sequence 10 was captured as additional data 13 to the image database D10.

[0068] The work terminal 400 is a device for annotating image sequences 10 managed by the image database D10. The work terminal 400 reads out the image sequences 10 from the image database D10. Then, an annotator 1 operates the work terminal 400 to annotate the read out image sequences 10. The work terminal 400 uploads the annotation work results 20 to the image database D10. do.

[0069] The user interface 500 provides an interface for the user of the annotation verification device 100. For example, the user interface 500 is configured by input devices such as a keyboard, a mouse, a touch panel, etc., and output devices such as a display, a speaker, etc.

[0070] The annotation verification device 100 according to this embodiment reads an image sequence 10 managed by an image database D10 and executes a process of verifying the work result 20 for the read image sequence 10 using the annotation verification method described above. The image sequence 10 to be read is determined, for example, by a user via a user interface 500. Alternatively, the annotation verification device 100 may be configured to refer to the image database D10 and sequentially read image sequences 10 to which the work result 20 has been added. The verification result by the annotation verification device 100 is, for example, transmitted to the image database D10 and added to the corresponding image sequence 10. Alternatively, the verification result is provided to the user via the user interface 500.

[0071] The annotation verification device 100 according to this embodiment is a computer including one or more processors 110 (hereinafter simply referred to as "processors 110") and one or more storage devices 120 (hereinafter simply referred to as "storage devices 120"). The processor 110 executes various processes. The processor 110 may be configured, for example, as a CPU (Central Processing Unit) including an arithmetic unit, registers, etc. The storage device 120 is connected to the processor 110 and stores various information required for the processor 110 to execute the processes. The storage device 120 may be configured, for example, as a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive).

[0072] The storage device 120 stores a computer program 121 and a regression model 122 .

[0073] The computer program 121 is stored on a computer-readable recording medium. The computer program 121 includes a plurality of instructions configured to cause the processor 110 to execute various processes. The processor 110 operates in accordance with the plurality of instructions, thereby realizing the execution of the various processes by the processor 110.

[0074] 1-3. Processing The following describes the processing executed by the annotation verification device 100, more specifically, the processing executed by the processor 110.

[0075] Fig. 11 is a flowchart showing an example of processing executed by the processor 110. The processing shown in Fig. 11 starts, for example, when a request to start execution is received and the target image sequence 10 is read.

[0076] In step S100, the processor 110 executes initialization processing. In the initialization processing, the processor 110 acquires various information such as the additional data 13 and the work result 20, determines the initial target image, checks the verification direction, and so on.

[0077] Next, in step S110, the processor 110 selects a first image sequence 12 adjacent to the target image, and obtains a verified first operation result 22 for the first image sequence 12.

[0078] Next, in step S120, processor 110 acquires reference information. At least in step S120, processor 110 acquires first reference information from first work result 22 acquired in step S110. Furthermore, in step S120, processor 110 may acquire second reference information, third reference information, or fourth reference information from additional data 13 or first work result 22.

[0079] Next, in step S130, the processor 110 calculates a predicted position of the bounding box 21 in the target image based on the reference information obtained in step S120.

[0080] Next, in step S140, the processor 110 obtains the actual specified position of the bounding box 21 in the target image.

[0081] Next, in step S150, the processor 110 calculates the degree of overlap between the bounding box 21 of the predicted position and the bounding box 21 of the actual specified position.

[0082] Next, in step S160, the processor 110 determines whether the degree of overlap calculated in step S150 is smaller than a predetermined threshold value. The predetermined threshold value may be appropriately set depending on the environment in which this embodiment is applied.

[0083] If the degree of overlap is smaller than the predetermined threshold (step S160; Yes), processor 110 determines that there is an abnormality in work result 20 for the target image (step S170). For example, processor 110 sets the abnormality determination flag corresponding to the target image to TRUE. Thereafter, the process proceeds to step S180.

[0084] If the degree of overlap is equal to or greater than the predetermined threshold value (step S160; No), it is determined that there is no abnormality in the work result 20 for the target image, and the process proceeds to step S180.

[0085] In step S180, processor 110 determines whether to end the verification. For example, processor 110 determines to end the verification on the condition that verification of operation results 20 for all images included in image sequence 10 has been completed.

[0086] If it is determined that the verification should be ended (step S180; Yes), the process ends.

[0087] If it is determined that the verification should not be completed (step S180; No), processor 110 shifts the target image (step S190). After that, processor 110 repeats the process from step S110 again.

[0088] As described above, the functions of the annotation verification device 100 according to this embodiment are realized by the processor 110 executing the processes. Furthermore, the annotation verification method according to this embodiment is implemented by the processor 110 executing the processes in this manner. Furthermore, the annotation verification program according to this embodiment is realized by the computer program 121 that causes the processor 110 to execute the processes in this manner.

[0089] 2. Second embodiment The second embodiment will be described below, with the overlapping parts of the above description being omitted as appropriate.

[0090] 2-1. Overview When annotating an image sequence 10, multiple annotators 1 may annotate the same image sequence 10. In this case, the annotation work result 20 for the image sequence 10 includes the work results 20 of each of the multiple annotators 1. It is generally expected that the work results 20 of each of the multiple annotators 1 will differ from one another depending on the skills, tendencies, etc. of each annotator 1.

[0091] The annotation verification method according to the second embodiment is applied when multiple annotators 1 are annotating the same image sequence 10. Below, an overview of the annotation verification method according to the second embodiment will be described with reference to Fig. 12. Fig. 12 conceptually shows work results 20 of three annotators 1, #1, #2, and #3, on the same image sequence 10.

[0092] In the annotation verification method according to the second embodiment, the reliability of each annotation by a plurality of annotators 1 is managed. The reliability of each annotator 1 is a value that estimates the correctness of the annotation by that annotator 1. In FIG. 12, the reliability of annotators 1 #1, #2, and #3 is given as 90%, 80%, and 60%, respectively. In other words, the annotation by annotator 1 #1 is estimated to be 90% correct, and the annotation by annotator 1 #3 is estimated to be 60% correct. Note that the expression of reliability is not limited to percentage. For example, reliability may be expressed as a decimal.

[0093] The reliability of each annotator 1 can be given by updating based on the verification results of the annotation verification method, as will be described later. In this case, the initial value of the reliability of each annotator 1 may be suitably given. Alternatively, the reliability of each annotator 1 may be given using indicators such as the number of annotations performed, years of experience, etc.

[0094] The annotation verification method according to the second embodiment sequentially verifies the work results 20 of each of a plurality of annotators 1 for each image included in an image sequence 10 in a predetermined verification direction. In Fig. 12, the verification direction is ascending order. Fig. 12 explains the case where image #i is the target image.

[0095] In the annotation verification method according to the second embodiment, first, a first work result 22 is obtained. In the second embodiment, the first work result 22 includes verified work results 20 of each of a plurality of annotators 1 on a first image sequence 12.

[0096] Next, in the annotation verification method according to the second embodiment, the reliability of each of the multiple annotators 1 is obtained. Then, in the annotation verification method according to the second embodiment, first reference information is obtained from the first work result 22 and the reliability of each of the multiple annotators 1.

[0097] In the second embodiment, the first reference information is a weighted average position 23 of the positions of bounding boxes 21 designated by multiple annotators 1 for each image included in the first image sequence 12. In particular, the weights used in calculating the weighted average position 23 are the reliability of each of the multiple annotators 1.

[0098] In FIG. 12, the weighted average position 23 is indicated by a vector wk (k=1, 2, . . . , M). For example, w1 is a vector whose elements are the weighted average of the coordinate positions of the four corners and the center of gravity of the bounding box 21 specified by annotators 1 #1, #2, and #3 for the image #iM. For example, the vector wk (k=1, 2, . . . , M) can be expressed by the following equation: where α1, α2, and α3 are the reliability of annotators 1 #1, #2, and #3, respectively. Also, v1k, v2k, and v3k are vectors whose elements are the coordinate positions of the four corners and the center of gravity of the bounding box 21 specified by annotators #1, #2, and #3, respectively, for the corresponding image.

[0099]

number

[0100] Next, in the annotation verification method according to the second embodiment, a predicted position of the bounding box 21 in the target image is calculated based on reference information including the first reference information. The calculation of the predicted position may be the same as in the first embodiment. That is, the predicted position may be calculated using a regression model 122 that uses reference information including the weighted average position 23 as the first reference information as an explanatory variable. The reference information may also include the second reference information, the third reference information, or the fourth reference information.

[0101] Next, in the annotation verification method according to the second embodiment, the actual designated position of the bounding box 21 in the target image is acquired for each of the multiple annotators 1 from the work results 20 of each of the multiple annotators 1.

[0102] In the annotation verification method according to the second embodiment, the work result 20 for the target image is verified by comparing the bounding box 21 of the predicted position in the target image with the bounding box 21 of the actual specified position for each of the multiple annotators 1. The bounding box 21 of the predicted position is common to the verification of each of the multiple annotators 1. The comparison method for the verification of each of the multiple annotators 1 may be the same as that of the first embodiment. In particular, the comparison for the verification of each of the multiple annotators 1 may be performed using the degree of overlap between the bounding box 21 of the predicted position in the target image and the bounding box 21 of the actual specified position as an index.

[0103] When the verification of the work result 20 for the target image for each of the multiple annotators 1 is completed, the target image is shifted along the verification direction and the above steps are repeated.

[0104] In this way, in the annotation verification method according to the second embodiment, the work results 20 for each image included in the image sequence 10 are verified sequentially for each of the multiple annotators 1.

[0105] In the annotation verification method according to the second embodiment, the reliability of each of the multiple annotators 1 is updated based on the verification results. Typically, the reliability of an annotator 1 whose verification results determine that the work result 20 is abnormal is reduced. Furthermore, the reliability of an annotator 1 whose verification results determine that the work result 20 is normal (the work result 20 is normal) may be increased. The reliability may be updated each time verification of the work result 20 for a target image is completed, or may be updated when verification of the work result 20 for all images is completed.

[0106] When updating the reliability of the annotator 1, the amount of change in the reliability may be adjusted based on the difficulty of the annotation. The difficulty of the annotation is a value that represents the difficulty of the annotator 1 to correctly perform the annotation.

[0107] The difficulty of annotation can be determined based on the size of the target object. For example, if the target object is small, such as a pebble, the difficulty of annotation is set high.

[0108] The degree of annotation difficulty can be determined based on the degree of freedom of the target object's movement. For example, if the target object is a flying object such as a bird or an airplane, the annotation difficulty is set high.

[0109] The difficulty of annotation can be determined by the ease of distinguishing the target object in the image. For example, when the classification of the target object is easily confused with other classifications that are not the target object (e.g., vans and SUVs, sidewalks and bicycle paths, etc.), the difficulty of annotation is set high.

[0110] The difficulty of annotation can be determined based on the appearance of the image. For example, the lower the brightness or contrast of the image, the higher the difficulty of annotation is set.

[0111] The degree of difficulty of annotation can be determined by the degree of difference in the work results 20 between multiple annotators 1. In this case, Krippendorff's alpha can be used to measure the degree of difference in the work results 20 between multiple annotators 1. For example, the larger the alpha coefficient, that is, the smaller the difference in the work results 20 between multiple annotators 1, the higher the difficulty of annotation is set.

[0112] The degree of difficulty of annotation may also be calculated using a trained machine learning model that uses an image as input. In this case, the machine learning model may be configured, for example, by a convolutional neural network (CNN).

[0113] The degree of difficulty of annotation may be set for the image sequence 10, or may be set individually for each image included in the image sequence 10.

[0114] The amount of change in reliability depending on the difficulty of the annotation is adjusted, for example, as shown in the table below. However, the amount of change in reliability may be adjusted in more stages or continuously depending on the difficulty of the annotation. By adjusting the amount of change in reliability depending on the difficulty of the annotation in this way, the reliability of each of the multiple annotators 1 can be more accurately managed.

[0115] [Table 2]

[0116] As described above, according to the annotation verification method of the second embodiment, a predicted position of a bounding box 21 in a target image is calculated based on reference information including a weighted average position 23 as first reference information. Then, for each of the multiple annotators 1, the bounding box 21 at the predicted position in the target image is compared with the bounding box 21 at the actual specified position, thereby verifying the work result 20 on the target image. Furthermore, by shifting the target image along a predetermined verification direction, the work result 20 on each image included in the image sequence 10 is sequentially verified for each of the multiple annotators 1. This makes it possible to verify the work result 20 while maintaining verification efficiency when multiple annotators 1 annotate the same image sequence 10. In particular, since the weighted average position 23 is calculated using the reliability of each of the multiple annotators 1 as a weight, the work result 20 of each of the multiple annotators 1 can be verified taking into account the skill of each annotator 1.

[0117] Furthermore, according to the annotation verification method of the second embodiment, the reliability of each of the multiple annotators 1 is updated based on the verification result. This makes it possible to dynamically manage the reliability of each of the multiple annotators 1.

[0118] 2-2. Annotation verification device The annotation verification device 100 for implementing the annotation verification method according to the second embodiment will be described below.

[0119] 13 is a block diagram showing an example of the configuration of an annotation verification device 100 according to the second embodiment. In the annotation verification device 100 according to the second embodiment, reliability information 123 is stored in a storage device 120, as compared with the first embodiment.

[0120] The reliability information 123 manages the reliability of each annotator 1. For example, the reliability information 123 is data in which identification information of each annotator 1 is linked to the reliability of each annotator 1. The reliability information 123 is updated and managed by processing executed by the processor 110.

[0121] 2-3. Processing The following describes the processing executed by the annotation verification device 100 according to the second embodiment, more specifically, the processing executed by the processor 110.

[0122] The processing executed by the processor 110 according to the second embodiment may be the same as the processing shown in Fig. 11. However, the processing according to steps S140 to S170 is executed for each of the multiple annotators 1. In addition, in step S120, the processor 110 according to the second embodiment executes at least the following processing shown in Fig. 14.

[0123] In step S210, the processor 110 acquires the reliability of each of the multiple annotators 1. For example, the processor 110 acquires the reliability of each annotator 1 by referring to the reliability information 123 using the identification information of each annotator 1.

[0124] Next, in step S220, the processor 110 calculates a weighted average position 23 as first reference information based on the first work result 22 obtained in step S110 and the reliability of each of the multiple annotators 1 obtained in step S210. [Explanation of symbols]

[0125] 1 Annotator 10 Image Sequences 11 Image data 12 First Image Sequence 13 Additional Data 20 Work Results 21 Bounding Box 22 First Work Result 23 Weighted average position 100 Annotation Verification Device 110 processors 120 Storage device 121 Computer Programs 122 Regression Models 123 Reliability Information D10 Image Database

Claims

1. An annotation verification method for verifying an annotation work result for an image sequence, comprising: The annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence; The annotation verification method is executed by a computer, Obtaining a first operation result, which is a verified operation result for a first image sequence included in the image sequences; obtaining first reference information from the first operation result regarding the position of the target object region in each image included in the first image sequence; calculating a predicted position of the target object region in a target image, the target image being an image adjacent to the first image sequence, based on reference information including the first reference information; acquiring an actual designated position, which is the position of the target object region actually designated in the target image, from the operation result for the target image; verifying the operation result for the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position; Contains Annotation validation methods.

2. 2. The annotation verification method according to claim 1, Verifying the work result on the target image includes: calculating a degree of overlap between the target object region at the predicted position and the target object region at the actual specified position; When the degree of overlap is smaller than a predetermined threshold, it is determined that there is an abnormality in the work result for the target image; Contains Annotation validation methods.

3. 2. The annotation verification method according to claim 1, obtaining second reference information related to a classification of the target object; The reference information further includes the second reference information. Annotation validation methods.

4. 2. The annotation verification method according to claim 1, the image sequence is captured by a predetermined camera; The annotation verification method further includes obtaining third reference information related to a distance from the camera to the target object in each image included in the first image sequence; The reference information further includes the third reference information. Annotation validation methods.

5. 2. The annotation verification method according to claim 1, the image sequence is captured by a camera mounted on a moving object; The annotation verification method further includes obtaining fourth reference information related to a speed of the moving object when each image included in the first image sequence was captured; The reference information further includes the fourth reference information. Annotation validation methods.

6. 6. The annotation verification method according to claim 1, further comprising: the annotations are made by multiple annotators; Obtaining the first reference information includes: obtaining a confidence level for the annotation of each of the plurality of annotators; calculating a weighted average position of the positions of the object regions designated by each of the plurality of annotators for each image included in the first image sequence, using the confidence as a weight; Including, The first reference information is the weighted average position in each image included in the first image sequence. Annotation validation methods.

7. 7. The annotation verification method according to claim 6, and updating the reliability based on a result of verifying the operation result for the target image. Annotation validation methods.

8. 8. The annotation verification method according to claim 7, Updating the confidence level includes: obtaining a difficulty level of the annotation; adjusting a change amount of the reliability at the time of updating according to the difficulty level; Contains Annotation validation methods.

9. An annotation verification device for verifying annotation results for an image sequence, comprising: one or more processors; The annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence; the one or more processors: obtaining a first operation result, the first operation result being a verified operation result for a first image sequence included in the image sequences; obtaining first reference information relating to the position of the target object region in each image included in the first image sequence from the first operation result; calculating a predicted position of the target object region in a target image that is an image adjacent to the first image sequence based on reference information including the first reference information; A process of acquiring an actual designated position, which is the position of the target object region actually designated in the target image, from the operation result for the target image; a process of verifying the result of the operation on the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position; is configured to run Annotation verification device.

10. An annotation verification program that causes a computer to execute a process for verifying an annotation work result for an image sequence, comprising: The annotation is an operation in which an annotator specifies a target object region surrounding a target object for each image included in the image sequence; The annotation verification program obtaining a first operation result, the first operation result being a verified operation result for a first image sequence included in the image sequences; obtaining first reference information relating to the position of the target object region in each image included in the first image sequence from the first operation result; calculating a predicted position of the target object region in a target image that is an image adjacent to the first image sequence based on reference information including the first reference information; A process of acquiring an actual designated position, which is the position of the target object region actually designated in the target image, from the operation result for the target image; a process of verifying the result of the operation on the target image by comparing the target object region at the predicted position in the target image with the target object region at the actual specified position; configured to cause the computer to execute An annotation verification program characterized by:

Images