Annotation assistance device and control program

Abstract

The present invention easily corrects labels of a plurality of feature points that are symmetric with respect to a symmetry axis.　This annotation assistance device 10 comprises: an input unit 115 which inputs, to an annotation target having a symmetry axis and a plurality of portions that are symmetric with respect to the symmetry axis, information about a first feature point which is included in a first portion and of which a label is to be corrected; and a label rewriting unit 117 which rewrites the label of the first feature point to a label of a second portion that is symmetric to the first portion with respect to the symmetry axis.

Description

Annotation support device and control program

[0001] The present invention relates to an annotation support device and a control program.

[0002] There is a technology that uses a deep neural network to detect a person's joint points and estimates the pose based on the detected joint points (see, for example, Patent Literature 1). Accurately detecting a person's joint points requires a large amount of training data for training the deep neural network. To efficiently create and label this training data, there is a technology that auto-annotates captured images. For example, annotation tools such as CVAT (Computer Vision Annotation Tool) and Supervisely are capable of auto-annotating the joint points of a photographed person.

[0003] However, auto-annotation does not always assign correct labels, and errors can occur. For example, when auto-annotating joint points from an image of a walking person, the upper body may be labeled relatively correctly, but the lower body may be reversed.

[0004] Japanese Patent Application Laid-Open No. 2020-86819

[0005] Correcting such incorrect labels is costly, especially when there are multiple joint points. For example, if multiple joint points making up the right and left feet are annotated in a mirrored manner, multiple points must be corrected, resulting in high correction costs.

[0006] The annotation tools described above have a function for automatically annotating feature points such as articulation points, but do not provide a function for easily correcting these annotations.

[0007] The present invention has been made in consideration of the above circumstances, and aims to provide an annotation support device and a control program that can easily modify the labels of multiple feature points that are symmetrical with respect to an axis of symmetry, such as bilateral symmetry.

[0008] The above object of the present invention can be achieved by the following means.

[0009] (1) An annotation support device that supports labeling of feature points of an annotation target included in an image, the annotation support device comprising: an input unit that inputs information about a first feature point included in a first part of the annotation target, the first feature point being the label of which should be corrected, for the annotation target having an axis of symmetry and a plurality of parts symmetrical to the axis of symmetry; and a label rewriting unit that rewrites the label of the first feature point to the label of a second part symmetrical to the first part with respect to the axis of symmetry.

[0010] (2) The annotation support device described in (1) above, wherein a plurality of the first feature points are registered as the first region, and the label rewriting unit rewrites the labels of the plurality of first feature points included in the first region in response to input of the first region or input of information on at least any of the first feature points to the input unit.

[0011] (3) The annotation support device described in (1) above, wherein the input of information about the first feature point by the input unit includes at least one of specifying the position of the first feature point on a display image displaying the annotation target and the feature point, specifying an area including the first feature point, specifying the axis of symmetry regarding the first feature point, and specifying a point on the axis of symmetry regarding the first feature point.

[0012] (4) The annotation support device according to (1), wherein information on the first feature point is input to the input unit from a voice analysis unit that analyzes input voice.

[0013] (5) The annotation support device according to (1), wherein the label rewriting unit rewrites the labels for a plurality of frames at once.

[0014] (6) An annotation support device described in any one of (1) to (5) above, wherein a second feature point symmetrical to the first feature point with respect to the axis of symmetry is set as a pair with the first feature point, and the label rewriting unit swaps the labels of the first feature point and the second feature point.

[0015] (7) The annotation support device according to (6), wherein the label rewriting unit swaps the positions of the first feature point and the second feature point.

[0016] (8) The annotation support device according to (6) above, further comprising a target designation unit that designates the annotation target, wherein the label rewriting unit swaps the labels of the first feature point and the second feature point for the designated annotation target.

[0017] (9) The annotation support device described in (6) above, wherein the annotation target is a person, and the pairs of the first and second feature points include at least one of left shoulder and right shoulder, left elbow and right elbow, left wrist and right wrist, left fingertip and right fingertip, left hip and right hip, left knee and right knee, left ankle and right ankle, left toe and right toe, left eye and right eye, and left ear and right ear.

[0018] (10) A control program that operates in an annotation support device that supports labeling of feature points of an annotation target included in an image, the control program for executing a process including: an input step of inputting information on a first feature point, the label of which is to be corrected, included in a first part of the annotation target having an axis of symmetry and a plurality of parts symmetrical to the axis of symmetry; and a label rewriting step of rewriting the label of the first feature point to the label of a second part symmetrical to the first part with respect to the axis of symmetry.

[0019] (11) The control program described in (10) above, wherein a plurality of the first feature points are registered as the first part, and in the label rewriting step, labels of the plurality of first feature points included in the first part are rewritten in accordance with input of the first part in the input step.

[0020] (12) The control program according to (10) above, wherein the input of information about the first feature point in the input step includes at least one of specifying the position of the first feature point on a display image displaying the annotation target and the feature point, specifying an area including the first feature point, specifying the axis of symmetry regarding the first feature point, and specifying a point on the axis of symmetry regarding the first feature point.

[0021] (13) The control program according to (10), wherein in the input step, information on the first feature point is input from a voice analysis unit that analyzes input voice.

[0022] (14) The control program according to (10), wherein the label rewriting step rewrites the labels for a plurality of frames at once.

[0023] (15) The control program according to any one of (10) to (14), wherein a second feature point symmetrical to the first feature point with respect to the axis of symmetry is set as a pair with the first feature point, and in the label rewriting step, labels of the first feature point and the second feature point are swapped.

[0024] (16) The control program according to (15), wherein in the label rewriting step, the positions of the first feature point and the second feature point are swapped.

[0025] (17) The control program according to (10), further comprising: a target designation step of designating the annotation target; and in the label rewriting step, swapping labels of the first feature point and the second feature point for the designated annotation target.

[0026] (18) The control program also described in (15) above, wherein the annotation target is a person, and the pairs of the first and second feature points include at least one of left and right shoulders, left and right elbows, left and right wrists, left and right fingertips, left and right hips, left and right knees, left and right ankles, left and right toes, left and right eyes, and left and right ears.

[0027] The annotation support device according to the present invention includes an input unit for inputting information on a first feature point included in a first part and whose label is to be corrected, for an annotation target having a symmetry axis and a plurality of parts symmetrical with respect to the symmetry axis, and a label rewriting unit for rewriting the label of the first feature point to the label of a second part symmetrical to the first part with respect to the symmetry axis. In this way, the labels of the plurality of feature points in a symmetrical relationship can be easily corrected.

[0028] 7 is a diagram showing a schematic configuration of an annotation support device according to the present embodiment. FIG. 1 is a table showing a pair relationship of labels that are symmetrical with respect to an axis of symmetry. FIG. 2 is a flowchart showing an outline of the processing procedure of the annotation support method. FIG. 3 is a schematic diagram showing a state in which correct labels have been assigned by annotation. FIG. 4 is a table showing the relationship between the parts and labels at each feature point in FIG. 4. FIG. 5 is a table showing the relationship between the parts and labels at each feature point in a modified example. FIG. 6 is a schematic diagram explaining rewriting of an incorrect label. FIG. 7 is a table for explaining rewriting of the parts and labels at each feature point corresponding to FIG. 7. FIG. 8 is a subroutine flowchart showing the rewriting process of step S08 of FIG. 3 in a first method. FIG. 9 is a diagram showing a procedure for specifying a first feature point or a first part on a display screen. FIG. 10 is a subroutine flowchart showing the rewriting process of step S08 of FIG. 3 in a second method. FIG. 11 is a diagram showing a procedure for specifying a first feature point or a first part on a display screen. FIG. 12 is a subroutine flowchart showing the rewriting process of step S08 of FIG. 3 in a third method. 16 is a diagram showing a procedure for specifying a center point on a symmetry axis on a display screen. FIG. 17 is a subroutine flowchart showing the rewriting process of step S08 of FIG. 3 in a fourth technique. FIG. 18 is a diagram showing a procedure for specifying a center point on a symmetry axis on a display screen. FIG. 19 is a table for explaining rewriting of parts and labels at each feature point corresponding to FIG. 16. FIG. 19 is a table for explaining rewriting of each feature point corresponding to FIG. 8 in a modified example. FIG. 20 is a schematic diagram for explaining an example of collective rewriting in time-series frames.

[0029] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are designated by the same reference numerals, and redundant description will be omitted. Advantages and features provided by one or more embodiments of the present invention will be fully understood from the detailed description set forth below and the accompanying drawings. Furthermore, the accompanying drawings are provided for illustrative purposes only and are not intended to define the scope of the present invention.

[0030] 1 is a diagram showing a schematic configuration of an annotation support device 10 according to this embodiment. An information system 300 includes the annotation support device 10 (hereinafter simply referred to as the support device 10) and an imaging device 20. The support device 10 and the imaging device 20 are connected via communication. The annotation support device 10 is a PC (personal computer) or a server. When the annotation support device 10 is configured as a server, some of its functions may be performed by a cloud server on a network.

[0031] The imaging device 20 includes a control unit, a communication unit, a camera, and the like. The camera is, for example, a visible light camera. The visible light camera captures an image of a predetermined area by receiving visible light reflected by objects within the predetermined area using a CMOS sensor. The camera may also be configured using a near-infrared camera instead of a visible light camera. The imaging device 20 captures a video consisting of multiple temporally consecutive captured images (frames) at a frame rate of, for example, 15 fps to 60 fps. The captured images are sent to the assistance device 10.

[0032] The support device 10 includes a control unit 11, a memory unit 12, a communication unit 13, a display unit 14, a keyboard 15, a mouse 16, and a microphone 17. The components within the support device 10 are electrically connected to one another, but the connections are not shown in FIG.

[0033] The control unit 11 is composed of a CPU, a GPU, a RAM, a ROM, etc. The control unit 11 functions as an image acquisition unit 111 in cooperation with the communication unit 13. The control unit 11 also functions as an articulation point extraction unit 112, an annotation unit 113, a pair setting unit 114, a feature point input unit 115, a target designation unit 116, a label rewriting unit 117, and a voice analysis unit 118 in cooperation with other components such as the display unit 14 and the mouse 16. Each of these functions will be described later.

[0034] The storage unit 12 is configured with an SSD, HDD, etc. The storage unit 12 stores various programs, a machine-learned model that extracts articulation points, a table describing label pair relationships (see FIG. 2, which will be described later), and the like.

[0035] The communication unit 13 is an interface circuit (for example, a LAN card) for communicating with other devices such as the image capture device 20 via a network.

[0036] The display unit 14 is configured with a display such as an LCD, and displays a display screen (GUI: Graphical User Interface) used for inputting the first feature point, etc. The display unit 14 may also include a touch screen.

[0037] The keyboard 15 and mouse 16 are used for inputting instructions from the user, and the microphone 17 is used for inputting voice from the user.

[0038] (Image Acquisition Unit 111) The image acquisition unit 111 acquires images captured by the imaging device 20. The captured images are linked to the time of capture and stored in the storage unit 12. The image acquisition unit 111 may also acquire images from other devices connected via a network.

[0039] (Articulation Point Extraction Unit 112) The articulation point extraction unit 112 detects skeletal information of an object, such as a person, included in the image acquired by the image acquisition unit 111. The articulation point extraction unit 112 detects the articulation points using a trained model for detecting the articulation points of a person from a rectangle including a person (hereinafter also referred to as a "person rectangle"). The person rectangle is an area that contains the articulation points and nodes of the person in the image. Known examples of such trained models include OpenPose (https: / / arxiv.org / abs / 1812.08008) and DeepPose (https: / / arxiv.org / abs / 1312.4659).

[0040] (Annotation Unit 113) The annotation unit 113 performs auto-annotation on the joint points extracted by the joint point extraction unit 112. For example, the auto-annotation is performed using CVAT. Examples of labels for each joint point (also called feature point) assigned by annotation will be described later (e.g., FIGS. 4 and 5).

[0041] (Pair Setting Unit 114) The pair setting unit 114 sets pairs of two feature points that are symmetrically positioned with respect to the axis of symmetry of the object. In the case of a person, the axis of symmetry is the central axis of the body. FIG. 3 is a table showing pair relationships. This table is set by the user and stored in advance in the storage unit 12. In FIG. 3, the object is a person, and example feature points are shown as joint points. As shown in FIG. 3, the right shoulder and left shoulder are in a pair relationship. Similarly, the right elbow and left elbow are in a pair relationship. FIG. 3 is merely an example, and other feature points may be included, or only some of them may be included. Furthermore, with respect to the axis of symmetry, one is a first feature point (or a first region), and the other is a second feature point (or a second region). In the example shown in FIG. 3 and hereinafter, unless otherwise specified, the left side of the left and right sides will be referred to as the first feature point or the first region, and the right side will be referred to as the second feature point or the second region.

[0042] (Feature Point Input Unit 115) The feature point input unit 115 accepts input of information about feature points whose labels are to be rewritten, via input via the mouse 16, keyboard 15, etc. to a display screen (such as display screen s1 in FIG. 10A described later) displayed on the display unit 14. The "feature point information" here includes the following information for identifying one or more feature points whose labels are to be rewritten: (1a) Information specifying the position of the feature point (the example in FIG. 10A described later). (1b) Information specifying the area including the feature point (the example in FIG. 10B described later). (1c) Specification of a symmetry axis related to the feature point. (1d) Specification of a point on the symmetry axis related to the feature point (the example in FIG. 14 described later).

[0043] Furthermore, when one feature point is designated by inputting feature point information, the feature point whose label is to be rewritten may have the following cases: (2a) When only that feature point is rewritten to a paired label (example in FIG. 16 described later). (2b) When that feature point and other feature points belonging to the same region are rewritten to paired labels (example in FIG. 15 described later). (2c) When the labels of that feature point and the feature point to which the paired label is assigned are mutually swapped (example in FIG. 9 described later). (2d) When the labels of that feature point and other feature points belonging to the same region are swapped with the feature point to which the paired label is assigned (example in FIG. 12 described later).

[0044] (Target Designation Unit 116) The target designation unit 116 accepts designation of an annotation target, the label of which is to be rewritten, by input via the mouse 16, the keyboard 15, etc. to an operation-required display screen (such as a display screen s1 in FIG. 10 , which will be described later) displayed on the display unit 14. For the designated target, the labels of feature points in a specific range are replaced with paired labels (see FIG. 15 , which will be described later).

[0045] (Label Rewriting Unit 117) The label rewriting unit 117 rewrites labels based on information about feature points or information about specified annotation targets input by the feature point input unit 115 or the target designation unit 116. This label rewriting includes both rewriting the label of a first feature point to the label of the other pair (example in FIG. 17 described later) and mutually swapping the labels of the first and second feature points (e.g., FIG. 8 described later). As another example, the label rewriting unit 117 may also perform processing to swap the positions of feature points to which paired labels have been assigned (example in FIG. 18 described later).

[0046] (Speech Analysis Unit 118) When the label rewrite mode is running, the speech analysis unit 118 analyzes the speech input from the user through the microphone 17 and obtains text data through speech recognition processing. Well-known processing can be applied to the speech recognition processing. By comparing the text data with dictionary data related to commands, instructions are issued to the feature point input unit 115 or the target designation unit 116.

[0047] (Annotation Support Method) The annotation support method will be described with reference to Fig. 3 to Fig. 15. Fig. 3 is a flowchart showing the annotation support method.

[0048] (Step S01) The annotation support device 10 acquires image data captured from the imaging device 20. For example, the image acquisition unit 111 acquires, from the imaging device 20, time-series frame images obtained by continuous imaging.

[0049] (Step S02) The joint point extraction unit 112 detects an object from the image using a trained model trained by machine learning. For example, a person (figure) is detected from the image as the object. Note that, although the following description will be given of a case where the annotation target (object) is a person, the present invention is not limited to this and may be applied to animals other than humans that have a skeleton, such as cats and dogs.

[0050] (Step S03) The joint point extraction unit 112 extracts (estimates) joint points of the person detected in step S02 using a known trained model such as Deep Pose. The annotation unit 113 assigns labels for auto-annotation to the joint points.

[0051] (Step S04) The control unit 11 displays the labeled image on the display unit 14. Fig. 4 is a schematic diagram showing the state in which correct labels have been assigned by the processing up to this point. Fig. 5 is a table showing the relationship between the parts and labels at each feature point in Fig. 4.

[0052] FIG. 4 is a schematic diagram showing an image of a person preparing to bat a ball in front of them, with annotation labels superimposed on the image. In FIG. 4, p11 to p26 indicate feature points, which correspond to the feature points in the table in FIG. 5. In FIG. 5, for example, feature point p11 is shown as a second feature point, a feature point of the right wrist, which belongs to the right arm. Feature points p11 to p13 are a group of multiple feature points that belong to the same part of the right arm and are labeled right wrist, right elbow, and right shoulder, respectively. As described above, the feature points on the left side are referred to as first feature points or first parts, and the feature points on the right side are referred to as second feature points or second parts. However, this is merely an example, and the right side may be the first feature point or first part instead.

[0053] Furthermore, in the example of Fig. 5, a single body part is described as including multiple feature points, but a single body part may include only one feature point. For example, Fig. 6 is a table showing the relationship between each body part and feature points in a modified example. In the example of Fig. 6, the right shoulder body part includes only the feature point labeled right shoulder.

[0054] (Step S05) The user can check whether the correct label has been assigned by viewing the content shown in Fig. 4 displayed on the display surface of the display unit 14. If the correct label has been assigned as in the example of Fig. 4, no correction is required (NO), and the process ends.

[0055] On the other hand, if label correction is necessary (YES), the user starts up the correction application and transitions to label rewriting mode, which causes the control unit 11 to proceed to step S06.

[0056] (Step S06) The pair setting unit 114 sets pairs of feature points that are symmetrical to each other. This is performed using the table shown in FIG. 2. For example, in the examples of FIGS. 4 and 5, the pair setting unit 114 pairs feature point p11 labeled with the right wrist with feature point p21 labeled with the left wrist that is paired with the label of the left wrist. Similarly, in the example of FIG. 5, feature point p12 and feature point p22 are set as a pair. Similarly, feature points p13 to p16 and feature points p23 to p26 are also set as pairs, respectively.

[0057] (Step S07) The control unit 11 displays the content as shown in FIG. 4 on the display surface of the display unit 14, and also displays a display screen that accepts input of information on feature points by the user.

[0058] (Step S08) Here, the control unit 11 performs processing to rewrite the labels of the feature points based on the information of the feature points input by the user.

[0059] There are four methods for inputting information about the user's feature points, which will be described later. Each method will be described later, but the overall outline is as follows.

[0060] 7 is a schematic diagram for explaining how to rewrite an incorrect label, and a table for explaining how to rewrite the label and the parts at the feature points corresponding to FIG.

[0061] 7A and 8A show the state where an incorrect label has been assigned, and Fig. 7B and 8B show the state after relabeling. The area surrounded by a dashed line in Fig. 7 indicates the range of feature points to which an incorrect label has been assigned, and similarly, the area surrounded by a dashed line in Fig. 8 indicates the corresponding feature points.

[0062] The feature point input unit 115 accepts input of feature points p21 to p23 (or feature points p11 to p13) from the user, which causes the label rewriting unit 117 to refer to the labels paired by the pair setting unit 114 (FIG. 2).

[0063] The label rewriting unit 117 then extracts these feature points p21 to p23 and the feature points p11 to p13 that were paired in step S06, and swaps their labels with each other. Alternatively, the label rewriting unit 117 extracts feature points p21 to p23 to which labels (right wrist, right elbow, right shoulder) that are paired with the labels (left wrist, left elbow, left shoulder) that are assigned to these feature points p11 to p13. The label rewriting unit 117 then swaps the labels of these feature points with each other. That is, the label rewriting unit 117 rewrites the left wrist label of feature point p11 to the right wrist label (see FIG. 2), and rewrites the label of the extracted feature point p21 (which has been assigned the right wrist label) to the left wrist. The label rewriting unit 117 performs the same process on the selected feature points p12 and p13, and rewrites the labels of feature points p12, p13, p22, and p23. The above is an outline of the label rewriting process. Next, we will explain how to input feature point information using each method.

[0064] (First Method) In the first method, the user designates each feature point to be rewritten. Fig. 9 is a subroutine flowchart showing the rewriting process of step S08 in Fig. 3 in the first method. Figs. 10A and 10B are diagrams showing the operation procedure for designating a first feature point on a display screen (GUI) displayed on the display unit 14 or the like.

[0065] (Step S101) The user refers to the label of the annotation target (person) displayed on the display screen and specifies the feature point requiring label modification. For example, the feature point is specified by clicking on it on the display screen s1 in Fig. 10A. Note that in the display screen s1 in Fig. 10A, the left and right feature points are shown as labels distinguishable by line type, density, color, mark shape, etc., but the labels may also be displayed in text (the same applies to other display screens such as Fig. 10B). Furthermore, by moving the cursor close to one of the feature points, the text data of the label for each feature point may be displayed.

[0066] The feature point input unit 115 receives input of a specified feature point in response to a click operation by the user. For example, assume that the user has selected feature point p22 as shown in FIG.

[0067] (Step S102) The label rewriting unit 117 recognizes the label (right elbow) assigned to the feature point p22 input by the user. This label is an incorrect label that the user wants to correct. The label rewriting unit 117 references the table of FIG. 2 set by the pair setting unit 114 and extracts the feature point p12 to which the label (left elbow) paired with this label (right elbow) has been assigned. The label rewriting unit 117 then swaps the labels of the paired labels assigned to the feature points p22 and p12 with each other. That is, the label of the feature point p22 is rewritten to "left elbow," and the label of the feature point p12 is rewritten to "right elbow" (see FIG. 8B). The processes of steps S101 and S102 are repeated each time the user specifies a feature point. Note that, as is clear from the above description, the same rewriting process is performed even if the user specifies the feature point p12 instead of the feature point p22.

[0068] The user may specify feature points by clicking with the mouse, or by specifying an area as shown in FIG. 10B. On the display screen s2 shown in FIG. 10B, the user specifies an area including feature points p21 to p23 whose labels he wishes to modify by dragging the mouse. This causes the feature point input unit 115 to accept the input of feature points p21 to p23. The label rewriting unit 117 then performs processing to swap the labels of feature points p21, p22, and p23 with those of their paired feature points p11, p12, and p13, respectively. Note that when specifying a range, the user may also select feature points p11 to p13 all at once. In this case, the same processing result is obtained.

[0069] (Second Method) A first body part may include multiple first feature points. For example, in the example shown in FIG. 5 , the left arm includes feature points p21 to p23 labeled as the left wrist, left elbow, and left shoulder. In the second method, the user designates one of the first feature points to be rewritten, and the first body part is treated as being designated. Alternatively, in the second method, all of the first feature points included in the first body part to which the designated feature point belongs are treated as being designated. In other words, the support device 10 collectively rewrites the labels of multiple feature points included in the first body part to which the designated first feature point belongs. Note that the center point of multiple feature points belonging to the first body part may be registered. In this case, when a drag operation described below is performed on the center point, the first body part is treated as being selected. Furthermore, for center points, the label name may include words that can identify the axis of symmetry and the feature point, such as center, right, or left, so that the feature point input unit 115 can automatically determine that it is a center point.

[0070] Fig. 11 is a subroutine flowchart showing the rewriting process in step S08 of Fig. 3 in the second method. Fig. 12 is a diagram showing an operation procedure for specifying the first part on the display screen (GUI) displayed on the display unit 14 or the like.

[0071] (Step S201) The user refers to the label of the annotation target (person) displayed on the display screen and specifies a first body part that requires label correction. For example, in the example shown in FIG. 12 , on the display screen s3, the user drags and moves feature point p22 to overlap another feature point p12. This operation causes the feature point input unit 115 to treat the first body part (right arm or left arm) to which feature point p22 belongs as having been input. Alternatively to the example shown in FIG. 12 , a button for switching between body part selection and feature point selection may be placed on the display screen, allowing the user to specify body part selection using the button. In this case, similar to the operation shown in FIG. 10A , when one of the feature points of the first body part is selected by clicking, the feature point input unit 115 treats the first body part as having been selected.

[0072] (Step S202) The label rewriting unit 117 replaces the labels of the multiple first feature points p21, p22, and p23 belonging to the first region with the labels of the multiple second feature points p11, p12, and p13 belonging to the paired second region (see, for example, FIG. 8).

[0073] (Third Method) In the third method, a symmetry axis or a point on the symmetry axis related to the first feature point is registered and displayed as an icon on the display screen. When this icon is selected by the user, the first and second feature points are treated as having been selected. In other words, the assistance device 10 collectively rewrites the labels of multiple feature points linked to the symmetry axis.

[0074] Fig. 13 is a subroutine flowchart showing the rewriting process in step S08 of Fig. 3 in the third method. Fig. 14 is a diagram showing the operating procedure for specifying a point on the axis of symmetry on the display screen (GUI) displayed on the display unit 14 or the like.

[0075] (Step S301) The user refers to the label of the annotation target (person) displayed on the display screen, recognizes a first feature point that requires label correction, and operates an icon indicating a point on the symmetry axis associated with the first feature point. For example, in the example shown in FIG. 14 , the icon indicating a point on the symmetry axis (hereinafter referred to as the axis icon) on the display screen s5 is an icon of two rotating arrows with a double circle inside. The user selects the axis icon by clicking on it. With this operation, the feature point input unit 115 treats the multiple first feature points (feature points p11 to p13 or feature points p21 to p23) associated with the symmetry axis of the axis icon as having been selected.

[0076] (Step S302) The label rewriting unit 117 replaces the labels of the plurality of first feature points p21, p22, and p23 associated with the symmetry axis of the axis icon with the labels of the plurality of second feature points p11, p12, and p13 paired with each other (see, for example, FIG. 8).

[0077] (Fourth Method) The fourth method involves accepting the specification of an annotation target rather than specifying feature points, and treating the specification as specifying a plurality of first feature points in a specific range related to the annotation target. The specific range may be (1) all first feature points, (2) first feature points in the upper body, or (3) first feature points in the lower body of the annotation target (person). For example, this selection may be made by the user using a selection button arranged on a display screen (not shown). In the following description, the specific range is assumed to be (2) the first feature points in the upper body.

[0078] FIG. 15 is a subroutine flowchart showing the rewriting process in step S08 of FIG. 3 in the fourth technique.

[0079] (Step S401) The target designation unit 116 accepts designation of a target for label correction, for example, by clicking on a person on the display screen s1 shown in FIG.

[0080] (Step S402) The label rewriting unit 117 replaces the labels of the first feature points p21 to p23 of the upper body (specific range) of the annotation target with the labels of the paired second feature points p11 to p13, respectively.

[0081] As described above, the annotation support device according to this embodiment includes an input unit for inputting information about a first feature point included in a first part whose label should be corrected for an annotation target having a symmetry axis and multiple parts symmetrical with respect to the symmetry axis, and a label rewriting unit for rewriting the label of the first feature point to the label of a second part symmetrical to the first part with respect to the symmetry axis. This makes it easy to correct the labels of multiple feature points that are symmetrical with respect to the symmetry axis, such as left-right symmetry. This reduces the cost of correcting labels for left-right reversal.

[0082] (Processing When Only One Feature Point is Extracted) In the above explanation, an example was shown in which both the first and second feature points were extracted, but there are cases in which one of the feature points is hidden and cannot be seen due to the shooting range of the imaging device 20 and the positional relationship or posture of the annotation target. In this case, only one of the feature points will be extracted.

[0083] Fig. 16 is a diagram showing the procedure for specifying the center point on the axis of symmetry on the display screen. Fig. 17 is a table for explaining the rewriting of the parts and labels at each feature point corresponding to Fig. 16. In the examples shown in Figs. 16 and 17, some feature points p14 and p21 to p24 are detected, and other feature points are not detected.

[0084] On the display screen s6 in FIG. 16 , the user recognizes that feature points p21 to p23 are feature points whose labels need to be modified, and selects an axis icon. The feature point input unit 115 treats the multiple first feature points p21 to p23 linked to the symmetry axis of the axis icon as selected. The label rewriting unit 117 rewrites the labels of the multiple first feature points p21, p22, and p23 linked to the symmetry axis of the axis icon into paired labels. The paired labels are set in the table in FIG. 2 . The display screen s7 in FIG. 16 and FIG. 7(B) show the state after the labels have been rewritten into paired labels. Note that while the example in FIG. 16 illustrates an example of designating feature points using the fourth method, the other first to third methods may also be used for designation.

[0085] (First Modification) In the above description, an example has been described in which the label rewriting unit 117 rewrites or swaps the labels of input feature points into paired labels. However, the present invention is not limited to this, and the label rewriting unit 117 may swap the positions of feature points. For example, as shown in FIG. 18 , the label rewriting unit 117 may swap the positions of feature points to reassign labels. The first modification also provides the same effects as the above-described embodiment.

[0086] (Second Modification) When acquiring a moving image from the imaging device 20, the annotation support device 10 may also perform label rewriting processing on consecutive frames all at once.

[0087] 19 is a schematic diagram illustrating an example of batch rewriting of time-series frames. These are images captured during a predetermined period (time t-α to t+β). For example, if one second of video data is captured with a 15 or 30 FPS camera, one piece of time-series data will consist of 15 or 30 frames (still images).

[0088] For example, when a user performs label rewriting processing on a still image at time t using the annotation assistance method described above with reference to FIGS. 1 to 15, the user also performs label rewriting processing on the images before and after the time t all at once. The range of images before and after the time t may be within a predetermined time period (a predetermined number of images), or may be images in which the same object is detected within the predetermined time period. Alternatively, it may be limited to images in which the same number of feature points are detected for the same object. In this way, in the second modified example, labels are rewritten for both the images before and after the time t all at once, making it easier to correct multiple feature points.

[0089] The configuration of the annotation support device 10 described above is a description of the main configuration in explaining the features of the above embodiment, but is not limited to the above configuration and can be variously modified within the scope of the claims. Furthermore, configurations that are included in general devices are not excluded. The scope of the present invention should be interpreted by the scope of the attached claims.

[0090] Furthermore, the means and methods for performing various processes in the annotation support device 10 according to the above-described embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The above program may be provided, for example, by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The above program may also be provided as standalone application software, or may be incorporated into the software of the device as a function of the device.

[0091] 300 Information system 10 Annotation support device 11 Control unit 111 Image acquisition unit 112 Articulation point extraction unit 113 Annotation unit 114 Pair setting unit 115 Feature point input unit 116 Target designation unit 117 Label rewriting unit 118 Audio analysis unit 12 Storage unit 13 Communication unit 14 Display unit 15 Keyboard 16 Mouse 17 Microphone

Claims

1. An annotation support device that provides support for labeling feature points to be annotated included in an image, the annotation support device comprising: an input unit that inputs information of a first feature point whose label is to be corrected, the first feature point being included in a first part of the annotation target having a symmetry axis and a plurality of parts symmetric with respect to the symmetry axis; and a label rewriting unit that rewrites the label of the first feature point to the label of a second part symmetric to the first part with respect to the symmetry axis.

2. The plurality of first feature points are registered as the first part, and the label rewriting unit rewrites the labels of the plurality of first feature points included in the first part in response to the input of the first part to the input unit or the input of information of at least any one of the first feature points. The annotation support device according to claim 1.

3. The input of the information of the first feature point by the input unit includes at least any one of: specifying the position of the first feature point on a display image on which the annotation target and the feature point are displayed; specifying a region including the first feature point; specifying the symmetry axis related to the first feature point; and specifying a point on the symmetry axis related to the first feature point. The annotation support device according to claim 1.

4. Information of the first feature point from a voice analysis unit that analyzes the input voice is input to the input unit. The annotation support device according to claim 1.

5. The label rewriting unit rewrites the label in a batch for a plurality of frames. The annotation support device according to claim 1.

6. A second feature point symmetric to the first feature point with respect to the symmetry axis is set as a pair for the first feature point, and the label rewriting unit exchanges the labels of the first feature point and the second feature point. The annotation support device according to any one of claims 1 to 5.

7. The label rewriting unit exchanges the positions of the first feature point and the second feature point. The annotation support device according to claim 6.

8. The annotation support device according to claim 6, further comprising a target specifying unit that specifies the annotation target, wherein the label rewriting unit exchanges the labels of the first feature point and the second feature point for the specified annotation target.

9. The annotation support device according to claim 6, wherein the annotation target is a person, and the pair of the first feature point and the second feature point includes at least any one of a left shoulder and a right shoulder, a left elbow and a right elbow, a left wrist and a right wrist, a left fingertip and a right fingertip, a left waist and a right waist, a left knee and a right knee, a left ankle and a right ankle, a left toe and a right toe, a left eye and a right eye, and a left ear and a right ear.

10. A control program that operates on an annotation support device that supports labeling of feature points of an annotation target included in an image, the control program having a symmetry axis and including: an input step of inputting information of a first feature point whose label should be corrected, the first feature point being included in a first part of the annotation target having a plurality of parts symmetric with respect to the symmetry axis; and a label rewriting step of rewriting the label of the first feature point to a label of a second part symmetric to the first part with respect to the symmetry axis.

11. The control program according to claim 10, wherein a plurality of the first feature points are registered as the first part, and in the label rewriting step, the labels of the plurality of first feature points included in the first part are rewritten in response to the input of the first part in the input step.

12. The control program according to claim 10, wherein the input of the information of the first feature point in the input step includes at least any one of specifying the position of the first feature point, specifying an area including the first feature point, specifying the symmetry axis related to the first feature point, and specifying a point on the symmetry axis related to the first feature point on a display image on which the annotation target and the feature point are displayed.

13. The control program according to claim 10, wherein in the input step, the information of the first feature point from a voice analysis unit that analyzes the input voice is input.

14. In the label rewriting step, the label is rewritten in a batch for a plurality of frames. The control program according to claim 10.

15. A second feature point symmetric to the first feature point with respect to the symmetry axis is set as a pair for the first feature point. In the label rewriting step, the labels of the first feature point and the second feature point are swapped. The control program according to any one of claims 10 to 14.

16. In the label rewriting step, the positions of the first feature point and the second feature point are swapped. The control program according to claim 15.

17. Including a target designation step of designating the annotation target. In the label rewriting step, the labels of the first feature point and the second feature point are swapped for the designated annotation target. The control program according to claim 10.

18. The annotation target is a person, and the pair of the first feature point and the second feature point includes at least any one of the left shoulder and the right shoulder, the left elbow and the right elbow, the left wrist and the right wrist, the left fingertip and the right fingertip, the left waist and the right waist, the left knee and the right knee, the left ankle and the right ankle, the left toe and the right toe, the left eye and the right eye, and the left ear and the right ear. The control program according to claim 15.

Images