Annotation support device and control program

Abstract

To easily correct the labels of a plurality of feature points that are in a symmetrical relationship with respect to a symmetry axis. The annotation support device 10 includes an input unit 115 for inputting information on a first feature point, the label of which is to be corrected, that is included in a first part of an annotation target having a symmetry axis and a plurality of parts symmetrical with respect to the symmetry axis, and a label rewriting unit 117 for rewriting the label of the first feature point to a label of a second part symmetrical to the first part with respect to the symmetry axis.

Description

[Technical field]

[0001] The present invention relates to an annotation support device and a control program. [Background technology]

[0002] There is a technology that uses a deep neural network to detect a person's joint points and estimates the posture based on the detected joint points (for example, Patent Document 1). In order to correctly detect a person's joint points, a large amount of training data is required to train the deep neural network. In order to efficiently create and label this training data, there is a technology that performs auto-annotation on captured images. For example, annotation tools such as CVAT (Computer Vision Annotation Tool) and Supervisely are capable of performing auto-annotation on the joint points of a photographed person.

[0003] However, auto-annotation does not always give correct labels and can make mistakes. For example, when auto-annotating joint points from an image of a walking person, the upper body may be labeled relatively correctly, but the lower body may be inverted. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-86819 A Summary of the Invention [Problem to be solved by the invention]

[0005] Correcting such incorrect labels is costly, especially when there are multiple joint points. For example, if multiple joint points constituting the right and left feet are all annotated with the left and right reversed, multiple points must be corrected, which is costly.

[0006] The annotation tools described above have a function for automatically annotating feature points such as joint points, but do not provide a function for easily correcting the annotation.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an annotation support device and a control program that can easily modify the labels of multiple feature points that are symmetrical about an axis of symmetry, such as left-right symmetry. [Means for solving the problem]

[0008] The above object of the present invention can be achieved by the following means.

[0009] (1) Annotation target included in the image A person or animal as An annotation support device that supports labeling of feature points of an input unit for inputting information on a first feature point, the label of which is to be corrected, that is included in a first portion of the annotation target having a symmetry axis and a plurality of portions symmetrical with respect to the symmetry axis; a label rewriting unit that rewrites the label of the first feature point to a label of a second portion that is symmetric to the first portion with respect to the axis of symmetry; An annotation support device comprising:

[0010] (2) a plurality of the first feature points are registered as the first region; the label rewriting unit rewrites labels of the plurality of first feature points included in the first region in response to an input of the first region or an input of information of at least any of the first feature points to the input unit. An annotation support device according to (1) above.

[0011] (3) The input of the information on the first feature point by the input unit includes: The annotation support device described in (1) above, comprising at least one of specifying the position of the first feature point on a display image in which the annotation target and the feature point are displayed, specifying an area including the first feature point, specifying the axis of symmetry regarding the first feature point, and specifying a point on the axis of symmetry regarding the first feature point.

[0012] (4) information on the first feature point is input to the input unit from a voice analysis unit that analyzes an input voice; An annotation support device according to (1) above.

[0013] (5) the label rewriting unit rewrites the labels for a plurality of frames at once; An annotation support device according to (1) above.

[0014] (6) a second feature point symmetrical with respect to the axis of symmetry is set as a pair with the first feature point; The annotation support device according to any one of (1) to 5 above, wherein the label rewriting unit swaps the labels of the first feature point and the second feature point.

[0015] (7) The annotation support device according to (6) above, wherein the label rewriting unit swaps positions of the first feature point and the second feature point.

[0016] (8) A target designation unit for designating a target for annotation, The annotation support device according to (6) above, wherein the label rewriting unit swaps the labels of the first feature point and the second feature point for the specified annotation target.

[0017] (9) before The annotation support device described in (6) above, wherein the pairs of the first and second feature points include at least any of left shoulder and right shoulder, left elbow and right elbow, left wrist and right wrist, left fingertip and right fingertip, left hip and right hip, left knee and right knee, left ankle and right ankle, left toe and right toe, left eye and right eye, and left ear and right ear.

[0018] (10) Annotation targets included in images A person or animal as A control program that operates on an annotation support device that supports labeling of feature points of an input step of inputting information on a first feature point, the label of which is to be corrected, that is included in a first portion of the annotation target having a symmetry axis and a plurality of portions symmetrical with respect to the symmetry axis; a label rewriting step of rewriting the label of the first feature point to a label of a second part symmetric to the first part with respect to the axis of symmetry; A control program for executing the process including the steps of:

[0019] (11) A plurality of the first feature points are registered as the first region; in the label rewriting step, labels of the plurality of first feature points included in the first region are rewritten in response to the input of the first region in the input step; The control program according to (10) above.

[0020] (12) The input of the information on the first feature point in the input step includes: The control program according to (10) above, comprising at least any one of specifying a position of the first feature point on a display image in which the annotation target and the feature point are displayed, specifying a region including the first feature point, specifying the axis of symmetry with respect to the first feature point, and specifying a point on the axis of symmetry with respect to the first feature point.

[0021] (13) In the input step, information on the first feature point is input from a voice analysis unit that analyzes an input voice. The control program according to (10) above.

[0022] (14) In the label rewriting step, the labels are rewritten simultaneously for a plurality of frames. The control program according to (10) above.

[0023] (15) A second feature point symmetrical with respect to the axis of symmetry is set as a pair with the first feature point; 15. The control program according to any one of (10) to 14, wherein in the label rewriting step, labels of the first feature point and the second feature point are swapped.

[0024] (16) The control program according to (15) above, wherein in the label rewriting step, positions of the first feature point and the second feature point are swapped.

[0025] (17) A target designation step of designating the annotation target, The control program according to (10) above, wherein in the label rewriting step, labels of the first feature point and the second feature point are swapped for the designated annotation target.

[0026] (18) before The control program also described in (15) above, wherein the pairs of the first and second feature points include at least any of left shoulder and right shoulder, left elbow and right elbow, left wrist and right wrist, left fingertip and right fingertip, left hip and right hip, left knee and right knee, left ankle and right ankle, left toe and right toe, left eye and right eye, and left ear and right ear. Effect of the Invention

[0027] The annotation support device according to the present invention includes an input unit for inputting information on a first feature point, the label of which is to be corrected, that is included in a first part of an annotation target having an axis of symmetry and a plurality of parts symmetrical with respect to the axis of symmetry, and a label rewriting unit for rewriting the label of the first feature point to the label of a second part symmetrical to the first part with respect to the axis of symmetry. In this way, the labels of the plurality of feature points that are in a symmetrical relationship can be easily corrected. [Brief description of the drawings]

[0028] [Figure 1] 1 is a diagram showing a schematic configuration of an annotation support device according to an embodiment of the present invention. [Diagram 2]1 is a table showing a pair relationship of labels that are symmetrical with respect to an axis of symmetry. [Diagram 3] 1 is a flowchart showing an outline of a processing procedure of an annotation assistance method. [Figure 4] FIG. 13 is a schematic diagram showing a state in which a correct label has been added by annotation. [Diagram 5] 5 is a table showing the relationship between the parts and the labels at each feature point in FIG. 4. [Figure 6] 13 is a table showing the relationship between the parts and the labels at each feature point in the modified example. [Figure 7] FIG. 13 is a schematic diagram illustrating rewriting of an incorrect label. [Figure 8] 8 is a table for explaining rewriting of the parts and labels at each feature point corresponding to FIG. 7. [Figure 9] 4 is a subroutine flowchart showing the rewriting process in step S08 of FIG. 3 in the first technique. [Figure 10A] 6A to 6C are diagrams showing a procedure for designating a first feature point or a first portion on a display screen. [Figure 10B] 11A to 11C are diagrams showing a procedure for designating a first feature point or a first portion on a display screen. [Figure 11] 4 is a subroutine flowchart showing the rewriting process in step S08 of FIG. 3 in the second technique. [Figure 12] 11A to 11C are diagrams showing a procedure for designating a first feature point or a first portion on a display screen. [Figure 13] 10 is a subroutine flowchart showing the rewriting process in step S08 of FIG. 3 in the third technique. [Figure 14] 13A to 13C are diagrams showing a procedure for specifying a center point on an axis of symmetry on a display screen. [Figure 15] 13 is a subroutine flowchart showing the rewriting process in step S08 of FIG. 3 in the fourth technique. [Figure 16] 13A to 13C are diagrams showing a procedure for specifying a center point on a symmetrical axis on a display screen. [Figure 17] 17 is a table for explaining rewriting of the parts and labels at each feature point corresponding to FIG. 16. [Figure 18] 9 is a table for explaining rewriting of each feature point corresponding to FIG. 8 in the modified example. [Figure 19] FIG. 13 is a schematic diagram illustrating an example of collectively rewriting frames in a time series. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings. However, the scope of the present invention is not limited to the disclosed embodiment. In the description of the drawings, the same elements are given the same reference numerals, and duplicated descriptions are omitted. Advantages and features provided by one or more embodiments of the present invention will be fully understood from the detailed description set forth below and the attached drawings. In addition, the attached drawings are provided for the purpose of explanation only, and are not intended to define the scope of the present invention.

[0030] 1 is a diagram showing a schematic configuration of an annotation support device 10 according to this embodiment. An information system 300 includes the annotation support device 10 (hereinafter also simply referred to as the support device 10) and an imaging device 20. The support device 10 and the imaging device 20 are connected for communication. The annotation support device 10 is a PC (personal computer) or a server. When configured as a server, some of the functions may be carried out by a cloud server on a network.

[0031] The imaging device 20 includes a control unit, a communication unit, a camera, and the like. The camera is, for example, a visible light camera. The visible light camera captures an image of a predetermined area by receiving visible light reflected by an object in the predetermined area using a CMOS sensor. The camera may be configured using a near-infrared camera instead of a visible light camera. The imaging device 20 captures a video consisting of a plurality of captured images (frames) that are consecutive in time, for example, at a frame rate of 15 fps to 60 fps. The captured images are sent to the support device 10.

[0032] The support device 10 includes a control unit 11, a memory unit 12, a communication unit 13, a display unit 14, a keyboard 15, a mouse 16, and a microphone 17. The components in the support device 10 are electrically connected to each other, but the connections are not shown in FIG.

[0033] The control unit 11 is composed of a CPU, a GPU, a RAM, a ROM, etc. The control unit 11 functions as an image acquisition unit 111 in cooperation with the communication unit 13. The control unit 11 also functions as a joint point extraction unit 112, an annotation unit 113, a pair setting unit 114, a feature point input unit 115, a target designation unit 116, a label rewriting unit 117, and a voice analysis unit 118 in cooperation with other components such as the display unit 14 and the mouse 16. Each of these functions will be described later.

[0034] The storage unit 12 is composed of an SSD, a HDD, etc. The storage unit 12 stores various programs, a machine-learned model that extracts articulation points, a table that describes the pair relationships of labels (see FIG. 2, which will be described later), and the like.

[0035] The communication unit 13 is an interface circuit (for example, a LAN card) for communicating with other devices such as the imaging device 20 via a network.

[0036] The display unit 14 is configured with a display such as an LCD, and displays a display screen (GUI: Graphical User Interface) used for inputting the first feature point, etc. The display unit 14 may also include a touch screen.

[0037] The keyboard 15 and mouse 16 are used for inputting instructions from the user, and the microphone 17 receives voice input from the user.

[0038] (Image acquisition unit 111) The image acquisition unit 111 acquires an image captured by the imaging device 20. The captured image is associated with the capture time and stored in the storage unit 12. The image acquisition unit 111 may also acquire an image from another device connected via a network.

[0039] (Joint point extraction unit 112) The joint point extraction unit 112 detects skeletal information of an object such as a person included in the image acquired by the image acquisition unit 111. The joint point extraction unit 112 detects the joint points using a trained model for detecting the joint points of a person from a rectangle including a person (hereinafter also referred to as a "person rectangle"). The person rectangle is an area that contains the joint points and nodes of a person in an image. As such trained models, for example, models such as OpenPose (https: / / arxiv.org / abs / 1812.08008) and Deep Pose (https: / / arxiv.org / abs / 1312.4659) are known.

[0040] (Annotation section 113) The annotation unit 113 performs auto-annotation on the joint points extracted by the joint point extraction unit 112. For example, the auto-annotation is performed using CVAT. An example of the label of each joint point (also called a feature point) assigned by annotation will be described later (see FIG. 5 ,figure 6 etc.).

[0041] (Pair setting unit 114) The pair setting unit 114 sets a pair of two feature points that are in a symmetrical positional relationship with respect to the axis of symmetry of the object. In the case of a person, the axis of symmetry is the central axis of the body. 2is a table showing pair relationships. This table is set by the user and stored in advance in the storage unit 12. 2 In this example, the object is a person, and the feature points are joint points. 2 As shown in the figure, the right shoulder and the left shoulder are in a paired relationship. Similarly, the right elbow and the left elbow are in a paired relationship. 2 are merely examples, and other feature points may be included, or only some of them may be included. In addition, one side with respect to the axis of symmetry is a first feature point (or a first portion), and the other side is a second feature point (or a second portion). 2 In the example shown in FIG. 1 and hereinafter, unless otherwise specified, the left side of the left and right sides will be referred to as the first feature point or first region, and the right side will be referred to as the second feature point or second region.

[0042] (Feature point input unit 115) The feature point input unit 115 accepts input of information on feature points for which labels are to be rewritten, input via the mouse 16, the keyboard 15, etc. to a display screen (such as a display screen s1 in FIG. 10A described below) displayed on the display unit 14. The "information on feature points" here includes the following information for specifying one or more feature points for which labels are to be rewritten: (1a) Information specifying the positions of feature points (an example of FIG. 10A described later). (1b) Information designating an area including feature points (an example of FIG. 10B described below). (1c) Specifying an axis of symmetry with respect to a feature point. (1d) Specifying a point on the axis of symmetry for a feature point (example in Figure 14 below).

[0043] When one feature point is specified by inputting feature point information, the feature point whose label is to be rewritten may be one of the following: (2a) When only the feature point is rewritten as the label of the pair (example in Figure 16 described later). (2b) The feature point and other feature points belonging to the same part are rewritten as a pair of labels (see the example in Figure 15 below). (2c) The labels of the feature point and its paired labeled feature point are swapped (see the example in Figure 9 below). (2d) The label of that feature point and other feature points belonging to the same part are replaced with the label of the feature point that has the paired label assigned to it (example in Figure 12 below).

[0044] (Target designation section 116) The target designation unit 116 accepts designation of an annotation target for which a label is to be rewritten by input via the mouse 16, the keyboard 15, etc. to a display screen for an operation (such as a display screen s1 in FIG. 10 described later) displayed on the display unit 14. For the designated target, the labels of feature points in a specific range are replaced with paired labels (see FIG. 15 described later).

[0045] (Label rewriting unit 117) The label rewriting unit 117 rewrites the labels based on the information on the feature points input by the feature point input unit 115 or the target designation unit 116 or the information on the designated annotation target. This label rewriting includes both rewriting the label of a first feature point to the label of the other pair (an example in FIG. 17 described later) and mutually swapping the labels of the first and second feature points (FIG. 8 and the like described later). As another example, the label rewriting unit 117 may perform processing to swap the positions of feature points to which labels that form a pair are assigned (an example in FIG. 18 described later).

[0046] (Audio Analysis Unit 118) When the label rewrite mode is being executed, the voice analysis unit 118 analyzes the voice of the user input from the microphone 17, and obtains text data by voice recognition processing. Well-known processing can be applied for the voice recognition processing. By matching the text data with dictionary data related to commands, instructions are given to the feature point input unit 115 or the target designation unit 116.

[0047] (Annotation support method) The annotation support method will be described with reference to Fig. 3 to Fig. 15. Fig. 3 is a flowchart showing the annotation support method.

[0048] (Step S01) The annotation support device 10 acquires image data captured by an image capture device 20 or the like. For example, the image acquisition unit 111 acquires image data captured by continuous capture from the image capture device 20. was Acquire frame images in time series.

[0049] (Step S02) The joint point extraction unit 112 detects an object from an image using a trained model trained by machine learning. For example, a person (person) is detected from an image as the object. Note that, although the following description will be given of a case where a person is used as the annotation object (object), the present invention is not limited to this and may be applied to animals other than humans that have a skeleton, such as cats and dogs.

[0050] (Step S03) The joint point extraction unit 112 extracts (estimates) joint points of the person detected in step S02 using a known trained model such as Deep Pose. The annotation unit 113 assigns labels for auto-annotation to the joint points.

[0051] (Step S04) The control unit 11 displays the image with the labels added on the display unit 14. Fig. 4 is a schematic diagram showing a state in which correct labels have been added by the processing up to this point. Fig. 5 is a table showing the relationship between the parts and the labels at each feature point in Fig. 4.

[0052] FIG. 4 is a schematic diagram showing an image of a person about to hit a ball in front of him with a bat, and an annotation label superimposed on the image. In FIG. 4, p11 to p26 indicate feature points, which correspond to the feature points in the table in FIG. 5. In FIG. 5, for example, feature point p11 is shown to be a second feature point and a feature point of the right wrist belonging to the part of the right arm. Feature points p11 to p13 are a group of multiple feature points belonging to the same part of the right arm, and are labeled with the right wrist, right elbow, and right shoulder, respectively. As described above, the feature point on the left side is referred to as the first feature point or the first part, and the feature point on the right side is referred to as the second feature point or the second part, but this is merely an example, and the right side may be the first feature point or the first part.

[0053] In addition, in the example of Fig. 5, a single part is described as including multiple feature points, but a single part may include only one feature point. For example, Fig. 6 is a table showing the relationship between each part and feature points in a modified example. In the example of Fig. 6, the right shoulder part includes only the feature point labeled right shoulder.

[0054] (Step S05) The user can check whether the correct label has been added by seeing the content as shown in Fig. 4 displayed on the display surface of the display unit 14. If the correct label has been added as in the example of Fig. 4, no correction is required (NO), and the process ends.

[0055] On the other hand, if label correction is necessary (YES), the user launches the correction application and transitions to the label rewriting mode, which causes the control unit 11 to proceed to step S06.

[0056] (Step S06) The pair setting unit 114 sets pairs of feature points that are in a symmetrical relationship. This is performed using the table shown in Fig. 2. For example, in the examples of Figs. 4 and 5, the pair setting unit 114 pairs feature point p11, whose label is the right wrist, with feature point p21, whose label is paired with the label of the left wrist. Similarly, in the example of Fig. 5, feature point p12 and feature point p22 are set as a pair. Similarly, feature points p13 to 16 and feature points p23 to 26 are each set as a pair.

[0057] (Step S07) The control unit 11 causes the display unit 14 to display the content as shown in FIG. 4 on its display surface, and also causes the display unit 14 to display a display screen for accepting input of information on feature points by the user.

[0058] (Step S08) Here, the control unit 11 performs a process of rewriting the labels of the feature points based on the user's input of information about the feature points.

[0059] There are four methods for inputting information about the user's feature points: Methods 1 to 4, which will be described later. Each method will be described later, but the overall outline is as follows.

[0060] Fig. 7 is a schematic diagram for explaining rewriting of an incorrect label, and a table for explaining rewriting of the parts and labels at each feature point corresponding to Fig. 7.

[0061] Fig. 7(A) and Fig. 8(A) show the state where the wrong label is assigned, and Fig. 7(B) and Fig. 8(B) show the state after the label is changed. The area surrounded by the dashed line in Fig. 7 indicates the range of feature points to which the wrong label is assigned, and similarly, the area surrounded by the dashed line in Fig. 8 indicates the corresponding feature points.

[0062] The feature point input unit 115 accepts input of feature points p21 to p23 (or feature points p11 to p13) from the user. In response to this, the label rewriting unit 117 refers to the labels paired by the pair setting unit 114 (FIG. 2).

[0063] Then, the label rewriting unit 117 extracts the feature points p21 to p23 and the feature points p11 to p13 that were set as pairs in step S06, and exchanges the labels with each other. Alternatively, the label rewriting unit 117 extracts the feature points p21 to p23 to which the labels (left wrist, left elbow, left shoulder) that are paired with the labels (right wrist, right elbow, right shoulder) that are assigned to the feature points p11 to p13 are assigned. Then, the labels of these feature points are exchanged with each other. That is, the label rewriting unit 117 rewrites the left wrist label of the feature point p11 to the right wrist label (see FIG. 2), and rewrites the label of the extracted feature point p21 (to which the right wrist label is assigned) to the left wrist. The label rewriting unit 117 performs the same process on the selected feature points p12 and p13, and rewrites the labels of the feature points p12, p13, p22, and p23. The above is an overview of the label rewriting process. Next, input of feature point information using each method will be described.

[0064] (First method) In the first method, the user designates each feature point to be rewritten. Fig. 9 is a subroutine flowchart showing the rewriting process in step S08 in Fig. 3 in the first method. Figs. 10A and 10B are diagrams showing the operation procedure for designating the first feature point on a display screen (GUI) displayed on the display unit 14 or the like.

[0065] (Step S101) The user refers to the label of the annotation target (person) displayed on the display screen and specifies the feature point that requires label correction. For example, the feature point is specified by clicking on the feature point on the display screen s1 of FIG. 10A. Note that in the display screen s1 of FIG. 10A, the left and right feature points are shown as label marks by line type, density, color, mark shape, etc., so that they can be distinguished from each other, but in addition to this, the labels may be displayed in text (the same applies to other display screens such as FIG. 10B). Also, by bringing the cursor close to one of the feature points, the text data of the label of each feature point may be displayed.

[0066] The feature point input unit 115 receives an input of a specified feature point in response to a click operation by the user. For example, it is assumed that the user has selected feature point p22 as shown in FIG 10A.

[0067] (Step S102) The label rewriting unit 117 recognizes the label (right elbow) given to the feature point p22 input by the user. This label is an incorrect label that the user wants to correct. The label rewriting unit 117 refers to the table of FIG. 2 set by the pair setting unit 114, and extracts the feature point p12 to which the label (left elbow) paired with this label (right elbow) is given. The label rewriting unit 117 then exchanges the labels of the feature points p22 and p12 to which the paired labels are given. That is, the label of the feature point p22 is rewritten to “left elbow”, and the label of the feature point p12 is rewritten to “right elbow” (see FIG. 8(B)). Each time the user specifies a feature point, the processes of steps S101 and S102 are repeated. Note that, as is clear from the above explanation, the same rewriting process is performed even if the user specifies the feature point p12 instead of the feature point p22.

[0068] The user may specify feature points by clicking with the mouse, or by specifying an area as shown in FIG. 10B. On the display screen s2 shown in FIG. 10B, the user specifies an area including feature points p21 to p23 whose labels are to be modified by dragging the mouse. This causes the feature point input unit 115 to accept the input of feature points p21 to p23. Then, the label rewriting unit 117 performs processing to exchange the labels of feature points p21, p22, and p23 with those of their pairs, p11, p12, and p13, respectively. Note that when specifying a range, the user may also select feature points p11 to p13 collectively. In this case, the same processing result is obtained.

[0069] (Second method) A first part may include a plurality of first feature points. For example, in the example shown in FIG. 5, the part of the left arm includes feature points p21 to p23 labeled with the left wrist, left elbow, and left shoulder. In the second method, the user designates one of the first feature points to be rewritten, and the first part is treated as being designated. Alternatively, in the second method, all of the first feature points included in the first part to which the feature point belongs are treated as being designated. That is, the support device 10 collectively rewrites the labels of a plurality of feature points included in the first part to which the designated first feature point belongs. Note that the center point of a plurality of feature points belonging to the first part may be registered. In this case, when a drag operation described below is performed on the center point, the first part is treated as being selected. In addition, the label name of the center point may include a word that can identify the axis of interest and the feature point, such as center, right, and left, so that the feature point input unit 115 can automatically determine that it is the center point.

[0070] Fig. 11 is a subroutine flowchart showing the rewriting process in step S08 in Fig. 3 in the second method. Fig. 12 is a diagram showing an operation procedure for designating the first part on the display screen (GUI) displayed on the display unit 14 or the like.

[0071] (Step S201) The user refers to the label of the annotation target (person) displayed on the display screen and specifies the first body part that needs label correction. For example, in the example shown in FIG. 12, the user drags and moves the feature point p22 on the display screen s3 to overlap with another feature point p12. By this operation, the feature point input unit 115 treats the first body part (right arm or left arm) to which the feature point p22 belongs as having been input. Note that, instead of the example shown in FIG. 12, a button for switching between body part selection / feature point selection may be placed on the display screen so that the body part selection can be specified by the button. In this case, as in the operation of FIG. 10A, when any of the feature points of the first body part is selected by a click operation, the feature point input unit 115 treats the first body part as having been selected.

[0072] (Step S202) The label rewriting unit 117 replaces the labels of the first feature points p21, p22, and p23 belonging to the first region with the labels of the second feature points p11, p12, and p13 belonging to the second region of the pair (see FIG. 8, for example).

[0073] (Third method) The third method is to register a symmetry axis or a point on the symmetry axis with respect to the first feature point and display it as an icon on the display screen. When the icon is selected by the user, the first and second feature points are treated as having been selected. In other words, the support device 10 collectively rewrites the labels of multiple feature points linked to the symmetry axis.

[0074] Fig. 13 is a subroutine flowchart showing the rewriting process of step S08 in Fig. 3 in the third method. Fig. 14 is a diagram showing an operation procedure for specifying a point on the axis of symmetry on the display screen (GUI) displayed on the display unit 14 or the like.

[0075] (Step S301) The user refers to the label of the annotation target (person) displayed on the display screen, recognizes a first feature point that needs label correction, and operates an icon indicating a point on the symmetric axis associated with the first feature point. For example, in the example shown in FIG. 14, the icon indicating a point on the symmetric axis on the display screen s5 (hereinafter referred to as the axis icon) is an icon of two rotating arrows and a double circle inside the arrows. The axis icon is selected by the user clicking the axis icon. With this operation, the feature point input unit 115 treats the multiple first feature points (feature points p11 to p13, or feature points p21 to p23) associated with the symmetric axis of the axis icon as having been selected.

[0076] (Step S302) The label rewriting unit 117 replaces the labels of the first feature points p21, p22, and p23 associated with the symmetry axis of the axis icon with the respective labels of the second feature points p11, p12, and p13 that are paired with the first feature points (see FIG. 8, for example).

[0077] (Fourth method) The fourth method does not specify feature points, but accepts the specification of an annotation target, and handles the specification as specifying a plurality of first feature points in a specific range related to the annotation target. The specific range may be any of (1) all first feature points, (2) first feature points in the upper body, or (3) first feature points in the lower body of the annotation target (person). For example, this selection may be made by the user using a selection button arranged on a display screen (not shown). In the following description, it is assumed that the specific range is (2) the first feature points in the upper body.

[0078] FIG. 15 is a subroutine flowchart showing the rewriting process in step S08 of FIG. 3 in the fourth technique.

[0079] (Step S401) The target designation unit 116 accepts the designation of a target for label correction. For example, the target designation unit 116 accepts a click operation on a person on a display screen s1 as shown in FIG.

[0080] (Step S402) The label rewriting unit 117 replaces the labels of the first feature points p21 to p23 of the upper body (specific area) of the annotation target with the labels of the paired second feature points p11 to p13, respectively.

[0081] As described above, the annotation support device according to this embodiment includes an input unit for inputting information on a first feature point, the label of which is to be corrected, that is included in a first part of an annotation target having an axis of symmetry and a plurality of parts symmetrical with respect to the axis of symmetry, and a label rewriting unit for rewriting the label of the first feature point to the label of a second part symmetrical to the first part with respect to the axis of symmetry. This makes it easy to correct the labels of a plurality of feature points that are in a symmetrical relationship with respect to the axis of symmetry, such as left-right symmetry. It is possible to reduce the cost of correcting the labels for left-right inversion.

[0082] (Processing when only one feature point is extracted) In the above explanation, an example in which both the first and second feature points are extracted has been shown, but one of the feature points may be hidden and invisible due to the shooting range of the imaging device 20 and the positional relationship or posture of the annotation target. In this case, only one of the feature points will be extracted.

[0083] Fig. 16 is a diagram showing a procedure for specifying a center point on a symmetrical axis on a display screen. Fig. 17 is a table for explaining rewriting of the parts and labels at each feature point corresponding to Fig. 16. In the examples shown in Figs. 16 and 17, some feature points p14, p21 to p24 are detected, and other feature points are not detected.

[0084] On the display screen s6 of FIG. 16, the user recognizes that the feature points p21 to p23 are feature points that require label correction, and selects an axis icon. The feature point input unit 115 handles the multiple first feature points p21 to p23 linked to the symmetric axis of the axis icon as selected. The label rewriting unit 117 rewrites each of the multiple first feature points p21, p22, and p23 linked to the symmetric axis of the axis icon to a pair label. The pair label is set in the table of FIG. 2. The display screen s7 of FIG. 16 and FIG. 7(B) show a state after the label is rewritten to the pair label. Note that, although an example of designating feature points by the fourth method is shown in the example of FIG. 16, designation by the other first to third methods may also be used.

[0085] (First Modification) In the above description, an example has been described in which the label rewriting unit 117 performs a process of rewriting or replacing the label of an input feature point with a pair of labels. However, the present invention is not limited to this, and the label rewriting unit 117 may replace the positions of feature points. For example, as shown in FIG. 18, the label rewriting unit 117 may replace the labels by replacing the positions of feature points. The first modification also provides the same effects as the above embodiment.

[0086] (Second Modification) When the annotation support device 10 acquires a video from the imaging device 20, the annotation support device 10 may perform label rewriting processing on consecutive frames all at once.

[0087] 19 is a schematic diagram for explaining an example of rewriting time-series frames all at once. These are images captured during a predetermined period (time t-α to t+β). For example, in the case of one second of video data captured by a camera at 15 or 30 FPS, one piece of time-series data is made up of 15 or 30 frames (still images).

[0088] For example, when a user performs label rewriting processing on a still image at time t using the annotation assistance method described above with reference to Figs. 1 to 15, the user performs label rewriting processing on the images before and after the still image at the same time. The range of images before and after the still image may be within a predetermined time period (a predetermined number of images), or may be images in which the same object is detected within a predetermined time period. Alternatively, the range may be limited to images in which the same number of feature points are detected for the same object. In this way, in the second modified example, the labels of the images before and after the still image are rewritten at the same time, making it easier to correct multiple feature points.

[0089] The configuration of the annotation support device 10 described above is a main configuration for explaining the features of the above embodiment, but is not limited to the above configuration and can be modified in various ways within the scope of the claims. Furthermore, configurations that are included in general devices are not excluded. The scope of the present invention should be interpreted by the scope of the attached claims.

[0090] In addition, the means and methods for performing various processes in the annotation support device 10 according to the above-described embodiment can be realized by either a dedicated hardware circuit or a programmed computer. The above-mentioned program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded in the computer-readable recording medium is usually transferred to and stored in a storage unit such as a hard disk. In addition, the above-mentioned program may be provided as a standalone application software, or may be incorporated into the software of the device as one of its functions. [Explanation of symbols]

[0091] 300 Information Systems 10 Annotation Support Device 11 Control section 111 Image acquisition unit 112 Joint Point Extraction Unit 113 Annotation Section 114 Pair Setting Section 115 Feature point input section 116 Target Designation Section 117 Label rewriting unit 118 Voice Analysis Section 12 Storage section 13. Communications Department 14 Display section 15 Keyboard 16 Mouse 17. Mike

Claims

1. An annotation support device that supports labeling of feature points of a person or an animal as an annotation target included in an image, comprising: an input unit for inputting information on a first feature point, the label of which is to be corrected, that is included in a first portion of the annotation target having a symmetry axis and a plurality of portions symmetrical with respect to the symmetry axis; a label rewriting unit that rewrites the label of the first feature point to a label of a second portion symmetrical to the first portion with respect to the axis of symmetry; An annotation support device comprising:

2. A plurality of the first feature points are registered as the first region, the label rewriting unit rewrites labels of the plurality of first feature points included in the first region in response to an input of the first region or an input of information of at least any of the first feature points to the input unit; The annotation support device according to claim 1 .

3. The input of the information on the first feature point by the input unit includes:

2. The annotation support device according to claim 1, further comprising at least any one of: specifying a position of the first feature point on a display image on which the annotation target and the feature point are displayed; specifying an area including the first feature point; specifying the axis of symmetry with respect to the first feature point; and specifying a point on the axis of symmetry with respect to the first feature point.

4. The input unit receives information on the first feature point from a voice analysis unit that analyzes an input voice. The annotation support device according to claim 1 .

5. the label rewriting unit rewrites the labels for a plurality of frames at once; The annotation support device according to claim 1 .

6. a second feature point symmetrical with respect to the axis of symmetry is set as a pair with the first feature point; The annotation support device according to claim 1 , wherein the label rewriting unit exchanges labels of the first feature point and the second feature point.

7. The annotation support device according to claim 6 , wherein the label rewriting unit swaps positions of the first feature point and the second feature point.

8. a target designation unit for designating the annotation target, The annotation support device according to claim 6 , wherein the label rewriting unit exchanges labels of the first feature point and the second feature point for the specified annotation target.

9. The annotation support device described in claim 6, wherein the pairs of the first feature point and the second feature point include at least any of left shoulder and right shoulder, left elbow and right elbow, left wrist and right wrist, left fingertips and right fingertips, left hip and right hip, left knee and right knee, left ankle and right ankle, left toes and right toes, left eye and right eye, and left ear and right ear.

10. A control program that operates on an annotation support device that supports labeling of feature points of a person or an animal as an annotation target included in an image, the control program comprising: an input step of inputting information on a first feature point, the first feature point being included in a first portion of the annotation target and the label of which is to be corrected, for the annotation target having a symmetry axis and a plurality of portions symmetrical with respect to the symmetry axis; a label rewriting step of rewriting the label of the first feature point to a label of a second part symmetric to the first part with respect to the axis of symmetry; A control program for executing the process including the steps of:

11. A plurality of the first feature points are registered as the first region, In the label rewriting step, labels of the plurality of first feature points included in the first region are rewritten in response to the input of the first region in the input step. The control program according to claim 10.

12. The input of the information on the first feature point in the input step includes:

11. The control program according to claim 10, further comprising at least one of specifying a position of the first feature point on a display image in which the annotation target and the feature point are displayed, specifying a region including the first feature point, specifying the axis of symmetry with respect to the first feature point, and specifying a point on the axis of symmetry with respect to the first feature point.

13. In the input step, information on the first feature point is input from a voice analysis unit that analyzes an input voice. The control program according to claim 10.

14. In the label rewriting step, the labels are rewritten simultaneously for a plurality of frames. The control program according to claim 10.

15. a second feature point symmetrical with respect to the axis of symmetry is set as a pair with the first feature point; 15. The control program according to claim 10, wherein in the label rewriting step, labels of the first feature point and the second feature point are swapped.

16. The control program according to claim 15 , wherein in the label rewriting step, positions of the first feature point and the second feature point are swapped.

17. a target designation step of designating the annotation target; The control program according to claim 10 , wherein in the label rewriting step, labels of the first feature point and the second feature point are swapped for the designated annotation target.

18. The control program described in claim 15, wherein the pairs of first and second feature points include at least any of left shoulder and right shoulder, left elbow and right elbow, left wrist and right wrist, left fingertips and right fingertips, left hip and right hip, left knee and right knee, left ankle and right ankle, left toes and right toes, left eye and right eye, and left ear and right ear.