Information processing device, information processing method, and program

Abstract

Provided are an information processing device, an information processing system, an information processing method, and a program with which it is possible for a user to efficiently add annotation information to images. An information processing device (1) comprises: a configuration information acquisition unit (2) for acquiring object configuration information that includes the definition of each of one or more elements added to an image in order to indicate an object included in the image and the definition of the order of the elements; a user interface unit (3) for providing, in order to accept an input for designating the elements with regard to the object, a user interface having been set on the basis of the definition of the elements indicated by the object configuration information, in accordance with the order of the elements that is defined in the object configuration information; and a storage control unit (4) for exercising control so that information of the elements designated by the input is stored in association with the image.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, an information processing method, and a non-transitory computer-readable medium, and more particularly to an information processing device, an information processing system, an information processing method, and a non-transitory computer-readable medium for a user to perform an operation of adding annotations to an image. [Background technology]

[0002] In recent years, machine learning models have been used in various technical fields, including technical fields related to authentication such as iris recognition and face recognition. Training data is used for training a machine learning model. When training a machine learning model using image data as training data, it is necessary to prepare training data in which annotation information is added to the image data. In this disclosure, annotation information refers to metadata added to image data. In other words, annotation information is supplementary information for expressing the content of the image data. In this regard, for example, Patent Document 1 discloses the following technology.

[0003] Patent Document 1 discloses a device that acquires character information, generates character images containing graphics corresponding to the character information, and adds the character information to the generated character images as annotation information to generate training data. This makes it possible to efficiently generate training data used in training a machine learning model for reading license plates of automobiles photographed by a camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-170213 Summary of the Invention [Problem to be solved by the invention]

[0005] Generally, when adding annotation information to image data, a user (worker) who views the displayed image performs an operation on the software to specify the annotation information to be added to the image, thereby generating training data.

[0006] However, because the amount of training data used to train machine learning models is enormous, the more operations required to annotate a single image, the greater the amount of work required to prepare a sufficient amount of training data required for training.

[0007] In contrast, the technology disclosed in Patent Document 1 automatically generates an image based on input character information and uses this character information as annotation information. Therefore, this technology does not support efficient user operations for adding annotation information to images.

[0008] Therefore, one of the objectives that the embodiments disclosed in this specification aim to achieve is to provide an information processing device, an information processing system, an information processing method, and a program that enable users to efficiently add annotation information to images. [Means for solving the problem]

[0009] An information processing device according to a first aspect of the present disclosure includes: a setting information acquisition means for acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface means for providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; a storage control means for controlling the storage of information on the element designated by the input in association with the image; It has.

[0010] An information processing system according to a second aspect of the present disclosure includes: an information processing device; Terminal device and Equipped with The information processing device includes: a setting information acquisition means for acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface means for providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; a storage control means for controlling the storage of information on the element designated by the input in association with the image; and The terminal device an output control means for outputting the user interface provided by the information processing device to an output device; an input receiving means for receiving an input to the user interface; It has.

[0011] In an information processing method according to a third aspect of the present disclosure, obtaining object setting information including definitions of one or more elements to be added to an image to indicate an object included in the image and a definition of the order of the elements; providing a user interface configured based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; Control is performed so that information on the element designated by the input is stored in association with the image.

[0012] A program according to a fourth aspect of the present disclosure includes: a setting information acquisition step of acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface step of providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information in order to receive an input for specifying the elements of the object; a storage control step of controlling the storage of information of the element designated by the input in association with the image; to be executed by the computer. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of an information processing device according to an outline of an embodiment. [Figure 2] FIG. 10 is a schematic diagram showing an example of annotation information about an object included in an image. [Figure 3] FIG. 10 is a schematic diagram showing an example of annotation information when the entire image is considered as one object. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment; [Figure 5] 1 is a block diagram illustrating an example of a hardware configuration of an information processing device according to a first embodiment. [Figure 6] 4 is a flowchart showing an example of an operation of the information processing device according to the first embodiment. [Figure 7] 3 is a schematic diagram showing an example of a user interface screen that is displayed on an output device by a user interface unit. FIG. [Figure 8] 10A and 10B are schematic diagrams illustrating an example of guide information that a user interface unit displays on an output device. [Figure 9] 3 is a schematic diagram showing an example of a user interface screen that is displayed on an output device by a user interface unit. FIG. [Figure 10]3 is a schematic diagram showing an example of a user interface screen that is displayed on an output device by a user interface unit. FIG. [Figure 11] 3 is a schematic diagram showing an example of a user interface screen that is displayed on an output device by a user interface unit. FIG. [Figure 12] 3 is a schematic diagram showing an example of a user interface screen that is displayed on an output device by a user interface unit. FIG. [Figure 13] FIG. 10 is a block diagram showing an example of a functional configuration of an information processing device according to a second embodiment. [Figure 14] 10 is a flowchart illustrating an example of an operation of the information processing device according to the second embodiment. [Figure 15] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a third embodiment. [Figure 16] 11 is a flowchart showing an example of an operation of the information processing device according to the third embodiment. [Figure 17] FIG. 10 is a block diagram illustrating an example of a configuration of an information processing system according to a fourth embodiment. [Figure 18] FIG. 11 is a block diagram showing an example of a functional configuration of a terminal device according to a fourth embodiment. [Figure 19] FIG. 10 is a block diagram illustrating an example of a hardware configuration of an information processing device according to a fourth embodiment. [Figure 20] FIG. 11 is a block diagram showing an example of a hardware configuration of a terminal device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] <Outline of the embodiment> First, an outline of the embodiment will be described. 1 is a block diagram showing an example of a configuration of an information processing device 1 according to an outline of an embodiment. As shown in FIG. 1, the information processing device 1 includes a setting information acquisition unit 2, a user interface unit 3, and a storage control unit 4.

[0015] The information processing device 1 is a device used to generate annotation information about an object included in an image. Note that the object included in the image may be an object included in a part of the image, or may refer to the entire image (i.e., the image itself). Each component of the information processing device 1 will be described below.

[0016] The setting information acquisition unit 2 acquires object setting information. Here, the object setting information includes definitions of one or more elements added to an image to indicate an object included in the image, and a definition of the order of the elements. These definitions are determined in advance by an administrator or the like, and can be defined arbitrarily.

[0017] Here, the element may be any information used to indicate an object, and may be, for example, a shape (such as a point, line, rectangle, triangle, circle, ellipse, or area) or text information. For example, if the object is an eye (iris), the object setting information defines, as elements, a plurality of points representing the outline of the eye, a circle representing the pupil, and a circle representing the iris, and also defines the order of these elements. Of course, the above example is merely an example, and other elements, such as text information indicating the characteristics of the eye (such as the type of person's eye the eye belongs to), may be defined for this object. Furthermore, the object is not limited to an eye. In the object setting information, elements according to the object of interest are defined.

[0018] The user interface unit 3 provides a user interface for accepting input for specifying elements of an object included in an image. This user interface is used to accept input from a user, but it can also be said to be used to accept input from an input device, such as a pointing device or keyboard, operated by the user. This image is an image to which annotation information should be added and is also referred to as a target image. Here, the user interface unit 3 provides user interfaces set based on the definitions of elements indicated in the object setting information acquired by the setting information acquisition unit 2, in the order of the elements defined in the object setting information. For example, the user interface unit 3 provides a user interface set based on the definition of the first element, and then provides a user interface set based on the definition of the second element. In this way, the user interface unit 3 provides user interfaces according to the elements in order. The user uses this user interface to perform input operations to specifically specify elements of an object included in the target image.

[0019] The storage control unit 4 controls the storage of information on elements designated by user input via the user interface provided by the user interface unit 3 in association with the target image.

[0020] The above describes the information processing device 1. As described above, the information processing device 1 automatically provides a user interface corresponding to each element in sequence based on the definition in the object setting information. This allows the user to immediately start input for specifically specifying an element without having to perform an operation such as selecting the type of element. Therefore, the information processing device 1 can efficiently add annotation information to an image.

[0021] 1, the above description has been made with respect to the information processing device 1 having the configuration shown in Fig. 1, but the manner in which the above-described effects can be obtained is not limited to the device. For example, the same effects can be obtained with an information processing method including the above-described processing of the information processing device 1, a program that performs the above-described processing of the information processing device 1, or a non-transitory computer-readable medium on which the program is stored.

[0022] <Preparation for explaining the embodiment> Here, as a preparation for explaining the details of the embodiment, annotations will be considered.

[0023] Fig. 2 is a schematic diagram showing an example of annotation information about objects in an image, i.e., objects included in an image. In an image 900 shown in Fig. 2, there are, as examples, a human face 910, a sun 920, and a star 930 as objects. For such an image 900, the following annotation information can be considered as annotations about the objects:

[0024] With respect to the face 910, for example, a rectangle 911 (rectangular frame) surrounding the face 910 is added to the image 900 as annotation information. Furthermore, in order to explain the face 910, a label "Mr. A" is added to the image 900 as annotation information, as an identifier such as a name indicating whose face the face 910 belongs to. With respect to the sun 920, for example, a circle 921 (circular frame) surrounding the sun 920 is added to the image 900 as annotation information. With respect to the sun 920, for example, a label "morning" is added to the image 900 as annotation information to explain the sun 920. With respect to the star 930, for example, points 931, 932, 933, and 934 indicating the vertices of the star 930 are added to the image 900 as annotation information. In this way, annotation information about objects included in an image is classified into shapes such as the above-mentioned rectangles, circles, and points, and labels that describe the objects, such as "morning" and "Mr. A." Note that the shapes and labels of the annotation information shown in FIG. 2 are merely examples, and other shapes and labels may be used. For example, shapes may include points, lines, rectangles, triangles, circles, ellipses, areas, etc.

[0025] In the example shown in FIG. 2, annotation information about an object included in a portion of the image 900 is shown, but the entire image 900 can also be considered as a single object. That is, the image 900 itself may be considered as an object and annotation information may be added to the image 900. FIG. 3 is a schematic diagram showing an example of annotation information when the entire image is considered as a single object. When the entire image 900 is considered as a single object, for example, the following annotation information can be added to the image 900. For the entire image, for example, a rectangle 901 (a rectangular frame) surrounding the entire image is added to the image 900 as annotation information. Furthermore, to explain the entire image, a label "abnormal phenomenon" is added to the image 900 as annotation information indicating the characteristics of the entire image. Therefore, annotation information may be used for an object included in a portion of the image, or the entire image (i.e., the image itself) may be considered as the object and used for the object.

[0026] Here, annotation information will be further considered. Annotation information can be classified into elements that must be added and elements that are optional. For example, the annotation information for the entire image described above may be annotation information that is added only when the image is in an abnormal state. On the other hand, the other annotation information shown in FIGS. 2 and 3 may be defined as annotation information that must be added to the image 900.

[0027] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. In addition, in each drawing, similar components are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0028] <First Embodiment> 4 is a block diagram showing an example of a functional configuration of an information processing device 100 according to the first embodiment. The information processing device 100 is a device used to generate annotation information about an object included in an image (an object included in a part of the image, or an object represented by the entire image). The information processing device 100 is a device corresponding to the information processing device 1 of FIG. 1. As shown in FIG. 4, the information processing device 100 includes a setting information storage unit 101, a target image storage unit 102, an annotation storage unit 103, a setting information acquisition unit 104, a user interface unit 105, and a storage control unit 106.

[0029] The setting information storage unit 101 stores various setting information such as object setting information and task setting information, which will be described later. In this embodiment, the setting information storage unit 101 stores setting information that has been created in advance. For example, this setting information may be information created by another device or may be a data file in a predetermined file format. In the configuration shown in FIG. 4, the setting information storage unit 101 is included in the information processing device 100, but the setting information storage unit 101 may also be realized in another device that is communicably connected to the information processing device 100 via a network or the like. The setting information storage unit 101 may also be configured as a database.

[0030] The target image storage unit 102 stores a target image, which is an image to which annotation information should be added. In this embodiment, the target image storage unit 102 specifically stores image data of the target image and an identifier (such as an image ID) for identifying the target image. In the configuration shown in FIG. 4, the target image storage unit 102 is included in the information processing device 100, but the target image storage unit 102 may be realized in another device communicably connected to the information processing device 100 via a network or the like. Furthermore, the target image storage unit 102 may be configured as a database.

[0031] The annotation storage unit 103 stores annotation information specified by a user. In this embodiment, the annotation storage unit 103 specifically stores an identifier of a target image and the annotation information in association with each other. In the configuration shown in FIG. 4, the annotation storage unit 103 is included in the information processing device 100, but the annotation storage unit 103 may be realized in another device communicably connected to the information processing device 100 via a network or the like. The annotation storage unit 103 may also be configured as a database.

[0032] Here, a description will be given of setting information used for processing by the information processing device 100. In this embodiment, the information processing device 100 uses object setting information and task setting information as setting information.

[0033] First, the object setting information will be described. The object setting information is setting information related to an object for which annotation information is to be created, and is setting information defined for each object. For example, this object may be various objects to be detected from an image, such as an eye (iris), a face, a person, a cat, a dog, or an animal. The object may also be the entire image.

[0034] In this embodiment, the target object setting information specifically includes the following data, for example: Objectname data Figures data Labels data ElementOrder data GuideSample data Multi-Instance Data Instance number data

[0035] Here, Objectname data is data indicating the name of an object. Furthermore, Figures data and Labels data are data representing definitions of elements (i.e., elements placed on the image) that are added to an image as annotation information to indicate the object. As described above, annotation information about an object included in an image is classified into figures for specifying the position of the object in the image and labels for describing the object. Therefore, in this embodiment, object setting information may include Figures data for defining elements classified as figures (hereinafter also referred to as figure elements) and Labels data for defining elements classified as labels (hereinafter also referred to as label elements).

[0036] The Figures data defines a set of figures used as annotation information for an object (more specifically, the object indicated by the Objectname data). If an object is specified by multiple figures, the Figures data defines a set including multiple figure elements. If no figure elements are required, the Figures data will be an empty set. The Labels data defines a set of labels used as annotation information for an object (more specifically, the object indicated by the Objectname data). If an object is described by multiple labels, the Labels data defines a set including multiple label elements. For example, if the object is an eye, the Labels data may define a set consisting of two label elements as follows: For example, the first label element may be a label describing whether it is a child's eye or an adult's eye, and the second label element may be a label describing whether it is a male's eye or a female's eye. If no label elements are required, the Labels data will be an empty set.

[0037] Furthermore, each element included in the Figures data and Labels data has flag information indicating whether it is an element that must be added to the target image. That is, each element has a parameter indicating whether it is an element that is required or optional as annotation information. In this way, the target setting information includes a definition indicating whether an element must be added to an image.

[0038] The ElementOrder data in the object setting information is data that defines the order of elements used as annotation information for an object (more specifically, an object indicated by ObjectName data). In particular, the ElementOrder data defines the order of elements that are required as annotation information. The ElementOrder data is data that defines the order in which the element designation operation should be performed when a user designates elements for an object in order to add annotation information to a target image. Note that the ElementOrder data defines the order of, for example, graphic elements, but is not limited to defining the order of graphic elements and may also define the order of label elements.

[0039] The GuideSample data is guide information for explaining to the user how to specify an element when the user performs an element specification operation via a user interface provided by the user interface unit 105 (described later). The GuideSample data may be text data explaining how to perform the specification operation, audio data explaining how to perform the specification operation, or an image explaining how to perform the specification operation. Here, the image explaining how to perform the specification operation is, for example, an image depicting a sample object and an element placed in an exemplary position relative to the object. The image explaining how to perform the specification operation may be a still image or a moving image. The GuideSample data may also be data that combines two or more types of data from among text data, audio data, and image data. The GuideSample data may also be data in a format other than text data, audio data, and image data.

[0040] The MultiInstance data is data that defines whether the number of objects (specifically, the objects identified by the Objectname data) included in one target image is unknown. In this embodiment, when the value of the MultiInstance data is set to "True," this means that a target image in which the number of objects is unknown is processed by the information processing device 100. In contrast, when the value of the MultiInstance data is set to "False," this means that a target image in which the number of objects is a fixed number is processed by the information processing device 100. Therefore, for example, when it is expected that only target images in which a predetermined number of predetermined objects are processed by the information processing device 100, the value of the MultiInstance data is set to "False."

[0041] The Instance number data is data that represents the fixed number described above. In other words, it is data that represents how many objects identified by the Object name data exist in one target image. For this reason, in this embodiment, when the value of the MultiInstance data is "False," the Instance number data is set to an integer greater than or equal to 1. On the other hand, when the value of the MultiInstance data is "True," the Instance number data is set to "Null."

[0042] The object setting information is not limited to the above-described data and may include other data. For example, the object setting information may include constraint data that defines constraint conditions for the placement of elements on the target image. Including the constraint data in the object setting information can encourage each element to be placed so as to satisfy predetermined placement rules, as described below. The constraint data may be a constraint focusing on one element or multiple elements. That is, the constraint data may be data that restricts the relative relationship of multiple elements on the target image. More specifically, the constraint data may be data that restricts a geometric relationship. The geometric relationship may be, for example, the positional relationship of multiple elements, an inclusion relationship between multiple elements, or a magnitude relationship between multiple elements. By using constraint data that restricts the relative relationship of multiple elements on the target image, it is possible to prevent the occurrence of inappropriate element placement, where the relative relationship does not satisfy predetermined rules. For example, it is possible to prevent a point representing the center of the eyelid from being placed at a position that deviates from the bisector between the outer corner and inner corner of the eye. It is also possible to prevent a circle representing the pupil from being placed so as to encompass a circle representing the iris.

[0043] Next, the task setting information will be described. The task setting information is setting information that indicates a set of one or more objects, and defines which objects should have annotation information added to one image. In other words, the task setting information is setting information that defines which objects should be used in the task of adding annotation information to a target image. The task setting information may define that annotation information should be added to one type of object, or that annotation information should be added to multiple types of objects, for one image.

[0044] In this embodiment, the task setting information specifically includes the following data, for example: Task name data Object data ObjectOrder data

[0045] Here, Taskname data is data indicating the name of a task. Object data is data defining which object annotation information should be added to an image. In other words, Object data is data defining a set of objects defined by object setting information. Specifically, Object data is expressed as a set of the above-mentioned Objectnames.

[0046] The ObjectOrder data in the task setting information is data that defines the order of objects indicated by the Object data. In other words, the ObjectOrder data in the task setting information is data that defines the order in which a designation operation is to be performed for each object indicated by the Object data when an element designation operation is performed to create annotation information. For example, assume that the ObjectOrder data in the task setting information defines the order of objects as object A, object B, and object C. In this case, the user first designates an element for object A, then designates an element for object B, and then designates an element for object C.

[0047] In this way, when ObjectOrder data is set in the task setting information, the user performs an operation to add annotation information to each object in the set order. At that time, the object is automatically switched in the defined order. That is, when the designation operation for one object is completed, the target of the designation operation is switched to the next object specified in the defined order.

[0048] In the above description, the task setting information includes a definition of the order of multiple objects, but the task setting information does not necessarily have to include such a definition. That is, the task setting information does not have to have ObjectOrder data. In this case, the user needs to specify, via a user interface provided by the user interface unit 105 (described later), which object the operation for adding annotation information to should be performed for. That is, if the order of objects is not defined in the setting information, the user needs to input an instruction to select an object.

[0049] In this embodiment, the setting information storage unit 101 of the information processing device 100 stores the above-described object setting information and task setting information in advance. Note that the setting information storage unit 101 may store a plurality of pieces of task setting information. For example, various pieces of task setting information may be stored, such as task setting information for creating annotation information about animals included in a target image, task setting information for creating annotation information about human eyes included in the target image, etc. In this way, various pieces of task setting information and object setting information are stored in the setting information storage unit 101, so that the information processing device 100 can be used to add various pieces of annotation information.

[0050] The setting information acquisition unit 104 corresponds to the setting information acquisition unit 2 in FIG. 1 . In this embodiment, the setting information acquisition unit 104 acquires object setting information and task setting information stored in the setting information acquisition unit 104. The setting information acquisition unit 104 acquires task setting information selected in response to a user instruction, and also acquires object setting information for an object set in the task setting information. For example, a user who wants to create annotation information for an eye included in a target image inputs an instruction to the information processing device 100 to select task setting information for creating annotation information for the eye. In response to this instruction, the setting information acquisition unit 104 acquires such task setting information from the setting information storage unit 101, and also acquires object setting information for an object indicated in the Object data of the task setting information from the setting information storage unit 101. Note that the instruction from the user does not necessarily have to be an instruction to select task setting information, but may also be an instruction to specify what type of annotation information the user wants to create. In this case, the setting information acquisition unit 104 may simply acquire task setting information corresponding to the type of annotation information designated by the user from the setting information storage unit 101. In this way, the setting information acquisition unit 104 acquires setting information corresponding to an instruction from the user from among various setting information for creating annotation information.

[0051] The user interface unit 105 corresponds to the user interface unit 3 in Fig. 1. The user interface unit 105 reads a target image stored in the target image storage unit 102. The user interface unit 105 also outputs a user interface that receives input from a user for creating annotation information for the read target image, based on the setting information acquired by the setting information acquisition unit 104. Specifically, for example, the user interface unit 105 displays a user interface screen on the output device 150, which will be described later. That is, the user interface unit 105 provides a GUI (Graphical User Interface).

[0052] The user interface unit 105 provides a user interface that includes only UI (user interface) components necessary for creating annotation information about an object indicated in the task setting information acquired by the setting information acquisition unit 104. In other words, the user interface unit 105 provides a user interface that does not include UI components necessary for creating annotation information about other objects not indicated in the task setting information acquired by the setting information acquisition unit 104. Therefore, the user interface provided by the user interface unit 105 differs depending on the setting information acquired by the setting information acquisition unit 104. In other words, the user interface provided by the user interface unit 105 differs depending on the user's purpose. For example, if a user wishes to create annotation information about an animal, that is, if the object is an animal, the user interface unit 105 provides a user interface specialized for creating annotation information about the animal. On the other hand, if a user wishes to create annotation information about a human eye, that is, if the object is an eye, the user interface unit 105 provides a user interface specialized for creating annotation information about the eye. More specifically, for example, if the object is an animal, the user interface unit 105 provides the following user interface. In this case, a user interface is provided that includes UI components for accepting input for elements defined in the object setting information for the animal, but does not include UI components for accepting input for other elements. For example, assume that only a rectangle is defined in the Figures data of the object setting information for the animal. In this case, a user interface is provided that includes a UI component for specifying the rectangle on the object image, but does not include UI components for specifying a point, a circle, etc. Similarly, for example, if the object is an eye, the user interface unit 105 provides the following user interface.In this case, a user interface is provided that includes UI components for accepting input for elements defined in the object setting information for the eye, but does not include UI components for accepting input for other elements. For example, suppose that only points and circles are defined in the Figures data of the object setting information for the eye. In this case, a user interface is provided that includes UI components for specifying points and UI components for specifying circles, but does not include UI components for specifying rectangles, etc.

[0053] When the acquired task setting information indicates multiple types of objects, the user interface unit 105 provides a user interface specialized for one of the objects. That is, in this case, the user interface unit 105 references the object setting information of one of the objects and provides a user interface in which only UI components necessary for creating annotation information for the object are arranged. When the task setting information includes a definition of the order of multiple objects (ObjectOrder data), the user interface unit 105 references object setting information selected according to the order of the multiple objects defined in the task setting information and provides a user interface. This configuration allows user interfaces specialized for each object to be provided sequentially without relying on a user's switching instruction. This enables more efficient work. On the other hand, when the task setting information does not include a definition of the order of multiple objects, the user interface unit 105 may provide a user interface by reference to object setting information selected according to a user's instruction. That is, the user may specify which object to input. This configuration allows the user to freely switch objects, improving convenience.

[0054] Furthermore, the user interface unit 105 provides a user interface in accordance with the order of elements defined in the object setting information acquired by the setting information acquisition unit 104. Specifically, the user interface unit 105 provides a user interface in accordance with the order of the defined elements as follows: The user interface unit 105 provides a user interface for accepting a designation for an element that is n-th in the defined order, and then, upon accepting a designation from the user, provides a user interface for accepting a designation for an element that is n+1-th, where n is an integer equal to or greater than 1. More specifically, after providing a user interface for accepting a designation for an element that is n-th in the defined order, when the user interface unit 105 accepts a designation from the user, it accepts this designation as a designation for the n-th element. Therefore, the user does not need to input an indication of the order of the element being designated.

[0055] The user interface unit 105 identifies the type of the nth element and provides a user interface specialized for accepting input specifying the identified type of element as a user interface for accepting specification for the nth element. In other words, the user interface unit 105 identifies the type of the nth element and provides a user interface of an input mode corresponding to the identified type as a user interface for accepting specification for the nth element. For example, assume that the type of the nth element is a point and the type of the n+1th element is a circle. In this case, the user interface unit 105 provides a user interface specialized for accepting specification of the position of a point to be placed in the target image, and upon receiving specification from the user, provides a user interface specialized for accepting specification of the position of a circle. In other words, the user interface unit 105 provides a user interface for a point input mode, and then provides a user interface for a circle input mode without receiving a mode change instruction from the user. In this way, the user interface unit 105 automatically switches the user interface upon receiving an instruction to specify an element.

[0056] As described above, the object setting information may also define elements that may be added to the target image as annotation information. In this embodiment, the order of such elements is not set by the ElementOrder data. Therefore, when the object setting information defines such elements, the user interface unit 105 provides the following user interface to enable the user to receive input from the user to specify the elements. That is, in this case, the user interface unit 105 provides a user interface that includes a UI component for switching to an input mode for inputting the specification for the elements. This allows the user to input the specification for the elements that may be added as needed.

[0057] The above has described the user interface unit 105. As can be seen from the above description, the user interface unit 105 provides a user interface by referring to the object setting information of any of the objects included in the set indicated by the task setting information acquired by the setting information acquisition unit 104. This makes it possible to provide a user interface that enables efficient creation of annotation information for each of various objects included in the same target image.

[0058] The storage control unit 106 corresponds to the storage control unit 4 in FIG. 1 . The storage control unit 106 controls the storage of information on elements specified by user input via a user interface provided by the user interface unit 105 as annotation information in association with the target image. Specific graphic information is obtained for each element through the user input via the user interface. If a label element is also specified, label information on the object identified by the specified element is also obtained. The storage control unit 106 controls the storage of the obtained information in association with the target image as annotation information in the annotation storage unit 103. In this embodiment, the storage control unit 106 stores a pair of the image ID of the target image and the annotation information for the target image in the annotation storage unit 103. The graphic information is information that specifies the arrangement of a graphic in the target image, and may include coordinate information indicating a position within the target image or information that defines geometric characteristics such as the radius. The label information is information that indicates the content of a label.

[0059] 5 is a block diagram showing an example of a hardware configuration of the information processing device 100. As shown in FIG. 5, the information processing device 100 includes an output device 150, an input device 151, a storage device 152, a memory 153, and a processor 154.

[0060] The output device 150 is an output device such as a display that outputs information to the outside. The display may be, for example, a flat panel display such as a liquid crystal display, a plasma display, or an organic EL (Electro-Luminescence) display. The output device 150 is not limited to a display and may be other devices. For example, the output device 150 may be a projection device such as a projector. The output device 150 may also include a speaker. The output device 150 displays a user interface provided by the user interface unit 105.

[0061] The input device 151 is a device for allowing a user to input via a user interface, and is, for example, an input device such as a pointing device or a keyboard. Examples of pointing devices include a mouse, a trackball, a touch panel, and a pen tablet. The input device 151 and the output device 150 may be integrated into a touch panel.

[0062] The storage device 152 is a non-volatile storage device such as a hard disk, a flash memory, etc. The setting information acquisition unit 104, the target image storage unit 102, and the annotation storage unit 103 described above are realized by, for example, the storage device 152, but may be realized by other storage devices.

[0063] The memory 153 is configured by, for example, a combination of a volatile memory and a non-volatile memory. The memory 153 is used to store software (computer programs) including one or more instructions executed by the processor 154, data used for various processes of the information processing device 100, and the like.

[0064] The processor 154 reads and executes software (computer programs) from the memory 153 to perform the processes of the setting information acquisition unit 104, the user interface unit 105, and the storage control unit 106. The processor 154 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU (Central Processing Unit). The processor 154 may include multiple processors. In this way, the information processing device 100 has the functionality of a computer.

[0065] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0066] Next, the operation of the information processing device 100 will be described with reference to a flowchart. Fig. 6 is a flowchart showing an example of the operation of the information processing device 100. Hereinafter, the flow of the operation of the information processing device 100 will be described with reference to Fig. 6.

[0067] In step S100, the setting information acquisition unit 104 acquires the object setting information and the task setting information stored in the setting information acquisition unit 104. As described above, the setting information acquisition unit 104 acquires the setting information according to the purpose of the annotation desired by the user. That is, the setting information acquisition unit 104 acquires the setting information according to the type of annotation information that the user wants to create.

[0068] Next, in step S101, the user interface unit 105 provides a user interface based on the setting information acquired in step S100 and accepts input from the user. Specifically, the user interface unit 105 reads the target image stored in the target image storage unit 102. Then, the user interface unit 105 outputs to the output device 150, based on the setting information acquired by the setting information acquisition unit 104, a user interface that accepts input from the user for creating annotation information for the read target image. At this time, the user interface unit 105 provides a different user interface for each definition of the target object (i.e., for each definition of an element to be added to the image to indicate the target object). Then, the user interface unit 105 accepts input via the input device 151. If the task setting information defines that annotation information is to be added to multiple types of targets, the user interface unit 105 accepts an operation to specify an element for each target object.

[0069] Next, in step S102, the user interface unit 105 determines whether the user's task of creating annotation information for the target image currently being handled by the user interface unit 105 is complete. For example, if the elements defined in the object setting information are only elements that must be added, the user interface unit 105 determines that the creation task is complete when the user inputs instructions for all elements defined in the object setting information. Furthermore, if the elements defined in the object setting information include elements that are optional to add, the user interface unit 105 determines that the creation task is complete when the user inputs an instruction indicating that the creation task is complete. Note that if multiple objects are specified in the task setting information, the user interface unit 105 determines that the creation task for the target image currently being handled is complete when operations for all objects have been completed. This step is repeated until the creation task is complete. When the creation task is complete, the process proceeds to step S103.

[0070] In step S103, the storage control unit 106 controls the storage of information on an element designated by a user input via a user interface provided by the user interface unit 105 as annotation information in association with the target image, thereby achieving the addition of annotation information to one target image.

[0071] Next, in step S104, the user interface unit 105 determines whether the annotation information creation work for all target images included in the predetermined image group has been completed. If there are target images for which the annotation information creation work has not been completed, the process returns to step S101, and the above-mentioned process is repeated. On the other hand, if the annotation information creation work for all target images included in the predetermined image group has been completed, the process ends.

[0072] Next, specific examples of setting information and specific examples of the user interface provided by the user interface unit 105 will be shown.

[0073] The setting information exemplified below is setting information for creating annotation information about the eyes (irises) included in the target image.

[0074] In this case, the object setting information includes, for example, the following data: Object name data: eye Figures data: {"F1":Point,···,"F8":Point,"I9":Circle,"I10":Circle} Labels data: {} ·ElementOrder data: {“F1”, “F2”, “F3”, “F4”, “F5”, “F6”, “F7”, “F8”, “I9”, “I10”} GuideSample data: {EyeGuideSampledata} MultiInstance data: False Instance number data: 1 ·Constraint condition data: “F3”:Point->constmode=BISECTOR,p1=“F1”,p2=“F2” ···etc.

[0075] The above object setting information defines an eye as the object. The object name is "eye" according to the Objectname data.

[0076] The Figures data also specifies that eight points and two circles are used as shapes to identify eyes in a target image. In other words, the Figures data defines that the elements added to a target image to indicate the eyes contained in the image are eight points and two circles. In the Figures data, F1, F2, ..., F8, I9, and I10 are element identifiers. Furthermore, "Point" and "Circle" written after the identifier indicate the type of shape. These 10 elements are defined as elements that must be added to a target image.

[0077] In the above object setting information, the Labels data is an empty set. Therefore, in the above example, no labels are created as annotation information. However, Labels data may be defined. For example, labels to describe the eyes, such as whether they are child's eyes or adult's eyes, may be defined.

[0078] The ElementOrder data also defines an order for the above-mentioned 10 elements, so that the designation operation for each of these 10 elements is performed in the order indicated by the ElementOrder data.

[0079] Furthermore, the GuideSample data defines that EyeGuideSampledata is referenced as guide information for creating annotation information for the eyes.

[0080] Furthermore, since the value of the MultiInstance data is "False," it is defined that annotation information is added to a target image that contains a predetermined number of eyes. This predetermined number is 1 because the value of the Instance number data is 1. In other words, this object setting information is used when creating annotation information for an image with one eye as the target image.

[0081] The above object setting information also includes constraint condition data, which defines, for example, that the position of the point represented by identifier F3 must be on the bisector between the point represented by identifier F1 (the point at the inner corner of the eye) and the point represented by identifier F2 (the point at the outer corner of the eye).

[0082] The task setting information includes, for example, the following data: Task name data: eyeAnnotationTask Object data: eye ·ObjectOrder data:{eye}

[0083] The above task setting information defines a task for creating annotation information for eyes. The Taskname data specifies the task name as eyeAnnotationTask. The Object data defines that annotation information should be added to one target image by referencing the object setting information whose Objectname data is "eye." While this example shows the addition of annotation information for one object, as mentioned above, the task setting information can also specify multiple objects, and in that case, the order of these objects can also be defined. The ObjectOrder data specifies the order of the objects, but since there is only one object in this example, the ObjectOrder data only includes "eye."

[0084] Next, specific examples of the user interface provided by the user interface unit 105 will be shown. Fig. 7 and Fig. 9 to Fig. 12 are schematic diagrams showing examples of user interface screens that the user interface unit 105 displays on the output device 150. Fig. 8 is a schematic diagram showing examples of guide information that the user interface unit 105 displays on the output device 150. Fig. 7 to Fig. 12 show examples in which the setting information acquisition unit 104 acquires task setting information whose Taskname data is eyeAnnotationTask and object setting information whose Objectname data is eye in response to a user instruction.

[0085] The user interface unit 105 first provides a user interface as shown in FIG. 7. Each UI component of the user interface shown in FIG. 7 will be described. The file button 801 is a button that provides a menu for specifying a target image. For example, this file button 801 provides a menu for specifying a folder that stores a target image to which eye annotation information should be added. The folder text box 802 is an area that displays the file path of the folder that stores the image. The load button 803 is a button for loading an image in a folder specified by the file path displayed in the folder text box 802 as the target image.

[0086] Image window 804 is an area that displays the loaded target image. The user performs an operation to specify elements defined in the target object setting information for the target image displayed in image window 804. In this example, image window 804 displays a target image that includes one eye.

[0087] The image change button 805 and the image change button 806 are buttons that allow the user to forcibly switch the target image to which annotation information is to be added. Pressing the image change button 805 switches the target image to the previous image. Similarly, pressing the image change button 806 switches the target image to the next image. Note that the user interface unit 105 automatically switches the target image to the next image once the annotation information creation work for one target image is completed. In particular, if the value of the MultiInstance data in the object setting information is "False," the target image is switched as follows. In this case, the user interface unit 105 automatically switches the target image to the next image once the element designation operation for each of the objects indicated by the Instance number data is completed. In this example, the user interface unit 105 automatically switches the target image to the next image once the designation operation for eight points and two circles is completed for one target image. In addition, if the task setting information defines the addition of annotation information for multiple types of objects, the user interface unit 105 switches the target image to the next image once the element specification operation for all of those objects has been completed.

[0088] The status area 807 is an area that displays the progress of the annotation information creation work. In this area, the user interface unit 105 indicates, for example, the number of images for which annotation information creation work has been completed out of all images in the folder identified by the file path displayed in the folder text box 802. The user interface unit 105 may also display the total number of images for which annotation information creation work has been completed.

[0089] The guide display button 808 is a button for outputting guide information. When the user operates this button, guide information defined in the GuideSample data of the object setting information is output. In this example, information indicated by EyeGuideSampledata is output as guide information for the task of creating annotation information for the eyes. FIG. 8 shows an example of the information indicated by EyeGuideSampledata. In the example shown in FIG. 8, an image 850 showing an example of the arrangement of each graphic element defined in the object setting information and text 851 explaining how to arrange each element are displayed as guide information. As described above, the guide information may be audio or the like.

[0090] In this way, the user interface unit 105 may output guide information that guides input for specifying elements of an object included in an image. With this configuration, the user can make input while referring to the guide information, thereby improving user convenience.

[0091] The task notification area 809 is an area that notifies the user of the task that the user should currently perform. The user interface unit 105 determines the content of the notification by referring to the object setting information. In this example, the ElementOrder data of the object setting information defines that a designation operation should first be performed on the point with identifier F1. Therefore, as shown in FIG. 7, the user interface unit 105 displays the first notification in the task notification area 809, saying, "Please designate the eye feature point F1." In this way, the user interface unit 105 switches the notification of the task that the user should perform, depending on the definition of the element indicated in the object setting information.

[0092] The Object Control area 810 is an area that displays UI components for selecting an object for which annotation information creation work is to be performed. The user interface unit 105 displays only UI components for objects defined in the acquired task setting information. In the example shown here, only eyes are defined as objects in the task setting information. Therefore, as shown in FIG. 7, the user interface unit 105 displays only the object button 811 for eyes in the Object Control area 810. In other words, buttons for objects not defined in the acquired task setting information, such as the object button for dogs and the object button for cats, are not displayed. This prevents unnecessary UI components from being displayed, providing a less cluttered UI screen. Then, the user interface unit 105 provides a user interface in which the object button 811 for eyes is pre-selected, as shown in FIG. 7.

[0093] If multiple objects are specified in the task setting information, buttons for each of the multiple objects are arranged in the Object Control area 810. In this case, the user can select one of the buttons to switch the object to which annotation information is to be added. However, if ObjectOrder data is defined in the acquired task setting information, the user interface unit 105 automatically switches the objects in the defined order, regardless of the user's operation on the Object Control area 810. In other words, when the specification operation for one object is completed, the user interface unit 105 switches to the next object specified in the defined order.

[0094] The figure control area 820 displays UI components for selecting which graphic element to specify. The user interface unit 105 displays only UI components for graphic elements defined in the acquired object setting information. In this way, the user interface unit 105 switches the type of instruction it can accept depending on the definition of the element indicated in the object setting information. In the example shown here, only points and circles are defined as graphic elements in the object setting information. Therefore, as shown in FIG. 7 , the user interface unit 105 displays only a graphic element button 821 for points and a graphic element button 822 for circles in the figure control area 820. In other words, buttons for graphic elements not defined in the acquired object setting information, such as a graphic element button for a rectangle or a graphic element button for a line, are not displayed. In other words, the user interface unit 105 provides a user interface that allows only either a point input mode or a circle input mode. This prevents the display of unnecessary UI components, thereby providing a UI screen with reduced clutter.

[0095] The user can switch which graphic element to specify by selecting a button displayed in the Figure Control area 820. However, for graphic elements defined by the ElementOrder data in the acquired object setting information, the user interface unit 105 can automatically switch the graphic elements according to the order defined by the ElementOrder data. In other words, the user interface unit 105 switches the input mode according to the ElementOrder data in the object setting information. As described above, in this embodiment, the ElementOrder data defines the order of elements that must be added.

[0096] In this way, the input target element is automatically switched in the order defined by the ElementOrder data, which reduces the user's operations for specifying the input target, thereby making the element specification operation more efficient.

[0097] In this example, the ElementOrder data of the object setting information defines that a designation operation should be performed on the point with identifier F1 first. Therefore, the user interface unit 105 first provides a user interface specialized for receiving input designating a point. Therefore, as shown in FIG. 7, the user interface unit 105 first provides a user interface for a point input mode.

[0098] In the example shown here, the object setting information defines only the graphic elements that must be added, but the object setting information may also include definitions of graphic elements that are optional to add. In this case, the user interface for such graphic elements may be provided in response to a user instruction to switch the user interface. In other words, the input mode may be switched in response to a user instruction.

[0099] In this way, the user interface unit 105 sequentially provides user interfaces set based on the definitions of elements that must be added in the order defined in the object setting information. On the other hand, the user interface unit 105 provides user interfaces corresponding to definitions of elements that do not must be added in response to an instruction from the user to switch the user interface. According to this processing by the user interface unit 105, even when elements that do not must be added can be added to the target image, it is possible to efficiently specify required elements while also accepting specification operations for optional elements as needed.

[0100] Furthermore, Figure Control area 820 has a cancel button 823 for canceling an operation. By operating this button, the user can cancel the designation operation that was just input.

[0101] The Label Control area 830 is an area that displays UI components for specifying label elements. The user interface unit 105 displays only UI components for label elements defined in the acquired object setting information. In the example shown here, no label elements are defined in the object setting information. Therefore, the display of label elements is omitted in the Label Control area 830. Therefore, unnecessary UI components are not displayed, and a less cluttered UI screen is provided.

[0102] When label elements are defined in the object setting information, UI components for determining the value of each label element are arranged in Label Control area 830. Such UI components may be any UI component for specifying a value, and may be a radio button, a check box, or a text box.

[0103] If the ElementOrder data in the object setting information defines the order of label elements, the user interface unit 105 may sequentially provide user interfaces for accepting the values ​​of each label element in the defined order. The user interfaces for accepting the values ​​of label elements may be set based on the definitions of the label elements indicated in the object setting information. For example, the user interface unit 105 may repeatedly provide a user interface for accepting an nth input for specifying the value of a label element as an input for specifying the value of the nth label element in the defined order, the number of times corresponding to the number of label elements whose order is defined. Here, n is an integer greater than or equal to 1. Alternatively, the user interface unit 105 may provide a user interface including a UI component for accepting the value of the nth label element, and then provide a user interface including a UI component for accepting the value of the n+1th label element. That is, the user interface unit 105 may sequentially provide UI components corresponding to the label elements whose order is defined. The object setting information may include definitions of label elements that may be added optionally. In this case, the user interface for such a label element may be provided in response to a user instruction to switch the user interface. That is, the user interface for specifying the label element may be switched in response to a user instruction.

[0104] In this way, the user interface unit 105 can provide user interfaces set based on the definition of elements that must be added for label elements in the order defined in the object setting information. The user interface unit 105 can also provide user interfaces corresponding to the definition of elements that do not necessarily need to be added for label elements in response to a user instruction to switch the user interface. Therefore, even when elements that do not necessarily need to be added can be added to the target image, the operation of specifying required elements can be made more efficient, while also accepting an operation of specifying optional elements as needed.

[0105] The Image View Control area 840 is an area where UI components for adjusting the display of the target image displayed in the image window 804 are arranged. In the example shown in FIG. 7, a Zoom slider 841 for specifying the enlargement or reduction of the target image and a Gamma slider 842 for adjusting the gamma value are arranged. Note that UI components for adjusting other items, such as a UI component for adjusting image rotation or a UI component for adjusting image brightness, may also be arranged in the Image View Control area 840. The type of UI component for adjusting an item to be arranged may also be defined in the object setting information or task setting information. Note that it is preferable that the adjustment results be maintained even when the target image to be displayed changes, but they may also be reset for each target image. However, if the display adjustment results are maintained even when the target image to be displayed changes, the user does not need to make adjustments for each target image, thereby improving convenience.

[0106] The above has described the screen of the user interface shown in Fig. 7, but the user interface may further include buttons not shown in Fig. 7. For example, when the value of the MultiInstance data is set to "True," that is, when the number of objects is unknown, there may be a Complete button that is operated by the user when work on all objects included in the target image is completed.

[0107] 9 is a schematic diagram showing the user interface immediately after a designation operation for a point with identifier F1 and a designation operation for a point with identifier F2 have been performed. Since the object setting information defines that a designation operation for a point with identifier F3 should be performed after the point with identifier F2, a notification stating "Please designate eye feature point F3" is displayed in the task notification area 809.

[0108] In the example shown here, as described above, the object setting information includes constraint data that restricts the position of feature point F3. When constraints are set on the placement of elements on the image, the user interface unit 105 may support the user's input for specifying elements based on the constraints. This configuration can encourage the placement of each element so as to satisfy predetermined rules for placement.

[0109] For example, if the input received from the user does not satisfy the constraints, the user interface unit 105 may correct the position of the graphic element specified by the input to a position that satisfies the constraints. For example, if the input received from the user does not satisfy the constraints, the user interface unit 105 may automatically move the position of the graphic element specified by the input to a position that is near the position specified by the input and satisfies the constraints. In this example, the constraint data requires that the position of the point represented by identifier F3 be on the bisector between the point represented by identifier F1 and the point represented by identifier F2. Therefore, for example, if the position of the point represented by identifier F3 specified by the user is not on the bisector between the point represented by identifier F1 and the point represented by identifier F2, the user interface unit 105 may automatically move the position of the point represented by identifier F3 to the following position: That is, the user interface unit 105 may move the position of the point represented by identifier F3 to a position near the position specified by the user and on the bisector. The position on the bisector that is near the position specified by the user may be the position of the intersection of the bisector and a perpendicular line drawn from the position specified by the user to the bisector. In this way, the user interface unit 105 corrects the position of the graphic element specified by input to a position that satisfies the constraints, thereby making it possible to easily place the graphic element at a position that satisfies the constraints.

[0110] Furthermore, the support for the user's input to specify an element may be as follows. For example, the user interface unit 105 may output a warning when the input received from the user does not satisfy the constraint conditions. For example, when the input received from the user does not satisfy the constraint conditions, the user interface unit 105 outputs a warning notifying the user that the input position does not satisfy the constraint conditions. This warning may also include a notification guiding the user to a position that satisfies the constraint conditions. Note that the output of the warning may be a visual output or an audio output. In this way, by the user interface unit 105 outputting a warning when the position of a graphic element specified by input does not satisfy the constraint conditions for the placement of the graphic element on the image, it is possible to prevent the graphic element from being placed in an inappropriate position.

[0111] 10 is a schematic diagram showing the user interface immediately after a designation operation for points with identifiers F1 to F8 and a designation operation for a circle with identifier I9 has been performed. The object setting information defines that a designation operation for a circle with identifier I10 should be performed after the circle with identifier I9, so a notification saying "Please designate the eye circle I10" is displayed in the task notification area 809. Furthermore, because the next designation operation to be performed is to designate the position of the circle, a circle input mode is automatically set, as shown in the Figure Control area 820 of FIG. 10.

[0112] Here, it is assumed that the user has placed the circle I9 in the wrong position. In this case, the user can cancel the designation operation that was just input by operating the cancel button 823 for canceling the operation. FIG. 11 shows the user interface when the user operates the cancel button 823 and returns to the state before the placement of the circle I9. In this case, the designation operation for the circle I9 needs to be performed again, and therefore a notification that reads, "Please designate the eye circle I9" is displayed in the task notification area 809.

[0113] FIG. 12 is a schematic diagram showing a user interface at the time when the designation operation for all graphic elements defined in the object setting information is completed. When the designation operation for the elements set in the object setting information is completed, the user interface unit 105 may accept a correction for the designated element. Therefore, in the example shown in FIG. 12, the user interface unit 105 displays a notification saying "Please confirm" in the task notification area 809 and accepts a correction operation by the user. For example, the user corrects the position of an already placed graphic element by performing a drag operation, etc. Note that the user interface unit 105 may accept a correction for information on an element that has already been input at any timing. In this way, the user interface unit 105 may accept a correction for information on an element specified by input. By the user interface unit 105 accepting a correction from the user, input of incorrect information can be corrected, thereby improving convenience.

[0114] The above has described the first embodiment. The information processing device 100 provides a user interface specialized for the work procedures that the user should perform. Therefore, the information processing device 100 can efficiently add annotation information to images.

[0115] The training data used to train a machine learning model varies depending on the intended use of the machine learning model. In other words, the type of annotation varies depending on the intended use of the machine learning model. Therefore, the work required to assign annotation information varies depending on the intended use of the machine learning model. Generally, a user interface for various purposes includes various UI components. However, a user only uses a limited number of UI components for a single purpose, and does not necessarily use all of the UI components. Using a user interface with unnecessary UI components reduces work efficiency. In contrast, the information processing device 100 provides a user interface that can be used for various purposes and is specialized for the user's purpose. Therefore, the information processing device 100 can enable efficient work while addressing various purposes.

[0116] <Embodiment 2> Next, a second embodiment will be described. This embodiment differs from the first embodiment in that the information processing device further has a function for generating setting information. FIG. 13 is a block diagram showing an example of the functional configuration of an information processing device 100a according to the second embodiment. The information processing device 100a according to the second embodiment differs from the information processing device 100 according to the first embodiment in that it further has a setting information generation unit 107. The setting information generation unit 107 is realized, for example, by the processor 154 reading and executing software (computer program) from the memory 153. Below, differences from the first embodiment will be specifically described, and overlapping descriptions will be omitted as appropriate.

[0117] The setting information generation unit 107 generates setting information based on input received from a user. Based on the user input, the setting information generation unit 107 may generate only object setting information, only task setting information, or both. The setting information generation unit 107 stores the generated setting information in the setting information storage unit 101. Therefore, in this embodiment, the setting information acquisition unit 104 acquires the object setting information generated by the setting information generation unit 107. Note that, also in this embodiment, the setting information acquisition unit 104 may acquire setting information not generated by the setting information generation unit 107, i.e., setting information acquired from another device. The setting information generation unit 107 may provide a user interface for determining the content of the object setting information or the task setting information. This user interface may be a GUI.

[0118] Fig. 14 is a flowchart showing an example of the operation of the information processing device 100a according to the second embodiment. The flowchart shown in Fig. 14 differs from the flowchart shown in Fig. 6 in that step S200 is performed before step S100 is performed.

[0119] In step S200, the setting information generation unit 107 generates setting information based on the input received from the user, and stores the generated setting information in the setting information storage unit 101. After step S200, the steps following step S100 described above are performed.

[0120] The above has described the second embodiment. In this embodiment, the information processing device 100a further has a function for generating setting information. This allows the user to freely design the configuration of annotation information and the task of creating annotation information, thereby further improving convenience.

[0121] <Third Embodiment> Next, a third embodiment will be described. This embodiment differs from the first embodiment in that the information processing device manages the progress of the annotation information creation work for each user. FIG. 15 is a block diagram showing an example of the functional configuration of an information processing device 100b according to the third embodiment. The information processing device 100b according to the third embodiment differs from the information processing device 100 according to the first embodiment in that it further includes a progress management unit 108 and a management information storage unit 109. The progress management unit 108 is realized, for example, by the processor 154 reading and executing software (computer program) from the memory 153. The management information storage unit 109 is realized, for example, by the storage device 152. Differences from the first embodiment will be specifically described below, and overlapping descriptions will be omitted as appropriate. Note that the features shown in this embodiment can also be combined with the features shown in the second embodiment.

[0122] The progress management unit 108 manages the progress of the annotation information creation work by the users for each user. That is, the progress management unit 108 manages the progress of the input for specifying elements of objects included in images for each user. For example, the progress management unit 108 may manage the number of images for which the annotation information creation work has been completed from the start of the work (e.g., the work start time or the work start date) to the present time for each user. In other words, the progress management unit 108 may manage for each user how many images the input for specifying elements of objects included in the images has been made for from the start of the work to the present time for each user. Alternatively, the progress management unit 108 may manage for each user how much time is required to create annotation information for one image. In other words, the progress management unit 108 may manage for each user how much time is required to input for specifying elements of objects included in one image for each user. In this way, the progress management unit 108 manages the work pace for each user.

[0123] The progress management unit 108 may manage the following information. For example, the progress management unit 108 may manage which user is responsible for creating annotation information for which image. The progress management unit 108 may also manage the number of operations required to create annotation information for one image. The progress management unit 108 may output the managed information as log data.

[0124] The progress management unit 108 generates progress management information that represents the progress status of each user, and stores the information in the management information storage unit 109. The management information storage unit 109 stores the progress management information generated by the progress management unit 108. In the configuration shown in FIG. 15, the management information storage unit 109 is included in the information processing device 100b, but the management information storage unit 109 may be realized in another device that is communicably connected to the information processing device 100b via a network or the like. The management information storage unit 109 may also be configured as a database.

[0125] In this embodiment, the user interface unit 105 outputs guide information for a user based on the user's work pace identified from the progress indicated in the progress management information. For example, the user interface unit 105 may output guide information to a user whose work pace does not meet a predetermined standard, regardless of whether the guide display button 808 is operated, i.e., regardless of whether there is a request from the user. In contrast, the user interface unit 105 may output guide information to a user whose work pace meets the predetermined standard only upon a request from the user. Furthermore, the user interface unit 105 may provide guide information with more information to a user whose work pace does not meet the predetermined standard than guide information for a user whose work pace meets the predetermined standard. Specifically, the user interface unit 105 may provide the following guide information to a user whose work pace does not meet the predetermined standard. For example, the user interface unit 105 may highlight and display an element corresponding to the element that the user should next place on the target image, among the elements described in an image showing an example of the placement of each graphic element (see FIG. 8 ). In contrast, the user interface unit 105 may omit such highlighting for a user whose work pace meets a predetermined standard. Also, for example, the user interface unit 105 may provide audio guide information to a user whose work pace does not meet the predetermined standard, and may omit providing such audio guide information to a user whose work pace meets the predetermined standard.

[0126] Fig. 16 is a flowchart showing an example of the operation of the information processing device 100b according to the third embodiment. The flowchart shown in Fig. 16 differs from the flowchart shown in Fig. 6 in that steps S300 and S302 are added and step S301 is included instead of step S101.

[0127] In this embodiment, after step S100, the process proceeds to step S300. However, step S300 may be performed before step S100. In step S300, if progress management information of the user who will be creating annotation information is stored in the management information storage unit 109, the user interface unit 105 acquires the progress management information from the management information storage unit 109. Note that if this is the first time the user is performing the work, this step may be skipped.

[0128] In this embodiment, after step S300, step S301 is performed instead of step S101. In step S301, the user interface unit 105 outputs guide information according to the user's work pace identified from the progress indicated in the progress management information, in addition to the processing of step S101 shown in FIG. 6. After step S301, the processing proceeds to step S102. Then, in this embodiment, after the processing of steps S102 and S103, the processing proceeds to step S302.

[0129] The progress management unit 108 monitors user operations on the user interface. When the process proceeds to step S302, the progress management unit 108 stores progress management information for the user in the management information storage unit 109. If progress management information for the user has not yet been stored in the management information storage unit 109, the progress management unit 108 generates new progress management information for the user and stores it in the management information storage unit 109. On the other hand, if progress management information for the user has already been stored in the management information storage unit 109, the progress management unit 108 updates the progress management information for the user in the management information storage unit 109. After step S302, the process proceeds to step S104. In the flowchart shown in FIG. 16, the storage process (update process) of the progress management information is performed after step S103, but the storage process (update process) may be performed at any timing. For example, the storage process (update process) of the progress management information may be performed after a Yes determination result is obtained in step S104.

[0130] The above describes the third embodiment. In this embodiment, the information processing device 100b manages the progress of the annotation information creation work for each user and outputs guide information based on the user's progress. Therefore, it is possible to provide appropriate guide information according to the user's level of proficiency.

[0131] <Fourth Embodiment> Next, a fourth embodiment will be described. In the first embodiment, an example of a configuration has been shown in which a user creates annotation information using the input device 151 and output device 150 of the information processing device 100. However, the user does not necessarily have to use the input device 151 and output device 150 of the information processing device 100. In particular, when the functions of the information processing device 100 are realized in a server device, the input device and output device of a client terminal communicably connected to the server device via a network may be used to create annotation information. Hereinafter, as the fourth embodiment, an embodiment in which the same functions as those of the first embodiment are provided by a server device and a client device will be described. Note that the configuration described below is not limited to the first embodiment, and can also be realized in the other embodiments described above.

[0132] FIG. 17 is a block diagram showing an example of the configuration of an information processing system 10 according to a fourth embodiment. As shown in FIG. 17, the information processing system 10 includes an information processing device 100c as a server and one or more terminal devices 200 as clients. The information processing device 100c and the terminal devices 200 are communicatively connected via wired or wireless communication. Each terminal device 200 can be used by a different user. Here, the user of each terminal device 200 may use the information processing device 100c to create annotation information about different objects. That is, the information processing device 100c can process various annotation information creation tasks. In this case, as described in the above-described embodiment, the information processing device 100c provides a user interface specialized for each task.

[0133] The functional configuration of the information processing device 100c is the same as the functional configuration of the information processing device 100 according to the first embodiment shown in Fig. 4, and therefore description thereof will be omitted. However, in this embodiment, the user interface unit 105 provides a user interface via the output device 150 of the terminal device 200, and accepts input to the user interface via the input device 151 of the terminal device 200. That is, in the first embodiment, the user interface unit 105 also has the functions of the output control unit 201 and the input accepting unit 202 described below, but in this embodiment, these functions are provided in the terminal device 200.

[0134] 18 is a block diagram showing an example of the functional configuration of the terminal device 200. As shown in FIG.

[0135] The output control unit 201 outputs a user interface provided from the information processing device 100c to the output device 150 of the terminal device 200. Furthermore, the input receiving unit 202 receives input from the user to the provided user interface. That is, the input receiving unit 202 receives input performed in the task of creating annotation information, such as an operation of specifying an element. Specifically, the input receiving unit 202 receives input via the input device 151 of the terminal device 200.

[0136] Fig. 19 is a block diagram showing an example of the hardware configuration of the information processing device 100c. As shown in Fig. 19, the information processing device 100c includes a storage device 152, a memory 153, a processor 154, and a network interface 155. In this way, the information processing device 100c has the functions of a computer. The storage device 152, the memory 153, and the processor 154 have already been described, so a description thereof will be omitted here.

[0137] The network interface 155 is used to communicate with other devices (specifically, the terminal device 200). The network interface 155 may include, for example, a network interface card (NIC). For example, the user interface unit 105 of the information processing device 100c provides a user interface to the terminal device 200 via the network interface 155.

[0138] Fig. 20 is a block diagram showing an example of the hardware configuration of the terminal device 200. As shown in Fig. 20, the terminal device 200 includes a network interface 156, an output device 150, an input device 151, a memory 157, and a processor 158. The output device 150 and the input device 151 shown in Fig. 20 are the same as those shown in Fig. 5, and therefore a description thereof will be omitted here.

[0139] The network interface 156 is used to communicate with other devices (specifically, the information processing device 100c). The network interface 156 may include, for example, a network interface card (NIC). For example, the output control unit 201 of the terminal device 200 receives a user interface via the network interface 156. Furthermore, the input receiving unit 202 transmits input received from a user to the information processing device 100c via the network interface 156.

[0140] The memory 157 is configured, for example, by a combination of volatile memory and non-volatile memory. The memory 157 is used to store software (computer programs) including one or more instructions executed by the processor 158, data used for various processes of the terminal device 200, and the like.

[0141] The processor 158 reads and executes software (computer programs) from the memory 157 to perform the processes of the output control unit 201 and the input receiving unit 202. The processor 158 may be, for example, a microprocessor, an MPU (Micro Processor Unit), or a CPU (Central Processing Unit). The processor 158 may include multiple processors. In this way, the terminal device 200 has the functionality of a computer.

[0142] The above describes the fourth embodiment. In this embodiment, the task of creating annotation information can be performed by the information processing system 10 including the information processing device 100c as a server and the terminal device 200 as a client. Therefore, the user only needs to have the terminal device 200 to create annotation information, thereby improving convenience.

[0143] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0144] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Appendix 1) a setting information acquisition means for acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface means for providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; a storage control means for controlling the storage of information on the element designated by the input in association with the image; An information processing device having the above. (Appendix 2) The setting information acquisition means further acquires task setting information indicating a set of one or more objects; The user interface means provides the user interface by referring to the object setting information of any of the objects included in the set indicated by the task setting information. 10. The information processing device according to claim 1. (Appendix 3) The user interface means provides the user interface by referring to the object setting information selected in accordance with the instruction. 3. The information processing device according to claim 2. (Appendix 4) the task setting information includes a definition of an order of the plurality of objects included in the set; The user interface means provides the user interface by referring to the object setting information selected in accordance with the order of the plurality of objects defined in the task setting information. 3. The information processing device according to claim 2. (Appendix 5) The object setting information further defines constraints on the placement of the elements on the image; The user interface means supports input for specifying the element based on the constraints. 5. The information processing device according to any one of claims 1 to 4. (Appendix 6) The constraints restrict the relative relationships of the elements on the image. 6. The information processing device according to claim 5. (Appendix 7) The user interface means outputs a warning when the received input does not satisfy the constraint. 7. The information processing device according to claim 5 or 6. (Appendix 8) When the received input does not satisfy the constraint, the user interface means corrects the position of the element specified by the input to a position that satisfies the constraint. 8. An information processing device according to any one of appendices 5 to 7. (Appendix 9) the object setting information includes a definition indicating whether the element must be added to the image; The object setting information defines an order in which the elements must be added; The user interface means sequentially provides user interfaces set based on the definitions of the elements that must be added in accordance with the order, and provides user interfaces corresponding to the definitions of the elements that are not required to be added in response to an instruction to switch the user interfaces. 9. An information processing device according to any one of appendices 1 to 8. (Appendix 10) The user interface means accepts a correction of the information of the element designated by the input. 10. An information processing device according to any one of appendices 1 to 9. (Appendix 11) The user interface means outputs guide information that guides input for specifying the element of the object. 11. The information processing device according to any one of Supplementary Notes 1 to 10. (Appendix 12) The method further includes a progress management unit that manages the progress of input for specifying the element of the object for each user, The user interface means outputs the guide information for the user based on the user's work pace identified from the progress. 12. The information processing device according to claim 11. (Appendix 13) The apparatus further includes a setting information generating means for generating the object setting information based on the received input, The setting information acquisition means acquires the object setting information generated by the setting information generation means. 13. An information processing device according to any one of appendices 1 to 12. (Appendix 14) an information processing device; Terminal device and Equipped with The information processing device includes: a setting information acquisition means for acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface means for providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; a storage control means for controlling the storage of information on the element designated by the input in association with the image; and The terminal device an output control means for outputting the user interface provided by the information processing device to an output device; an input receiving means for receiving an input to the user interface; have Information processing system. (Appendix 15) The setting information acquisition means further acquires task setting information indicating a set of one or more objects; The user interface means provides the user interface by referring to the object setting information of any of the objects included in the set indicated by the task setting information. 15. The information processing system of claim 14. (Appendix 16) obtaining object setting information including definitions of one or more elements to be added to an image to indicate an object included in the image and a definition of the order of the elements; providing a user interface configured based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; Controlling the storage of information about the element designated by the input in association with the image. Information processing methods. (Appendix 17) a setting information acquisition step of acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface step of providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information in order to receive an input for specifying the elements of the object; a storage control step of controlling the storage of information of the element designated by the input in association with the image; A non-transitory computer-readable medium storing a program that causes a computer to execute the program. [Explanation of symbols]

[0145] 1. Information processing equipment 2. Setting information acquisition section 3 User Interface 4 Memory control unit 10 Information Processing Systems 100 Information processing device 100a Information processing device 100b Information processing device 100c Information processing equipment 101 Setting information storage unit 102 Target image storage unit 103 Annotation Storage Unit 104 Setting information acquisition unit 105 User Interface Section 106 Memory control unit 107 Configuration information generation unit 108 Progress Management Department 109 Management information storage unit 150 Output Device 151 Input Device 152 Storage device 153 memory 154 processors 155 Network Interface 156 Network Interface 157 memory 158 processors 200 Terminal Device 201 Output control section 202 Input reception unit

Claims

1. a setting information acquisition means for acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface means for providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive an input for specifying the elements of the object; a storage control means for controlling the storage of information on the element designated by the input in association with the image; and the object setting information includes a definition indicating whether the element must be added to the image; The object setting information defines an order in which the elements must be added; The user interface means sequentially provides user interfaces set based on the definitions of the elements that must be added in accordance with the order, and provides user interfaces corresponding to the definitions of the elements that are not required to be added in response to an instruction to switch the user interfaces. Information processing device.

2. The setting information acquisition means further acquires task setting information indicating a set of one or more objects; The user interface means provides the user interface by referring to the object setting information of any of the objects included in the set indicated by the task setting information. The information processing device according to claim 1 .

3. The user interface means provides the user interface by referring to the object setting information selected in accordance with the instruction. The information processing device according to claim 2 .

4. the task setting information includes a definition of an order of the plurality of objects included in the set; The user interface means provides the user interface by referring to the object setting information selected in accordance with the order of the plurality of objects defined in the task setting information. The information processing device according to claim 2 .

5. The object setting information further defines constraints on the placement of the elements on the image; The user interface means supports input for specifying the element based on the constraints. The information processing device according to claim 1 .

6. The constraints restrict the relative relationships of the elements on the image. The information processing device according to claim 5 .

7. The user interface means outputs a warning when the received input does not satisfy the constraint.

7. The information processing device according to claim 5 or 6.

8. When the received input does not satisfy the constraint, the user interface means corrects the position of the element specified by the input to a position that satisfies the constraint. The information processing device according to claim 5 .

9. A computer acquires object setting information including a definition of each of one or more elements to be added to an image to indicate an object contained in the image and a definition of the order of the elements; the computer provides a user interface configured based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information, in order to receive input for specifying the elements of the object; controlling the computer to store information about the element designated by the input in association with the image; the object setting information includes a definition indicating whether the element must be added to the image; The object setting information defines an order in which the elements must be added; In providing the user interfaces, the user interfaces set based on the definition of the element that must be added are provided in sequence in the order, and the user interfaces corresponding to the definition of the element that is not required to be added are provided in response to an instruction to switch the user interfaces. Information processing methods.

10. a setting information acquisition step of acquiring object setting information including definitions of one or more elements added to an image to indicate an object included in the image and a definition of the order of the elements; a user interface step of providing a user interface set based on definitions of the elements indicated in the object setting information in accordance with the order of the elements defined in the object setting information in order to receive an input for specifying the elements of the object; a storage control step of controlling the storage of information of the element designated by the input in association with the image; on the computer, the object setting information includes a definition indicating whether the element must be added to the image; The object setting information defines an order in which the elements must be added; In the user interface step, the user interfaces set based on the definitions of the elements that must be added are provided in order according to the order, and the user interfaces corresponding to the definitions of the elements that are not required to be added are provided in response to an instruction to switch the user interfaces. program.

Images