Information processing device, information processing method, and information processing program

Abstract

Efficiently add annotation information to images with intuitive operation. [Solution] The information processing device 4 generates annotation information to be applied to the target image. The information processing device 4 includes a display device 44 that displays an image with a predetermined area in space as the display area, a display control unit 413 that controls the display device 44 so that the target image is displayed in the predetermined area, an imaging device 46 that captures the movements of a worker or the movements of a work tool in accordance with said movements with respect to the target image displayed in the predetermined area, an motion analysis unit 414 that analyzes the movements of the worker or the movements of the work tool captured by the imaging device 46, and an annotation processing unit 415 that generates annotation information to be applied to objects in the target image based on the analysis results of the motion analysis unit 414.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program for processing information related to images.

Background Art

[0002] Conventionally, there has been known an apparatus configured to add annotation information to an image by labeling or the like (see, for example, Patent Document 1). In the apparatus described in Patent Document 1, audio information input via a microphone is added to an image as annotation information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the method of adding annotation information to an image by voice as in the apparatus described in Patent Document 1 mentioned above has difficulty in correcting erroneous inputs and the like.

Means for Solving the Problems

[0005] An information processing apparatus according to one aspect of the present invention is an information processing apparatus that generates annotation information to be added to a target image, and includes a display device that displays an image with a predetermined region in space as a display region, a display control unit that controls the display device such that the target image is displayed in the predetermined region, an imaging device that images the motion of a worker or the movement of a work tool following the motion with respect to the target image displayed in the predetermined region, an analysis unit that analyzes the motion of the worker or the movement of the work tool imaged by the imaging device, and an annotation processing unit that generates annotation information to be added to an object in the target image based on the analysis result of the analysis unit.

[0006] Another aspect of the present invention is an information processing device that generates annotation information to be applied to a target image, comprising: a display device that displays a three-dimensional virtual space based on a target image; a detection unit that detects the real-space position of a work tool whose real-space position is calibrated with the virtual-space position; an analysis unit that analyzes changes in the virtual-space position of a work tool held by an operator based on the real-space position detected by the detection unit; and an annotation processing unit that generates annotation information to be applied to objects in the virtual space displayed based on the target image based on the analysis results of the analysis unit.

[0007] An information processing device in yet another aspect of the present invention is an information processing device for generating annotation information to be applied to a target image, comprising: a wearable display device that displays a three-dimensional virtual space based on a target image; an imaging device calibrated to the coordinate system of the virtual space; a detection unit that detects the movements of an operator in real space based on an image of the operator wearing the display device acquired by the imaging device; an analysis unit that analyzes the movements of the operator in the virtual space based on the movements in real space detected by the detection unit; and an annotation processing unit that generates annotation information to be applied to objects in the virtual space displayed based on the target image based on the analysis results of the analysis unit. [Effects of the Invention]

[0008] According to the present invention, annotation information can be efficiently added to images through intuitive operation. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram schematically showing the overall configuration of an information processing system including an information management device according to an embodiment of the present invention. [Figure 2] A block diagram illustrating the main components of the control device shown in Figure 1. [Figure 3A] A diagram showing the data recorded in the image database. [Figure 3B] A diagram showing the data recorded in the annotator database. [Figure 4] A block diagram illustrating the main components of the annotator terminal 2 shown in Figure 1. [Figure 5] A flowchart showing an example of the process executed by the controller in Figure 2. [Figure 6] A flowchart showing an example of the process in step S106 of Figure 5. [Figure 7] A flowchart showing an example of the process in step S107 of Figure 5. [Figure 8] A flowchart showing an example of the process executed by the controller in Figure 4. [Figure 9] A diagram showing an example of a target image that includes a vehicle. [Figure 10A] Figure 4 shows an example of the target image displayed on the display device. [Figure 10B] Figure 4 shows another example of the target image displayed on the display device. [Figure 10C] Figure 4 shows another example of the target image displayed on the display device. [Figure 11] A block diagram illustrating another example of the overall configuration of an information processing system. [Figure 12] A block diagram illustrating the main components of the information processing device shown in Figure 11. [Figure 13] A diagram illustrating annotation operations. [Modes for carrying out the invention]

[0010] Embodiments of the invention will now be described with reference to the drawings. An information management device (hereinafter simply referred to as the management device) according to an embodiment of the present invention is a device for supporting the annotation work of an annotator. Figure 1 is a schematic block diagram showing an example of the overall configuration of an information processing system 100 including the management device 1 according to this embodiment. As shown in Figure 1, the information processing system 100 includes the management device 1 and a communication terminal (hereinafter referred to as the annotator terminal) 2 used by the annotator.

[0011] The management device 1 and the annotator terminal 2 are communicably connected via a communication network 3. The communication network 3 includes not only public wireless communication networks represented by the Internet network and mobile phone networks, but also closed communication networks provided for each predetermined management area, such as wireless LAN, Wi-Fi (registered trademark), and Bluetooth (registered trademark).

[0012] The annotator terminal 2 is a communication terminal such as a tablet terminal. Although one annotator terminal 2 is illustrated in FIG. 1, a plurality of annotator terminals 2 may be connected to the management device 1 via the communication network 3.

[0013] FIG. 2 is a block diagram illustrating the main configuration of the management device 1 in FIG. 1. The management device 1 includes a controller 10 and a communication unit 13.

[0014] The communication unit 13 is a communication interface that connects the management device 1 to the communication network 3. The management device 1 transmits and receives information to and from the annotator terminal 2 and the like connected to the communication network 3 via the communication unit 13.

[0015] The controller 10 is configured to include a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and a RAM, and other peripheral circuits (not shown) such as an I / O interface. The arithmetic unit 11 functions as a task management unit 111, a task setting unit 112, a quality analysis unit 113, and a learning unit 115 by executing programs stored in the storage unit 12.

[0016] The storage unit 12 includes an image DB (database) 121 and an annotator DB 122. The image DB 121 stores image data to be subjected to annotation (hereinafter referred to as target image data, or simply a target image), and image data for which annotation has been completed (hereinafter referred to as processed image data, or simply a processed image).

[0017] The target images are classified into groups based on at least one of the following: the imaging environment, image features (such as brightness), and the type of object in the image, and stored in the image database 121. Processed images are images that do not require additional annotation and can be used as training data for machine learning models. For each image (target image and processed image), the image database 121 stores the image data representing the image, status information, difficulty information, and assigned annotation information, associated with the image ID. Figure 3A shows the data recorded in the image database 121.

[0018] Status information indicates whether or not annotation information has been added to the image, that is, whether or not annotation information has been added to all objects in the image that should be annotated. If the image contains objects of different types, status information for each type of object is stored, associated with the image ID of each target image, as shown in Figure 3A. Difficulty information indicates the difficulty level of annotation on the image.

[0019] The annotator DB122 stores information about the annotator (hereinafter referred to as annotator information). The annotator DB122 also stores work time statistics and quality information, which will be described later. Figure 3B shows the data recorded in the annotator DB122.

[0020] As shown in Figure 3B, the annotator information includes work history information indicating the annotator's work history and characteristic information indicating the annotator's work characteristics, associated with the annotator's identification information (hereinafter referred to as the annotator ID). The annotator information further includes condition information indicating the annotator's condition (such as physical condition and concentration level).

[0021] As shown in Figure 3B, the work history information includes identification information (hereinafter referred to as image ID) for each target image that the annotator has annotated. The work history information also includes pass count information, indicating the number of times the annotator performed pass operations on each target image, and work time information, indicating the annotator's work time on each target image. Pass operations will be described later.

[0022] As shown in Figure 3B, the characteristic information includes skill information indicating the annotator's work skill level and tendency information indicating the types of work (images) the annotator struggles with. Images that the annotator struggles with are those that require more than a predetermined number of path operations or those with low annotation accuracy. The tendency information includes the image ID or image group to which images that require more than a predetermined number of path operations belong among the images the annotator works with, and the image ID of images whose annotation accuracy is below a predetermined level. The tendency information may also include the types of work (images) the annotator excels at. Images that the annotator excels at are, for example, images with short annotation tasks or images with high annotation accuracy.

[0023] The task management unit 111 manages the tasks assigned to the annotator. The task is to add annotation information to objects in an image. The annotation information includes segmentation information, which represents the boundaries of the object with polygons, and labeling information, which indicates the type of object (vehicle, pedestrian, traffic sign, traffic light, etc.).

[0024] The task management unit 111 detects changes in the annotator's work time based on work time statistics stored in the annotator DB 122. The task management unit 111 also detects changes in the annotator's work quality based on quality information stored in the annotator DB 122. Furthermore, the task management unit 111 generates condition information based on the detected changes in work time or work quality and stores it in the annotator DB 122.

[0025] When the task management unit 111 receives a login request (authentication request) from the annotator terminal 2, it reads the target image to be sent to the annotator terminal 2 from the image DB 121 based on the annotator ID associated with the login request. Specifically, the task management unit 111 first obtains annotator information, or more precisely characteristic information, stored in association with the annotator ID of the annotator from the annotator DB 122.

[0026] The task management unit 111 determines the task (work content) to be assigned to the annotator based on the annotator information, or more specifically, the characteristic information, stored in the annotator DB 122.

[0027] The task management unit 111 recognizes the annotator's annotation skill level based on the skill information included in the characteristic information. Based on the difficulty information stored in the image database 121, the task management unit 111 reads a series of target images (for example, 25 images) from the image database 121 whose annotation difficulty corresponds to the annotator's annotation skill level. This allows the annotator to be assigned target images with a difficulty level appropriate to their annotation skill level.

[0028] At this time, the task management unit 111 extracts a series of target images from the image DB 121 based on the trend information included in the characteristic information, so as to exclude images that the annotator is not good at. This prevents images that the annotator is not good at from being assigned to the annotator as target images.

[0029] The task management unit 111 may classify target images based on the imaging environment in which the image was acquired (e.g., a highway), the characteristics of the image (e.g., brightness (luminance)), and the type of object contained in the image (e.g., a vehicle), and extract target images of the same classification as a series of target images. By assigning target images of the same classification to the annotator in this way, the work efficiency of the annotator can be improved.

[0030] Furthermore, when classifying target images based on the type of object, the task management unit 111 applies a preliminary segmentation as a preprocessing step to each target image stored in the image database 121 in advance, and recognizes the type of object contained in each target image. Preliminary segmentation is, for example, semantic segmentation.

[0031] The task management unit 111 transmits a series of target images read from the image database 121 to the annotator terminal 2 via the communication unit 13. In this way, a task is assigned to the annotator. At this time, the task management unit 111 transmits the series of target images along with window setting information and mask setting information, which will be described later.

[0032] The task setting unit 112 sets the tasks assigned to the annotator so that annotation of one target image is performed on one object within the image. Specifically, the task setting unit 112 sets a window (hereinafter referred to as the work window) for the target image that indicates the area to which annotation information is to be applied. More specifically, the task setting unit 112 generates information (hereinafter referred to as window setting information) that specifies the shape, size, transparency, and scanning method of the work window. The shape of the work window is assumed to be rectangular.

[0033] The task setting unit 112 prioritizes applying the smallest window from among several pre-prepared windows of different sizes to the work window. The annotator may also specify the size of the window to be applied to the work window from among the multiple windows of different sizes. Furthermore, the task setting unit 112 may set the size of the work window based on the size of objects assumed to exist within the target image.

[0034] For example, if the target image is an image captured by an onboard camera of a vehicle traveling on a highway, that is, if the environment in which the target image was captured is a highway, then it is assumed that a vehicle is present in the image. Therefore, in that case, the task setting unit 112 sets the size of the work window based on the maximum vertical and horizontal size of the vehicle (vehicle ahead or oncoming vehicle) estimated within the field of view of the onboard camera.

[0035] Furthermore, the task setting unit 112 may determine a typical size for the work window and apply that size to the work window that will be superimposed on the target image. The typical size for the work window may be determined by processing the processed image stored in the image DB 121 to calculate the distribution of the shape and size of the bounding box applied to the objects in the image, or by other methods.

[0036] Furthermore, the task setting unit 112 may classify the processed images stored in the image DB 121 according to the type of object to which annotation information has been assigned, and apply the size and shape most frequently applied to processed images classified in the same group to the work window.

[0037] Furthermore, the task setting unit 112 sets the tolerance for annotation. The further back an object is in the image (the farther away it is from the image viewpoint), the fewer pixels make up the region corresponding to that object (hereinafter referred to as the number of constituent pixels). If the number of constituent pixels is small, the contours and fine features of the object cannot be sufficiently represented, making precise segmentation difficult. Therefore, the task setting unit 112 sets a larger tolerance for annotation for objects with fewer constituent pixels, and sets a lower annotation accuracy required of the annotator. Annotation accuracy includes segmentation accuracy, which is evaluated by the amount of segmentation overhang and the size of the margins, and labeling accuracy, which is evaluated by labeling errors and labeling omissions.

[0038] Furthermore, the task setting unit 112 sets a mask (Figure 10C, described later) on the target image so that the areas of objects to which annotation information has already been applied cannot be seen. More specifically, the task setting unit 112 generates information (hereinafter referred to as mask setting information) that specifies the shape, size, and display position of the mask corresponding to the object, based on the annotation information attached to the object in the target image.

[0039] Furthermore, the task setting unit 112 sets the next tasks for the annotator that performed the path operation so that target images whose annotation was skipped due to the path operation are not assigned to that annotator. More specifically, the task setting unit 112 sets the next tasks for the annotator that performed the path operation so that target images belonging to the same group as the target images whose annotation was skipped are not assigned to that annotator.

[0040] The task setting unit 112 determines the difficulty level of annotation for each target image based on the number of pass operations performed on the target image, or the number of annotators who performed pass operations on the target image. The task setting unit 112 stores the determined difficulty level information for each target image, along with the image ID of each target image, in the image DB 121. A pass operation is an operation to skip annotation on a target image that is currently being displayed, and is input by the annotator via the annotator terminal 2. Pass operations will be described later.

[0041] The quality analysis unit 113 monitors the annotation by the annotator. More specifically, the quality analysis unit 113 monitors the time required for the annotator's work and the quality of the annotation information. This monitoring may be performed in real time or at predetermined intervals.

[0042] The quality analysis unit 113 monitors the annotator's working time based on the annotator information recorded in the annotator DB 122, and more specifically, the work history information. The quality analysis unit 113 also monitors the quality of the annotation information (segmentation information and labeling information) applied to the target image by the annotator using ground truth data created with computer graphics (CG), etc. Specifically, the quality analysis unit 113 evaluates the quality of the segmentation information as higher the amount of segmentation overhang and the size of the margins are smaller compared to the ground truth data. The quality analysis unit 113 also evaluates the quality of the labeling information as higher the number of detected labeling errors and missing labels is lower.

[0043] The quality analysis unit 113 stores work time statistics, which show the monitoring results of the annotator's work time, and quality information, which shows the monitoring results of the quality of the annotation information, in the annotator DB 122. Furthermore, the quality analysis unit 113 generates skill information, which indicates the level of the annotator's annotation skill, based on the monitoring results of the quality of the annotation information, and stores it in the annotator DB 122. The quality analysis unit 113 evaluates the annotator's annotation skill more highly the higher the quality of the annotation information attached to the target image by the annotator.

[0044] Figure 4 is a block diagram illustrating the main components of the annotator terminal 2 shown in Figure 1. The annotator terminal 2 includes a controller 20, a communication unit 23, a display device 24, and an operation unit 25.

[0045] The communication unit 23 is a communication interface that connects the annotator terminal 2 to the communication network 3. The annotator terminal 2 sends and receives information with the management device 1 and other devices connected to the communication network 3 via the communication unit 13.

[0046] The display device 24 is comprised of a liquid crystal display, organic EL display, or the like, equipped with a touch panel. The operation unit 25 is comprised of the touch panel of the display device 24 and is configured to allow various instructions from the user to be input. The operation unit 25 may also be comprised of input devices such as switches and buttons.

[0047] The controller 20 is comprised of a computer having an arithmetic unit 21 such as a CPU, a storage unit 22 such as ROM and RAM, and other peripheral circuits not shown such as an I / O interface. The arithmetic unit 21 functions as an operation detection unit 211, an acquisition unit 212, a display control unit 213, and an annotation processing unit 214 by executing a program stored in the storage unit 22.

[0048] The operation detection unit 211 detects annotator operations (hereinafter sometimes referred to as user operations) input via the operation unit 25.

[0049] When the annotator starts work, it launches a predetermined application (hereinafter referred to as the work application) that is pre-installed in the storage unit 22 via the operation unit 25. When the work application is launched, the login screen (not shown) for the work application is displayed on the display device 24. The annotator performs a login operation on the login screen via the operation unit 25, including entering the annotator ID.

[0050] When the operation detection unit 211 detects a login operation, the acquisition unit 212 sends a login request to the management device 1 via the communication unit 13, along with the input annotator ID. In response to the login request, the acquisition unit 212 receives a series of target images, window setting information, and mask setting information transmitted from the task management unit 111 of the management device 1.

[0051] After logging in, when the annotator presses the start button (not shown) displayed on the display device 24 via the operation unit 25, the display control unit 213 starts displaying the target image on the display device 24. The display control unit 213 displays a series of target images acquired by the acquisition unit 212 on the display device 24, switching between them. The annotator adds annotation information, namely segmentation information and labeling information, to the target images displayed sequentially on the display device 24 via the operation unit 25.

[0052] The display control unit 213 switches the display to the next target image when a predetermined time has elapsed since the start of displaying the target image, or when a pass operation is detected by the operation detection unit 211 while the target image is being displayed. A pass operation is a predetermined operation to skip annotation on the currently displayed image, such as a flick.

[0053] Furthermore, when the display control unit 213 displays the target image on the display device 24, it overlays the work window on the target image based on the window setting information. The display control unit 213 then scans the work window on the target image. The scanning method may be a sliding window method that scans from the top left to the bottom right of the image, or other methods may be used.

[0054] Furthermore, when the display control unit 213 displays the target image on the display device 24, it overlays a mask composed of single-color pixels onto the position of an object in the target image, based on the mask setting information, so that objects to which annotation information has already been assigned cannot be seen.

[0055] When the annotation processing unit 211 detects a segmentation operation, the annotation processing unit 214 generates segmentation information based on the display position of the work window at the time of detection. More specifically, the annotation processing unit 214 acquires the position coordinates of the corners and center of the work window as segmentation information. A segmentation operation is a predetermined operation for inputting segmentation information, such as tapping inside the work window while sliding it across the target image. Alternatively, the annotator may input the contour of an object on the display device or place a bounding box.

[0056] When a series of target images acquired by the acquisition unit 212 are classified based on the type of object contained in the images, the annotation processing unit 214 acquires that type as labeling information. Furthermore, when a series of target images are classified based on the imaging environment, the annotation processing unit 214 may acquire the type of object expected from that imaging environment as labeling information. For example, if the imaging environment is a "highway," "vehicle" is acquired as labeling information.

[0057] Furthermore, when the type of object is input by the annotator via the operation unit 25, the annotation processing unit 214 may acquire the input type of object as labeling information. In this case, the annotator inputs the type of object by operating the label selection screen (not shown) via the operation unit 25. The label selection screen is displayed near the work window by the display control unit 213 when a segmentation operation is detected by the operation detection unit 211. Multiple types of objects are displayed on the label selection screen, and when any of them are tapped, the annotation processing unit 214 acquires the type of the tapped object as labeling information.

[0058] When the annotation processing unit 214 detects a work completion operation by the operation detection unit 211, it transmits a work completion notification to the management device 1 via the communication unit 13. At this time, along with the work completion notification, the annotation processing unit 214 transmits performance information indicating the annotator's work performance to the management device 1. A work completion operation is a predetermined operation that indicates the end of work, for example, pressing a work completion button (not shown) displayed on the display device 24.

[0059] The performance information includes work time information, which indicates the time it took the annotator to add annotation information (the time from when one target image is displayed to when the next target image is displayed), associated with the image ID of each target image in a series, and path operation history information, which indicates whether or not path operations were performed. The performance information also includes the annotation information attached to the target images.

[0060] Figure 5 is a flowchart showing an example of a process executed by the controller 10 of the management device 1 according to a predetermined program. The process shown in this flowchart starts when the controller 10 is activated and is repeated at predetermined intervals.

[0061] First, in step S101, the controller 10 determines whether or not it has received a login request from the annotator terminal 2. If the result in step S101 is negative, the controller 10 terminates processing. If the result in step S101 is positive, in step S102, the controller 10 performs login authentication based on the annotator ID and password associated with the login request. If successful, it retrieves the annotator's characteristic information from the annotator DB. If login authentication fails, the controller 10 terminates processing.

[0062] The controller 10 recognizes the characteristics of the annotator based on the acquired characteristic information. Specifically, the controller 10 recognizes the annotator's annotation accuracy based on the skill information included in the characteristic information. In addition, the controller 10 recognizes the types of work (images) that the annotator struggles with based on the tendency information included in the characteristic information.

[0063] In step S103, the controller 10 reads a series of target images from the image DB 121, where the difficulty of annotation corresponds to the annotator's level of annotation skill. At this time, the controller 10 extracts a series of target images based on the tendency information contained in the characteristic information, so as to exclude images that the annotator is not good at, or to include images that the annotator is good at.

[0064] In step S104, the controller 10 transmits the series of target images acquired in step S103 to the annotator terminal 2. In step S105, the controller 10 determines whether the annotator's work on the series of target images transmitted in step S104 is complete. At this time, the controller 10 determines that the work is complete when it receives a work completion notification from the annotator terminal 2. The work completion notification is accompanied by performance information indicating the annotator's work performance.

[0065] The process in step S105 is repeated until a positive result is obtained. If a positive result is obtained in step S105, the controller 10 performs the image saving process (step S106) and the quality analysis process (step S107), and then terminates the process.

[0066] Figure 6 is a flowchart showing an example of the processing (image saving process) in step S106 of Figure 5. The controller 10 executes the processing in steps S61 to S64 for each of the series of target images acquired in step S103, that is, the series of target images that are to be annotated.

[0067] First, in step S61, the controller 10 determines, based on the performance information, whether or not new annotation information has been added to the target image, in other words, whether or not a path operation has been performed on the target image. If the result in step S61 is positive, the controller 10 proceeds to the process in step S63.

[0068] If the result in step S61 is positive, in step S62, the annotation information attached to the target image, which is included in the performance information, is associated with the image ID of the target image and saved in the image DB 121. At this time, if annotation information corresponding to that image ID is already stored in the image DB 121, the controller 10 updates the annotation information stored in the image DB 121.

[0069] In step S63, the controller 10 determines whether the target image satisfies the annotation completion conditions. The annotation completion conditions are determined by whether annotation information has been applied to all objects in the target image, more specifically, to all objects that belong to the types predetermined as targets for annotation information. The annotation completion conditions may also be determined by whether all predetermined work window sizes have been applied.

[0070] If the result in step S63 is negative, the controller 10 proceeds to step S65. If the result in step S63 is positive, in step S64, the controller 10 updates the status information corresponding to the target image stored in the image DB 121 to "Completed". The initial value of the status information is "Incomplete". Target images whose status information has been updated to "Completed" are treated as processed images thereafter.

[0071] In step S65, the controller 10 determines whether or not there are any unprocessed target images. If the result in step S65 is positive, the controller 10 repeats the processing in steps S61 to S64 for the unprocessed target images. If the result in step S65 is negative, the controller 10 terminates the processing.

[0072] Figure 7 is a flowchart showing an example of the process (quality analysis process) in step S107 of Figure 5.

[0073] First, in step S71, the controller 10 updates the work history information recorded in the annotator DB122 based on the work time information for each target image included in the performance information received in step S105.

[0074] In step S72, the controller 10 uses pre-created ground truth data to analyze the quality of annotation information for each target image included in the performance information received in step S105. In step S73, the controller 10 updates the annotator skill information stored in the annotator DB 122 based on the analysis results in step S72. In step S74, the controller 10 determines whether or not there is a target image for which a path operation has been performed, based on the path operation history information included in the performance information.

[0075] If the result in step S74 is positive, the controller 10 updates the pass count information recorded in the annotator DB 122 based on the pass operation history information and proceeds to step S76. If the result in step S74 is negative, in step S75, the controller 10 determines whether or not there are target images with annotation accuracy below a predetermined level based on the analysis results in step S72. If the result in step S75 is negative, the process ends. If the result in step S75 is positive, in step S76, the controller 10 updates the annotator trend information stored in the annotator DB 122 and ends the process.

[0076] Figure 8 is a flowchart showing an example of a process executed by the controller 20 of the annotator terminal 2 according to a predetermined program. The process shown in this flowchart starts when the work application is launched and is repeated at predetermined intervals.

[0077] First, in step S201, the controller 20 determines whether or not it has received a login request via the operation unit 25. In step S202, the controller 20 sends a login request (authentication request) to the management device 1 along with the annotator ID and password entered in the login operation. In step S203, the controller 20 determines whether or not authentication was successful. If the authentication is denied in step S201 or S203, the process terminates.

[0078] If affirmed in step S203, in step S204, the controller 20 acquires a series of target images sent from the management device 1 in response to the login request. The controller 20 also acquires window setting information and mask setting information associated with the series of target images. Here, we will use the example where an image including a vehicle is acquired as the series of target images. Figure 9A shows an example of a target image including a vehicle. Figure 9A shows an image IM acquired by an onboard camera of a vehicle traveling on a highway HW. As shown in Figure 9A, the target image includes the vehicles ahead VH1, VH2, and VH3.

[0079] Once a series of target images have been acquired, the controller 20 displays a work start screen (not shown) on the display device 24. The work start screen includes a work start button and a message explaining the work to the annotator (for example, "Tap the window when the car is included within the square window."). The annotator presses the work start button via the operation unit 25 when starting the work.

[0080] In step S205, the controller 20 determines whether the start button has been pressed. The process in step S205 is repeated until a positive determination is made. If a positive determination is made in step S205, in step S206, the controller 20 displays the target image on the display device 24 with a work window of the specified shape and size superimposed on it, based on the window setting information. The controller 20 also scans the target image with the work window according to the specified scanning method, based on the window setting information. Furthermore, the controller 20 superimposes a mask onto the target image as needed, based on the mask setting information.

[0081] Figures 10A to 10C show examples of target images displayed on the display device 24. Figure 10A shows a target image with a work window WD superimposed on it. As shown in Figure 10A, the area other than the work window WD is displayed with lower transparency than the work window. This allows the annotator to easily recognize the work window WD superimposed on the target image IM. The area other than the work window WD is displayed with a transparency of a predetermined value or higher so that the annotator can confirm the contents of the target image. That is, the area other than the work window WD is displayed with a transparency of a predetermined value or higher and lower than the work window.

[0082] Furthermore, as shown in Figure 10A, the work window WD slides across the target image IM. The white arrows and dashed lines in the figure schematically represent the sliding movement of the work window WD. When the scanning method is the sliding window method, it slides from the upper left to the lower right of the target image IM.

[0083] In step S207, the controller 20 determines whether or not a path operation has been received via the operation unit 25. If the result in step S207 is positive, the controller 20 proceeds to the process in step S210. If the result in step S207 is negative, in step S208, it determines whether or not a tap operation on the work window has been detected.

[0084] If step S208 is rejected, the controller 20 returns to the process in step S207. If step S208 is accepted, in step S209, the controller 20 generates segmentation information based on the position coordinates, shape, and size of the work window at the time the tap operation is detected. The controller 20 also generates labeling information indicating the type of object in the work window (in this example, a vehicle). Furthermore, the controller 20 generates annotation information based on the segmentation information and the labeling information.

[0085] Figure 10B shows the state when the work window WD slides to the position of vehicle VH1. In the display state of Figure 10B, when the annotator taps the work window WD, annotation information corresponding to vehicle VH1 is generated based on the position coordinates, shape, and size of the work window WD. In subsequent annotations on the target image IM, the area corresponding to vehicle VH1 for which annotation information has already been generated will be masked. Figure 10C shows the target image IM with the mask MR superimposed on the area corresponding to vehicle VH1.

[0086] In step S210, the controller 20 determines whether or not there are any unprocessed target images. If the result in step S210 is positive, the controller 20 returns to the process in step S206. If the result in step S210 is negative, in step S211, the controller 20 sends a work completion notification to the management device 1.

[0087] According to the embodiments described above, the following effects and advantages can be obtained. (1) The management device 1 includes a task management unit 111 that assigns the task of adding annotation information to objects in an image to an annotator (worker), a storage unit 12 that stores target images which are images to be annotated by the annotator, and a task setting unit 112 that sets a task to add annotation information to one object included in the target image. The task management unit 111 stores the task information set by the task setting unit 112 in the storage unit 12 in association with the target image. As a result, the annotator only needs to add annotation information to one object for each target image, so annotation can proceed smoothly.

[0088] (2) The task setting unit 112 sets a task so that annotation information is added to multiple target images stored in the memory unit 12, each containing an object of the same type. This allows the annotator to process objects of the same type consecutively. As a result, a reduction in the cognitive load on the annotator is expected.

[0089] (3) The task setting unit 112 sets the task so that annotation information is applied to objects within a predetermined size work window superimposed on the target image. This eliminates the need for the annotator to check areas outside the work window, allowing them to concentrate on objects within the work window.

[0090] (4) The task setting unit 112 further sets the task so that areas outside the work window are displayed with lower transparency than the work window. This allows the annotator to see areas outside the work window and proceed with annotation while understanding the scene in which the target image was captured.

[0091] (5) The task setting unit 112 sets tasks so that multiple windows of different sizes are applied to a single target image. This makes it easier to detect objects by using windows that are appropriate for the size of the objects.

[0092] (6) The task setting unit 112 further sets the task so that areas in the target image other than the area corresponding to the object to which annotation information has been added are masked. This reduces the area that is subject to annotation.

[0093] (7) The annotator terminal 2 includes a communication unit 23 that communicates with the management device 1. The annotator terminal 2 includes a display device 24, an acquisition unit 212 that acquires a series of target images stored in the storage unit 12 of the management device 1 via the communication unit 23, a display control unit 213 that displays the series of target images acquired by the acquisition unit 212 on the display device 24 while switching between them, and an operation detection unit 211 that detects operations performed by the annotator on the target images displayed on the display device 24 (hereinafter referred to as user operations). The display control unit 213 switches the display to the next target image when a predetermined time has elapsed since the display of a target image, or when a predetermined user operation is detected by the operation detection unit 211 while a target image is being displayed. As a result, the annotator can skip annotation of images that are difficult to work with, and can proceed with the work at a good pace.

[0094] (8) When the display control unit 213 switches the display according to a predetermined user operation, the task setting unit 112 sets the next tasks to be assigned to the annotator so that the target image that was displayed before the display switch does not become the subject of annotation information to be added by the annotator. This prevents the annotator from being presented with images they are not comfortable with again, and is expected to improve the annotator's work efficiency.

[0095] (9) Each target image is classified into groups based on at least one of the imaging environment and the type of object in the image, and stored in the storage unit 12. The task setting unit 112 sets the tasks to be assigned to the annotator for the next time onward so that target images classified into the same group as the target image before the display switch are not to be subject to annotation information by the annotator. This prevents the annotator from being presented with images they are not comfortable with again, and is expected to improve the annotator's work efficiency.

[0096] (10) The task setting unit 112, acting as a determination unit, determines the difficulty level of adding annotation information to each target image based on the number of display switching operations performed for each target image, or the number of annotators who performed predetermined user operations for each target image. The task setting unit 112 further stores difficulty information indicating the difficulty level of each target image in the storage unit 12. By managing the difficulty level for each target image in this way, an appropriate task can be assigned to an annotator.

[0097] (11) The memory unit 12 further includes a quality analysis unit 113 as a generation unit that stores work history information showing the annotator's work history and generates characteristic information showing the annotator's work characteristics based on the work history information stored in the memory unit. The characteristic information includes information indicating the level of the annotator's work skills. The quality analysis unit 113 stores the characteristic information in the memory unit 12 in association with the work history information. By managing the annotator's work characteristics in this way, appropriate tasks can be assigned to the annotator.

[0098] (12) The quality analysis unit 113 further analyzes the quality of the annotation information attached to the target image. The quality analysis unit 113 stores the quality information indicating the quality analysis results in the storage unit 12. This allows tasks to be managed based on the quality analysis results.

[0099] (13) The task management unit 111 further detects changes in the annotator's work time based on work history information, detects changes in the annotator's work quality based on quality information, and generates condition information indicating the annotator's work condition based on the detected changes in work time or work quality, and stores it in the memory device. In this way, changes in the annotator's state can be detected by monitoring the annotator's state. As a result, the annotator can recognize its own condition (physical condition and concentration) by continuing the annotation work.

[0100] (14) The task management unit 111 assigns high-difficulty target images to annotators with high work skills based on the characteristic information and difficulty information stored in the memory unit 12. This is expected to improve the efficiency of annotation.

[0101] Incidentally, if the annotator is not familiar with operating devices such as tablet terminals or PCs (personal computers), it becomes difficult to smoothly carry out annotation work using such devices. Furthermore, if the annotator is unfamiliar with annotation itself, it becomes even more difficult. Therefore, in order to address these problems, the information processing system 100 is configured as follows.

[0102] Figure 11 is a schematic block diagram showing another example of the overall configuration of the information processing system 100, including the management device 1. The information processing system 100 in Figure 11 includes the management device 1 and an information processing device 4 that is communicatively connected to the management device 1 via a communication network 3. Although Figure 11 illustrates one information processing device 4, the management device 1 may be connected to multiple information processing devices 4 via the communication network 3. The information processing system 100 may also include the annotator terminal 2 shown in Figure 1.

[0103] Figure 12 is a block diagram illustrating the main components of the information processing device 4 shown in Figure 11. The information processing device 4 includes a controller 40, a communication unit 43, a display device 44, an operation unit 45, an imaging device 46, and an audio input device 47.

[0104] The communication unit 43 is a communication interface that connects the information processing device 4 to the communication network 3. The information processing device 4 sends and receives information with the management device 1 and other devices connected to the communication network 3 via the communication unit 13.

[0105] The display device 44 is a projection device (projector) that projects images. The display device 44 has its internal parameters (such as focal length) and external parameters (such as position and orientation) pre-calibrated so that an image is projected onto a predetermined area on a wall surface in space.

[0106] The imaging device 46 has an image sensor such as a CCD or CMOS, and acquires an image by converting light into an electrical signal. The imaging device 46 captures the operation of the annotator or the movement of a work tool in accordance with the operation of the annotator with respect to the target image projected (displayed) in a predetermined area by the display device 44. The work tool is, for example, a pointer.

[0107] The audio input device 47 is, for example, a microphone, which inputs the sound from the annotator as an audio signal. The audio signal input by the audio input device 47 is output to the arithmetic unit 41 as audio data via an A / D converter (not shown).

[0108] The controller 40 is comprised of a computer having an arithmetic unit 41 such as a CPU, a storage unit 42 such as ROM and RAM, and other peripheral circuits (not shown) such as an I / O interface. The arithmetic unit 41 functions as an operation detection unit 411, an acquisition unit 412, a display control unit 413, an operation analysis unit 414, and an annotation processing unit 415 by executing a program stored in the storage unit 42. The storage unit 42 stores operation information, which will be described later.

[0109] The motion analysis unit 414 analyzes the motion of the annotator captured by the imaging device 46. More specifically, the motion analysis unit 414 analyzes the motion of a predetermined part of the annotator (e.g., a fingertip) or the movement of a work tool in accordance with the motion of the annotator, based on the captured images acquired by the imaging device 46.

[0110] The operation detection unit 411, acquisition unit 412, display control unit 413, and annotation processing unit 415 have the same configuration as the operation detection unit 211, acquisition unit 212, display control unit 213, and annotation processing unit 214 in Figure 4. However, the operation detection unit 411, display control unit 413, and annotation processing unit 415 differ from the operation detection unit 211, display control unit 213, and annotation processing unit 214 in Figure 4 in the following respects.

[0111] The operation detection unit 411 detects annotation operations corresponding to the annotator's actions or the movement of the work tool, in addition to annotator operations (such as login operations) input via the operation unit 45. Specifically, the operation detection unit 411 detects annotation operations corresponding to the annotator's actions or the movement of the work tool based on the operation information stored in the storage unit 42 and the analysis results of the motion analysis unit 414.

[0112] Action information is information that associates annotation operations with the actions of the annotator (such as tapping, double-tapping, swiping, or dragging) or the movements of the work tools. Annotation operations include segmentation operations and labeling (attribute assignment) operations.

[0113] Segmentation operations include operations to cut out and adjust the contours of objects. For example, when an annotator traces the contour of an object in a target image with a finger or pointer, the motion analysis unit 414 tracks the position of the fingertip or the tip of the pointer. The operation detection unit 411 detects the contour cutting operation performed by the annotator based on the analysis results (tracking results) of the motion analysis unit 414.

[0114] Figure 13 is a diagram illustrating the annotation operation. Figure 13 shows how the annotator AT performs annotation on the target image IM projected onto the wall surface WL by the display device 44. As shown in Figure 13, when the annotator AT traces the outline of an object in the target image IM with its finger, the motion analysis unit 414 analyzes the movement of the fingertip, and the operation detection unit 411 detects the outline cutting operation.

[0115] Segmentation may be performed by adjusting the shape and size of bounding boxes (hereinafter simply referred to as "boxes") superimposed on the target image so that they enclose the objects. In this case, the segmentation operation includes box adjustment operations. The operation detection unit 411 detects gestures of the annotator AT that enlarge or reduce the boxes as box adjustment operations.

[0116] Furthermore, segmentation may be performed by inputting a box that encloses an object within the target image. In this case, the segmentation operation includes the input operation of the box. The operation detection unit 411 detects the annotator AT touching two points on the target image with a finger or pointer as the input operation of the box. In this case, the two touched points are designated as the upper right and lower left corners of the box.

[0117] The display control unit 413 outputs the target image to the display device 44 and controls the display device 44 so that the target image is projected onto a predetermined area. Furthermore, when the operation detection unit 411 detects a segmentation operation, the display control unit 413 projects a label selection screen (not shown) overlaid on the target image.

[0118] When the operation detection unit 411 detects a segmentation operation, the annotation processing unit 415 generates segmentation information based on the display position of the region specified by the contour cutting operation or box input operation.

[0119] Furthermore, when the annotator touches (tap) one of the multiple object types displayed on the label selection screen, the annotation processing unit 415 acquires the type of the touched object as labeling information. The annotator AT may also specify one of the multiple object types displayed on the label selection screen by voice. In this case, the annotation processing unit 415 recognizes the voice input from the annotator AT via the voice input device 47 and generates labeling information based on the recognition result.

[0120] Furthermore, when the operation detection unit 411 detects the contour cutting operation of the annotator AT, the display control unit 413 may superimpose the trajectory of the annotator AT's finger or pointer, tracked by the motion analysis unit 414, onto the target image. This allows people around the annotator AT to check the quality of the contour cutting. Additionally, when the annotator AT holds its palm over the superimposed trajectory on the target image, the display control unit 413 may remove the superimposed display of the trajectory included in the area where the palm is held.

[0121] Furthermore, when the display control unit 413 projects a target image onto a predetermined area, it may project a work window overlaid on the target image projected onto the predetermined area based on the window setting information described above, and may also scan the work window on the target image. In this case, the annotation processing unit 214 generates segmentation information based on the display position of the work window when a segmentation operation is detected. In addition, when the display control unit 413 projects a target image onto a predetermined area, it may project a mask overlaid on the target image projected onto the predetermined area based on the mask setting information described above.

[0122] According to the embodiments described above, the following effects and advantages can be obtained. (1) The information processing device 4 generates annotation information to be applied to the target image. The information processing device 4 includes a display device 44 that displays an image with a predetermined area in space as the display area, a display control unit 413 that controls the display device 44 so that the target image is displayed in the predetermined area, an imaging device 46 that captures the movements of the worker or the movements of a work tool in accordance with those movements with respect to the target image displayed in the predetermined area, a motion analysis unit 414 that analyzes the movements of the worker or the movements of the work tool captured by the imaging device 46, and an annotation processing unit 415 that generates annotation information to be applied to objects in the target image based on the analysis results of the motion analysis unit 414. As a result, the annotator can perform annotation on a target image projected onto a wall or floor.

[0123] (2) The predetermined area is an area on the wall surface in space, the display device 44 is an image projection device, the display control unit 413 controls the display device 44 so that the target image is projected onto the predetermined area, and the motion analysis unit 414 analyzes the movement of a predetermined part of the annotator based on the captured image acquired by the imaging device 46. As a result, the target image is projected onto the wall surface, allowing people in the vicinity to see the target image and the annotation information attached to it. Furthermore, by performing segmentation on the large target image projected onto the wall surface, it is expected that the accuracy of segmentation will be improved.

[0124] (3) The information processing device 4 further includes an operation detection unit 411 that detects an operation corresponding to the annotator's operation or the movement of a work tool, based on operation information that associates multiple operations related to annotation with multiple actions of the operator, and the analysis results of the operation analysis unit 414. The annotation processing unit 415 generates annotation information to be applied to objects in the target image according to the operation detected by the operation detection unit 411. In this way, multiple actions of the annotator can be associated with different annotation operations, improving the operability of the annotator.

[0125] The above embodiment can be modified into various forms. Modifications will be described below. In the above embodiment, the task setting unit 112, as a setting unit, sets a task to add annotation information to one object included in the target image. However, the setting unit may set a task to add annotation information to a predetermined number of objects included in the target image. In this case, the setting unit includes information specifying the number of work windows in the work window setting information so that a number of work windows corresponding to the predetermined number are superimposed on the target image. In this case, the process in step S208 of Figure 8 is not affirmed until all work windows are tapped. At this time, the superimposed display of the tapped work windows may be canceled, or a predetermined mark indicating that annotation has been added may be superimposed on the tapped work windows. This eliminates the need for the annotator to determine how much work is needed, even if a single image contains multiple objects. The predetermined number of work windows superimposed on the target image may have different sizes from each other.

[0126] Furthermore, in the above embodiment, when the annotator taps a rectangular work window displayed on the target image, the annotation processing unit 214 acquires the position coordinates of the corners and center of the work window as segmentation information. However, the annotator may adjust the shape of the work window after tapping it to match the contour of an object within the work window. More specifically, the annotator may adjust the position and number of vertices of the work window by tapping multiple points along the contour of the object. In this case, the annotation processing unit generates segmentation information based on the adjusted shape of the work window.

[0127] Furthermore, in the above embodiment, the task setting unit 112, which acts as a setting unit, sets a work window for the target image. However, the setting unit may set a temporary contour line for the target image instead of a work window. A temporary contour line is an outline that roughly encloses the objects in the image. In this case, the annotator adjusts the segmentation by changing the positions of the vertices of the temporary contour line.

[0128] Furthermore, in the above embodiment, the task setting unit 112, acting as a setting unit, determines the difficulty level of annotation for the target image based on the number of pass operations performed on the target image, or the number of annotators who performed pass operations on the target image. However, the setting unit may also provide incentives to the target image based on the determined difficulty level. For example, the unit may set a higher unit price for annotation work on target images with higher difficulty levels. In this case, the setting unit may store the incentive information set for the target image in the image DB 121, associating it with the image ID of the target image.

[0129] Furthermore, in the above embodiment, the motion analysis unit 414, acting as an analysis unit, was shown to analyze the movement of the pointer as a work tool based on the captured image acquired by the imaging device 46. However, tools other than the pointer may be used as work tools.

[0130] For example, a tool that emits electromagnetic waves of a specific wavelength may be used as a work tool. In this case, the analysis unit analyzes the movement of the electromagnetic waves emitted by the work tool, which is captured by the imaging device 46.

[0131] As a specific example, a pulsed laser gun may be used as a work tool. In this case, the annotator AT operates the laser gun so that the laser is irradiated in a dotted manner along the contour of an object on the target image. The motion analysis unit 414 acts as a reach detection unit and detects, based on the image captured by the imaging device 46, the arrival of the laser emitted from the laser gun to a predetermined area on the wall surface WL, more specifically, the time of arrival and the arrival position. When the reach detection unit detects the arrival of the laser, the display control unit 413 controls the display device 44 to superimpose a predetermined image onto the area including the laser's arrival position. More specifically, the display control unit 413 controls the display device 44 to project an image simulating a bullet hole (hereinafter referred to as a bullet hole image) onto the laser's arrival position. The annotation processing unit 415 generates segmentation information based on each of the laser's arrival positions.

[0132] The display control unit 413 may project an image (hereinafter referred to as the droplet image) that mimics a droplet flowing downward from a bullet hole, along with the bullet hole image, and superimpose the droplet image onto the target image. In this case, the display control unit 413 may change the droplet image so that the droplet extends downward according to the elapsed time since the laser arrived.

[0133] Alternatively, a device that launches game balls (so-called pachinko balls) may be used instead of a laser gun. In this case, the motion analysis unit 414, which acts as the arrival detection unit, detects the arrival of the game ball to the wall surface WL (the time of arrival and the arrival position) based on the image captured by the imaging device 46, and when the arrival of the game ball is detected, the display control unit 413 superimposes a predetermined image (for example, a colored circle the same size as the game ball) onto the arrival position.

[0134] Alternatively, for example, darts may be used as a work tool. When darts are used as a work tool, the wall surface WL is made of a material into which the darts can penetrate. When darts are used as a work tool, the motion analysis unit 414, which acts as a reach detection unit, detects the position where the darts reach the wall surface WL based on the image captured by the imaging device 46, and the annotation processing unit 415 generates segmentation information based on each of the dart's reach positions.

[0135] Furthermore, when the dart's arrival is detected, the display control unit 413 may superimpose a predetermined image (for example, an image simulating light emission) onto the arrival location. Also, if the laser gun is equipped with communication and vibration functions, when the laser's arrival at the wall is detected, the display control unit 413 may output a vibration command to the laser gun via the communication unit 43, causing the laser gun to vibrate.

[0136] Furthermore, the display control unit 413 may also function as an acoustic control unit and output sound effects when it detects that a projectile such as a laser, game ball, or dart has reached a wall. In this case, the acoustic control unit controls the generation of sound by controlling an acoustic generator (not shown) that generates sound corresponding to the input acoustic data. The motion analysis unit 414, acting as a reach detection unit, detects that a projectile such as an electromagnetic wave emitted from a laser gun has reached a predetermined area, and when the detection unit detects that the projectile has reached the area, the acoustic control unit outputs acoustic data corresponding to a predetermined sound such as a sound effect to the acoustic generator.

[0137] As described above, the method of generating segmentation information based on the projectile's arrival position allows multiple annotators to perform segmentation on a single target image or on a single object within a target image. As a result, multiple annotators can compete in segmentation time or segmentation accuracy. For example, the annotator with the highest segmentation accuracy can be declared the winner, or the annotator who comes closest to a pre-set segmentation accuracy can be declared the winner, adding entertainment and game-like elements to the annotation process. Furthermore, when multiple annotators perform segmentation, the entertainment and game-like elements of the annotation process can be further enhanced by having them compete on the number and accuracy of their labels.

[0138] Furthermore, in the above embodiment, the internal and external parameters of the display device 44, which acts as an image projection device, are calibrated so that the target image is projected onto a predetermined area on the wall surface. However, the internal and external parameters of the image projection device may also be calibrated so that the target image is projected onto a predetermined area on the floor.

[0139] In this case, the annotator moves by walking or running along the contour of the object in the target image projected onto the floor. The motion analysis unit 414 tracks the annotator's movement trajectory, and the operation detection unit 411 detects the contour extraction operation for the object in the target image based on the tracking results. Alternatively, instead of moving along the contour of the object, the annotator may trace the contour with a rod-shaped member such as a pointer. The motion analysis unit 414 tracks the position of the tip of the rod-shaped member, and the operation detection unit 411 detects the contour extraction operation for the object in the target image based on the tracking results.

[0140] The annotator may also use a brush with a handle to cut out the outline of an object by scrubbing the entire inside area of ​​the object in the projected image on the floor, similar to sweeping in curling. In this case, the display control unit may superimpose a predetermined image (e.g., brush marks) onto the areas where the brush has been scrubbed. By adding a competitive element to the annotation work in this way, it is expected that people who do not know each other will cooperate in the annotation work.

[0141] Furthermore, as described above, if gamification or competitive elements are added to the annotation work, the annotation processing unit 415 may act as a measurement unit to measure the work time required to generate annotation information for each annotator and compare the measured work times of each annotator. The incentive provision unit may then provide incentives to each annotator based on the comparison results. For example, a ranking of each annotator may be determined according to their work time. More specifically, annotators with shorter work times may be ranked higher. This can further enhance the entertainment and gamification aspects of the annotation work.

[0142] Furthermore, in the above embodiment, the display device 44, as an image projection device, was used as an example when it projects a target image onto a wall surface. However, instead of a wall surface, the target image may be projected onto an electronic device having an input area that detects the approach or contact of an input device. For example, the target image may be projected onto an electronic device such as a touch panel or an electronic whiteboard that is large enough to allow image projection by the image projection device. In this case, the display device 44 is configured so that the position of the display area is set in advance so that the input area of ​​the electronic device becomes the display area. The annotator performs operations such as cutting out the outline of an object on the target image projected onto the electronic device using a dedicated pen such as a stylus. The motion analysis unit 414, as a position detection unit, detects the approach or contact position of the input device in the input area. For example, when the input device is a touch panel, the position detection unit detects the approach or contact position of the input device based on the detection value of the touch panel's sensor. The motion analysis unit 414 further acts as an analysis unit, analyzing the changes in the detected approach or contact position. The annotation processing unit 415 generates annotation information to be applied to objects in the target image based on the analysis results of the analysis unit. More specifically, the operation detection unit 411 detects the annotator's operation on the target image projected onto the touch panel based on the analysis results of the analysis unit, and the annotation processing unit 415 generates annotation information to be applied to objects in the target image based on the detection results of the operation detection unit 411.

[0143] Alternatively, the target image may be projected onto a whiteboard instead of a wall. In this case, the annotator uses a marker (magic pen) to perform operations such as cutting out the outline. The motion analysis unit 414, acting as an analysis unit, tracks the position of the tip of these pens, and the operation detection unit 411 detects the annotator's operations based on the tracking results.

[0144] Furthermore, in the above embodiment, the display device 44, as an image projection device, was used as an example in which the target image is projected onto a two-dimensional wall surface. However, the display device may also be a wearable display device equipped with virtual reality (VR) functionality that displays a three-dimensional virtual space based on the target image.

[0145] The wearable display device is an HMD (Head-Mounted Display) such as VR goggles. In this case, the display control unit 413 generates a VR image using binocular stereoscopic viewing based on two target images simultaneously captured by two imaging devices, and the display device plays back the VR image. The method of generating the VR image is not limited to this, and may also be generated using a target image captured by one imaging device and the sensor value of a distance sensor (not shown). The motion analysis unit 414, as a detection unit, detects the real-space position of a VR work tool (such as a VR hand controller or VR gloves) whose real-space position has been calibrated with the virtual-space position. The motion analysis unit 414 further, as an analysis unit, analyzes the change in the virtual-space position of the work tool held by the annotator based on the detected real-space position. More specifically, the analysis unit maps the detected real-space position to the virtual space through a predetermined coordinate transformation and tracks the position of the work tool in the virtual space. The annotation processing unit 415 generates annotation information to be applied to objects within the virtual section based on the analysis results of the analysis unit.

[0146] This allows an annotator wearing VR goggles to perform annotation work in a three-dimensional virtual space. When the target image is a two-dimensional image, some areas within the image may be invisible due to occlusion, etc. However, by using the above HMD, it is possible to check areas that are difficult to see on a two-dimensional image, and an improvement in the accuracy of annotation by the annotator is expected.

[0147] The display control unit 413 may generate VR images so that the scale of objects in the VR space matches the scale of the real space. This allows the annotator to see objects at their actual size, which is expected to reduce the cognitive load on the annotator. Alternatively, the display control unit 413 may generate VR images so that the scale of objects in the VR space is larger than the scale of the real space. This makes it possible to segment objects down to their finest details, which is expected to further improve the accuracy of annotation by the annotator.

[0148] Alternatively, instead of tracking the virtual space position of the VR work tool, the motion analysis unit 414 may analyze the movements of an annotator wearing an HMD such as VR goggles, as captured by the imaging device 46. In this case, the camera coordinate system of the imaging device 46 is calibrated with the coordinate system of the virtual space. The imaging device 46 is also installed in a predetermined position so as to capture the finger movements and gestures of the annotator wearing the HMD. The imaging device 46 may also be a built-in camera mounted on the lower front, left side, or right side of the VR goggles.

[0149] In this case, the motion analysis unit 414, acting as a detection unit, detects the annotator's movements in real space (such as finger movements and gestures) based on the captured image of the annotator wearing the HMD, acquired by the imaging device 46. The motion analysis unit 414 further acts as an analysis unit, analyzing the annotator's movements in virtual space based on the detected movements of the annotator in real space. More specifically, the analysis unit maps the detected movements in real space to movements in virtual space through a predetermined coordinate transformation, and tracks the annotator's movements in virtual space.

[0150] From another perspective, the information processing device of the above embodiment can also be configured as an information processing method for generating annotation information to be applied to a target image. That is, it can also be configured as an information processing method that includes the steps of: controlling a display device that displays an image with a predetermined area in space as the display area so that the target image is displayed in a predetermined area; analyzing the movements of a worker or a work tool captured by an imaging device that captures the movements of a worker or a work tool in accordance with said movements in relation to the target image displayed in the predetermined area; and generating annotation information to be applied to objects in the target image based on the results of the analysis.

[0151] Furthermore, the information processing method can also be configured to include the steps of: detecting the real-world position of a work tool, which has been calibrated against the position in a three-dimensional virtual space displayed by a display device based on a target image; analyzing the change in the virtual space position of the work tool held by the worker based on the detected real-world position; and generating annotation information to be applied to objects in the virtual space based on the target image based on the analysis results.

[0152] Furthermore, the information processing method can also be configured to include the steps of: detecting the worker's movements in real space based on the coordinate system of a three-dimensional virtual space displayed on the display device based on the target image and an image captured by an imaging device calibrated with the display device; an analysis unit that analyzes the worker's movements in virtual space based on the detected movements in real space; and generating annotation information to be applied to objects in the virtual space displayed on the target image based on the analysis results.

[0153] Furthermore, the present invention can be constructed by replacing the above information processing method with a program that causes a computer to execute processes to support the annotation work of an annotator. Even further, the present invention can be constructed by replacing this with a computer-readable storage medium on which this program is recorded.

[0154] The above description is merely an example, and the present invention is not limited by the embodiments and modifications described above, as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and to combine modifications with each other. [Explanation of Symbols]

[0155] 1 Management device, 2 Annotator terminal, 10,20 Controller, 11,21 Calculation unit, 12,22 Storage unit, 13,23 Communication unit, 24 Display device, 25 Operation unit, 47 Voice input device, 111 Task management unit, 112 Task setting unit, 113 Quality analysis unit, 121 Image DB, 122 Annotator DB, 211,411 Operation detection unit, 212,412 Acquisition unit, 213,413 Display control unit, 214,415 Annotation processing unit, 414 Motion analysis unit

Claims

1. An information processing device that generates annotation information to be applied to a target image, A display device that displays an image using a predetermined area in space as the display area, A display control unit controls the display device so that the target image is displayed in the predetermined area, An imaging device that captures the movements of an operator or the movements of a work tool in accordance with such movements in relation to the target image displayed in the predetermined area, An analysis unit that analyzes the movements of the worker or the movements of the work tool captured by the imaging device, An information processing apparatus comprising: an annotation processing unit that generates annotation information to be applied to objects in the target image based on the analysis results of the analysis unit; and an annotation processing unit.

2. In the information processing apparatus according to claim 1, The predetermined region is a region on the wall surface in the space, The aforementioned display device is an image projection device, The display control unit controls the image projection device so that the target image is projected onto the predetermined area. The analysis unit is characterized by analyzing the movement of a predetermined part of the worker based on the captured image acquired by the imaging device.

3. In the information processing apparatus according to claim 1, The system further includes an operation detection unit that detects the operation corresponding to the operator's operation or the movement of the work tool, based on operation information that associates multiple operations related to annotation with multiple actions of the operator, and the analysis results of the analysis unit. The annotation processing unit is characterized by generating annotation information to be applied to the object in the target image in accordance with the operation detected by the operation detection unit.

4. In the information processing apparatus according to claim 1, The aforementioned work tool is a tool that emits electromagnetic waves of a specific wavelength, The analysis unit is characterized by analyzing the movement of the electromagnetic waves emitted by the work tool, which are captured by the imaging device.

5. In the information processing apparatus according to claim 4, The system further comprises a reach detection unit that detects the arrival of the electromagnetic waves emitted from the work tool into the predetermined region, The information processing apparatus is characterized in that, when the arrival of the electromagnetic wave is detected by the arrival detection unit, the display control unit controls the display device to superimpose and display a predetermined image in a region including the arrival position of the electromagnetic wave.

6. In the information processing apparatus according to claim 4, A sound generating device that generates sound corresponding to the input sound data, The sound control unit controls the generation of sound by the sound generating device, The system further comprises a reach detection unit that detects the arrival of the electromagnetic waves emitted from the work tool into the predetermined region, The acoustic control unit is characterized in that, when the arrival of the electromagnetic wave is detected by the arrival detection unit, it outputs the acoustic data corresponding to a predetermined sound to the sound generating device.

7. In the information processing apparatus according to claim 1, An information processing device further comprising an incentive granting unit that measures the work time required for generating the annotation information for each worker, compares the measured work time for each worker, and grants an incentive to each worker based on the results of the comparison.

8. An information processing device that generates annotation information to be applied to a target image, An electronic device having an input area for detecting the approach or contact of an input device, A display device in which the position of the display area is predetermined so that the input area becomes the display area, A position detection unit for detecting the proximity or contact position of the input device in the input area, An analysis unit that analyzes the detected change in the approach position or the contact position, An information processing apparatus comprising: an annotation processing unit that generates annotation information to be applied to objects in the target image based on the analysis results of the analysis unit;

9. An information processing device that generates annotation information to be applied to a target image, A display device that displays a three-dimensional virtual space based on the aforementioned target image, A detection unit for detecting the real-space position of a work tool whose real-space position has been calibrated with the virtual-space position, An analysis unit analyzes the change in the position of the work tool held by the worker in the virtual space based on the position in the real space detected by the detection unit, An information processing apparatus comprising: an annotation processing unit that generates annotation information to be applied to objects in the virtual space displayed based on the target image, based on the analysis results of the analysis unit; and an annotation processing unit.

10. An information processing device that generates annotation information to be applied to a target image, A wearable display device that displays a three-dimensional virtual space based on the aforementioned target image, The coordinate system of the virtual space and the calibrated imaging device, A detection unit that detects the worker's movements in real space based on the captured image of the worker wearing the display device acquired by the imaging device, Based on the actions in the real space detected by the detection unit, an analysis unit analyzes the actions of the worker in the virtual space, An information processing apparatus comprising: an annotation processing unit that generates annotation information to be applied to objects in the virtual space displayed based on the target image, based on the analysis results of the analysis unit; and an annotation processing unit.

Images