Creating training data variability for object annotation in images in machine learning

Abstract

An image annotation system (100) is described, the system comprising a support (2), a digital camera (1), a processing and control device (5) and a digital display (4); the support (2) is used for an object (3) to be annotated; the digital camera (1) is configured to capture multiple images of a scene including the object (3); the processing and control device (5) is configured to receive the images and generate corresponding annotation data (21-24, L1-L4) associated with the object (3); the digital display (4) is associated with the support (2) and connected to the processing and control device (5) to selectively display a group of images including the following: The additional images (7-13) are selected from: a first image (7-11) which is a background for the plurality of images and introduces a certain degree of variability in the scene; a second image (12) which indicates the position and / or orientation of the object (3) placed on the support (2) by the user; a third image (13) which is captured by the digital camera (1) and provided to the processing and control device (5) for evaluating the position of the digital camera (1) relative to the digital display (4); and a fourth image which is captured by the digital camera (1) and provided to the processing and control device (5) for evaluating at least one of the position, orientation and 3D shape of the object (3).

Description

Technical Field

[0001] The invention relates to an image annotation system. Background Art

[0002] Machine learning systems, such as neural networks, must be trained using data, and the availability of such data is critical. This is especially true for data-driven models, where the basic data type is images. According to known techniques, images used to train such models are manually annotated by humans.

[0003] Image annotation means generating content and associating it with images.

[0004] For example, the content may be a series of digital boxes surrounding objects of interest present in the image. Further examples of annotations are to "color" each pixel differently or to provide a textual description in a manner that is germane to the content of the scene (eg, the objects therein).

[0005] Examples of annotated images are shown in the paper “YOLO9000: better, faster, stronger” by A. Redmon et al., IEEE conference on computer vision and pattern recognition, 2017 (http: / / pjreddie.com / yolo9000 / ).

[0006] Furthermore, it has been noted that in order to properly train data-driven models using annotated images of objects of interest, it is necessary to rearrange the objects or change the image background in which they are placed, thus introducing a certain degree of variability in the scene. Introducing this variability in datasets, especially for synthetically generated images, is referred to in the literature as domain randomization.

[0007] In 2017, P. Tobin et al., "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World" (IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), 2017) explained that domain randomization is the basis for training effective models. In particular, in the above paper, domain randomization is used with synthetic data, i.e., a 3D renderer that simulates real scenes.

[0008] Domain randomization is also discussed in the paper “LOOP: Leveraging on a generic Object detector for Orientation Prediction” (Annex-1, 2018) by D. De Gregorio et al., where this technique is applied to real data.

[0009] Applicants have noted that training state-of-the-art deep learning models may require thousands of annotated images, and it is necessary to retrain / reconfigure these models whenever working conditions change (e.g., objects of interest, lighting conditions of the scene, tabletop texture, etc.). For example, consider a situation where five thousand images must be annotated with reference to twelve different objects of interest. An average user would take about four minutes to draw the twelve boxes in the image (considering that the category to which each box belongs must be carefully selected); the total work time would exceed 300 hours (4 minutes × 5000).

[0010] The above examples show that the image annotation process performed according to known techniques is time consuming and expensive.

[0011] The following further prior art documents are cited in the following description to specifically explain aspects of the present invention:

[0012] [1] FARedmon et al., “YOLO9000: better, faster, stronger”, IEEE conference on computer vision and pattern recognition, 2017.

[0013] [2] C. Liu et al., “Ssd: Single shot multibox detector”, European conference on computer vision, 2016.

[0014] [3] D. Xiang et al., “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scene”, Arxiv, 2017.

[0015] [4] LVRad et al., “BB8: a scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth,” Proceedings of the IEEE International Conference on Computer Vision, 2017.

[0016] [5] BGJ Godard et al., “Unsupervised monocular depth estimation with left-right consistency,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

[0017] [6] YALChen et al., “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, 2018.

[0018] [7] GRHe et al., “Mask r-CNN,” Proceedings of the IEEE international conference on computer vision, 2017.

[0019] [8] LS Kalashnikov et al., “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation,” Arxiv, 2018. Summary of the invention

[0020] The invention solves the problem of providing an image annotation system which allows reducing human intervention in the annotation process.

[0021] According to a first object, the present invention relates to an image annotation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Further features and advantages will become more apparent from the following description of a preferred embodiment and its alternatives given as examples, with reference to the accompanying drawings, in which:

[0023] - Figure 1 An embodiment of an image annotation system including a digital camera, a digital display, and a processing and control device is schematically shown;

[0024] - Figure 2 An example of annotation data obtainable by the image annotation system is shown;

[0025] - Figure 3 shows an example of a background image produced by the digital display of the image annotation system;

[0026] - Figure 4 shows an example of a placeholder generated by the digital display according to a particular embodiment of the system;

[0027] - Figure 5 shows an example of a 2D bounding box generated by the system according to two different perspectives;

[0028] - Figure 6 Another embodiment of the system is directed to, wherein the digital camera is tracked using a marker;

[0029] - Figure 7 Another embodiment of the system is concerned, wherein a mechanical manipulator equipped with a gripper is used. DETAILED DESCRIPTION

[0030] Figure 1 A first embodiment of an image annotation system 100 is schematically shown, comprising: a digital camera 1, a support 2 for at least one object 3 to be annotated, and a digital display 4. The image annotation system 100 (hereinafter, for simplicity, also referred to as "system 100") is also provided with a processing and control device 5 connected to the digital camera 1 and the digital display 4 (in a wireless and / or wired connection).

[0031] In particular, the processing and control device 5 may include one or more computers with a suitable human-machine interface, a graphical interface, and a memory storing software instructions. The software instructions stored in the processing and control device 5 allow the system 100 to execute an object labeling program.

[0032] For the purposes of this invention, “image annotation” means associating an image with content that may be needed to train a machine learning system.

[0033] The digital camera 1 is configured to capture a plurality of images of one or more objects 3 located on the support 2. For example, the digital camera 1 may be a video camera, a still camera, a thermal imaging camera, an infrared camera or a 3D camera. The digital camera 1 is preferably movable so as to take different positions and orientations relative to the object 3 in space.

[0034] In particular, the support 2 may be a transparent screen which may be part of the digital display 4. The digital display 4 may be of a known type and preferably it is an E-INK display which shows that it can well simulate a surface similar to paper since it has no backlight.

[0035] Preferably, the system 100 can be configured to perform automatic annotation of objects of interest. In this case, the digital camera 1 is a calibrated camera, i.e. the optical parameters associated with the digital camera 1 are known and can be stored in a memory of the processing and control device 5. Furthermore, in order to perform automatic annotation, the position and orientation of the digital camera 1 in space are known; for example, the corresponding data are stored in a memory of the processing and control device 5. Furthermore, the position and orientation of the object 3 are also known, in particular stored in a memory of the processing and control device 5. Furthermore, according to a preferred example, the 3D shape of the object 3 is also known data (e.g., as a 3D CAD model), or they can be reconstructed by known motion recovery structure techniques based on the knowledge of the position and orientation of the digital camera 1.

[0036] Figure 1 As shown exemplarily, the system 100 may be provided with a movable support 6 for supporting and displacing the digital camera 1. The movable support 6 may be a mechanical manipulator, which is particularly capable of moving the digital camera 1 and tracking its position and orientation in a common reference frame originating from the base of the mechanical manipulator itself. The movable support 6 is controlled by the processing and control device 5.

[0037] The mechanical manipulator may be a robot having at least one degree of freedom. For example, the manipulator may be a robot selected from the following group: a Cartesian / Gantry robot, a cylindrical robot, a spherical robot, a SCARA robot, an articulated robot (robotic arm), a parallel robot or a passive arm.

[0038] The robot is able to control the position and orientation of a digital camera 1 with sub-millimeter precision. Passive arms, basically robots without motors (i.e. moved by an operator), could represent a portable solution with an accuracy comparable to that of a robot.

[0039] Alternatively, the digital camera 1 may be hand-supported (i.e. without using the movable support 6), but in this case the digital camera 1 preferably has an embedded positioning capability, i.e. the capability to locate itself in the environment by means of on-board sensors. For example, the digital camera 1 is provided with an inertial measurement unit (IMU) similar to those included in currently available smartphones and specific software (resident on the digital camera 1 or in the processing and control means 5) that allows the position and orientation of the digital camera 1 to be calculated, also taking into account the data provided by the IMU together with the acquired images.

[0040] According to another embodiment, the digital camera 1 may be held in a fixed position and the digital display 4 may be moved by a mechanical manipulator, which may be similar to that described above.

[0041] For example, the digital display 4 is mounted in front of the movable support 6 in such a way that each screen point of the digital display 4 has a known position; this can be represented in a common reference frame. The digital display 4 can accommodate an object 3 on its transparent screen, which will then have a known position and orientation relative to the mechanical manipulator (i.e. the digital camera 1).

[0042] The digital camera 1 is configured to capture at least one image of a scene comprising a digital display 4 and one or more objects 3 placed on a support 2. The processing and control means 5 are configured (using suitable software instructions) to receive said image and to generate annotation data associated with the objects 3 as represented in the captured image of the captured scene.

[0043] refer to Figure 2 , the annotation data can be selected from the following data groups: a text description of the scene represented in the captured image (not shown in the figure), a 2D bounding box 21, a 3D bounding box and 3D pose information 23 (i.e., numerical data). Other types of annotation data include: depth maps and semantic maps (in Figure 2 d) of ). In addition, other types of bounding contours can be used instead of boxes, such as 2D bounding circles 22 or 2D bounding polygons 24. It is also noted that other 3D bounding shapes can be used instead of 3D bounding boxes. The 2D bounding box 21 is a rectangular digital box around the object 3 represented in the captured image. For example, as described in the above-cited prior art documents [1] and [2], the 2D bounding box 21 can be used to train an object detector.

[0044] For example, as described in the above-referenced documents [3] and [4], the 3D bounding box and 3D pose information 23 (relative to the camera coordinate system) of each object in the scene can be used to train a pose estimator.

[0045] A depth map is an image in which each pixel represents the distance of that point from the digital camera 1 , and as described according to the example in the previously cited document [5], the depth map can be used to train monocular stereo.

[0046] Semantic Maps Figure 2 Part d) of the above method can achieve a higher level of detail, i.e., each pixel in the image is "colored" with the category to which the underlying object belongs. As described in the above-mentioned paper [6], semantic maps can be used to train general semantic segmenters. In addition, as described in the previously listed paper [7], semantic maps can be used to train fine-grained object detectors.

[0047] Note that the generated annotation data is not necessarily displayed by the digital display 4 and is only for the sake of clarity. Figure 2 , as visualized on a digital display 4 .

[0048] Note that the digital display 4 is configured to display additional images to be employed in the annotation process of the object 3 .

[0049] According to a first example, the additional image plays the role of a background image for a plurality of images of the object 3 to be captured by the digital camera 1. In detail, the digital display 4 is configured to selectively display a different background image for each captured scene comprising the object 3.

[0050] In particular, the digital display 4 may change the color and / or texture (ie pattern) of the displayed background image. More particularly, the digital display 4 may repeatedly change the color and / or texture (ie pattern) of the displayed background image to introduce the desired variability.

[0051] The possibility to selectively display different background images allows to achieve true domain randomization. The importance and advantages of performing true domain randomization have been discussed in the above cited prior art papers of P. Tobin et al. (Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World) and D. De Gregorio et al. (LOOP: Leveraging on a generic Object detector for Orientation Prediction).

[0052] Figure 3 Examples of different background images 7 - 11 that may be displayed by the digital display 4 are shown.

[0053] According to an example of the operation of the system 100, a user places one or more objects 3 on a transparent screen of a digital display 4. According to an example, the user provides the orientation and position data of the objects 3 to the processing and control device 5, or according to another example, the orientation and position data of the objects 3 can be automatically evaluated. The processing and control device 5 manages the subsequent steps by controlling the digital display 4, the digital camera 1 and the mechanical manipulator (if provided).

[0054] In particular, the digital display 4 displays a first background image 7 ( Figure 2 ) and the mechanical manipulator assumes a first position. Then, the digital camera 1 captures a first image of a scene comprising the object 3 with a first background image 7.

[0055] If automatic annotation is performed, the processing and control means 5 processes the first image (taking into account the known position and orientation data of the digital camera 1 and the objects 3) and associates annotation data with each object 3. For example, a 2D bounding box 21 is generated around each object 3 and the corresponding coordinates are stored in association with each part of the image comprising a specific object 3.

[0056] If the annotation is not performed automatically, an operator can act on the processing and control means 5 (for example via a suitable interface) to introduce the data required to generate the specific annotation data.

[0057] Furthermore, in a further exemplary step, the digital display 4 changes the visualization background and displays a second background image 8, while the object 3 and the digital camera 1 remain in the assumed position and orientation. The digital camera 1 captures a second image of the scene including the object 3 on the second background image 8. The processing and control device 5 processes the second image in a manner similar to that described above and associates corresponding annotation data.

[0058] The steps described above can be repeated for other background images 9-11 to annotate the object 3 under different background conditions. Preferably, the annotation described above can also be performed by capturing images of multiple scenes so as to collect a large amount of annotation data associated with different images of the scene including the object 3, wherein the digital camera 1 and the object 3 have taken different relative positions and orientations in the multiple scenes.

[0059] The plurality of captured images and the annotated data associated with each object 3 together form a data set to be used in a training process. In particular, the data set obtained with the system 100 can be used for training processes in the field of artificial intelligence, in particular machine learning and deep learning, where the availability of data for executing models such as neural networks is crucial. More specifically, the obtained data set can be used to train a data-driven model (i.e. a model that trains / reconfigures itself using a data set rather than a handcrafted algorithm), in which the basic data type is an image.

[0060] Figure 4 A second example of an additional image that can be generated by the digital display 4 (in addition to the background image 7-11 or instead of the background image 7-11) is concerned. According to this second example, the digital display 4 is controlled in such a way as to display a placeholder 12, i.e. an image indicating the position and preferably the direction in which each object is placed on the transparent screen of the digital display 4. For example, the placeholder 12 represents the projection of the object 3 on the plane corresponding to the transparent screen with a continuous line. In particular, the placeholder 12 can be generated starting from the shape data of the object 3 acquired by the processing and control device 5.

[0061] The visualization of the placeholder 12 makes it possible to avoid the processing and control device 5 having to calculate the position and orientation of the object 3 on the digital display 4 from the image captured by the digital camera 1 .

[0062] Figure 4 Part a) shows the digital display 4 without any visualization; Figure 4 Part b) shows the displayed placeholder 12; Figure 4 Part d) of FIG. 1 shows objects 3 (of different types) placed in corresponding placeholders 12 .

[0063] It is observed that the exact position of each pixel of the digital display 4 relative to the digital camera 1 is known to the processing and control means 5 and this allows the generation of relevant data (ie annotation data). In particular, Figure 4 Part c) also shows the annotation data generated by the processing and control device 5 (ie the associated data in the form of a 2D bounding box).

[0064] It is further noted that due to the use of the placeholder 12 and the knowledge of the relative position between the digital camera 1 and the digital display 4, associated content can be generated that is consistently and correctly superimposed on the object 3 of the scene. Figure 5 An example of such consistent and correctly superimposed associated content (i.e., annotation data) is shown in FIG. 4 , for example, due to the different positions of the digital camera 1 relative to the digital display 4, different 2D bounding boxes ( Figure 5 Part a) and Figure 5 b) of the Regulations).

[0065] Figure 6 A third example relates to additional images that may be generated by the digital display 4 (in addition to or instead of the background image 7 - 11 and the placeholder 12 ).

[0066] According to this third example, the processing and control means 5 are configured to control the digital display 4 to display a marking 13, i.e. a specific pattern, preferably with high distinguishing power. Such a marking 13 can be used by the digital camera 1 and the processing and control means 5 to locate the digital camera 1 by means of a suitable tracking algorithm. The marking 13 is particularly useful when the position and orientation of the digital camera 1 cannot be obtained from a mechanical manipulator or directly from the camera itself.

[0067] In particular, the markers 13 are displayed at a specific frequency during the annotation process to avoid that such markers 13 may appear in the captured images and thus jeopardize the collected dataset. More particularly, the markers 13 are displayed at a selected frequency so that frames including the markers 13 are used for tracking and frames not including the markers 13 are used to form the dataset.

[0068] According to a fourth example, the digital display 4 can generate (in addition to or instead of the background images 7-11 and the markings 13) further types of images. Such images are captured by the digital camera 1 after the object 3 has been placed on the support 2 and provided to the processing and control means 5. The images thus captured by the digital camera 1 are processed by the processing and control means 5 to allow such means to evaluate (through suitable software, executing corresponding algorithms) at least one of the following data of the object 3: the position, orientation, 3D shape of the object. In particular, the image adopted according to the fourth example can be a specific color adopted by the digital display 4, which allows to illuminate the object 3. It is known to the skilled person to allow to reconstruct the position, orientation, 3D shape of an object from digital images of the same object under different lighting conditions.

[0069] Another embodiment relates to a system 100 that employs a robot as described above as a mechanical manipulator. In this case, the mechanical manipulator is provided with a gripper 14 configured to grasp an object 3. The mechanical manipulator with the gripper 14 can automatically grasp and rearrange the object 3 (to capture more images) because it knows the exact position and orientation on the digital display 4.

[0070] This additional feature allows the system 100 to be fully automated and also removes the burden on the operator to have to rearrange the scene from time to time. Furthermore, the use of the fixture 14 allows the system 100 to generate hybrid data that includes images, annotated data, and physical robot actions. Such hybrid data is very useful for training modern systems based on reinforcement learning. Reinforcement learning is described in [8].

[0071] The described image annotation system 100 allows a significant reduction in human intervention in the annotation process. The image annotation system 100 using the digital display 4 that generates the background image according to the first example allows a dataset of annotated images to be generated according to domain randomization in an efficient manner, thereby limiting the time and cost of the process. In addition, the described image annotation system 100 is compatible with automatic annotation that further reduces the time and cost of the annotation process.

[0072] As indicated in the above description, further advantages are provided by the examples related to the placeholder 12, the marking 13 and the fixture 14. The possibility of assessing the position and / or orientation and / or the 3D shape of the object 3, due to the image generated by the digital display 4 highlighting the object 3 placed on the support 2, represents a further advantage that the system 100 can provide.

Claims

1. An image annotation system (100), comprising: A support (2) for an object (3) to be marked; a digital camera (1) configured to capture a plurality of images of a scene including the object (3); a processing and control device (5) configured to receive the plurality of images and generate corresponding annotation data associated with the object (3); a digital display (4) associated with the support (2) and connected to the processing and control means (5) for selectively displaying an additional image selected from the following images: a first image (7-11) in the form of a background for said plurality of images and introducing a degree of variability in the scene to be captured by said digital camera (1) and processed by said processing and control means (5); a second image (12) indicating a position and / or orientation according to which the user places the object (3) on the support (2) so as to be captured by the digital camera (1); a third image (13) captured by the digital camera (1) and provided to the processing and control device (5) for evaluating the position of the digital camera (1) relative to the digital display (4); A fourth image, which is generated when the object (3) has been placed on the support (2), is captured by the digital camera (1) and provided to the processing and control device (5) to evaluate at least one of the following data of the object (3): position, orientation, 3D shape.

2. The system (100) according to claim 1, wherein: The processing and control device (5) is configured to cause the digital display (4) to selectively display the first image (7-11), wherein the first image (7-11) includes a first background image having a corresponding color and texture and a second background image having a relative color and / or texture different from the color and texture of the first background image.

3. The system (100) of claim 1, wherein: The digital camera (1) is movable relative to the support (2); The processing and control device (5) is configured to store the following data: optical parameters of the digital camera (1), position data and orientation data of the digital camera (1), and position data and orientation data of the object (3).

4. The system (100) of claim 1, wherein: The corresponding annotation data is selected from the following data group: text description of the scene represented in the captured image, 2D boundary contours (21, 22, 24), 3D boundary shapes, 3D pose information (23), depth maps, semantic maps (L1-L4).

5. The system according to claim 2, further comprising a movable support (6) for supporting and moving the digital camera (1) or the digital display (4) to capture the scene according to different viewing angles.

6. The system according to claim 5, wherein: The movable support (6) is a mechanical manipulator selected from the group consisting of a rectangular coordinate robot, a gantry robot, a cylindrical coordinate robot, a spherical robot, a SCARA robot, a multi-joint robot and a parallel robot.

7. The system according to claim 5, wherein: The movable support (6) is a passive arm.

8. The system according to claim 2, wherein: The digital camera (1) is handheld and comprises a plurality of sensors to provide position and orientation data of the digital camera (1) to the processing and control device (5).

9. The system according to claim 1, wherein: The processing and control device (5) is configured to display the third image (13) at a pre-established frequency to avoid the third image (13) appearing in the plurality of images of the scene including the object (3).

10. The system according to claim 6, wherein: The mechanical manipulator also includes a gripper (14) configured to grip the object (3).

Images