Semantic segmentation learning device and learning method

Abstract

To provide a semantic segmentation learning device and a semantic segmentation learning method, capable of reducing a load on creation of semantic segmentation learning data.SOLUTION: A semantic segmentation learning device comprises: a mobile robot 5 installed with a camera 3; an annotation image storage unit 7 which stores an annotation image; a photographic information storage unit 9 which stores photographic information S of the camera; a photographic information conversion unit 11 which creates conversion photographic information of the camera 3; a control unit 13 which moves the mobile robot 5 on the basis of the conversion photographic information and photographs an image by the camera 3; an image storage unit 15 which stores photographed images G; a view point conversion image generation unit 17 which generates a view point conversion image from a plurality of photographed images G; a view point conversion storage unit 19 which stores the generated view point conversion image; and a semantic segmentation learning unit 21 which learns semantic segmentation using the annotation image and the view point conversion image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a semantic segmentation learning device and learning method. [Background technology]

[0002] Various developments are underway to realize autonomous driving and mobile robots. For environmental recognition while driving, it is important to separate and recognize roads, vehicles, people, etc. from images, and a technology called semantic segmentation using deep learning is used (Non-Patent Documents 1 and 2). Semantic segmentation uses camera images and training data in which each pixel in the image is labeled to build a deep learning model. Training this semantic segmentation requires a process called annotation, in which a label is assigned to each pixel, which takes an enormous amount of time. Furthermore, because semantic segmentation is deep learning, a large amount of training data is required for training.

[0003] When a mobile robot equipped with a camera acquires images and performs environmental recognition using semantic segmentation, recognition accuracy can deteriorate if the surface condition has changed since the previous deep learning model was built, such as when road lines have been repainted or when an object is wet due to rain.In addition to the change in the object's surface, there may be a slight deviation from the original photograph position due to driving errors of the mobile robot. To improve the recognition accuracy of images after changes in road surface conditions, it is necessary to annotate and re-learn the images, but because annotation images for each pixel are required, there is a restriction that the original annotation image cannot be directly used if the image has shifted from the current shooting position.If the original image cannot be directly used, it will be necessary to re-annotate the newly obtained image, which poses the problem of a very large workload. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] TensorFlow & Keras, “TenosorFlow: Road Image Segmentation for Autonomous Driving”, [online], [Retrieved May 31, 2022], Internet <URL:https: / / tensorflow.classcat.com / 2017 / 06 / 27 / tensorflow-segnet-for-image-segmentation / > [Non-patent document 2] Cityscapes dataset, “Examples”, [online], [Retrieved May 31, 2022], Internet<URL:https: / / www.cityscapes-dataset.com / examples / > [Non-patent document 3] A* algorithm, "A* shortest path search", Python program, [online], [Retrieved May 31, 2022], Internet<URL:https: / / myenigma.hatenablog.com / entry / 20140503 / 1399080847> [Non-patent document 4] pure pursuit, “Development of vehicle control algorithms in Autoware”, [online], [Retrieved May 31, 2022], Internet<URL:https: / / tech.tier4.jp / entry / 2019 / 03 / 29 / 173543> [Non-patent document 5] 3D data generation from images using SfM / MVS, “3D data generation from images using SfM / MVS”, [online], [Retrieved May 31, 2022], Internet<URL:https: / / aicam.jp / tech / sfmmvs> [Non-patent document 6] Autoencoder (Denoising Autoencoder), "What is AE (Autoencoder)? Easy-to-understand explanation with illustrations!!", [online], [Retrieved May 31, 2022], Internet<URL:https: / / nisshingeppo.com / ai / whats-autoencorder / > [Non-Patent Document 7] GAN, "GAN: What is a Generative Adversarial Network? Image Generation Using Unsupervised Learning", [online], [Retrieved May 31, 2022], Internet <URL:https: / / www.imagazine.co.jp / gan%EF%BC%9A%E6%95%B5%E5%AF%BE%E7%9A%84%E7%94%9F%E6%88%90%E3%83%8D%E3%83%83%E3%83%88%E3%83%AF%E3%83%BC%E3%82%AF%E3%81%A8%E3%81%AF%E4%BD%95%E3%81%8B%E3%80%80%EF%BD%9E%E3%80%8C%E6%95%99%E5%B8%AB / > Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and the problem that the present invention aims to solve is to provide a semantic segmentation learning device and learning method that reduces the burden of creating learning data for semantic segmentation by using past annotation images to transform the viewpoint of new images and create learning data when learning semantic segmentation. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention employs the following means. That is, the semantic segmentation learning device of the present invention comprises a camera, a mobile robot on which the camera is installed, an annotation image storage unit that stores pre-annotated annotation images, a shooting information storage unit that stores shooting information including the position and orientation of the camera when the annotation image was captured, a shooting information conversion unit that changes the position and orientation of the camera stored in the shooting information storage unit to create converted shooting information including a new position and orientation, a control unit that moves the mobile robot based on the converted shooting information, determines the position and orientation of the camera, and captures images using the camera, an image storage unit that stores images captured by the camera, a viewpoint converted image generation unit that generates a viewpoint converted image reconstructed from a plurality of images captured based on the converted shooting information to an image with the same viewpoint as the annotation image, a viewpoint converted image storage unit that stores the viewpoint converted image, and a semantic segmentation learning unit that performs semantic segmentation learning using the annotation image and the viewpoint converted image with the same viewpoint as the annotation image.

[0007] According to the present invention, semantic segmentation is learned using an annotation image and a viewpoint-converted image reconstructed into an image with the same viewpoint as the annotation image, thereby reducing the burden of creating training data for semantic segmentation.

[0008] In one aspect of the present invention, the viewpoint conversion image generation unit includes: a memory unit; an original image input unit that inputs the annotation image from the annotation image storage unit and stores it in the memory unit as original image data; a new image input unit that inputs the image from the image storage unit and stores it in the memory unit as new image data; a camera position and attitude estimation unit that inputs the original image data and the new image data from the memory unit, calculates the position and attitude of the camera at the time of shooting, and stores the camera position and attitude in the memory unit as camera position and attitude data; a three-dimensional shape estimation unit that inputs the new image data and the camera position and attitude data from the memory unit, calculates a three-dimensional shape, and stores the three-dimensional shape in the memory unit as three-dimensional shape data; and a reconstructed image generation unit that inputs the original image data, the new image data, the camera position and attitude data, and the three-dimensional shape data from the memory unit, generates reconstructed image data by reconstructing the RGB values of each pixel of the original image data with the RGB values of the new image data, and stores the generated reconstructed image data in the memory unit.

[0009] In one aspect of the present invention, the image processing device further includes a noise removal unit that removes noise from the viewpoint converted image generated by the viewpoint converted image generation unit. In this aspect, noise is removed from the viewpoint conversion image, making it possible to create appropriate training data for semantic segmentation.

[0010] In one aspect of the present invention, the image processing device further includes a foreign matter removal unit that removes foreign matter from the viewpoint converted image generated by the viewpoint converted image generation unit. In this aspect, foreign objects are removed from the viewpoint conversion image, making it possible to create appropriate training data for semantic segmentation.

[0011] The semantic segmentation learning method of the present invention includes preparing a camera and a mobile robot on which the camera is installed, saving a pre-annotated annotation image, saving shooting information including the position and orientation of the camera when the annotation image was captured, changing the saved position and orientation of the camera to create converted shooting information including a new position and orientation, moving the mobile robot based on the converted shooting information, determining the position and orientation of the camera, and capturing images using the camera, saving the images captured by the camera, generating a viewpoint-converted image reconstructed from a plurality of the images captured based on the converted shooting information to an image with the same viewpoint as the annotation image, saving the viewpoint-converted image, and learning semantic segmentation using the annotation image and the viewpoint-converted image with the same viewpoint as the annotation image.

[0012] According to the present invention, semantic segmentation is learned using an annotation image and a viewpoint-converted image reconstructed into an image with the same viewpoint as the annotation image, thereby reducing the burden of creating training data for semantic segmentation.

[0013] In one aspect of the present invention, generating the viewpoint-converted image includes saving the annotation image as original image data, saving the image captured by the camera as new image data, calculating the position and orientation of the camera at the time of capture from the original image data and the new image data and saving the position and orientation of the camera as camera position and orientation data, calculating a three-dimensional shape from the new image data and the camera position and orientation data and saving the three-dimensional shape as three-dimensional shape data, generating reconstructed image data from the original image data, the new image data, the camera position and orientation data, and the three-dimensional shape data by reconstructing the RGB values of each pixel of the original image data with the RGB values of the new image data, and saving the generated reconstructed image data.

[0014] In one aspect of the present invention, noise in the viewpoint converted image is removed. In this aspect, noise is removed from the viewpoint conversion image, making it possible to create appropriate training data for semantic segmentation.

[0015] In one aspect of the present invention, the method includes removing foreign matter from the viewpoint converted image. In this aspect, foreign objects are removed from the viewpoint conversion image, making it possible to create appropriate training data for semantic segmentation. [Effects of the Invention]

[0016] According to the present invention, when learning semantic segmentation, it is possible to provide a semantic segmentation learning device and learning method that reduces the burden of creating learning data for semantic segmentation by using past annotation images to transform the viewpoint of new images and create learning data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing the configuration of a semantic segmentation learning device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a configuration of a shooting information conversion unit in the embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing a configuration of a viewpoint conversion image generation unit in the embodiment of the present invention. [Figure 4] 5A and 5B are schematic diagrams illustrating shooting information conversion in an embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an operation of a viewpoint conversion image generation unit in the embodiment of the present invention. [Figure 6] 1A and 1B are schematic diagrams illustrating camera position and orientation estimation according to an embodiment of the present invention. [Figure 7] 1 is a schematic diagram showing a relationship between points on an image of a camera and three-dimensional points in an embodiment of the present invention. [Figure 8] 5A and 5B are schematic diagrams illustrating estimation of corresponding points between cameras in the embodiment of the present invention. [Figure 9] 1 is a block diagram showing the configuration of a semantic segmentation learning device according to an embodiment of the present invention. [Figure 10] FIG. 2 is a block diagram showing a configuration of a noise removal unit during learning in an embodiment of the present invention. [Figure 11] FIG. 2 is a block diagram showing a configuration of a noise removal unit at the time of inference in an embodiment of the present invention. [Figure 12] 1 is a block diagram showing the configuration of a semantic segmentation learning device according to an embodiment of the present invention. [Figure 13] FIG. 2 is a block diagram showing a configuration of a foreign substance removal unit during learning in an embodiment of the present invention. [Figure 14] FIG. 2 is a block diagram showing a configuration of a foreign substance removal unit during learning in an embodiment of the present invention. [Figure 15] FIG. 2 is a block diagram showing a configuration of a foreign matter removal unit at the time of inference in an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. (First embodiment) Fig. 1 is a block diagram showing the configuration of a semantic segmentation learning device according to this embodiment. As shown in Fig. 1, the semantic segmentation learning device 1 according to this embodiment includes a camera 3, a mobile robot 5 on which the camera 3 is installed, an annotation image storage unit 7, an imaging information storage unit 9, an imaging information conversion unit 11, a control unit 13, an image storage unit 15, a viewpoint-converted image generation unit 17, a viewpoint-converted image storage unit 19, and a semantic segmentation learning unit 21.

[0019] The annotation image storage unit 7 stores annotation images that have been annotated in advance. The shooting information storage unit 9 stores shooting information including the position and posture of the camera 3 when the annotation image was captured. The shooting information conversion unit 11 changes the position and posture of the camera 3 stored in the shooting information storage unit 9 to create converted shooting information including a new position and posture. The control unit 13 moves the mobile robot 5 based on the converted shooting information created by the shooting information conversion unit 11, determines the position and posture of the camera 3, and captures an image using the camera 3. The image storage unit 15 stores images G captured by the camera 3. The viewpoint converted image generation unit 17 generates viewpoint converted images by reconstructing multiple images G captured based on the converted shooting information into images with the same viewpoint as the annotation image. The viewpoint converted image storage unit 19 stores the viewpoint converted images generated by the viewpoint converted image generation unit 17. The semantic segmentation learning unit 21 performs semantic segmentation learning using an annotation image and a viewpoint-converted image with the same viewpoint as the annotation image.

[0020] Here, examples of the mobile robot 5 include SLAM (Simultaneous Localization and Mapping: performing location identification and map creation simultaneously) type AGVs (Automated Guided Vehicles), etc. In this embodiment, the mobile robot 5 is capable of estimating its own position and creating a map by SLAM (Simultaneous Localization and Mapping) using sensors such as cameras and LiDAR (light detection and ranging).

[0021] The annotation image storage unit 7, the shooting information storage unit 9, the shooting information conversion unit 11, the control unit 13, the image storage unit 15, the viewpoint conversion image generation unit 17, the viewpoint conversion image storage unit 19, and the semantic segmentation learning unit 21 are implemented in an information processing device such as a personal computer or a cloud server. Here, the shooting information conversion unit 11, the control unit 13, the viewpoint conversion image generation unit 17, and the semantic segmentation learning unit 21 may be software executed by a CPU or a GPU of the information processing device. The annotation image storage unit 7, the shooting information storage unit 9, the image storage unit 15, and the viewpoint conversion image storage unit 19 may be configured as a storage device of the information processing device, such as a solid state drive (SSD) using a flash memory or a hard disk drive (HDD) using a magnetic disk.

[0022] 2 is a block diagram showing the configuration of the photographing information conversion unit 11 in this embodiment. As shown in Fig. 2, the photographing information conversion unit 11 includes a photographing information input unit 39, a position conversion unit 41, a posture conversion unit 43, a parameter setting unit 45, and a converted photographing information creation unit 47.

[0023] The shooting information input unit 39 retrieves shooting information including the position and orientation of the camera at the time of shooting, which is stored in the shooting information storage unit 9, and outputs the position data to the position conversion unit and the orientation data to the orientation conversion unit. The parameter setting unit 45 outputs arbitrarily set position ranges and orientation ranges to the position conversion unit 41 and the orientation conversion unit 43. The position conversion unit 41 creates new position data by varying the input position data within the position range. The created new position data is output to the converted shooting information creation unit 47. The orientation conversion unit creates new orientation data by varying the input orientation data within the orientation range. The created new orientation data is output to the converted shooting information creation unit 47. The converted shooting information creation unit 47 combines the new position data and orientation data to create converted shooting information and outputs it to the control unit 13.

[0024] 3 is a block diagram showing the configuration of the viewpoint conversion image generation unit 17 in this embodiment. As shown in Fig. 3, the viewpoint conversion image generation unit 17 includes a storage unit 23, an original image input unit 25, a new image input unit 27, a camera position and orientation estimation unit 29, a three-dimensional shape estimation unit 31, a reconstructed image generation unit 33, and a reconstructed image output unit 34.

[0025] The original image input unit 25 inputs the annotation image from the annotation image storage unit 7 and stores it as original image data in the memory unit 23. The new image input unit 27 inputs the image G from the image storage unit 15 and stores it as new image data in the memory unit 23. The camera position and orientation estimation unit 29 inputs the original image data and new image data from the memory unit 23, calculates the position and orientation of the camera 3 at the time of shooting, and stores the position and orientation of the camera 3 as camera position and orientation data in the memory unit 23. The three-dimensional shape estimation unit 31 inputs the new image data and camera position and orientation data from the memory unit, calculates a three-dimensional shape, and stores this three-dimensional shape as three-dimensional shape data in the memory unit 23. The reconstructed image generation unit 33 inputs the original image data, new image data, camera position and orientation data, and three-dimensional shape data from the memory unit 23, generates reconstructed image data by reconstructing the RGB values of each pixel of the original image data with the RGB values of the new image data, and stores the generated reconstructed image data in the memory unit 23. The reconstructed image output unit 34 retrieves the generated reconstructed image data from the storage unit 23 and outputs it to the viewpoint converted image storage unit 19 as a viewpoint converted image.

[0026] Next, the operation of the semantic segmentation learning device 1 configured as above will be explained. First, the mobile robot 5 is operated automatically (or manually) to take photos at each shooting point. The following explains the operation when the robot moves automatically to the shooting point. As mentioned above, the mobile robot 5 is assumed to be capable of estimating its own position and creating a map by SLAM using sensors such as a camera and LiDAR.

[0027] The photographing information for each photographing point is the position and orientation in three-dimensional space where the original image was photographed, and is stored in the photographing information storage unit 9. The position is stored as coordinates in a Cartesian coordinate system expressed as (x, y, z), and the orientation is stored as a three-dimensional vector expressed as (u, v, w). In other words, the position of the photographing information is the position of the origin that serves as the reference for photographing by the camera 3, and the orientation is expressed as a vector that represents the direction from the origin of the camera 3 toward the subject photographed by the camera 3. The reference for the origin and orientation of the Cartesian coordinate system is the initial position and orientation at the time of map creation. To perform viewpoint conversion, the mobile robot 5 is moved to a point where the position and orientation have been changed from the original photographing point, and a new image is photographed that partially overlaps the original image.

[0028] To move to the shooting point, the shooting information when the pre-annotated annotation image was captured is used. The mobile robot can reach the destination by reading the shooting information, generating a route, and following the route. Route generation can be achieved by using, for example, the A* algorithm (Non-Patent Document 3), and path following can be achieved by using, for example, pure pursuit (Non-Patent Document 4).

[0029] The photographic information conversion unit 11 converts the position and posture of the photographic information stored in the photographic information storage unit 9. The mobile robot 5 can capture multiple images while reducing the load by changing the position and posture of the photographic information and allowing it to move autonomously. Next, an image with the same gaze as the original annotation image is reconstructed from the multiple newly acquired images. After that, semantic segmentation learning is performed using a pair of annotation data of the original image and an image reconstructed to the same viewpoint as the original image by viewpoint conversion.

[0030] Next, the operation of the photographing information conversion unit 11 will be described. As shown in Fig. 2, the parameter setting unit 45 determines how much the position and posture are to be changed from the original photographing information. Fig. 4 shows an image of the change in position and posture. The position range P and posture range D are set so that even when the photographing information is changed most drastically based on the original photographing information S, when the mobile robot 5 is moved and photographed according to the changed photographing information, a part of the photograph overlaps with the original photograph. In the following, the mobile robot 5 is described as one that cannot move in the z-axis direction with respect to position, and can only rotate about the z-axis with respect to posture.

[0031] If the coordinates of the original photographic information are (x1, y1, z1), the position conversion unit generates a random number r within the ranges determined in the x and y axis directions. x ,r y The coordinates after transformation are (x1+r x ,y1+r y ,z1). If the posture of the original photograph information is (u,v,w), the posture after the change will be (ucosθ-vsinθ,usinθ+vcosθ,w), using a random number θ generated randomly within the range of posture change. In the above, a normal vehicle-type mobile robot is assumed, so there are restrictions on the changes in position and posture, but in the case of a mobile robot with more degrees of freedom, the position and posture can be changed freely within a range where the photographed image partially overlaps with the original photograph.

[0032] Next, we will explain the operation of the viewpoint conversion image generation unit 17 of this embodiment. The viewpoint conversion image generation unit 17 shown in Fig. 3 receives original image data and multiple frames of new image data as input, generates reconstructed image data by reconstructing the RGB values of each pixel of the original image data using the RGB values of the multiple frames of new image data, and outputs the reconstructed image data as a viewpoint conversion image. 5 is a flowchart showing the procedure for generating a reconstructed image. The reconstructed image is generated in the following procedure. (1) Image data input (S01) (2) Camera position and orientation estimation (S02) (3) Three-dimensional shape estimation (S03) (4) Reconstructed image generation (S04)

[0033] First, the camera position and orientation estimation (S02) will be described. The camera position and orientation is estimated using a method called SfM (Structure From Motion), which reconstructs 3D shapes from multi-viewpoint images. SfM involves the following two processes: (a) The camera position and orientation at the time of shooting are calculated using the feature points in the image as clues. (b) Generate low-density point cloud data reconstructed from the feature points by triangulation. For SfM, the method described in Non-Patent Document 5, for example, is used.

[0034] In this embodiment, original image data and multiple new image data are input, and the position and orientation of the camera 3 that captured the original image and each of the positions and orientations of the camera 3 that captured multiple new images are estimated by SfM. Fig. 6 is a schematic diagram illustrating the estimation of the position and orientation of the camera 3.

[0035] The position and orientation of the camera 3 is expressed as a coordinate transformation of the camera coordinate system with respect to the world coordinate system, which is arbitrarily set by SfM calculations. The origin of the world coordinate system is, for example, the position and orientation of the camera 3 in the original image data. The position of the camera 3 is the three-dimensional position (Xc, Yc, Zc) of the origin of the camera coordinate system, and the orientation of the camera 3 is a rotation transformation matrix of the camera coordinate system with respect to the world coordinate system, which is a 3 × 3 matrix. Therefore, in the present invention, the orientation of the camera 3 is expressed as each component (a 11 ,a 12 ,a 13 ,a 21 ,a 22 ,a 23 ,a 31 ,a 32 ,a 33 )

[0036] Next, the three-dimensional shape estimation (S03) will be described. The 3D shape estimation uses a method called MVS (Multi-ViewStereo) which generates high-density 3D point cloud data from multi-viewpoint images. MVS performs the following two processes. (a) Generate high-density point cloud data. (b) Generate a textured polygonal model. For MVS, a method described in, for example, Non-Patent Document 1 is used. In this embodiment, multiple new image data and the position and orientation data of the camera 3 are input, and high-density 3D point cloud data is generated by MVS, which is used as 3D shape data. The 3D shape data is a collection of 3D positions (X, Y, Z) of points on the surface of the photographed object relative to a world coordinate system arbitrarily set by SfM calculations.

[0037] Next, the generation of a reconstructed image (S04) will be described. In this embodiment, a reconstructed image is generated by reconstructing the RGB values of each pixel of the original image data using the RGB values of multiple new image data. The camera that captured the original image is camera 0, and its position and orientation is Σc0. The camera that captured a new image is camera i, and its position and orientation is Σci. First, find the point (X,Y,Z) on the surface of the object viewed by camera 0 from the point (x,y) on the image captured by camera 0.

[0038] When the coordinate transformation from the world coordinate system to the camera coordinate system is expressed as shown in Figure 6, the relationship between a point (x, y) on the image and a three-dimensional point (X, Y, Z) on the world coordinate system is generally as follows, based on the collinearity condition: f in the equation is the distance from the origin of the camera coordinate system to the image plane, or the so-called lens focal length.

[0039]

number

[0040] Here, it is assumed that the world coordinate system Σw has the same position and orientation as the original camera coordinate system Σc0, as shown in Figure 7. Then, each parameter of the coordinate transformation is as follows:

[0041]

number

[0042] To simplify the calculations, f = 1 is used when calculating the camera position and orientation for each camera in SfM. The values of each point cloud of the 3D shape are not actual size but are values that include scaling, but this does not cause any problems in calculating corresponding points between images taken by each camera. Substituting these camera parameters into the equation derived from the collinearity condition above, the transformation equation from a point (x,y) on the image taken by the original camera to a three-dimensional point (X,Y,Z) is as follows:

[0043]

number

[0044] Therefore, for the point (x, y) on the image taken by the original camera for which we want to find a corresponding point, we search for points that exist on the previously obtained 3D point cloud while searching Z from 0 to infinity. In this way, we find the 3D point (X, Y, Z) corresponding to the point (x, y) on the image taken by the original camera.

[0045] Next, from the 3D point (X,Y,Z), we find the point (x i ,y i ) is found. This can be calculated by substituting the camera parameters and the three-dimensional point (X, Y, Z) into the equation derived from the collinearity condition, using the following equation.

[0046]

number

[0047] By this procedure, it is possible to find the point (xi, yi) on the image captured by camera i that corresponds to the point (x, y) on the image captured by camera 0. A schematic diagram of the estimation of corresponding points between cameras is shown in Fig. 8.

[0048] Here, the point (x i ,y i ) pixel value at Ii(R i ,G i ,B iThe pixel value I' of a point on the reconstructed image is calculated by averaging the pixel values of corresponding points on the images from camera 1 to camera n.

[0049]

number

[0050] By performing inter-camera corresponding point estimation and pixel value averaging for all pixels on the reconstructed image, all pixel values on the reconstructed image are calculated, and a reconstructed image is generated. However, points whose corresponding points are outside the image range of the newly captured image are not used in the pixel value averaging.

[0051] As described above, the semantic segmentation learning device 1 in this embodiment changes the shooting information of a pre-annotated annotation image to generate new converted shooting information. Based on this converted shooting information, multiple new images are captured, including portions that overlap with the original annotation image. Next, an image with the same gaze as the original annotation image is reconstructed from the multiple new images acquired. Then, semantic segmentation learning is performed using a pair of annotation data for the original image and an image reconstructed to have the same viewpoint as the original image through viewpoint transformation. Therefore, semantic segmentation learning can be performed using the original annotation data without newly annotating the new image, thereby providing a semantic segmentation learning device and learning method that reduce the burden of creating semantic segmentation learning data.

[0052] (Second embodiment) 9 is a block diagram showing the configuration of a semantic segmentation learning device 10 according to a second embodiment of the present invention. In the description of this embodiment, components common to those in the first embodiment are given the same reference numerals in the figure, and descriptions thereof will be omitted.

[0053] As shown in Fig. 9, this embodiment includes a noise removal unit 35 that removes noise from the viewpoint converted image generated by the viewpoint converted image generation unit 17. When a reflective object is present during viewpoint conversion or when the texture of the subject is the same and it is difficult to obtain feature points, distortion and noise may occur in the reconstructed image, and this noise is processed by the noise removal unit 35. The noise removal unit 35 uses an autoencoder based on a deep learning method as described in Non-Patent Document 6.

[0054] Because autoencoders are deep learning techniques, they require neural network training. Figure 10 is a block diagram showing the configuration of the autoencoder 49 constituting the noise removal unit 35 during training. As shown in Figure 10, during training, a noise-free image G1 is input, and the neural network weights are trained to output the same noise-free image G2. This allows the intermediate layer of the neural network to acquire noise-free image features. Figure 11 is a block diagram showing the configuration of the noise removal unit 35 of this embodiment during inference. As shown in the figure, when performing noise removal, a reconstructed image G3 in which noise has occurred after viewpoint conversion is input. Since the autoencoder 49 has only learned noise-free features, it outputs a noise-free image G4. In this way, noise removal can be performed using a trained autoencoder.

[0055] Therefore, in this embodiment, in addition to the effects of the first embodiment, the noise removal unit 35 can remove distortion and noise from the reconstructed image that occurs during viewpoint conversion, thereby improving the quality of the images used when learning semantic segmentation and enabling appropriate semantic segmentation learning to be performed.

[0056] (Third embodiment) 12 is a block diagram showing the configuration of a semantic segmentation learning device 20 according to a third embodiment of the present invention. In the description of this embodiment, components common to those in the first embodiment are given the same reference numerals in the figure, and descriptions thereof will be omitted.

[0057] As shown in Fig. 12, this embodiment includes a foreign object removal unit 37 that removes foreign objects from the viewpoint converted image generated by the viewpoint converted image generation unit 17. Since a newly captured image may contain people, cars, etc. that were not in the original image, the foreign object removal unit 37 performs processing to remove foreign objects that were not in the original image, such as people and cars. The foreign object removal unit 37 uses a GAN (Generative Adversarial Network), a deep learning method described in Non-Patent Document 7.

[0058] 13 and 14 are block diagrams showing the configuration of a GAN during training. As shown in FIG. 13, the generator 51 receives an image G5 with a foreign object as input and learns to output an image G6 with the foreign object removed. As shown in FIG. 14, the classifier 53 learns to output "Real" when it receives an image G7 without a foreign object and an image G8 with a foreign object taken from the same location. As shown in FIG. 13, the classifier 53 learns to output "Fake" when it receives two images, the generated image G6 output from the generator 51 and the image G5 with a foreign object. With this configuration of the generator 51 and the classifier 53, the generator 51 outputs an image G6 with the foreign object removed to deceive the classifier 53, and the classifier 53 learns to improve the accuracy of distinguishing between the actually taken image G7 without a foreign object and the image G4 output from the generator. FIG. 15 is a block diagram showing the configuration of the foreign object removal unit 37 during inference. An image G9 with a foreign object is input to the trained generator 51. The generator 51 has been trained to the extent that it is difficult to distinguish between an actual image without foreign matter and the generated image, and therefore outputs an image G10 that is indistinguishable from an actual image without foreign matter.

[0059] Therefore, in this embodiment, in addition to the effects of the first embodiment, it is possible to perform processing to remove foreign objects that are not present in the original image, such as people or cars, that appear in the newly captured image by the foreign object removal unit 37. This foreign object removal improves the quality of the images used in learning semantic segmentation, making it possible to carry out appropriate semantic segmentation learning. [Explanation of symbols]

[0060] 1, 10, 20 Semantic Segmentation Learning Machine 3 Camera 5. Mobile robots 7 Annotation image storage section 9. Shooting information storage section 11. Shooting information conversion section 13 Control Unit 15 Image storage section 17 Viewpoint conversion image generation unit 19 Viewpoint conversion image storage unit 21 Semantic Segmentation Training Unit 23 Memory section 25 Original image input section 27 New image input section 29 Camera position and orientation estimation unit 31 3D shape estimation part 33 Reconstructed image generation unit 34 Reconstructed image output unit 35 Noise removal section 37 Foreign matter removal section G, G1 to G10 images S Shooting Information

Claims

1. A camera and a mobile robot equipped with the camera; an annotation image storage unit that stores pre-annotated annotation images; an imaging information storage unit that stores first imaging information including the position and orientation of the camera when the annotation image is captured; a photographing information creating unit that changes the position and orientation of the camera stored in the photographing information storing unit and creates second photographing information including a new position and orientation; a control unit that moves the mobile robot, determines the position and posture of the camera, and performs photography using the camera based on the second photography information; an image storage unit that stores images captured by the camera; an image generating unit that generates a reconstructed image by reconstructing the plurality of images captured based on the second imaging information into an image having the same viewpoint as the annotation image; a reconstructed image storage unit for storing the reconstructed image; a semantic segmentation learning unit configured to learn semantic segmentation using the annotation image and the reconstructed image from the same viewpoint as the annotation image.

2. The image generation unit A memory unit; an original image input unit that inputs the annotation image from the annotation image storage unit and stores the annotation image as original image data in the storage unit; a new image input unit that inputs an image from the image storage unit and stores it in the storage unit as new image data; a camera position and orientation estimation unit that receives the original image data and the new image data from the storage unit, estimates a position and orientation of the camera at the time of image capture, and stores the estimated position and orientation of the camera in the storage unit as camera position and orientation data; a three-dimensional shape estimation unit that receives the new image data and the camera position and orientation data from a storage unit, estimates a three-dimensional shape, and stores the three-dimensional shape in the storage unit as three-dimensional shape data; a reconstructed image generating unit that receives the original image data, the new image data, the camera position and orientation data, and the three-dimensional shape data from the storage unit, generates reconstructed image data by reconstructing RGB values of each pixel of the original image data using RGB values of the new image data, and stores the generated reconstructed image data in the storage unit; The apparatus for learning semantic segmentation according to claim 1 , comprising:

3. The semantic segmentation learning device according to claim 1 , further comprising a noise removal unit that removes noise from the reconstructed image generated by the image generation unit.

4. The semantic segmentation learning device according to claim 1 , further comprising a foreign object removal unit that removes foreign objects from the reconstructed image generated by the image generation unit.

5. A learning device including a camera and a mobile robot on which the camera is installed, saving the pre-annotated annotation image; storing first photographing information including the position and orientation of the camera when the annotation image was photographed; changing the stored position and orientation of the camera to create second photographing information including a new position and orientation; moving the mobile robot based on the second photographing information, determining the position and attitude of the camera, and photographing with the camera; storing images captured by said camera; generating a reconstructed image from the plurality of images captured based on the second imaging information, the reconstructed image being an image from the same viewpoint as the annotation image; storing the reconstructed image; performing semantic segmentation learning using the annotation image and the reconstructed image at the same viewpoint as the annotation image.

6. generating the reconstructed image includes: saving the annotation image as original image data; saving the image captured by the camera as new image data; calculating the position and orientation of the camera at the time of photographing from the original image data and the new image data, and storing the position and orientation of the camera as camera position and orientation data; calculating a three-dimensional shape from the new image data and the camera position and orientation data, and storing the three-dimensional shape as three-dimensional shape data; generating reconstructed image data by reconstructing the RGB values of each pixel of the original image data using the RGB values of the new image data from the original image data, the new image data, the camera position and orientation data, and the three-dimensional shape data, and storing the generated reconstructed image data; The method of claim 5 , comprising:

7. A semantic segmentation learning method as described in claim 5 or 6, wherein the learning device includes removing noise from the reconstructed image.

8. A semantic segmentation learning method as described in claim 5 or 6, wherein the learning device includes removing foreign objects from the reconstructed image.

Images