Teacher dataset generation system, estimation model generation unit, shape estimation device, and teacher dataset generation method

Abstract

Provided is a teacher dataset generation system comprising: a shape extraction device that acquires a distance measurement image to be learned and extracts a specific shape on the basis of the distance measurement image; and a teacher dataset generation device that associates an RGB image to be learned with the specific shape, thereby generating a teacher dataset.

Description

Teacher dataset generation system, estimation model generation unit, shape estimation device, and teacher dataset generation method

[0001] The present invention relates to a teacher dataset generation system, an estimation model generation unit, a shape estimation device, and a teacher dataset generation method.This application claims priority to Japanese Patent Application No. 2023-216436, filed in Japan on December 22, 2023, the contents of which are incorporated herein by reference.

[0002] To photograph a subject, a camera suited to the purpose of the photograph is used. For example, an RGB camera that captures images in the visible light range can be used to photograph the brightness and color of the subject. Furthermore, a ToF (Time of Flight) camera or the like can be used to measure the distance to the subject. For example, Patent Document 1 discloses a method of acquiring a depth image of a scene using a depth sensor and extracting a plane from the depth image.

[0003] In this way, for example, when acquiring brightness, color, and the shape of a plane of a subject, both the RGB camera and the ToF camera are used.

[0004] JP 2016-212086 A

[0005] However, since two cameras are used, it is more expensive than using one camera as in normal photography.An object of the present invention is to provide a teacher dataset generation system, an estimation model generation unit, a shape estimation device, and a teacher dataset generation method for estimating shape based on RGB images.

[0006] [1] One aspect of this embodiment is a teacher dataset generation system that includes a shape extraction device that acquires a ranging image of a learning object and extracts a specific shape based on the ranging image, and a teacher dataset generation device that generates a teacher dataset by associating an RGB image of the learning object with the specific shape.

[0007] According to this embodiment, the shape can be estimated based on the RGB image.

[0008] 1 is a diagram showing the configuration of a teacher dataset generation system according to the present embodiment. FIG. 2 is a diagram showing an example of the configuration of a camera according to the present embodiment. FIG. 3 is a diagram showing an example of the configuration of a camera according to the present embodiment. FIG. 4 is an example of an RGB image captured by a camera. FIG. 5 is a diagram showing an example of a method for extracting a plane. FIG. 6 is a diagram showing the configuration of a teacher dataset generation device according to the present embodiment. FIG. 7 is a flowchart showing the operation of the teacher dataset generation system according to the present embodiment. FIG. 8 is a diagram showing the configuration of an RGB image shape estimation model generation device according to the present embodiment. FIG. 9 is a flowchart showing the operation of the RGB image shape estimation model generation device according to the present embodiment. FIG. 10 is a top view when parallel light is incident on a flat surface and a curved surface. FIG. 11 is a front view when parallel light is incident on a flat surface and a curved surface. FIG. 12 is a diagram showing the configuration of a shape estimation system according to the present embodiment. FIG. 13 is a diagram showing the configuration of an RGB image shape estimation device according to the present embodiment. FIG. 14 is a flowchart showing the operation of the shape estimation system according to the present embodiment. FIG. 15 is an internal block diagram showing an example of the internal configuration of each device included in the teacher dataset generation system according to the present embodiment.

[0009] [System Overview] Conventionally, there have been detection systems that acquire ranging images including distance information from a ToF camera equipped with a ToF sensor, and extract the shape of a subject based on the distance information. Conventional detection systems configured in this manner are capable of extracting the shape of a subject. On the other hand, when acquiring brightness and color information of a subject, the conventional detection systems described above use two types of cameras, an RGB camera and a ToF camera, which complicates the system configuration and makes it difficult to reduce the cost of the detection system. If the shape of a subject could be detected based on an image captured by an RGB camera, the ToF camera could be omitted from the detection system. However, images captured by an RGB camera generally do not include the distance information captured by the ToF camera, and therefore, it may not be possible to accurately detect the shape of a subject based solely on the image captured by the RGB camera.

[0010] Therefore, in this embodiment, we propose to use machine learning or the like to learn the correspondence between a shape extracted from an image captured by a ToF camera and an image captured by an RGB camera that corresponds to that shape, thereby detecting a shape based on the image captured by the RGB camera with the same accuracy as when captured by a ToF camera. When performing machine learning, a common challenge is how to efficiently generate training data. In this embodiment, the training data is shape information of the learning object (e.g., the interior of a room). In this embodiment, the combination of the shape information as training data and the RGB image captured by the RGB camera is called a training dataset.

[0011] Below, we will explain the functional configuration of each in the following order: the teacher dataset generation system 1 that generates this teacher dataset; the RGB image shape estimation model generation device 20 that generates a shape estimation model using RGB images based on the teacher dataset generated by the teacher dataset generation system 1; and the RGB image shape estimation device 32 that estimates a shape based on the shape estimation model generated by the RGB image shape estimation model generation device 20.

[0012] <Teacher Data Set Generation> Fig. 1 is a diagram showing the configuration of a teacher dataset generation system 1 according to this embodiment. The teacher dataset generation system 1 is a system that generates a dataset in which RGB images of a learning object are associated with shape information. The teacher dataset generation system 1 includes a camera 10, a shape extraction device 12, and a teacher dataset generation device 14. Here, the learning object is, for example, the interior design of a room.

[0013] The teacher dataset generation system 1 according to this embodiment can be configured by adding a teacher dataset generation device 14 to a conventionally existing detection system.

[0014] The camera 10 captures RGB images and ranging images of the learning object. That is, the camera 10 has the functions of both an RGB camera and a ToF camera. The camera 10 outputs the captured RGB images to the teacher dataset generation device 14. The camera 10 outputs the captured ranging images to the shape extraction device 12.

[0015] The camera 10 preferably captures the RGB image and the distance measurement image on the same optical axis. Fig. 2 is a diagram showing an example of the configuration of the camera 10 according to this embodiment. The camera 10 includes a lens 101, a prism 102, an RGB sensor 103, a ToF sensor 104, an RGB processing unit 105, and a distance measurement processing unit 106. In the camera 10 shown in Fig. 2, light entering the lens 101 is split by the prism 102, and visible light is input to the RGB sensor 103, and infrared light is input to the ToF sensor 104. The RGB sensor 103 detects red, green, and blue light of the input light, and the RGB processing unit 105 processes the light to generate an RGB image.

[0016] The ToF sensor 104 detects the distance to the learning object by detecting the time it takes for the output infrared light to reflect and return. The distance measurement processing unit 106 generates a distance measurement image including distance information to the learning object. The distance measurement image is, for example, point cloud data, which is a collection of points including three-dimensional position information. Note that the ToF sensor 104 may be another type of sensor as long as it can acquire the distance to the learning object.

[0017] The camera 10 may capture the RGB image and the ranging image at a close angle so that the optical axis can be considered to be substantially the same. FIG. 3 is a diagram showing an example of the configuration of the camera 10 according to this embodiment. The camera 10 includes two lenses 101-1 and 101-2, an RGB sensor 103, a ToF sensor 104, an RGB processing unit 105, and a ranging processing unit 106. In FIG. 3, light entering the lens 101-1 is input to the RGB sensor 103. In FIG. 3, infrared light entering the lens 101-2 is input to the ToF sensor 104. The operations of the RGB sensor 103, the ToF sensor 104, the RGB processing unit 105, and the ranging processing unit 106 in FIG. 3 are the same as the operations of the RGB sensor 103, the ToF sensor 104, the RGB processing unit 105, and the ranging processing unit 106 in FIG. 2. The optical axis of the light incident on the RGB sensor 103 and the optical axis of the light incident on the ToF sensor 104 are oriented in the same direction and are close to each other. Therefore, they can be said to have substantially the same optical axis. Hereinafter, the term "same optical axis" will also include the case where the optical axis is substantially the same. Note that if the pixel correspondence of the images acquired by the RGB sensor 103 and the ToF sensor 104 is misaligned, the images may be corrected so that the pixel correspondence matches.

[0018] The shape extraction device 12 acquires a ranging image from the camera 10. The shape extraction device 12 extracts a shape based on the ranging image. The shape extraction device 12 outputs shape information to the teacher dataset generation device 14. The shape information is information regarding the type of extracted shape and the position of the extracted shape. The shape type is, for example, a plane or a curved surface. As a shape type, a curved surface may include the side surface of a sphere or a cylinder. The shape type may also include a solid formed by combining planes (such as a triangular prism or a triangular pyramid). The information regarding the position of the shape is expressed, for example, by coordinates in a predetermined reference system. The information regarding the position of the shape may be a set of points included in a specific shape, or a set of points forming the outline of a specific shape.

[0019] Fig. 4 is an example of an RGB image captured by camera 10. In the example shown in Fig. 4, the objects (learning targets) captured by camera 10 are the ceiling P1, first wall P2, floor P3, second wall P4, third wall P5, and cylinder C1 of the interior of a room. The RGB image shown in Fig. 4 is an image captured when camera 10 is facing directly toward third wall P5.

[0020] A method for extracting a plane from point cloud data will be described below with reference to FIG. 5 . First, two adjacent points are selected for one point A1. The two points selected here are selected so that the two adjacent points and the three points including point A1 are not on a straight line. The two selected points are designated as A2 and A3. Next, an infinite plane including the three points A1, A2, and A3 is considered. This infinite plane is designated as the reference plane. Next, the distance between point A4, which is adjacent to point A1, A2, or A3, and the reference plane is calculated. If the calculated distance is smaller than a predetermined value d, point A4 is determined to be in the same plane as points A1, A2, and A3. If the calculated distance is equal to or greater than the predetermined value d, point A4 is determined to not be in the same plane as points A1, A2, and A3. Similar calculations are performed on adjacent points other than point A4 to determine whether they are in the same plane as points A1, A2, and A3.

[0021] If it is determined that point A4 forms the same plane as points A1, A2, and A3, the reference plane is updated by setting the infinite plane that has the smallest sum of the distances from the four points A1, A2, A3, and A4 as the reference plane. After updating the reference plane, the distances from the reference plane to the points adjacent to the four points A1, A2, A3, or A4 are calculated, and it is determined whether they are smaller than a predetermined value d. By continuing the above process, eventually, there will be no adjacent points for which the distance to the reference plane needs to be calculated. This determines the plane that includes point A1. By sequentially performing this process on points that are not included in the determined plane, all planes included in the point cloud can be extracted.

[0022] It is also possible to set a lower limit on the number of points that constitute the same plane, and a plane that is constituted by a number of points below the lower limit need not be recognized as a plane. The shape extraction device 12 can extract planes from point cloud data using the above method. The plane extraction method is not limited to the above method, and other known methods (such as RANSAC) may also be used.

[0023] The shape extraction device 12 can extract both directly facing and non-directly facing planes using the above method. In the example shown in Figure 4, the shape extraction device 12 uses this method to extract as planes the directly facing plane P5, the non-directly facing ceiling P1, the first wall P2, the floor P3, and the second wall P4.

[0024] The shape extraction unit 12 extracts, for example, a set of points that have not been extracted as a plane as a curved surface. In the example shown in Fig. 4, the shape extraction unit 12 extracts the side surface of a cylinder C1 as a curved surface.

[0025] The teacher dataset generation device 14 generates a teacher dataset based on RGB images and shape information for the same learning target. Fig. 6 is a diagram showing the configuration of the teacher dataset generation device 14 according to this embodiment. The teacher dataset generation device 14 includes an RGB image acquisition unit 140, a shape information acquisition unit 142, a teacher dataset generation unit 144, and a teacher dataset output unit 146.

[0026] The RGB image acquisition unit 140 acquires an RGB image of the learning object from the camera 10 .

[0027] The shape information acquisition unit 142 acquires shape information of the learning object from the shape extraction device 12 .

[0028] The teacher dataset generation unit 144 generates a teacher dataset by associating the RGB image of the learning target acquired by the RGB image acquisition unit 140 with the shape information acquired by the shape information acquisition unit 142. By associating the RGB image with the shape information in the teacher dataset, the type of shape and the position of the shape of an object appearing in the RGB image are identified.

[0029] The training data set output unit 146 outputs the training data set, which is input to the RGB image shape estimation model generation device 20, which will be described later.

[0030] 7 is a flowchart showing the operation of the teacher dataset generation system 1 according to this embodiment. The camera 10 captures an RGB image and a ranging image of the learning target (step S101). The camera 10 outputs the RGB image to the teacher dataset generation device 14 (step S102). The camera 10 outputs the ranging image to the shape extraction device 12 (step S103). The teacher dataset generation device 14 acquires the RGB image output by the camera 10 (step S141). The shape extraction device 12 acquires the ranging image output by the camera 10 (step S121). The shape extraction device 12 extracts a shape based on the ranging image (step S122). The shape extraction device 12 outputs the shape information to the teacher dataset generation device 14 (step S123).

[0031] The teacher dataset generation device 14 acquires the shape information output by the shape extraction device 12 (step S142), generates a teacher dataset in which the RGB image and the shape information are associated (step S143), and outputs the teacher dataset (step S144).

[0032] As described above, the teacher dataset generation device 14 can generate a teacher dataset in which RGB images are associated with shape information. The teacher dataset generation system 1 can be created by adding a configuration for acquiring RGB images (the RGB sensor 103 and the RGB processing unit 105) and adding the teacher dataset generation device 14 to a conventional system that acquires ranging images and estimates the type and position of a shape from the ranging images.

[0033] Furthermore, by capturing the RGB image and the ranging image on the same optical axis using the camera 10, the shape shown by the RGB image in the training data and the shape extracted from the ranging image appear in the same position without any misalignment. This makes it possible to use the generated training data set to improve the accuracy of the estimation model that estimates shape information based on the RGB image.

[0034] 8 is a diagram showing the configuration of an RGB image shape estimation model generation device 20 according to this embodiment. The RGB image shape estimation model generation device 20 includes a teacher data set acquisition unit 200, an estimation model generation unit 202, and an estimation model output unit 204.

[0035] The teacher dataset acquisition unit 200 acquires a teacher dataset from the teacher dataset generation device 14. The estimation model generation unit 202 generates an RGB image shape estimation model by learning using the teacher dataset. The RGB image shape estimation model is a model that estimates the type and position of the shape of a subject by inputting an RGB image. The learning method is not particularly limited, and the RGB image shape estimation model is, for example, a neural network.

[0036] The estimated model output unit 204 outputs the generated RGB image shape estimation model.

[0037] 9 is a flowchart showing the operation of the RGB image shape estimation model generation device 20 according to this embodiment. The teacher data set acquisition unit 200 acquires a teacher data set from the teacher data set generation device 14 (step S201). The estimation model generation unit 202 generates an RGB image shape estimation model by learning using the teacher data set (step S202). The estimation model output unit 204 outputs the RGB image shape estimation model (step S203).

[0038] The appearance of an RGB image differs depending on whether the shape is flat or curved. When parallel light hits a flat surface, the angle with respect to the plane is the same, so the light energy per unit area on the plane is the same. Assuming that the light is diffused in the same direction, the luminance of the plane will be the same. When parallel light hits a curved surface, the angle varies depending on the location, so the light energy per unit area varies depending on the location, resulting in different luminance levels and unevenness. Therefore, by learning the positional information of whether the shape is flat or curved and the unevenness of luminance in the RGB image, it is believed possible to create a model that can estimate the position of whether the shape is flat or curved based on the RGB image.

[0039] FIG. 10 is a top view of parallel light incident on a flat surface and a curved surface. (a) of FIG. 10 shows the reflection when parallel light hits one face of a cube (flat surface), and (b) shows the reflection when parallel light hits the side of a cylinder (curved surface). In the example shown in (a) of FIG. 10, three parallel light rays are all incident on the flat surface at the same angle θ. Therefore, the light energy per unit area is equal, and assuming that the incident light is diffused in equal directions, the luminance at the flat surface is equal. On the other hand, in the example shown in (b) of FIG. 10, three parallel light rays are incident on the curved surface at angles α, β, and γ, respectively. Therefore, the light energy per unit area is different, and the luminance is different.

[0040] FIG. 11 shows front views of a plane and a curved surface illuminated with parallel light. FIG. 11(a) is a front view of a plane surface illuminated with parallel light, and FIG. 11(b) is a front view of a curved surface illuminated with parallel light. As shown in FIG. 11(b), when parallel light illuminates a curved surface, the brightness varies depending on the area illuminated by the light, resulting in uneven brightness in the RGB image of the curved surface. On the other hand, as shown in FIG. 11(a), when parallel light illuminates a plane surface, the brightness does not vary depending on the area illuminated by the light, resulting in no uneven brightness in the RGB image of the curved surface.

[0041] In this embodiment, since it is considered that unevenness in brightness in an RGB image is being learned, it is desirable that the brightness in the RGB image be greater than a predetermined value. If a value indicating the brightness in the RGB image (e.g., the average, maximum, or minimum brightness value) is equal to or less than a predetermined value, the estimation model generation unit 202 may not use training data including the RGB image for training. This allows RGB images of objects photographed that are black or have a non-reflective coating and have low reflectance to be excluded from the training training data set.

[0042] <Shape Estimation System> Fig. 12 is a diagram showing the configuration of a shape estimation system 3 according to this embodiment. The shape estimation system 3 includes an RGB camera 30 and an RGB image shape estimation device 32. The RGB camera 30 photographs an object to be estimated and captures an RGB image. The RGB camera 30 outputs the captured RGB image to the RGB image shape estimation device 32. The RGB image shape estimation device 32 estimates and outputs the type and position of the shape of the RGB image based on the RGB image. The shape estimation system 3 is used to photograph the interior of a room and estimate the positions of interior walls, etc.

[0043] 13 is a diagram showing the configuration of an RGB image shape estimation device 32 according to this embodiment. The RGB image shape estimation device 32 includes an RGB image acquisition unit 320, a shape estimation unit 322, a shape information output unit 324, and a storage unit 330. The storage unit 330 stores the RGB image shape estimation model output by the RGB image shape estimation model generation device 20.

[0044] The RGB image acquisition unit 320 acquires an RGB image from the RGB camera 30 .

[0045] The shape estimation unit 322 estimates the type and position of a shape based on an RGB image using an RGB image shape estimation model. The shape estimation unit 322 estimates the type and position of a shape by inputting an RGB image to the RGB image shape estimation model and outputting the estimation results of the type and position of the shape.

[0046] The shape information output unit 324 outputs the estimated shape information (type and position of the shape).

[0047] 14 is a flowchart showing the operation of the shape estimation system 3 according to this embodiment. The RGB camera 30 captures an RGB image of an object to be estimated (step S301). The RGB camera 30 outputs the RGB image to the RGB image shape estimation device 32 (step S302).

[0048] The RGB image acquisition unit 320 acquires an RGB image from the RGB camera 30 (step S321). The shape estimation unit 322 estimates the type and position of a shape based on the RGB image using an RGB image shape estimation model (step S322). The shape information output unit 324 outputs the estimated type and position of the shape (step S323).

[0049] As described above, the shape estimation system 3 can estimate the type of shape and position of the estimation object based on the RGB image. In addition to estimating the type of shape and position of the estimation object based on the RGB image, the shape estimation system 3 can also inspect the color of the estimation object based on the RGB image. The shape estimation system 3 can not only inspect the color of the estimation object but also estimate the shape by simply capturing the RGB image.

[0050] Other Embodiments One embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like can be made within the scope that does not deviate from the gist of the present invention.

[0051] <Internal Configuration> FIG. 15 is an internal block diagram showing an example of the internal configuration of each device included in the teacher dataset generation system according to this embodiment. As shown in the figure, at least some of the functions of each device (shape extraction device 12, teacher dataset generation device 14, RGB image shape estimation model generation device 20, or RGB image shape estimation device 32) included in the teacher dataset generation system 1 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. Furthermore, all or part of the functional units provided in each device may be realized using hardware such as an ASIC, a PLD, or an FPGA. Furthermore, all or part of the functional units may be realized by a combination of software and hardware.

[0052] The processing of the shape extraction device 12, teacher dataset generation device 14, RGB image shape estimation model generation device 20, or RGB image shape estimation device 32 in the above-described embodiments may be implemented by a computer using software. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded and executed by a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, and media that store programs for a fixed period of time, such as volatile memory within the computer systems that serve as servers or clients in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0053] According to the present invention, it is possible to estimate the shape based on the RGB image.

[0054] 1 Teacher dataset generation system, 10 Camera, 101 Lens, 102 Prism, 103 RGB sensor, 104 ToF sensor, 105 RGB processing unit, 106 Distance measurement processing unit, 12 Shape extraction device, 14 Teacher dataset generation device, 140 RGB image acquisition unit, 142 Shape information acquisition unit, 144 Teacher dataset generation unit, 146 Teacher dataset output unit, 20 RGB image shape estimation model generation device, 200 Teacher dataset acquisition unit, 202 Estimation model generation unit, 204 Estimation model output unit, 3 Shape estimation system, 30 RGB camera, 32 RGB image shape estimation device, 320 RGB image acquisition unit, 322 Shape estimation unit, 324 Shape information output unit, 330 Memory unit

Claims

1. A teacher dataset generation system comprising: a shape extraction device that acquires a ranging image of a learning object and extracts a specific shape based on the ranging image; and a teacher dataset generation device that generates a teacher dataset by matching an RGB image of the learning object with the specific shape.

2. The training dataset generation system according to claim 1, wherein the specific shape is a plane.

3. The teacher dataset generation system of claim 1, wherein the distance measurement image is point cloud data, and the shape extraction device extracts a plane as the specific shape based on a distance between a plane determined by points included in the point cloud data and other points.

4. The teacher dataset generation system described in claim 3, wherein the shape extraction device determines that a plane determined by points included in the point cloud data and another point constitute the plane when the distance between the other point and the plane is less than a predetermined value, and extracts the plane as the specific shape.

5. The teacher dataset generation system according to claim 4, wherein, when the shape extraction device determines that there are other points that constitute the plane, it resets the plane so that the distance from all points that constitute the plane is minimized.

6. A teacher dataset generation system according to any one of claims 1 to 5, wherein the RGB image and the ranging image are acquired on the same optical axis.

7. An estimation model generation unit that learns to extract planes from an RGB image using a teacher dataset generated by the teacher dataset generation system according to any one of claims 1 to 6.

8. A shape estimation device, which estimates the shape from an RGB image to be estimated using an RGB image shape estimation model trained using a teacher dataset generated by a teacher dataset generation system according to any one of claims 1 to 5.

9. A teacher dataset generation method comprising: a shape extraction step of acquiring a ranging image of a learning object and extracting a specific shape based on the ranging image; and a teacher dataset generation step of generating a teacher dataset by matching an RGB image of the learning object with the specific shape.

Images