Learning device

Abstract

A learning device (10) comprises a control unit (11) that sequentially learns a plurality of first RGB images representing a space in which a moving object and a stationary object are present from mutually different viewpoints to generate a model for outputting a predicted RGB image representing a corrected space excluding the moving object from a designated viewpoint and a predicted depth image of the moving object and the stationary object viewed from the designated viewpoint, generates, for each of the first RGB images, a second RGB image in which a region indicating the moving object is masked with respect to the corresponding first RGB image on the basis of a comparison between the predicted depth image and a corrected depth image viewed from the same viewpoint as the first RGB images, and updates the model using each of the second RGB images.

Description

Learning device

[0001] This disclosure relates to a learning device.

[0002] Conventionally, there exists a machine learning model for generating images called NeRF (Neural Radiance Fields). When training an NeRF model, if an image containing a moving object is used as is, noise will be generated in the image generated by the model, resulting in a decrease in image quality. Therefore, it is known to perform a preprocessing step to remove the moving object from the input image. For example, Non-Patent Document 1 discloses that it is necessary to annotate pixels corresponding to the object to be removed and pixels corresponding to the region not to be removed. Non-Patent Document 2 discloses that it is necessary to annotate the region to be removed.

[0003] A. Mirzaei, et al., "SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields", arXiv:2211.12254v2, 2023H. Liu, et al., "NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors", arXiv:2206.04901v1, 2022

[0004] The pre-processing step requires manual discrimination between moving and stationary objects, which is time-consuming.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the efficiency of learning a model that uses images containing moving objects as learning data.

[0006] A learning device according to one embodiment acquires a plurality of first RGB images representing a space in which a moving object and a stationary object exist from different viewpoints, acquires a modified depth image corresponding to each first RGB image and representing the depth of the stationary object as seen from the same viewpoint as the corresponding first RGB image, and generates a model that, when a viewpoint is specified, outputs a predicted RGB image representing the modified space obtained by excluding the moving object from the space, and a predicted depth image representing the depth of the moving object and the stationary object as seen from the specified viewpoint, based on a comparison between each modified depth image and the predicted depth image output by inputting the same viewpoint as the corresponding first RGB image to the model, generates a second RGB image in which the region representing the moving object is masked from the corresponding first RGB image, and updates the model using each second RGB image.

[0007] According to the present disclosure, it is possible to improve the efficiency of learning a model that uses images containing moving objects as learning data.

[0008] It is a block diagram showing the configuration of a learning device. It is a flowchart showing an example of the operation of the learning device. It is a diagram showing a first RGB image. It is a diagram showing a modified depth image. It is a diagram showing a predicted depth image. It is a diagram showing a second RGB image.

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0010] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0011] An overview of this embodiment will be described with reference to Figure 1. The learning device 10 according to this embodiment is, for example, a general-purpose computer such as a PC or tablet, a server computer such as a cloud server, or a dedicated computer. "PC" is an abbreviation for Personal Computer.

[0012] The learning device 10 acquires multiple first RGB images representing the space in which moving and stationary objects exist from different viewpoints. The learning device 10 acquires a modified depth image corresponding to each first RGB image, representing the depth of the stationary object as seen from the same viewpoint as the corresponding first RGB image. By sequentially learning each first RGB image, the learning device 10 generates a model that, when a viewpoint is specified, outputs a predicted RGB image representing the modified space (with the moving object removed) from the specified viewpoint, and a predicted depth image representing the depth of the moving and stationary objects as seen from the specified viewpoint. Based on a comparison of each modified depth image with the predicted depth image output by inputting the same viewpoint as the corresponding first RGB image into the model, the learning device 10 generates a second RGB image for the corresponding first RGB image, masking the region representing the moving object. The learning device 10 updates the model using each second RGB image.

[0013] Moving objects include, for example, people, animals, or vehicles. Stationary objects include, for example, buildings, the ground, a calm body of water, or plants. The first RGB image and the predicted RGB image are images in which each pixel has values ​​for the R, G, and B components. The corrected depth image and the predicted depth image are images in which each pixel has the distance to the subject as a depth value, and the depth value is represented by grayscale or color. According to this embodiment, the learning device 10 can construct a model that generates an image representing a corrected space in which only stationary objects exist, while automatically distinguishing between moving objects and stationary objects, without requiring preprocessing steps such as manually removing moving objects from the image. Therefore, it is possible to improve the efficiency of learning a model that uses images containing moving objects as learning data.

[0014] The configuration of a learning device 10 according to this embodiment will be described with reference to Fig. 1. The learning device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, and an output unit 15.

[0015] The control unit 11 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for Central Processing Unit. "GPU" is an abbreviation for Graphics Processing Unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for Field-Programmable Gate Array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for Application Specific Integrated Circuit. The control unit 11 controls each part of the learning device 10 and executes processes related to the operation of the learning device 10.

[0016] The memory unit 12 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, RAM, ROM, or flash memory. "RAM" is an abbreviation for Random Access Memory. "ROM" is an abbreviation for Read Only Memory. The RAM is, for example, SRAM or DRAM. "SRAM" is an abbreviation for Static Random Access Memory. "DRAM" is an abbreviation for Dynamic Random Access Memory. The ROM is, for example, EEPROM. "EEPROM" is an abbreviation for Electrically Erasable Programmable Read Only Memory. The flash memory is, for example, SSD. "SSD" is an abbreviation for Solid-State Drive. The magnetic memory is, for example, HDD. "HDD" is an abbreviation for Hard Disk Drive. The storage unit 12 functions, for example, as a main memory, auxiliary memory, or cache memory. The storage unit 12 stores information used for the operation of the learning device 10 and information obtained through the operation of the learning device 10.

[0017] The communication unit 13 includes at least one communication module. The communication module is, for example, a module compatible with a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE 802.11, or a mobile communication standard such as LTE, 4G, or 5G. "LAN" is an abbreviation for local area network. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 13 receives information used in the operation of the learning device 10 and transmits information obtained by the operation of the learning device 10. The communication unit 13 enables the learning device 10 to send and receive information to and from other devices via a network.

[0018] The network may include the Internet, at least one WAN, at least one MAN, or a combination thereof. "WAN" is an abbreviation for Wide Area Network. "MAN" is an abbreviation for Metropolitan Area Network. The network may include at least one wireless network, at least one optical network, or a combination thereof. The wireless network may be, for example, an ad-hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for Local Area Network.

[0019] The input unit 14 includes at least one input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, a touchscreen integrated with a display, or a microphone. The input unit 14 accepts operations to input data used for the operation of the learning device 10. Instead of being provided in the learning device 10, the input unit 14 may be connected to the learning device 10 as an external input device. Any connection method can be used, for example, USB, HDMI®, or Bluetooth®. "USB" is an abbreviation for Universal Serial Bus. "HDMI®" is an abbreviation for High-Definition Multimedia Interface.

[0020] The output unit 15 includes at least one output interface. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for Liquid Crystal Display. "EL" is an abbreviation for Electro Luminescence. The output unit 15 outputs data obtained by the operation of the learning device 10. Instead of being provided in the learning device 10, the output unit 15 may be connected to the learning device 10 as an external output device. Any connection method can be used, for example, USB, HDMI®, or Bluetooth®.

[0021] The functions of the learning device 10 are realized by executing the program according to this embodiment on the processor acting as the control unit 11. In other words, the functions of the learning device 10 are realized by software. The program causes the computer to perform the operations of the learning device 10, thereby causing the computer to function as the learning device 10. That is, the computer functions as the learning device 10 by performing the operations of the learning device 10 according to the program.

[0022] The program can be stored on a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include flash memory, magnetic recording devices, optical disks, magneto-optical recording media, and ROMs. The program can be distributed by selling, transferring, or lending portable media such as SD cards, DVDs, or CD-ROMs that store the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for Digital Versatile Disc. "CD-ROM" is an abbreviation for Compact Disc Read Only Memory. The program can also be distributed by storing it in the storage of a server and transferring it from the server to another computer. The program can also be provided as a program product.

[0023] A computer temporarily stores a program stored on a portable medium or transferred from a server in its main storage device. The computer then reads the program stored in the main storage device using a processor and executes processing in accordance with the read program. The computer may also read the program directly from the portable medium and execute processing in accordance with the program. The computer may also execute processing in accordance with the received program each time a program is transferred from the server to the computer. Processing may also be executed using a so-called ASP-type service that realizes functions simply by issuing execution instructions and obtaining results, without transferring the program from the server to the computer. "ASP" is an abbreviation for Application Service Provider. A program is information used for processing by a computer and includes something equivalent to a program. For example, data that is not a direct instruction to a computer but has properties that define computer processing falls under the category of "something equivalent to a program."

[0024] Some or all of the functions of the learning device 10 may be realized by a programmable circuit or a dedicated circuit as the control unit 11. In other words, some or all of the functions of the learning device 10 may be realized by hardware.

[0025] Next, the operation of the learning device 10 according to this embodiment will be described with reference to FIGS. 2 to 3D.

[0026] In step S101 of Figure 2, the control unit 11 acquires a plurality of first RGB images. In this embodiment, the plurality of first RGB images as training data are stored in the storage unit 12, but the plurality of first RGB images may also be received from an external server device via the communication unit 13. The plurality of first RGB images are RGB images that represent a space in which moving objects and stationary objects exist from different viewpoints. Each of the plurality of first RGB images may be captured by an RGB-D (RGB-Depth) camera or the like. Each of the plurality of first RGB images includes viewpoint information indicating the position and direction of the viewpoint.

[0027] In step S102, the control unit 11 acquires point cloud data representing a modified space in which only stationary objects exist. The point cloud data is, for example, data generated using the Structure From Motion (SfM) method or the like, based on multiple images taken at a location in the same space as the space represented by multiple first RGB images during a time period when there are no moving objects. The point cloud data may also be data generated using LiDAR (Light Detection and Ranging). Any method may be used to acquire the point cloud data. For example, the control unit 11 may receive point cloud data from an external server device, such as a mapmaking company, via the communication unit 13.

[0028] In step S103, the control unit 11 selects and acquires an image related to a predetermined viewpoint from the multiple first RGB images acquired in step S101. The control unit 11 may determine the predetermined viewpoint randomly or in a predetermined order. Fig. 3A shows a first RGB image related to a predetermined viewpoint acquired by the control unit 11. Referring to Fig. 3A, a space is shown in which a car C as a moving object and buildings B1 and B2 as stationary objects exist.

[0029] In step S104, the control unit 11 generates and acquires a modified depth image representing the depth of the stationary object as viewed from a predetermined viewpoint, i.e., the same viewpoint as the selected first RGB image, based on the point cloud data acquired in step S102 and viewpoint information included in the first RGB image selected in step S103. Any method may be used to acquire the modified depth image. In this embodiment, the control unit 11 projects the point cloud data onto a two-dimensional plane based on the viewpoint information. The control unit 11 calculates the depth, which is the distance from the viewpoint to each point, for each pixel on the two-dimensional plane to generate the modified depth image. FIG. 3B shows the modified depth image acquired by the control unit 11. The modified depth image of FIG. 3B is an image representing a modified space in which only buildings B1' and B2' exist as stationary objects in FIG. 3A. In the modified depth image of FIG. 3B, the farther from the viewpoint a pixel is, the darker the pixel color is displayed. However, this is not limited to this; the farther from the viewpoint a pixel is, the whiter the pixel color is displayed in the modified depth image.

[0030] In step S105, the control unit 11 learns the first RGB image selected in step S103, and when a viewpoint is specified, it generates or updates a model that outputs a predicted RGB image representing the modified space as seen from the specified viewpoint, and a predicted depth image representing the depth between moving and stationary objects as seen from the specified viewpoint. Any machine learning method may be used for learning, and the model includes any image generation model. The model according to this embodiment is a NeRF model. Details about NeRF are described in, for example, the following reference 1, so a detailed explanation is omitted. [Reference 1] B. Mildenhall, et al., “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, arXiv: 2003.08934v2, 2020

[0031] In this embodiment, the control unit 11, when generating or updating a model, uses a function that takes the viewpoint position and viewpoint direction related to the first RGB image as input and outputs the color and density of an arbitrary point in the modified space. This function is not limited to NeRF models but can be used with any model such as NSVF (Neural Sparse Voxel Fields), which is a derivative of NeRF. Based on the density and color of points on a virtual ray cast from the viewpoint position and viewpoint direction, the control unit 11 can generate a predicted RGB image and a predicted depth image by volume rendering.

[0032] In step S106, the control unit 11 inputs the viewpoint position and viewpoint direction indicated by the viewpoint information included in the first RGB image selected in step S103 into the model generated or updated in step S105 to obtain a predicted RGB image, and uses the first RGB image as the true value to obtain the first RGB loss L, which is the difference between the predicted RGB image and the first RGB image. Ｃ The control unit 11 calculates the first RGB loss Lc by a method that is generally used as a photometric loss, such as L1 loss, which takes the absolute value of the difference between an estimated value and a true value as the loss.

[0033] The control unit 11 inputs the viewpoint position and viewpoint direction indicated by the viewpoint information included in the first RGB image selected in step S103 into the model generated or updated in step S105 to obtain a predicted depth image, and calculates a depth loss L, which is the difference between the predicted depth image and the corrected depth image, using the corrected depth image generated in step S103 as a true value. Ｄ The control unit 11 calculates the depth loss L by a method commonly used for depth loss such as L1 loss. Ｄ Calculate.

[0034] The control unit 11 calculates the depth loss L Ｄ Pixels whose value is above a predetermined threshold are identified, and a mask is applied to the pixels of the first RGB image selected in step S103 at the same position as the identified pixels. Equation 1 below represents the pixel M to be masked. In Equation 1, L Ｄ (i) is the depth loss of pixel i, and τ is a predetermined threshold. The predetermined threshold may be set freely in advance. The control unit 11 may set a rectangle that surrounds all of the pixels M to be masked, and apply a mask having the rectangle. In this way, the control unit 11 generates a second RGB image in which the area indicating the moving object is masked from the first RGB image selected in step S103.

[0035] Figure 3C shows the predicted depth image. The depth loss L of the pixels representing the stationary objects, buildings B1'' and B2'', in the predicted depth image of Figure 3C is compared with the corrected depth image of Figure 3B. Ｄ The value is below a predetermined threshold. Therefore, the control unit 11 does not identify the pixels representing buildings B1'' and B2'' as pixels M to mask. The depth loss L of the pixels representing the moving object, car C'', in the predicted depth image of Figure 3C is compared with the corrected depth image of Figure 3B. Ｄ The value is above a predetermined threshold. Therefore, the control unit 11 identifies the pixel representing the vehicle C'' as the pixel M to be masked. The control unit 11 sets a rectangle MS that encloses all of the pixels M to be masked.

[0036] Figure 3D shows a second RGB image generated by the control unit 11 applying a mask MK, corresponding to the rectangle MS in Figure 3C, to the first RGB image in Figure 3A. Referring to Figure 3D, compared to Figure 3A, the region representing the moving object, the car C, is masked.

[0037] The control unit 11 uses the second RGB image as the true value and calculates the second RGB loss L, which is the difference between the predicted RGB image and the second RGB image. ＣＭ The control unit 11 calculates the first RGB loss L. Ｃ Using a similar method, the second RGB loss L ＣＭ may be calculated.

[0038] The control unit 11 calculates the first RGB loss L Ｃ、 Depth loss L Ｄ , and the second RGB loss L ＣＭ Sum these values ​​and calculate the loss function in Equation 2 below: L = L Ｃ +L Ｄ +L ＣＭ (Formula 2)

[0039] The control unit 11 determines whether the value of the loss function in Equation 2 is less than a first predetermined value. The first predetermined value may be set freely in advance. If it is less than the first predetermined value, the control unit 11 proceeds to step S108. If it is greater than or equal to the first predetermined value, the control unit 11 proceeds to step S107.

[0040] In step S107, the control unit 11 continues learning the model and adjusts the model parameters to minimize losses. After that, the control unit 11 returns to step S105, and the control unit 11 updates the model.

[0041] In step S108, the control unit 11 increments the value of the counter by 1 to determine whether the value of the counter is greater than or equal to a second predetermined value. The second predetermined value is, for example, the total number of first RGB images. This allows the control unit 11 to construct a model based on the first RGB images relating to all viewpoints. The second predetermined value may be freely set in advance. If the value of the counter is greater than or equal to the second predetermined value, the control unit 11 proceeds to step S109. If the value of the counter is less than the second predetermined value, the control unit 11 returns to step S103. Returning to step S103, the control unit 11 learns the first RGB images relating to different viewpoints than before, thereby constructing a model that can generate images of higher quality with less noise than when learning by erasing the region relating to moving objects in only a single image relating to one viewpoint.

[0042] In step S109, the control unit 11 outputs the model, after which the operation of the learning device 10 ends.

[0043] The output model can be used for various tasks depending on the user's application. For example, when the model is used in an image search task to find images similar to the input image, the model can generate modified images that do not include moving objects, thus improving search performance. For example, when the model is used in a task to estimate the camera's pose by matching feature points from a reference image similar to the input image and solving a PnP (Perspective-n-Point) problem, the model can improve the accuracy of camera pose estimation because it does not incorrectly acquire feature points of moving objects.

[0044] The following additional notes are provided regarding the above-described embodiments.

[0045] (Supplementary Item 1) A learning device comprising: a control unit that acquires a plurality of first RGB images representing a space in which moving objects and stationary objects exist from different viewpoints; acquires modified depth images corresponding to each of the first RGB images and representing the depth of the stationary objects as seen from the same viewpoint as the corresponding first RGB image; sequentially learns each of the first RGB images, and when a viewpoint is specified, generates a model that outputs a predicted RGB image representing a modified space in which the moving objects are removed from the space from the specified viewpoint, and a predicted depth image representing the depth of the moving objects and the stationary objects as seen from the specified viewpoint; generates a second RGB image in which an area representing the moving object is masked from the corresponding first RGB image based on a comparison between each modified depth image and a predicted depth image output by inputting the same viewpoint as the corresponding first RGB image into the model; and updates the model using each second RGB image. (Supplementary Item 2) The learning device according to Supplementary Item 1, wherein the control unit updates the model using a loss function that sums a first RGB loss in which each of the first RGB images is a true value, a depth loss in which each of the corrected depth images is a true value, and a second RGB loss in which each of the second RGB images is a true value. (Supplementary Item 3) The learning device according to Supplementary Item 1 or 2, wherein the control unit acquires point cloud data representing the corrected space and generates the corrected depth image based on the point cloud data. (Supplementary Item 4) The learning device according to any one of Claims 1 to 3, wherein the control unit learns the model using a function that receives as input a viewpoint position and a viewpoint direction related to each of the first RGB images and outputs the color and density of an arbitrary point in the corrected space.

[0046] The present disclosure is not limited to the above-described embodiments. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Two or more steps shown in the flowchart may be executed in parallel or in a different order, instead of being executed in chronological order as described, depending on the processing capabilities of the device executing each step, or as needed. Other modifications are possible within the scope of the present disclosure.

[0047] 10 Learning device 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Output unit

Claims

1. A learning device comprising: a control unit that acquires a plurality of first RGB images representing a space in which moving objects and stationary objects exist from different viewpoints; acquires a modified depth image corresponding to each first RGB image and representing the depth of the stationary objects as seen from the same viewpoint as the corresponding first RGB image; sequentially learns each of the first RGB images, thereby generating a model that, when a viewpoint is specified, outputs a predicted RGB image representing a modified space in which the moving objects are removed from the space from the specified viewpoint, and a predicted depth image representing the depth of the moving objects and the stationary objects as seen from the specified viewpoint; generates a second RGB image in which the area representing the moving object is masked from the corresponding first RGB image based on a comparison between each modified depth image and a predicted depth image output by inputting the same viewpoint as the corresponding first RGB image into the model; and updates the model using each second RGB image.

2. A learning device according to claim 1, wherein the control unit updates the model using a loss function that sums a first RGB loss in which each of the first RGB images is set to a true value, a depth loss in which each of the corrected depth images is set to a true value, and a second RGB loss in which each of the second RGB images is set to a true value.

3. A learning device according to claim 1 or 2, wherein the control unit acquires point cloud data representing the modified space, and generates the modified depth image based on the point cloud data.

4. A learning device according to claim 1 or 2, wherein the control unit learns the model using a function that takes as input the viewpoint position and viewpoint direction for each of the first RGB images and outputs the color and density of any point in the correction space.

Images