Processing device, processing method, and process program

Abstract

To provide a new technique for reproducing a three-dimensional structure, a processing device (1) comprises: a specification means that identifies a non-estimation target voxel group; an estimation / completion means that calculates the voxel values of estimable voxels and complements the voxel values of non-estimable voxels among voxels included in an estimation target voxel group; and an assignment means that assigns a confidence level to each voxel included in the estimation target voxel group.

Description

Processing device, processing method, and processing program

[0001] The present disclosure relates to a processing device, a processing method, and a processing program.

[0002] When a three-dimensional region is restored as a three-dimensional structure from a color image and / or a depth image captured of the three-dimensional region, some parts may not be restored as a three-dimensional structure due to factors such as the camera position, the accuracy of the device, textureless regions, etc. Under such circumstances, Patent Document 1, Non-Patent Document 1, and Non-Patent Document 2 disclose techniques for complementing parts that have not been restored as a three-dimensional structure.

[0003] Japanese Patent Application Publication No. 2019-28861

[0004] Angela Dai et. al., ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans, 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018 pp. 4578-4587.Wenbo Hu et. al., Cycle4Completion: Unpaired Point Cloud Completion using Cycle Transformation with Missing Region Coding, 2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021 pp. 14368-14377.

[0005] The technology described in Patent Document 1 interpolates areas that have not been reconstructed as a three-dimensional structure by referencing the labels of adjacent ranging points. This requires assigning a label to each ranging point according to the type of subject. Furthermore, the completion of the three-dimensional structure described in Non-Patent Documents 2 and 3 is achieved by learning using information on a complete three-dimensional structure, such as synthetic data. Indoor three-dimensional structures can be accurately reconstructed by using appropriate synthetic data (e.g., the SUNCG dataset). However, due to the wide variety of objects present outdoors and the wide variety of terrain included in the background, it is difficult to create synthetic data for use in learning for outdoor three-dimensional structures.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of the purpose is to provide a new technology for complementing areas that were not restored when restoring a three-dimensional structure from an image of a three-dimensional area.

[0007] A processing device according to one aspect of the present invention comprises: an identification means for identifying, in a voxel space representing a three-dimensional region, a group of voxels that are not included in the angle of view of any of one or more images obtained by imaging the three-dimensional region as a group of voxels that are not to be estimated; an estimation / complementation means for calculating, among voxels included in a group of voxels to be estimated from the voxel space excluding the group of non-estimation voxels, voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, by referring to the one or more images, and complementing voxel values ​​of non-estimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; and an assignment means for assigning a reliability to each voxel included in the group of voxels to be estimated, wherein the assignment means assigns a reliability to the non-estimable voxels that is lower than the reliability assigned to the estimable voxels.

[0008] A processing method according to one aspect of the present invention includes at least one processor: identifying, in a voxel space representing a three-dimensional region, a voxel group that is not included in the angle of view of any of one or more images obtained by capturing the three-dimensional region as a non-estimation target voxel group; calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space by referring to the one or more images; and complementing voxel values ​​of unestimable voxels whose voxel values ​​cannot be estimated from the one or more images by referring to the voxel values ​​of the estimable voxels; and assigning a reliability of the voxel value of each voxel included in the estimation target voxel group, and assigning a reliability lower than a reliability assigned to an unestimable voxel.

[0009] A program for causing a computer to function as a processing device according to one aspect of the present invention causes at least one processor of the computer to function as: identification means for identifying, in a voxel space representing a three-dimensional region, a group of voxels that are not included in the angle of view of any of one or more images obtained by imaging the three-dimensional region as a group of non-estimation target voxels; estimation / complement means for calculating, by referring to the one or more images, voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in the group of voxels to be estimated, excluding the group of non-estimation target voxels from the voxel space; and assignment means for assigning a reliability to the voxel value of each voxel included in the group of voxels to be estimated, wherein the assignment means assigns a reliability to the non-estimable voxels that is lower than the reliability assigned to the estimable voxels.

[0010] According to one aspect of the present invention, it is possible to provide a new technique for complementing parts that have not been restored when a three-dimensional structure is restored from an image of a three-dimensional region.

[0011] FIG. 1 is a block diagram showing the configuration of a processing device 1 according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of a processing method S1 according to exemplary embodiment 1 of the present invention. FIG. 3 is a diagram showing an example of processing by an assignment unit 13 provided in the processing device 1 according to exemplary embodiment 1 of the present invention. FIG. 4 is a block diagram showing the configuration of a processing device 1A according to exemplary embodiment 2 of the present invention. FIG. 5 is a flow diagram showing the flow of a processing method S1A according to exemplary embodiment 2 of the present invention. FIG. 6 is a block diagram showing the configuration of a processing device 1B according to exemplary embodiment 3 of the present invention. FIG. 7 is a flow diagram showing the flow of a processing method S1B according to exemplary embodiment 3 of the present invention. FIG. 8 is a block diagram showing the configuration of a processing program according to exemplary embodiment 4 of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0013] (Overview of Processing Device 1) When creating a group of voxels representing a three-dimensional region from a color image and / or depth image of the region, blind spots due to occlusions and areas without texture may not be restored as voxels, resulting in voxels without features. Even when performing tasks such as three-dimensional semantic segmentation and object detection on a group of voxels including voxels without features, the accuracy may be reduced. The processing device 1 according to this exemplary embodiment is a device for improving the accuracy of a task by, for example, complementing unrestored portions of a three-dimensional structure to be restored to perform the task.

[0014] (Configuration of Processing Device 1) The configuration of the processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the processing device 1.

[0015] 1, the processing device 1 includes an identifying unit 11, an estimating / complementing unit 12, and an assigning unit 13. In this exemplary embodiment, the identifying unit 11, the estimating / complementing unit 12, and the assigning unit 13 are configured to respectively realize an identifying means, an estimating / complementing means, and an assigning means.

[0016] The identification unit 11 is configured to identify a group of voxels that are not to be estimated in a voxel space that represents a three-dimensional region. As the group of voxels that are not to be estimated, a group of voxels that are not included in the angle of view of any one or more images obtained by capturing the three-dimensional region are identified. The identification unit 11 receives information about the angle of view of the images as an input.

[0017] The estimation / complement unit 12 is configured to calculate voxel values ​​of estimable voxels among voxels included in an estimation target voxel group, and to complement voxel values ​​of unestimable voxels. The estimation target voxel group is a voxel group obtained by excluding non-estimation target voxels from voxel space. An estimable voxel refers to a voxel included in the estimation target voxel group whose voxel value can be estimated from one or more images. The voxel value of the estimable voxel is calculated with reference to one or more images. An unestimable voxel refers to a voxel whose voxel value cannot be estimated from one or more images. The voxel value of the unestimable voxel is complemented with reference to the voxel value of the estimable voxel. The estimation / complement unit 12 receives one or more images as input.

[0018] The assigning unit 13 is configured to assign a reliability of the voxel value of each voxel included in the estimation target voxel group. The reliability assigned to the unestimable voxels is lower than the reliability assigned to the estimable voxels.

[0019] (Flow of Processing Method S1) The flow of processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of processing method S1.

[0020] As shown in FIG. 2, the processing method S1 includes a specifying process S11, an estimating / complementing process S12, and an assigning process S13.

[0021] The identification process S11 is a process for identifying a group of voxels that are not subject to estimation in a voxel space representing a three-dimensional region. The group of voxels that are not subject to estimation are identified as voxels that are not included in the angle of view of any of one or more images obtained by capturing the three-dimensional region.

[0022] The estimation / complementation process S12 is a process for calculating voxel values ​​of estimable voxels among voxels included in an estimation target voxel group obtained by excluding a non-estimation target voxel group from the voxel space, and for complementing voxel values ​​of unestimable voxels. An estimable voxel refers to a voxel included in an estimation target voxel group obtained by excluding a non-estimation target voxel group from the voxel space, whose voxel value can be estimated from one or more images. The voxel value of the estimable voxel is calculated with reference to one or more images. An unestimable voxel refers to a voxel whose voxel value cannot be estimated from one or more images. The voxel value of the unestimable voxel is complemented with reference to the voxel value of the estimable voxel.

[0023] The assigning process S13 is a process for assigning a reliability to the voxel value of each voxel included in the estimation target voxel group. The reliability assigned to the unestimable voxels is lower than the reliability assigned to the estimable voxels.

[0024] (Effects of the Processing Device 1) As described above, the device 1 according to this exemplary embodiment employs a configuration for complementing voxel values ​​of unestimable voxels in a group of voxels to be estimated. According to the processing device 1 according to this exemplary embodiment, the voxel values ​​of unestimable voxels can be complemented without labeling according to the type of object or learning based on previously created synthetic data before complementing the voxel values. This advantageously provides a new technique for complementing unreconstructed portions when restoring a three-dimensional structure from an image of a three-dimensional area. In particular, the present disclosure can restore three-dimensional structures for which labeling according to the type of object or creating synthetic data containing complete three-dimensional structure information in advance is difficult. Therefore, the present disclosure can be applied to processing outdoor three-dimensional areas, which have been difficult to apply conventional techniques to, and three-dimensional areas with many unknown objects. (Specific Example of Image) The image can be a color image and / or a depth image. Furthermore, for example, the image can be an image in which color information (RGB values) and depth information are assigned to each pixel.

[0025] (Specific Example of Identification Unit 11) The identification unit 11 can identify a group of voxels not subject to estimation by referring to the camera position and orientation of each camera that has acquired one or more images and the angle of view of the camera.

[0026] (Specific example of estimation / complementation unit 12) The voxel value of each voxel is a value including the three-dimensional coordinates of the voxel and a feature of the voxel determined based on the image. The feature of each voxel determined based on the image may be, for example, an RGB value, a normal value, etc. Examples of voxels that cannot be estimated include occluded voxels and missing voxels. An occluded voxel is a voxel corresponding to a point that is not included as a subject in one or more images. A missing voxel is a voxel corresponding to a point that has not been assigned a depth value in an image, or a point that is included as a subject in multiple images but has inconsistent depth values ​​between the images.

[0027] An occluded voxel refers to a voxel corresponding to a point that was not captured by any camera, such as a point that was not restored due to the movement or position of the camera when capturing each image, or a point that was not restored due to a blind spot caused by a foreground object. Also, for example, an occluded voxel may include a voxel corresponding to a point that is included as a subject in one of multiple color images. Since a depth value cannot be estimated for a point that is included in only one color image, the voxel value is not calculated as an estimable voxel, and the point can be interpolated as an occluded voxel included in the group of voxels to be estimated.

[0028] A missing voxel is a voxel that corresponds to a point in an image that has not been assigned a depth value, or a point that is included as a subject in multiple images but has inconsistent depth values ​​between the multiple images. Such a point in a three-dimensional region is, for example, a point that is included as a subject in multiple two-dimensional images but does not have texture assigned by edge processing or the like.

[0029] The voxel value of an unpredictable voxel can be complemented with, for example, the voxel value of a voxel that can be predicted that is closest to the unpredictable voxel.

[0030] (Specific example of assignment unit 13) For example, the assignment unit 13 can assign a reliability to an occluded voxel such that the reliability decreases as the distance from the occluded voxel to the closest estimable voxel to the occluded voxel increases.

[0031] An example of assigning reliability to occluded voxels will be described with reference to Fig. 3. Fig. 3 shows a cross-sectional view of a group of voxels to be estimated, generated from multiple two-dimensional images of a step as a three-dimensional region. Each square represents one voxel. In the upper part of Fig. 3, voxels shown in black represent estimable voxels, and voxels shown with diagonal lines represent occluded voxels. Voxels shown in white are voxels that do not correspond to the subject.

[0032] An example of the reliability assignment process in the assignment unit 13 is shown with reference to the lower part of Figure 3. The assignment unit 13 assigns a reliability of 1.0 to an estimable voxel. The assignment unit 13 assigns a reliability of 0.9 to an occluding voxel facing the estimable voxel, a reliability of 0.8 to an occluding voxel one square away, and a reliability of 0.7 to an occluding voxel two squares away, and does not assign a reliability to voxels that are further away. In this way, the assignment unit 13 assigns a reliability to an occluding voxel that decreases as the distance to the nearest estimable voxel increases.

[0033] The assigning unit 13 can assign a reliability to each missing voxel corresponding to a point where the depth values ​​are inconsistent between multiple images, with the reliability decreasing as the difference between the depth value set for the pixel corresponding to the missing voxel among the pixels constituting the image and the depth value to be set for the pixel estimated from the position of the missing voxel in voxel space increases. The depth value of each pixel of the image can be, for example, a depth value determined based on multiple color images, a depth value acquired together with the image by an RGB-D camera that captures a three-dimensional area, a depth value acquired by a lidar used at the same position and orientation as the color image, and a depth value of a depth image acquired by a lidar. The depth value to be set for the pixel estimated from the position of each missing voxel in voxel space can be determined by a projection method.

[0034] An example of assigning reliability to missing voxels will be described with reference to Fig. 4. Each diagram in Fig. 4 shows a projection plane when a group of voxels to be estimated, generated from multiple two-dimensional images captured as a three-dimensional region, is projected onto one of the multiple two-dimensional images. Here, the voxels to be estimated include missing voxels.

[0035] In the upper part of Figure 4, voxels shown in gray indicate pixels corresponding to estimable voxels, and voxels shown with diagonal lines indicate pixels corresponding to missing voxels. Among the missing voxels, voxels shown with thick diagonal lines indicate voxels corresponding to object X. Voxels shown in white are voxels that do not correspond to an object.

[0036] The middle part of Figure 4 shows the results of calculating the difference between the depth value of each pixel in a projected image projected onto a two-dimensional image and the depth value of that pixel in the depth image corresponding to that two-dimensional image, using numbers from 1 to 6. A larger value indicates a larger difference in depth values.

[0037] The bottom row of Figure 4 shows the results of assigning a reliability to each missing voxel based on the magnitude of the difference. Voxels with a difference value of 1 are assigned a reliability of 0.9, voxels with a difference value of 2 are assigned a reliability of 0.8, and voxels with a difference value of 3 are assigned a reliability of 0.7. Voxels with a difference value of 4 or greater are not assigned a reliability.

[0038] (Variation of Processing Device 1) In the processing device 1 described above, the estimation / completion unit 12 and the assignment unit 13 perform sequential and independent processing, but the present invention is not limited to this, and the estimation / completion unit 12 and the assignment unit 13 can perform processing in parallel, such as processing using a truncated signed distance function (TSDF). For example, the estimation / completion unit 12 and the assignment unit 13 calculate voxel values ​​of estimable voxels among voxels included in an estimation target voxel group obtained by excluding a non-estimation target voxel group from the voxel space, and in the process of complementing the voxel values ​​of unestimable voxels, can assign reliability to the unestimable voxels by referring to a score based on their distance from an estimable voxel that is nearby the unestimable voxel.

[0039]

[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.

[0040] 5, the processing device 1A includes an identification unit 11, an estimation / complementation unit 12, an assignment unit 13, a calculation unit 14, and a segmentation unit 15. In this exemplary embodiment, the identification unit 11, the estimation / complementation unit 12, the assignment unit 13, the calculation unit 14, and the segmentation unit 15 are configured to respectively realize an identification means, an estimation / complementation means, an assignment means, a calculation means, and a segmentation means.

[0041] (Configuration of Processing Device 1A) The specifying unit 11, the estimating / complementing unit 12, and the assigning unit 13 of the processing device 1A are components having the same functions as the specifying unit 11, the estimating / complementing unit 12, and the assigning unit 13 of the processing device 1.

[0042] The calculation unit 14 is configured to calculate a feature amount of each voxel included in the estimation target voxel group. The feature amount of each voxel is calculated by referring to the voxel value of the voxel and the reliability assigned to the voxel.

[0043] The segmentation unit 15 is configured to perform semantic segmentation of the voxel group to be estimated and determine the labels of the voxels. The semantic segmentation is performed by referring to the feature amounts of each voxel included in the voxel group to be estimated.

[0044] (Flow of Processing Method S1A) The flow of processing method S1A according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a flow chart showing the flow of processing method S1A.

[0045] As shown in FIG. 6, the processing method S1A includes a specification process S11, an estimation / complementation process S12, an assignment process S13, a calculation process S14, and a segmentation process S15.

[0046] The calculation process S14 is a process for calculating the feature amount of each voxel included in the estimation target voxel group. The feature amount of each voxel is calculated by referring to the voxel value of the voxel and the reliability assigned to the voxel.

[0047] The segmentation process S15 is a process for executing semantic segmentation of the voxel group to be estimated and determining the labels of the voxels. The semantic segmentation is performed by referring to the feature amounts of each voxel included in the voxel group to be estimated.

[0048] (Effects of the Processing Device 1A) The processing device 1A according to this exemplary embodiment employs a configuration for performing semantic segmentation on a group of voxels to be estimated after completion of voxels that cannot be estimated. Therefore, the processing device 1A according to this exemplary embodiment can achieve the effect of performing highly accurate semantic segmentation in addition to the effects achieved by the processing device 1 according to the first exemplary embodiment.

[0049] (Modification of the segmentation unit 15) The processing device 1A may be provided with a configuration for executing a task other than semantic segmentation, instead of or in addition to the segmentation unit 15. For example, the processing device 1A may be provided with a task execution unit for executing an object detection task, such as detecting a three-dimensional area around a target object by enclosing it in a bounding box. Furthermore, for example, the processing device 1A may be provided with a task execution unit for executing a completion task.

[0050]

[0033] A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are denoted by the same reference numerals, and their description will not be repeated.

[0051] 7, the processing device 1B includes an identification unit 11, an estimation / complementation unit 12, an assignment unit 13, a calculation unit 14, a segmentation unit 15, and a learning unit 16. In this exemplary embodiment, the identification unit 11, the estimation / complementation unit 12, the assignment unit 13, the calculation unit 14, the segmentation unit 15, and the learning unit 16 are configured to respectively realize an identification means, an estimation / complementation means, an assignment means, a calculation means, a segmentation means, and a learning means.

[0052] (Configuration of processing device 1B) The identification unit 11, estimation / completion unit 12, assignment unit 13, calculation unit 14, and segmentation unit 15 of processing device 1B are components that have the same functions as the identification unit 11, estimation / completion unit 12, and assignment unit 13 of processing device 1, and the calculation unit 14 and segmentation unit 15 of processing device 1A.

[0053] The learning unit 16 is configured to generate, by machine learning, a model used by the calculation unit 14 to calculate feature quantities. The learning unit 16 uses, for the machine learning, training data including, as feature quantities of unestimable voxels, estimated feature quantities estimated from the feature quantities of the estimable voxels present in the periphery of the unestimable voxels, as correct labels. (Flow of Processing Method S1B) The flow of processing method S1A according to this exemplary embodiment will be described with reference to FIG. 8. FIG. 8 is a flow chart showing the flow of processing method S1B.

[0054] As shown in FIG. 8, the processing method S1B includes a specification process S11, an estimation / complementation process S12, an assignment process S13, a calculation process S14, a segmentation process S15, and a learning process S16.

[0055] The learning process S16 is configured to generate a model used to calculate feature quantities by machine learning. In the learning process S16, training data including, as the feature quantities of unestimable voxels, estimated feature quantities estimated from the feature quantities of the estimable voxels present around the unestimable voxels as correct labels are used for the machine learning.

[0056] As described above, the processing device 1B according to this exemplary embodiment is configured to generate a model used to calculate features by machine learning. Therefore, the processing device 1B according to this exemplary embodiment has the effect of being able to obtain a model that can perform highly accurate semantic segmentation.

[0057] (Specific Example of the Learning Unit 16) The learning unit 16 uses, as training data, data to which correct labels are assigned as feature quantities of unestimable voxels. The training data is created from a group of voxels to be estimated, generated by the estimation / completion unit 12. The training data of estimable voxels included in the group of voxels to be estimated is assigned feature quantities of correct labels with reference to voxel values. Furthermore, the training data of an unestimable voxel included in the group of voxels to be estimated can be a value obtained by copying the correct label of the voxel nearest to the voxel, or a value obtained by reflecting the feature quantities of each of multiple neighboring voxels in the feature quantities of the unestimable voxel according to the distance therebetween, for example.

[0058] The learning unit 16 can update the model used by the calculation unit 14 to calculate the feature amounts based on a first deviation, which is the deviation between the generated training data and the feature amounts of each voxel calculated by the calculation unit 14. The training data and the feature amounts of each voxel calculated based on the estimable voxels and the unestimable voxels are used to calculate the first deviation. The learning unit 16 can update the model used by the calculation unit 14 to calculate the feature amounts, for example, so as to minimize the sum of the first deviation amounts.

[0059] The learning unit 16 can further perform learning using training data of images in which correct labels are assigned to each pixel included in each of one or more images used to generate the estimation target voxel group. For example, the learning unit 16 can update the model used by the calculation unit 14 to calculate the feature amounts based on a second discrepancy, which is the discrepancy between the training data of the two-dimensional image and the feature amounts of the voxels corresponding to the pixels calculated by the calculation unit 14. For example, the learning unit 16 can calculate the sum of the second discrepancies and use this to update the model used by the calculation unit 14 to calculate the feature amounts.

[0060] Here, for each of one or more images, the position of the two-dimensional image within the estimation target voxel group can be determined by referring to the estimation result of the camera position and orientation when the two-dimensional image was captured, thereby determining, for each pixel in the two-dimensional image, the voxel within the estimation target voxel group that corresponds to the pixel.

[0061] (Modification of Processing Device 1B) The calculation unit 14 can further calculate, for each of one or more images, the feature amount of each pixel in the two-dimensional image.

[0062] The segmentation unit 15 may have a configuration for performing semantic segmentation based on the feature amount of each voxel in the estimation target voxel group and the feature amount of each two-dimensional image.

[0063] The learning unit 16 can perform learning using training data of one or more images in which a correct label is assigned to each pixel included in each of the one or more images. The learning unit 16 can update a model used by the calculation unit 14 to calculate the feature amounts based on a third discrepancy, which is the discrepancy between the training data of the one or more images and the feature amounts of each pixel in the one or more images calculated by the calculation unit 14. For example, the learning unit 16 calculates the sum of the second discrepancies and uses this to update the model used by the calculation unit 14 to calculate the feature amounts. With this configuration, semantic segmentation can be performed by referring to both the feature amounts of each voxel and the feature amounts of each pixel, thereby achieving more robust semantic segmentation.

[0064] (Variant example of learning unit 16) When the processing device 1B has a task execution unit instead of the segmentation unit 15, the learning unit 16 can learn using training data created according to the task executed by the task execution unit.

[0065] [Example of Software Implementation] Some or all of the functions of the processing devices 1, 1A, 1B may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.

[0066] In the latter case, the processing devices 1, 1A, and 1B are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 9. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the processing devices 1, 1A, and 1B. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the processing devices 1, 1A, and 1B.

[0067] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0068] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0069] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0070] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0071] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.

[0072] (Supplementary Note 1) A processing device comprising: an identifying means for identifying, in a voxel space representing a three-dimensional region, a voxel group that is not included in the angle of view of any of one or more images obtained by imaging the three-dimensional region as a non-estimation target voxel group; an estimation / complementing means for calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space, by referring to the one or more images, and complementing voxel values ​​of unestimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; and an assigning means for assigning a reliability to the voxel value of each voxel included in the estimation target voxel group, wherein the assigning means assigns a reliability to the unestimable voxels that is lower than the reliability to the estimable voxels.

[0073] According to the above configuration, it is possible to provide a new technique for complementing parts that have not been restored when a three-dimensional structure is restored from an image of a three-dimensional region.

[0074] (Supplementary Note 2) The processing device described in Supplementary Note 1, wherein the unestimable voxels include occluded voxels corresponding to points that were not included as subjects in any of the one or more images, and missing voxels corresponding to points that were not assigned depth values ​​in any of the one or more images, or points that were included as subjects in the multiple images but had inconsistent depth values ​​between the images.

[0075] According to the above configuration, it is possible to specify in more detail the estimation-unavailable voxels based on the cause of their occurrence.

[0076] (Supplementary Note 3) The processing device according to Supplementary Note 2, wherein the assigning means assigns a reliability to each occluded voxel that decreases as the distance from the occluded voxel to the closest estimable voxel to the occluded voxel increases.

[0077] According to the above configuration, it is possible to assign pseudo labels to occluding voxels.

[0078] (Supplementary Note 4) The processing device described in Supplementary Note 2 or 3, wherein the assigning means assigns a reliability to each missing voxel corresponding to a point where depth values ​​are inconsistent between the images, the reliability being lower as the difference between the depth value set for a pixel corresponding to the missing voxel among the pixels constituting the image and the depth value that should be set for the pixel estimated from the position of the missing voxel in the voxel space increases.

[0079] According to the above configuration, it is possible to assign pseudo labels to missing voxels.

[0080] (Supplementary Note 5) The processing device according to any one of Supplementary Notes 1 to 4, further comprising: a calculation means for calculating a feature amount of each voxel included in the group of voxels to be estimated by referring to a voxel value of the voxel and a reliability assigned to the voxel; and a segmentation means for performing semantic segmentation of the group of voxels to be estimated by referring to the feature amount of each voxel included in the group of voxels to be estimated, and determining a label of the voxel.

[0081] According to the above configuration, highly accurate semantic segmentation can be performed.

[0082] (Supplementary Note 6) The processing device according to Supplementary Note 4, further comprising a learning means for generating a model by machine learning that is used by the calculation means to calculate the feature amount, wherein the learning means uses training data for the machine learning that includes, as a correct label, an estimated feature amount estimated from the feature amounts of the estimable voxels present in the periphery of the unestimable voxel.

[0083] According to the above configuration, a model capable of performing highly accurate semantic segmentation can be obtained.

[0084] (Supplementary Note 7) A processing method including: at least one processor identifying, in a voxel space representing a three-dimensional region, a group of voxels that are not included in an angle of view of any of one or more images obtained by imaging the three-dimensional region as a group of voxels that are not to be estimated; calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in a group of voxels that are to be estimated from the voxel space excluding the group of non-estimable voxels, by referring to the one or more images; and complementing voxel values ​​of unestimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; and assigning a reliability indicating a reliability of the voxel value of each voxel included in the group of voxels that are to be estimated, to each voxel that is to be estimated, and assigning a reliability that is lower than a reliability assigned to an estimable voxel, as a reliability assigned to an unestimable voxel.

[0085] According to the above configuration, it is possible to provide a new technique for complementing parts that have not been restored when a three-dimensional structure is restored from an image of a three-dimensional region.

[0086] (Appendix 8) a program for causing a computer to function as a processing device, the program causing at least one processor of the computer to function as: an identification means for identifying, in a voxel space representing a three-dimensional region, a voxel group that is not included in the angle of view of any of one or more images obtained by imaging the three-dimensional region as a non-estimation target voxel group; an estimation / complement means for calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space, by referring to the one or more images, and complementing voxel values ​​of unestimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; and an assignment means for assigning a reliability to each voxel included in the estimation target voxel group, the assignment means assigning a reliability to the unestimable voxels that is lower than the reliability assigned to the estimable voxels.

[0087] According to the above configuration, it is possible to provide a new technique for complementing parts that have not been restored when a three-dimensional structure is restored from an image of a three-dimensional region.

[0088] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.

[0089] a processing device comprising at least one processor, the processor executing: an identification process for identifying, in a voxel space representing a three-dimensional region, a voxel group that is not included in the angle of view of any of one or more images obtained by imaging the three-dimensional region as a non-estimation target voxel group; an estimation / complement process for calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space, by referring to the one or more images, and complementing voxel values ​​of unestimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; and an assignment means for assigning a reliability to the voxel value of each voxel included in the estimation target voxel group, wherein the assignment means assigns a reliability to the unestimable voxels that is lower than the reliability assigned to the estimable voxels.

[0090] The processing device may further include a memory that stores a program for causing the processor to execute the identification process, the estimation / complementation process, and the assignment process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0091] DESCRIPTION OF SYMBOLS 1, 1A, 1B Processing device 11 Identification unit 12 Estimation / complementation unit 13 Assignment unit 14 Calculation unit 15 Segmentation unit 16 Learning unit

Claims

1. an identification means for identifying, in a voxel space representing a three-dimensional region, a group of voxels that are not included in any angle of view of one or more images obtained by capturing the three-dimensional region as a group of voxels that are not to be estimated; an estimation / complementation means for calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space, by referring to the one or more images, and for complementing voxel values ​​of non-estimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; and an assigning means for assigning a reliability of a voxel value of each voxel included in the group of voxels to be estimated, wherein the assigning means assigns a lower reliability to an unestimable voxel than to an estimable voxel.

2. The processing device described in claim 1, wherein the unestimable voxels include occluded voxels corresponding to points that were not included as subjects in any of the one or more images, and missing voxels corresponding to points that were not assigned a depth value in any of the one or more images, or points that were included as subjects in the multiple images but whose depth values ​​were inconsistent between the multiple images.

3. 3. The processing device according to claim 2, wherein said assigning means assigns to each occluding voxel a reliability that decreases as the distance from the occluding voxel to the closest estimable voxel to the occluding voxel increases.

4. 4. The processing device according to claim 2 or 3, wherein the assigning means assigns a reliability to each missing voxel corresponding to a point where depth values ​​are inconsistent between the plurality of images, the reliability being lower as the difference between the depth value set for a pixel corresponding to the missing voxel among the pixels constituting the images and the depth value to be set for the pixel estimated from the position of the missing voxel in the voxel space increases.

5. a calculation means for calculating a feature amount of each voxel included in the estimation target voxel group by referring to a voxel value of the voxel and a reliability assigned to the voxel; The processing device according to claim 1 or 2, further comprising: segmentation means for performing semantic segmentation of the estimation target voxel group by referring to a feature amount of each voxel included in the estimation target voxel group, and determining a label of the voxel.

6. further comprising a learning means for generating a model by machine learning that is used by the calculation means to calculate the feature amount; 6. The processing device according to claim 5, wherein the learning means uses training data for the machine learning, the training data including, as a correct label, an estimated feature amount estimated from the feature amounts of the estimable voxels present around the unestimable voxel, as a feature amount of the unestimable voxel.

7. At least one processor identifies, in a voxel space representing a three-dimensional region, a group of voxels that are not included in any angle of view of one or more images obtained by capturing the three-dimensional region as a group of voxels that are not to be estimated; Among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space, calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images by referring to the one or more images, and complementing voxel values ​​of non-estimable voxels whose voxel values ​​cannot be estimated from the one or more images by referring to the voxel values ​​of the estimable voxels; A processing method including: assigning a reliability indicating the reliability of the voxel value of each voxel included in the group of voxels to be estimated, and assigning a reliability to an unestimable voxel that is lower than the reliability assigned to an estimable voxel.

8. A program for causing a computer to function as a processing device, the program comprising: an identification means for identifying, in a voxel space representing a three-dimensional region, a group of voxels that are not included in any angle of view of one or more images obtained by capturing the three-dimensional region as a group of voxels that are not to be estimated; an estimation / complementation means for calculating voxel values ​​of estimable voxels whose voxel values ​​can be estimated from the one or more images, among voxels included in an estimation target voxel group obtained by excluding the non-estimation target voxel group from the voxel space, by referring to the one or more images, and for complementing voxel values ​​of non-estimable voxels whose voxel values ​​cannot be estimated from the one or more images, by referring to the voxel values ​​of the estimable voxels; a processing program that functions as an assigning means that assigns a reliability of the voxel value of each voxel included in the group of voxels to be estimated, and the assigning means assigns a lower reliability to an unestimable voxel than to an estimable voxel.