Image processing device, method and program

Abstract

An image processing device according to one embodiment of the present invention comprises an acquisition unit that acquires first point cloud data without color information and second point cloud data having color information and positioned in the vicinity of the first point cloud data, and an estimation processing unit that inputs the first and second point cloud data acquired by the acquisition unit into a neural network to output point cloud data in which color information relating to the first point cloud data is estimated.

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to an image processing device, method, and program. [Background technology]

[0002] One way to represent the three-dimensional structure of an object or space is to create a point cloud, which can be generated by measuring the object or space using LiDAR (Light Detecting And Ranging) or a camera. However, the measured point cloud may contain missing data due to issues such as the performance of the LiDAR or camera, the measurement position, or the shape of the object or space, which may prevent the laser emitted by the LiDAR from reaching the target, or the camera being unable to measure the diffuse reflected light from the surface of the object or space.

[0003] Furthermore, when measuring spatial point clouds using LiDAR or a camera, there are cases where obstacles that were present at the time of measurement but that should not be included in the final point cloud data are included, such as temporarily parked vehicles or cardboard placed in a room.

[0004] However, when considering removing such obstacles in post-measurement processing, point clouds such as the road hidden by the obstacle in the shadow of a parked vehicle or the floor of a room hidden in the shadow of a cardboard box will become missing parts because they could not be measured due to the obstacle.

[0005] Various methods have been proposed as Point Cloud Completion techniques that estimate and complete these missing parts using machine-learned neural networks that directly handle point clouds.

[0006] Point Cloud Completion technology can be broadly classified into two types based on the output. The first is a neural network that takes A as input and outputs B', which is the result of estimating B, when a group of missing points to be complemented, i.e., a group of points with a missing portion, is designated as A and the missing portion is designated as B, as disclosed in Non-Patent Document 1, for example. The second is a neural network that takes A as input and outputs "A'+B'" which is the result of estimating the whole of A+B, as disclosed in Non-Patent Document 2, for example.

[0007] However, since all of the above Point Cloud Completion methods only handle the geometric shape completion of point clouds, even if the above point cloud A is a point cloud in which each point has geometric position information and color information, i.e., a so-called color point cloud, the estimated result B' or "A'+B'" will be a point cloud without color information, i.e., a so-called non-color point cloud, and it is not possible to restore the color information.

[0008] Furthermore, for example, Non-Patent Document 3 discloses a learning model generation method using a neural network and a generative adversarial network (GAN) that receives a non-color point cloud as input and outputs a color point cloud. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Huang, Zitian, et al. "Pf-net: Point fractal network for 3d point cloud completion." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020. [Non-patent document 2] Yuan, W., Khot, T., Held, D., Mertz, C., & Hebert, M. (2018, September). Pcn: Point completion network. In 2018 International Conference on 3D Vision (3DV) (pp. 728-737). IEEE. [Non-patent document 3] Liu, J., Dai, S., & Li, X. (2019, September). Pccn: Point cloud colorization network. In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 3716-3720). IEEE. Summary of the Invention [Problem to be solved by the invention]

[0010] The color point cloud output method disclosed in Non-Patent Document 3 has the following two problems. The first issue is that when only geometric information of a point cloud without color information is input, there may be multiple plausible colors from the geometric shape. For example, suppose the geometric shape of the point cloud for which color information is to be estimated is a "car." In this case, the color of the point cloud for the car's body estimated by the neural network of the prior art is likely to be multiple colors, such as white, red, or black, increasing the possibility that the correct color will not be determined. Furthermore, for example, when coloring a point cloud obtained by measuring an outdoor space, even if the space is in the same location and shape, the color of the space will be completely different in the morning and in the evening, so the correct color cannot be uniquely determined from the geometric shape alone.

[0011] This problem means that in the aforementioned Point Cloud Completion, when a non-colored point cloud "A'+B'" is generated after completion from a missing point cloud A and the color of this "A'+B'", i.e., color information, is estimated, the color of this estimated point cloud "A'+B'" will be completely different from the color of A.

[0012] The second issue is that it is difficult to estimate the color of a part of a geometric shape when the geometric shape is insufficient. For example, if the point cloud for which color information is to be estimated is part of the roof of a "car," and it is not possible to distinguish from the shape of this point cloud alone whether the point cloud is part of the car roof or part of a desk top, there are two possibilities: the point cloud may be estimated to have a color similar to that of a "car," or it may be estimated to have a color similar to that of a desk top.

[0013] This issue means that when a non-color point cloud B' of the complemented portion is generated from a missing point cloud A in the aforementioned Point Cloud Completion method, if only this non-color point cloud B' is input, it will be difficult to perform accurate color estimation if the non-color point cloud B' does not have a shape sufficient for color estimation on its own.

[0014] The present invention has been made in light of the above-mentioned circumstances, and its purpose is to provide an image processing device, method, and program that can appropriately estimate missing parts of point cloud data containing color information along with correct color information. [Means for solving the problem]

[0015] An image processing device according to one aspect of the present invention includes an acquisition unit that acquires first point cloud data that does not have color information and second point cloud data that is located near the first point cloud data and has color information, and an estimation processing unit that inputs the first and second point cloud data acquired by the acquisition unit to a neural network and outputs point cloud data in which color information is estimated for the first point cloud data. the neural network adds a vector of the number of elements to a tensor representing the first point cloud data, compresses the number of point cloud data in the result of the addition, adds the vector of the number of elements to a tensor representing the second point cloud data, compresses the number of point cloud data in the result of the addition, combines the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, with the result of combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to a first number of tensors representing the first point cloud data with the result of adding a second number of tensors representing the first point cloud data that is greater than the first number, and outputs point cloud data in which color information is estimated for the first point cloud data. .

[0016] An image processing device according to one aspect of the present invention includes an acquisition unit that acquires first point cloud data that does not have color information and second point cloud data that is located near the first point cloud data and has color information, and inputs the first and second point cloud data acquired by the acquisition unit into a neural network to generate a neural network representation of the first point cloud data. Follow an estimation processing unit that outputs point cloud data with estimated color information, wherein the neural network adds a vector of the number of elements to a tensor representing the first point cloud data and compresses the number of point cloud data in the result of the addition, adds a vector of the number of elements to a tensor representing the second point cloud data and compresses the number of point cloud data in the result of the addition, combines the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, and adds a second number of elements greater than the first number to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data. a parameter update unit that outputs point cloud data in which color information is estimated for the first point cloud data based on a result of combining results of adding numerical vectors, acquires point cloud data with color information, generates ground truth data that is point cloud data with color information, point cloud data without color information, and point cloud data in which a portion of the point cloud data with color information has been removed based on the point cloud data, inputs the generated point cloud data without color information and the point cloud data in which a portion of the point cloud data with color information has been removed to the neural network, acquires the portion of data in which color information is estimated, calculates a loss function for the neural network based on the acquired data and the ground truth data, and updates parameters of the neural network based on the calculated loss function.

[0017] An image processing method according to one aspect of the present invention is performed by an image processing device. circlean acquisition unit of the image processing device acquires first point cloud data that does not have color information and second point cloud data that is located in the vicinity of the first point cloud data and has color information; and an estimation processing unit of the image processing device inputs the first and second point cloud data acquired by the acquisition unit into a neural network, thereby outputting point cloud data in which color information is estimated for the first point cloud data. the neural network adds a vector of the number of elements to a tensor representing the first point cloud data and compresses the number of point cloud data in the result of the addition, adds the vector of the number of elements to a tensor representing the second point cloud data and compresses the number of point cloud data in the result of the addition, combines the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, with the result of combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to a first number of tensors representing the first point cloud data with the result of adding a second number of tensors representing the first point cloud data that is greater than the first number, and outputs point cloud data in which color information is estimated for the first point cloud data. .

[0018] An image processing method according to one aspect of the present invention is performed by an image processing device. circlean acquisition unit of the image processing device acquires first point cloud data that does not have color information and second point cloud data that is located near the first point cloud data and has color information; an estimation processing unit of the image processing device inputs the first and second point cloud data acquired by the acquisition unit into a neural network, thereby outputting point cloud data in which color information is estimated for the first point cloud data; the neural network adds a vector of the number of elements to a tensor representing the first point cloud data and compresses the number of point cloud data in the added result; adds a vector of the number of elements to a tensor representing the second point cloud data and compresses the number of point cloud data in the added result; combines the compressed result of the number of point cloud data obtained by adding the vector of the number of elements to the tensor representing the first point cloud data with the compressed result of the number of point cloud data obtained by adding the vector of the number of elements to the tensor representing the second point cloud data, with the same number of point cloud data; a parameter update unit of the image processing device acquires point cloud data with color information, and based on the point cloud data, generates ground truth data that is point cloud data with color information, point cloud data without color information, and point cloud data with a portion removed from the point cloud data with color information, inputs the generated point cloud data without color information and the generated point cloud data with a portion removed from the point cloud data with color information to the neural network, acquires the portion of data with estimated color information, and calculates a loss function for the neural network based on the acquired data and the ground truth data, and updates the parameters of the neural network based on the calculated loss function. [Effects of the Invention]

[0019] According to the present invention, it is possible to appropriately estimate missing portions of point cloud data having color information together with correct color information. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram showing a first example of inputs and outputs of a neural network used in one embodiment of the present invention. [Figure 2A] FIG. 2A is a diagram illustrating an example of an implementation of a multilayer perceptron (MLP) layer. [Figure 2B] FIG. 2B illustrates an example implementation of an MLP layer. [Figure 2C] FIG. 2C illustrates an example implementation of an MLP layer. [Figure 3] FIG. 3 is a diagram showing an example of implementation of the SA layer (Set Abstraction Layer). [Figure 4] FIG. 4 is a diagram illustrating a first example of an implementation of a Feature Propagation Layer (FP layer). [Figure 5] FIG. 5 is a diagram illustrating a second example of an implementation of the FP layer. [Figure 6] FIG. 6 is a diagram showing a second example of inputs and outputs of a neural network used in one embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing an example of implementation of a PTF layer (Point Transformer Layer). [Figure 8A] FIG. 8A is a diagram illustrating an example implementation of the PTF layer. [Figure 8B] FIG. 8B is a diagram illustrating an example implementation of the PTF layer. [Figure 9] FIG. 9 is a diagram illustrating an example of an implementation of the PTF layer. [Figure 10] FIG. 10 is a diagram illustrating a first example of an identification network. [Figure 11] FIG. 11 is a diagram illustrating an example of an implementation of the MLP layer. [Figure 12] FIG. 12 is a diagram illustrating a second example of an identification network. [Figure 13] FIG. 13 is a diagram illustrating a first example of a process for generating training data for a neural network. [Figure 14] FIG. 14 is a diagram illustrating a second example of the process of generating training data for a neural network. [Figure 15] FIG. 15 is a block diagram showing an example of the overall configuration of an image processing system according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a color estimation DNN (Deep Neural Network) used in the first embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of data storage in the learning point cloud DB. [Figure 18] FIG. 18 is a diagram illustrating an example of data storage in a color estimation DNN parameter DB. [Figure 19] FIG. 19 is a flowchart illustrating an example of a processing operation by the learning operation unit of the computing terminal according to the first embodiment. [Figure 20] FIG. 20 is a flowchart illustrating an example of processing operations in the color estimation DNN processing unit of the computing terminal according to the first embodiment. [Figure 21] FIG. 21 is a flowchart illustrating an example of a processing operation of the input operation unit of the computing terminal according to the first embodiment. [Figure 22] FIG. 22 is a diagram illustrating an example of the overall configuration of an image processing system according to the second embodiment. [Figure 23] FIG. 23 is a diagram illustrating an example of a color estimation DNN used in the second embodiment. [Figure 24] FIG. 24 is a diagram illustrating an example of a classification DNN used in the second embodiment. [Figure 25] FIG. 25 is a diagram illustrating an example of data storage of classification DNN parameters. [Figure 26] FIG. 26 is a flowchart illustrating an example of a processing operation by the learning operation unit of the computing terminal according to the second embodiment. [Figure 27] FIG. 27 is a flowchart illustrating an example of a procedure of processing operations in the classification DNN processing unit according to the second embodiment. [Figure 28] FIG. 28 is a block diagram showing an example of the hardware configuration of a computing terminal of an image processing system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] An embodiment of the present invention will be described below with reference to the drawings. In this embodiment, a colored point cloud (color point cloud) (also referred to as colored point cloud data) of N points having geometric information and color information present in the vicinity of a point cloud whose color information is to be estimated is expressed as N(x i , y i , z i , r i , g i , b i ) is written as In this case, i is "i=1, 2, 3, …, N" and x i , y i , z i is the geometric information of the point group, and r i , g i , b i is the color information of the point group.

[0022] In this embodiment, the colorless point group (non-color point group) of N' points that are the target of color estimation is N'(x' i´ , y´ i´ , z´ i´ ), and a point cloud with N' colorless points with correct color information for each point is written as N'(x' i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´ ) is written as In this case, i' is "i'=1, 2, 3, …, N'" and x' i´ , y´ i´ , z´i´ is the geometric information of the colorless point cloud, and r' i´ , g´ i´ , b´ i´ is the color information of the colorless point cloud.

[0023] In this embodiment, N(x i , y i , z i , r i , g i , b i ) and N' points, which is a non-color point group of N' points, N'(x' i´ , y´ i´ , z´ i´ ) is input, and this non-color point group N´(x´ i´ , y´ i´ , z´ i´ ) color point cloud N´(x´ i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´ This paper deals with a neural network that outputs estimated results of a neural network and a method for generating a learning model for the neural network. In this embodiment, the missing point group A in the above-mentioned Point Cloud Completion is a colored point group N(x i , y i , z i , r i , g i , b i ), and the non-color point group B' or "A'+B'" that is the result of complementing the missing parts corresponds to the colorless point group N'(x' i´ , y´ i´ , z´ i´ ) applies.

[0024] In this embodiment, in order to solve the above two problems of the conventional method, in addition to inputting the geometric information of a point cloud without color information as in the conventional method, we will explain how to extend the neural network algorithm so that the geometric information and color information of a colored point cloud near the point cloud that is the target of color estimation can also be input. In addition, this embodiment will also explain a method for generating a learning model of the neural network.

[0025] (Neural network algorithm (first example)) Here, we describe the first example of a neural network algorithm that can input not only the geometric information of a point cloud without color information, but also the geometric information and color information of a colored point cloud that exists near the point cloud whose color information is to be estimated.

[0026] FIG. 1 is a diagram showing a first example of inputs and outputs of a neural network used in one embodiment of the present invention. In Fig. 1, a colored point cloud N(x) of N points with geometric and color information and missing parts (symbol a in Fig. 1) is shown. i , y i , z i , r i , g i , b i ) and the colorless point cloud N'(x' i´ , y´ i´ , z´ i´ ) is input, and a colored point cloud N'(x' i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´ ) is the configuration of a neural network that outputs the estimation result.

[0027] This neural network is realized by combining three layers: an MLP layer, an SA layer, and an FP layer. The MLP layer is a neural network of a multilayer perceptron, and is composed of a fully connected linear layer (hereinafter referred to as Linear(a, b)) with a number of input elements a and a number of output elements b, and an activation function for each element such as ReLU.

[0028] In the example shown in Figure 1, the colored point cloud N(x i , y i , z i , r i , g i , b i ) is input to the output of MLP1, and SA, SA, and MLP2 are connected in series. i´ , y´ i´ , z´ i´ ) is input to the output of MLP1, to which SA, SA, and MLP2 are connected in series. The outputs of the two MLP2s are connected to FP, to the output of which four stages of FP are connected in series, and MLP3 is connected to the output of the final stage FP. Each layer from MLP1 to MLP2 is also called an encoder. Colored point cloud N(x i , y i , z i , r i , g i , b i ) is input to the output of MLP1, which is also connected to the FP at the fourth stage from the output of the FP connected to the output of each MLP2, and the colorless point cloud N'(x' i´ , y´ i´ , z´ i´ ) is also connected to the FP in the third stage from the FP connected to the output of each MLP2.

[0029] Colored point cloud N(x i , y i , z i , r i , g i , b i The output of the SA connected to the output of the MLP1 that receives the input is also connected to the FP in the second stage from the FP connected to the output of each MLP2, and the colorless point cloud N'(x'i´ , y´ i´ , z´ i´ The output of the SA connected to the output of the MLP1 that receives the input of the signal ) is also connected to the FP in the first stage from the FP connected to the output of each MLP2.

[0030] 2A, 2B, and 2C are diagrams illustrating an example of an implementation of an MLP layer. 2A, 2B, and 2C, MLP1, MLP2, and MLP3 are shown as example implementations of the MLP layer shown in FIG. In MLP1 shown in Figure 2A, N(x i , y i , z i , r i , g i , b i ) or N´(x´ i´ , y´ i´ , z´ i´ ) is input, and in this embodiment, the point groups N and N' are expressed as (N, 3 + 3), (N', 3 + 0), focusing on the number of elements in each layer of the tensor. In this case, the first "3" in the "3+3" and "3+0" in the second layer of the tensor represents the position information of each point (x, y, z), and the latter "3" and "0" in the "3+3" and "3+0" in the second layer represent the color information of each point or the length of the feature vector.

[0031] MLP1 in Figure 2A first assigns each input point to Linear(6, D) or Linear(3, D). The first half of the argument to be assigned corresponds to whether the input has color information or not; if it does not have color information, it will input "3", and if it does have color information, it will input "6". The D in the above Linear(6, D) or Linear(3, D) is an integer parameter determined according to the number of point groups, etc., and in the example of FIG. 2A, D=32. The output of the above Linear(6,D) or Linear(3,D) is (N,D) or (N',D) because a vector with D elements is given for each point.

[0032] Next, the activation function ReLU is applied to the output (N, D) or (N', D). Furthermore, the ReLU output (N, D) or (N', D) can be multiplied by Linear (D, D) to obtain the output (N, D) or (N', D).

[0033] Finally, (N, 3) and (N', 3) which contain the position information of the input N or N' are combined with this output (N, D) or (N', D) to produce the output (N, 3 + D) and (N', 3 + D). The configuration described here is one possible configuration, and if the Linear layer and ReLU layer after MLP1 are applied over multiple stages, or if the number of output elements in the output of the final Linear layer is D, changes such as increasing or decreasing the number of elements along the way become adjustment factors for the hyperparameters.

[0034] The MLP2 shown in Figure 2B corresponds to the final stage of the encoder shown in Figure 1. 、 After applying multiple SA layers to the inputs to the two MLP1s to reduce the number of input points. of Applies to the last stage.

[0035] Figure 1 shows an example where two SA layers are applied to the output of MLP1, resulting in N points compressed to 1 / 16 (N / 16, 3 + D), or N' points compressed to 1 / 16 (N' / 16, 3 + D), being input to MLP2. Also, in MLP2 shown in FIG. 1, D=128.

[0036] In Figure 1, there are two SA layers connected to each MLP1, but by repeatedly adapting the SA layers to three or four layers, the number of points in the point cloud data can be further compressed to 1 / 64 or 1 / 256, in which case the number of dimensions will increase to D = 256 or D = 512, etc. The number of SA layers is a parameter that is determined by the number of points N or N' to be treated.

[0037] As shown in Figure 2B, in MLP2, the above (N / 16, 3 + D) with the position information omitted is (N / 16, D), or (N / 16, 3 + D) with the position information omitted is (N' / 16, D), and this is input to Linear(D, D'). D' is a parameter determined according to the value of D, and is set to D'=256 when D=128.

[0038] After that, as with MLP1, the input and output of ReLU, Linear(D',D'), ReLU, Linear(D',D) is repeated, and (N / 16,3 + D) is output in which (N / 16,3) with position information of N / 16 is combined with the output of Linear(D',D) in the final stage, or (N' / 16,3 + D) in which (N' / 16,3) with position information of N' / 16 is combined with is output. The MLP2 configuration described here is one possible configuration, and adapting the subsequent Linear layer and ReLU layer within the MLP2 over multiple stages, or increasing or decreasing the number of elements along the way if the number of output elements in the output of the final Linear layer is D, are adjustment factors that function as hyperparameters.

[0039] MLP3 shown in Fig. 2C corresponds to the output of the final stage of the neural network shown in Fig. 1. D and D' in MLP2 are integer parameters, and in the example shown in Fig. 2C, D = 32 and D' = 64. Like MLP1 and MLP2, MLP3 also repeatedly applies a linear layer and a ReLU activation function, and the output of the final linear layer is output as "3" because it outputs color information.

[0040] If color information is expressed between 0 and 1, the result of applying a sigmoid activation function to the output of the final Linear layer can be applied as the output. The output of the final Linear layer and Sigmoid is (N', 3), where "3" corresponds to each element of color information such as r, g, and b.

[0041] Finally, the (N', 3) part of the MLP3 input (N', 3 + D), which corresponds to the position information of the point cloud N', is combined with the output (N', 3) of the final Linear or Sigmoid layer to obtain the final output (N', 3 + 3) consisting of the position information and color information of N'.

[0042] In MLP3, adjusting the number of input or output elements in the linear layer or adding more linear and ReLU layers is also a hyperparameter adjustment factor so that the final output number of elements in the linear layer is 3. Also, if color information is not expressed in the range of 0 to 1, it is possible to omit the sigmoid applied to the final linear layer.

[0043] Next, we will discuss the SA layer in Figure 1. Figure 3 is a diagram showing an example of the implementation of the SA layer. The SA layer acts as an encoder that compresses the number of points in the input point group N or point group N' while extracting the feature value of each point. In the example shown in Figure 1, the number of points is compressed to one-fourth in one SA layer, but the compression ratio is an adjustment factor of the hyperparameters, and it can also be set to one-half or one-eighth.

[0044] In the example shown in Figure 1, two SA layers are applied to each of the point cloud N and point cloud N'. The number of applications is also one of the hyperparameter adjustment factors, and the number of applications can be increased to three or four. Re That's fine.

[0045] In the example shown in Figure 1, the feature value of each point with its position information removed before and after the SA layer is doubled from 32 to 64, or from 64 to 128. However, determining the number of features when passing through each layer is also one of the hyperparameter adjustment factors, and can be adjusted by changing the numerical value.

[0046] In the example shown in Figure 3, the number of points in the input point cloud is N, the number of feature elements excluding position information is D, the number of points in the output point cloud is Nds, and the number of feature elements excluding position information is Dds. In the SA layer, the initially input number of points N is downsampled to the output Nds, and is output as a second-order tensor (Nds, 3).

[0047] In this down sampling, any Nds points may be randomly sampled from the N points, or farthest point sampling, which is a widely known general method, may be used.

[0048] Next, in the SA layer, for each sampled point Nds, the k points that are the closest neighbors are selected from the N input points. Then, the combination of the k points selected for each point Nds and the feature values ​​D possessed by each of the k points is output as a third-order tensor (Nds, k, D). k is a hyperparameter, and in Figure 3, k = 16.

[0049] Next, in the SA layer, the output (Nds, k, D) is subjected to 2D convolution with the number of input channels D, the number of output channels Dds, the kernel size 1x1, and the slide size 1 (hereinafter, these parameters are collectively referred to as Conv2D(D, Dds, 1)).

[0050] The third-order tensor (Nds, k, D) input to this two-dimensional convolution can be regarded as data of D channels, having a pixel count of Nds×k in length and width.

[0051] Next, batch normalization and the activation function ReLU are applied to the output of Conv2D(D, Dds, 1), and then Conv2D(Dds, Dds, 1), batch normalization, and ReLU are applied to it. The number of times these combinations of Conv2D, batch normalization, and activation functions are applied is a hyperparameter adjustment factor.

[0052] Furthermore, if the number of output channels of the final-stage Conv2D is Dds, the number of input / output channels of the middle-stage Conv2D is a hyperparameter and an adjustment factor.

[0053] Next, in the SA layer, a Maxpool layer is applied to the feature vector of Dds elements, which is the output of the final ReLU stage (Nds, k, Dds), and extracts only the largest feature among the k points corresponding to each point in Nds.

[0054] This Maxpool layer combines k feature vectors with Dds elements into a single feature vector with Dds elements, the largest for each element. Therefore, the third-order tensor (Nds, k, Dds) that is input to the Maxpool layer is output as a second-order tensor (Nds, Dds).

[0055] Finally, in the SA layer, the down-sampled Nds points and position information (Nds, 3) are combined with the output of the Maxpool layer (Nds, Dds), and finally output as (Nds, 3 + Dds).

[0056] Next, we will discuss the FP layer shown in Figure 1. Figure 4 is a diagram showing a first example of an implementation of the FP layer. The FP layer has two inputs and can be applied in two patterns: N', N' combination, in which both inputs are the results of applying the MLP layer or SA layer to the point cloud N', and N, N' combination, in which one input is the result of applying the MLP layer or SA layer to the point cloud N.

[0057] Furthermore, the number of FP layers is "O+1" if the number of SA layers is O. In other words, the number of FP layers is an adjustment parameter that is affected by the number of SA layers. Furthermore, the N, N' and N', N' connections are applied alternately, and the input point cloud is selected and applied in order from the output of the latter stage, where the number of points has been reduced by repeatedly applying the MLP layer and SA layer.

[0058] For the two inputs of the FP layer, the number of point clouds in input A is Nds, the number of elements in the feature vector of each point other than the location information is Dds, the number of point clouds in input B is Nup, and the number of elements in the feature vector of each point other than the location information is Dup.

[0059] In the N, N' combination, the result of applying an MLP layer or SA layer to point group N' is applied to input A, and the result of applying an MLP layer or SA layer to point group N is applied to input B. In addition, in N, N connections, the output of the MLP layer next to the final SA layer or the output of the previous FP layer is applied to input A, and the result of applying the MLP layer or SA layer to point cloud N' is applied to input B.

[0060] In the FP layer shown in Figure 4, first, for each point p i For (i=1, 2, …, Nds), the top k points p closest to input A ij (j=1, 2, …, k) is selected, and p i From p ij Each distance d to ij is calculated and a third-order tensor (Nup, k, 1) is output (see symbol a in Figure 4).

[0061] Next, distance d ij Each point p of Nup iThe weights w for the top k neighboring points of input A are calculated from ij is calculated using the following equation (1) (see symbol b in Figure 4).

[0062]

number

[0063] This weight w ij is the distance d ij is a variable that increases as the point gets shorter, and α is ij = 0, it is a hyperparameter that takes a small value to prevent indeterminacy. In the example shown in Figure 1, α = 1.0*10 -8 is.

[0064] Next, for each point p selected from Nds by input A, ij The feature vector v corresponding to ij (Number of elements Dds) for each point p ij The weight w corresponding to ij The product of w ij v ij is calculated (see symbol c in FIG. 4). Then, for each point p of Nup i The weighted feature vector w of k points selected from ij v ij The pair (Nup, k, Dds) is output.

[0065] Next, for each point p i The sum of each element of the weighted feature vector of k points selected from

[0066]

number

[0067] (feature vector v i The number of elements Dds) is calculated (see symbol d in Figure 4). And for each point p i and the extracted feature vector v iThe pair is output as a second-order tensor (Nup, Dds).

[0068] Next, the tensor (Nup, Dup) consisting of the features excluding the Nup point of input B and the position information at the time of input, and the output of the Maxpool layer (Nup, Dds) are combined to form (Nup, Dds + Dup), which is output to the next layer.

[0069] Next, one-dimensional convolution (hereinafter referred to as Conv1D(Dds + Dup, Dup, 1) including each parameter) of the input channel "Dds + Dup", output channel Dup, kernel size "1", and slide size "1" is performed on the above (Nup, Dup), and a combination of features with element count Dup for each point is output (Nup, Dup).

[0070] Furthermore, batch normalization and the activation function ReLU are applied to each element output by the above Conv1D(Dds + Dup, Dup, 1). This combination of Conv1D, batch normalization, and activation function may be applied multiple times, for example, two or three times, and the number of applications is an adjustment factor for the hyperparameters. In this case, it is sufficient that the input channel of the first Conv1D is "Dds + Dup" and the output channel of the last Conv1D is Dup.

[0071] Finally, the output of the last ReLU layer (Nup, Dup) is combined with each point in input B and its position information pair (Nup, 3), and the final output of the FP layer is (Nup, 3 + Dup).

[0072] Fig. 5 is a diagram showing a second example of implementation of the FP layer. In addition to the example shown in Fig. 4, the example shown in Fig. 5 is also possible for the FP layer. In the FP layer shown in FIG. 5, input A and input B are the same inputs as in the example shown in FIG. 4, and a Linear(Dds, Dup), batch normalization, and ReLU layer are applied to input A, and a Linear(Dup, Dup), batch normalization, and ReLU layer are applied to input B.

[0073] In the FP layer shown in Figure 5, a combination of three layers consisting of a Linear layer, a batch normalization layer, and a ReLU layer is first applied to both inputs, but if the number of output elements of the final Linear layer is Dup, the combination of the above three layers may be applied twice or three times, and the number of applications is an adjustment factor for the hyperparameters.

[0074] Next, for input A and input B, similar to the FP layer shown in Figure 4, each point p i For (i=1, 2, …, Nds), the top k nearest points p ij (j=1, 2, …, k) is selected, and p i From p ij The respective distances d to ij is calculated and a third-order tensor (Nup, k, 1) is output (see symbol a in Figure 5).

[0075] And the distance d ij Each point p of Nup i The weights w for the top k points among the neighboring points of input A are calculated from ij is calculated by the above formula (1) (see symbol b in FIG. 5).

[0076] Next, for each point p selected from Nds of input A, ij The feature vector v is the output of Linear, Batch Normalization, and ReLU. ij (Number of elements Dup) for each point p ij Corresponding to w ij The product of w ij v ij is calculated (see symbol c in FIG. 5). Then, for each point p of Nupi The weighted feature vector w of k points selected from ij v ij The pair (Nup, k, Dup) is output.

[0077] Next, similar to the FP layer shown in Figure 4, for each point p i The sum of each element of the weighted feature vector of k points selected from

[0078]

number

[0079] (feature vector v i The number of elements Dup is calculated (see symbol d in FIG. 5). And for each point p i and the extracted feature vector v i The pair is output as a second-order tensor (Nup, Dup) (see symbol e in Figure 5).

[0080] Next, we apply Linear, Batch Normalization, and ReLU to the input B, and output a second-order tensor T with the number of elements (Nup, Dup). in2 The second element of the second-order tensor, i.e., the sum of each element of the number of elements Dup, is calculated using the second-order tensor (Nup, Dup) whose number of elements was summed in the previous layer, and a new second-order tensor (Nup, Dup) is output.

[0081] Finally, the second-order tensor (Nup, 3) consisting of the position information for the input Nup points is combined with the second element of the second-order tensor whose sum was calculated in the previous layer, and (Nup, 3 + Dup) is finally output.

[0082] Finally, we will summarize and explain the main points of the neural network configuration shown in Figure 1 above. Known configurations can be applied to the MLP layer, SA layer, and FP layer in this neural network.

[0083] On the other hand, the greatest feature of this neural network is that "in addition to taking as input the geometric information of point clouds that do not have color information, it is also possible to input the geometric information and color information of colored point clouds that exist in the vicinity of the point cloud that is the target of color information estimation." In order to realize this configuration, the processing of the N, N' connections in the FP layer is a unique application that does not rely on known configurations.

[0084] (Neural network algorithm (second example)) Next, we will describe a second example of a neural network algorithm that can input not only the geometric information of a point cloud without color information, but also the geometric information and color information of a colored point cloud that exists near the point cloud whose color information is to be estimated.

[0085] FIG. 6 is a diagram showing a second example of inputs and outputs of a neural network used in one embodiment of the present invention. This second example of a neural network algorithm is an improvement over the first example of the neural network algorithm described above, resulting in improved accuracy.

[0086] The second example shown in Figure 6 differs from the first example shown in Figure 1 in that, except for the final MLP layer, a new PTF layer is sandwiched after each of the other MLP layers, each SA layer, and each FP layer. In this second example, the implementation of the MLP layer, SA layer, and FP layer is the same as the example shown in Figures 2 to 5 above. Also, as in the first example, the internal processing of the MLP layer, SA layer, and FP layer has parameters (such as k) that can be adjusted as hyperparameters, or degrees of freedom for expansion (such as the number of Linear layers and Conv layers applied).

[0087] In addition, in the example of Figure 6, the SA layer is applied "2 stages x 2 = 4 times" and the FP layer is applied "4 + 1 = 5 times." However, it is also possible to add more SA layers within the range of "X (variable) stages x 2" and the FP layer "(X x 2) + 1," for example, by applying the SA layer "3 stages x 2 = 6 times" and the FP layer "6 + 1 = 7 times," as in the first example.

[0088] Next, the details of the processing by the PTF layer will be explained. Figures 7, 8A, 8B and 9 are diagrams showing implementation examples of the PTF layer. The PTF layer receives input of a second-order tensor (N, 3 + D) that has the number of points N and the feature value of "position information + number of elements D" for each point.

[0089] First, as shown in Figure 7, in the (N, 3 + D) tensor input to the PTF layer, for each point p in the N point cloud, the k closest points are selected from the same N point cloud (see symbol a in Figure 7). Then, for each point p in the N points, a third-order tensor (N, k, 3 + D) consisting of the "position information + feature values" of the k neighboring points is output. k is a positive integer that is adjusted according to the input data, and in the example shown in Figure 7, k = 16. For the sake of explanation, the output of this layer (N, k, 3 + D) is defined as kNN.

[0090] Next, Linear(D, H) is applied to the input (N, 3 + D) to the PTF layer, with the position information removed (N, D). H is a hyperparameter that is adjusted according to the input data, and in the example shown in Figure 7, H=512.

[0091] Furthermore, three Linear(H, H) are applied separately, i.e., in parallel, to the second-order tensor (N, H) that is the output of Linear(D, H) from the previous layer. For the sake of explanation, the output of the second-order tensor (N, H) that is the output of the first Linear(H, H) is defined as Que. The second-order tensor (N, H) that is the output of the second Linear(H, H) is processed by referring to the kNN data. First, kNN (N, k, 3 + D) is a vector consisting of the k points closest to p "out of N points" for each point p on the N points, and the feature values ​​of this position and D elements (see symbol b in Figure 7), and the selected k points are points on the N points.

[0092] The output (N, H) of Linear(H, H) is a tensor consisting of features with H elements for each point p of N points. Therefore, it can be considered that there are features with H elements for each of the k points selected from the N points.

[0093] Therefore, in this example, the process is performed to replace the "3 + D" element of each k point in the above (N, k, 3 + D) with H. The result of this process is a third-order tensor of (N, k, H).

[0094] The same process is performed on the output of the third Linear(H, H), resulting in a third-order tensor of (N, k, H). In the example shown in Figure 7, (N, k, H) derived from the output of the second Linear(H, H) is defined as Key, and (N, k, H) derived from the third Linear(H, H) is defined as Value.

[0095] Next, in the PTF layer shown in Figure 8A, the data of (N, k, 3 + D) of kNN and (N, k, 3) and (N, 3) consisting of only the point and position information of (N, 3 + D) of input A are input, and a point p on N points is i =(x i , y i , z i ) (where i=1, 2, … ,N) and p i The neighboring point p of ij =(x ij , y ij , z ij ) (where j=1, 2, …, k) i -x ik , y i -y ik , z i -z ik ) is calculated (see symbol a in FIG. 8A). The above difference is composed of three elements, just like the position information, and the difference elements between all points on the N points and the neighboring k points are calculated and output as (N, k, 3).

[0096] Next, Linear(3,H), ReLU, and Linear(H,H) are applied sequentially to the previous layer (N, k, 3). In this example, two Linear layers are applied, but the number of layers applied is an adjustable hyperparameter. As long as the number of input elements in the first Linear layer is 3 and the number of output elements in the last Linear layer is H, the number of layers may be increased or the number of input / output elements in the intermediate layer may be changed from H. In the example shown in FIG. 8A, the output (N, k, H) of the last Linear layer is defined as POS_ENC for the sake of explanation.

[0097] Next, in the PTF layer shown in FIG. 8B, the Que, Key, and POS_ENC are input and the calculation "Que-Key+POS_ENC" is performed. In this case, Que is a second-order tensor of (N, H), Key is a third-order tensor of (N, k, H), and POS_ENC is a third-order tensor of (N, k, H). However, the shape of the tensors is adjusted by copying the same k values ​​for the feature vector with H elements at each point of N in Que and expanding it into a third-order tensor of (N, k, H).

[0098] Next, Linear(H,H), ReLU, and Linear(H,H) are applied to the output (N, k, H) resulting from "Que-Key+POS_ENC". In this process, two Linear layers are applied, but the number of layers applied is an adjustable parameter; as long as the number of input elements in the first Linear layer is H and the number of output elements in the last Linear layer is H, it is possible to increase the number of layers or change the number of input / output elements in the intermediate layer from H.

[0099] Next, for each element of (N, k, H), which is the output of the last Linear layer,

[0100]

number

[0101] A division by the value of is performed. Then, a Softmax layer is applied to the kth element of the division output. i and,p i k points p in the neighborhood of ij The feature vector v has H elements. ij =(v ij1 , v ij2 ,…, v ijh , …, v ijH ) exists, then v ijh The value v´ after the Softmax layer is applied. ijh is calculated using the following formula (2):

[0102]

number

[0103] For the sake of explanation, we define the third-order tensor (N, k, H) after a Softmax layer is applied to all elements of k as ATTENTION.

[0104] Next, the sum of Value and POS_ENC is calculated in the PTF layer shown in Figure 9. Since Value and POS_ENC are tensors with the same number of elements (N, k, H), this process is a simple addition of all element pairs.

[0105] Next, the result of "Value + POS_ENC" is multiplied by ATTENSION. Since the result of "Value + POS_ENC" and ATTENSION are tensors with the same number of elements (N, k, H), this process is a simple multiplication of all element pairs.

[0106] Here, in (N, k, H), which is the result of accumulating "Value + POS_ENC" and ATTENTION, a certain point p on the N point i and,p i k points p in the neighborhood of ijLet v´ be the feature vector with H elements. ij When we define i The sum of the features v´´ i of

[0107]

number

[0108] Then, in the PTF layer, all p i v´´ i The pair is calculated and output as a second-order tensor in (N, H).

[0109] Next, Linear(H, D) is applied to (N, H), the output of the previous layer. In this process, one Linear layer is applied, but the number of layers applied is an adjustable hyperparameter; if the number of input elements of the first Linear layer is H and the number of output elements of the last Linear layer is D, it is possible to increase the number of layers applied or change the number of input / output elements of the intermediate layer from H by inserting a ReLU between the first and second Linear layers.

[0110] Next, for the output (N, D) of the final Linear layer, the sum of the elements of (N, D) from input A (N, 3 + D) with the position information removed is calculated.

[0111] Finally, the output of the previous layer (N, D) is combined with input A (N, 3 + D), which contains only positional information (N, 3), and the final output is (N, 3 + D).

[0112] (Learning method for neural network algorithms (first and second examples)) Next, the learning of the above neural network algorithm, that is, the learning of the neural network parameters, will be explained. To train the first and second examples of the neural network algorithm above, three elements are required: setting the loss function, setting the optimization algorithm, and generating training data. In this embodiment, as a learning method, a method for generating loss functions and training data for first and second examples of a neural network algorithm will be described. Also, a setting method for the loss function using a generative adversarial network (GAN) will be described.

[0113] The optimization algorithm for the above learning may be any general optimization algorithm for neural networks, such as SGD (Stochastic Gradient Descent) or Adam (Adaptive Moment Estimation).

[0114] (Loss function of neural networks) The first and second examples of the neural network algorithm above use colored point cloud data N(x i , y i , z i , r i , g i , b i ) and colorless point cloud data N´(x´ i´ , y´ i´ , z´ i´ ) is input, and colored point cloud data N'(x' i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´ ) is an algorithm that outputs

[0115] At this time, the correct label (label) N'(x' i´ , y´ i´ , z´ i´ , R´ i´ , G´ i´ , B´ i´), the loss function in the first and second examples of the neural network algorithm is defined as the following equation (3):

[0116]

number

[0117] (Discrimination network of generative adversarial networks) In the first and second examples of the neural network algorithm, learning can be performed using the loss function shown in the above formula (3) and the training data described below. In this embodiment, however, in order to improve the learning accuracy, a learning method using a generative adversarial network (GAN) will be described.

[0118] To realize GAN, a classification network and a loss function that includes the classification network are required, but we will first discuss the classification network.

[0119] FIG. 10 is a diagram illustrating a first example of an identification network. The classification network uses the colored point cloud data N'(x' i´ , y´ i´ , z´ i´ , R´ i´ , G´ i´ , B´ i´ ) or the output result N´(x´ in the first and second examples of the neural network algorithm i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´ ) is a scalar value between 0 and 1, and the input data is N'(x' i´ , y´ i´ , z´ i´ , R´ i´ , G´ i´ , B´ i´ ), output "1" and input data is N'(x' i´ , y´ i´, z´ i´ , r´ i´ , g´ i´ , b´ i´ ) is a network that outputs "0".

[0120] As shown in Figure 10, the classification network is realized by combining the MLP layer, SA layer, and Maxpool layer. The MLP layer is shown in Figure 2. A and MLP4 shown in FIG. 10, and the SA layer is realized with the configuration shown in FIG. 3. In the example shown in FIG. 10, one SA layer compresses the number of points to one-fourth, but the compression rate is a hyperparameter adjustment factor and may be set to one-half or one-eighth. Also, in the example shown in FIG. 10, two SA layers are applied, but the number of times they are applied is a hyperparameter adjustment factor and may be increased to three or four times, etc.

[0121] In the example shown in Figure 10, the feature values ​​of each point with its position information removed before and after the SA layer are doubled from 32 to 64 and from 64 to 128. However, determining the number of features when passing through each layer is also one of the hyperparameter adjustment factors, and can be adjusted by changing the numerical value.

[0122] The Maxpool layer is applied to the output of the last SA layer. sa In the example shown in Figure 10, the number of feature elements other than the location information is D sa (In the example shown in Figure 10, 128) is entered, N sa Each point p on i (i=1, 2, 3, …, N sa ) corresponding feature vector

[0123]

number

[0124] When there exists a vector

[0125]

number

[0126] In maxpool, the function max returns the largest element among the input elements.

[0127]

number

[0128] is defined as:

[0129] In addition, the Maxpool layer can also apply average global pooling, in which case the function average returns the average of the input elements.

[0130]

number

[0131] is defined as: In the Maxpool layer, the number of elements D is the result of applying maxpool or average global pooling. sa is returned as output.

[0132] Fig. 11 is a diagram showing an example of an implementation of an MLP layer. In the MLP4 shown in Fig. 11, Linear(D, D'), ReLU, Linear(D', D''), ReLU, and Linear(D'', 1) are applied sequentially to the first-order tensor output of the Maxpool layer. D is the number of elements of the first-order tensor that is the output of the Maxpool layer, and in the example shown in FIG. 10, D=128. Furthermore, (D', D'') are hyperparameters that are adjustment factors, and in the example shown in FIG. 10, D'=64 and D''=32.

[0133] In the example shown in FIG. 11, three linear layers are applied, but the number of linear layers is one of the hyperparameters that serve as adjustment factors, and if the linear output of the final layer is "1", the number of linear layers may be reduced to two, or may be increased to three or five layers by inserting an activation function such as ReLU.

[0134] Finally, the Sigmoid function, a common activation function in neural networks, is applied to the output of the Linear layer, which is a first-order tensor with one element, to obtain the final output, a first-order tensor with one element consisting of scalar values ​​between 0 and 1.

[0135] FIG. 12 is a diagram illustrating a second example of an identification network. The example shown in Figure 12 is an example in which an additional PTF layer is inserted to improve the accuracy of the classification network shown in Figure 10. to The processing and adjustment parameters of each layer of the illustrated network are the same as those in the example shown in Fig. 10, but differ from the example shown in Fig. 10 in that a PTF layer is sandwiched after the MLP layer and SA layer except for the final layer. This PTF layer can have a configuration similar to that shown in Figs. 7 to 9, which were shown in the second example of the neural network algorithm.

[0136] (Loss functions including generative adversarial networks) The classification network is fed the correct labeled point cloud N'(x' i´ , y´ i´ , z´ i´ , R´ i´ , G´ i´ , B´ i´ ) is input, the value is D(R), and the classification network is input with the colored point group N'(x' i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´When the value of D(F) is input, the loss function of the classification network is defined by the following equation (4):

[0137] Loss D =-log(D(R))-log(1-D(F)) …Equation (4)

[0138] Furthermore, when learning by an adversarial network is applied, the loss function in the first and second examples of the neural network algorithm is defined by the following equation (5).

[0139] Loss main_gan =λLoss main +log(D(R))-log(1-D(F)) …Equation (5)

[0140] In the above formula (5), λ is a hyperparameter that determines whether to prioritize the difference between the colored point cloud generated by the neural network algorithm, i.e., the colored point cloud and the correct data, or whether the neural network was able to deceive the classification network. λ is set between 0 and 1.0. For example, λ = 0.5.

[0141] (Generating training data) In this embodiment, the neural network can estimate color information that cannot be complemented by conventional neural networks that complement point cloud shapes. Conventional neural networks that complement point cloud shapes include a first type of neural network, as disclosed in the above-mentioned Non-Patent Document 1, which takes A as input, outputs B', which is the result of estimating B when the missing point cloud is A and the missing part is B, and a second type of neural network, as disclosed in the above-mentioned Non-Patent Document 2, which takes A as input, and outputs A'+B', which is the result of estimating the whole of A+B.

[0142] Furthermore, the neural network in this embodiment is a neural network that takes A+B' or A'+B' as input information and outputs color information of B' or A'+B', and is capable of complementing the point cloud shapes of both of the above patterns.

[0143] Next, a method for generating training data that can be applied to the first and second examples of the neural network algorithm and that corresponds to each of the first and second patterns of neural networks will be described.

[0144] FIG. 13 is a diagram illustrating a first example of a process for generating training data for a neural network. First, define the original (number of points O) as a colored 3D point cloud with a complete shape for learning. Next, to align the number of points in the input point cloud, we set the number of points in the above original to a fixed number N pre Down sampling is performed so that In this Down Sampling, the number of points from point O to any N pre The points may be sampled randomly, or farthest point sampling, which is a widely known common method, may be used.

[0145] Next, the above score N pre is divided into two parts, one of which has N points pre The method of division differs depending on the shape to be complemented. For example, one point in the down-sampled point cloud is randomly selected, and the N neighboring points of this point are pre One example is to use the point as "Data2".

[0146] Finally, the result of removing the color information from "Data2" is defined as "Data3".

[0147] When training the neural network according to this embodiment, "Data1" and "Data3" are used as inputs to the neural network, and an estimation result "Data2'" is obtained. Furthermore, "Data2" is used as the correct label, and the estimation result "Data2'" by the neural network of this embodiment is calculated using the loss function Loss main is substituted into the equation and the calculation is performed. Furthermore, when training a generative adversarial network, we use "Data1+Data2" as the correct labeled point cloud N'(x' i´ , y´ i´ , z´ i´ , R´ i´ , G´ i´ , B´ i´ ), and “Data1+Data2′” is the colored point group N′(x′) estimated by the first or second example of the neural network algorithm according to this embodiment. i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´ i´ ) as Loss D or Loss main_gan The calculation is performed.

[0148] FIG. 14 is a diagram showing a second example of the process of generating training data for a neural network. In this second example, as in the first example, we first define a shape-complete colored 3D point cloud for learning as Original (number of points: O). Next, in order to align the number of points in the input point cloud, downsampling is performed so that two patterns of point cloud data "Data1" and "Data2" with N' points are created from the original. In this down sampling, any N' points may be randomly sampled from point O, or the above-mentioned farthest point sampling may be used. However, although overlapping point groups may exist in the generated "Data1" and "Data2", down sampling is performed so that they do not become exactly the same point groups.

[0149] In addition, to avoid creating identical point clouds, a small amount of random noise may be added to the position information of each point in "Data2." "Data1" may also be created by combining down sampling and partial division. For example, first, random sampling is performed from the Original to obtain "N'+A" points, then a specific point is randomly selected, and point A near this selected point is divided into parts and changed to point N'. Next, "Data2" is generated as "Data3" with only the color information removed.

[0150] When training the neural network according to this embodiment, "Data1" and "Data3" are input to the neural network so that an estimation result "Data2'" is obtained. In addition, "Data2" is used as the correct label, and the estimation result "Data2'" is calculated using the loss function Loss main is substituted into the equation and the calculation is performed.

[0151] Furthermore, when training the generative adversarial network, we use “Data2” as the point cloud N´(x´ i´ , y´ i´ , z´ i´ , R´ i´ , G´ i´ , B´ i´ ), and “Data2´” is the colored point cloud N´(x´ i´ , y´ i´ , z´ i´ , r´ i´ , g´ i´ , b´i´ ) as Loss D or Loss main_gan is calculated.

[0152] (Embodiment) Each embodiment will be described below. In the first embodiment, a neural network based on the configuration shown in FIG. 1 is trained using the loss function shown in the above formula (3) and training data generated by the method shown in FIG. 14, and a configuration for utilizing this is shown.

[0153] In the second embodiment, a neural network based on the configuration shown in FIG. 6 is trained using the classification network shown in FIG. 12, the loss function shown in equation (4) or (5), and training data generated by the method shown in FIG. 13, and a configuration is shown in which this is utilized.

[0154] (First embodiment) Next, the first embodiment will be described. In this embodiment, an example of a DNN without a PTF layer, without a GAN, and compatible with the data type disclosed in Non-Patent Document 2 will be described.

[0155] FIG. 15 is a block diagram showing an example of the overall configuration of the image processing system according to the first embodiment. The image processing system according to this first embodiment is composed of a computing terminal 100, which is an image processing device including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and memory, a recording device 200 capable of storing data, and an external input / output device 300 capable of sending and receiving data to and from the computing terminal 100.

[0156] The computing terminal 100 also includes a learning operation unit 101, a color estimation DNN processing unit 102, and an input operation unit 104 as processing programs, and further includes a color estimation DNN parameter DB (database) 103 as a variable storage area.

[0157] The recording device 200 also has a storage device such as a non-volatile memory, and this storage device has a training point cloud DB 201.

[0158] FIG. 16 is a diagram illustrating an example of a color estimation DNN used in the first embodiment. The color estimation DNN used in the first embodiment has the configuration shown in FIG. 16, which is based on the configuration shown in FIG. 1. The DNN shown in FIG. 16 receives a colored input point cloud of 2048 points and a colorless input point cloud of 2048 points, and estimates the colored input point cloud of 2048 points. the law of nature Output the output point cloud.

[0159] If we focus only on the position information, the colored input point cloud and the colorless input point cloud are not equivalent, while the colorless input point cloud and the colored output point cloud are equivalent point cloud data. Here, "equivalent" means that there is a pair of points with exactly the same position information between the two point clouds for every point.

[0160] The MLP layers of the neural network shown in Figure 16, i.e., MLP1, MLP2, and MLP3, are shown in Figure 2. A, Figure 2B, Figure 2C The SA layer uses the same configuration as shown in Figure 3, and the FP layer uses the same configuration as shown in Figure 4. In MLP1, D = 32, in MLP2, D = 128 and D' = 256, and in MLP3, D = 32 and D' = 64. In the SA layer, the variables N, Nds, D, and Dds excluding k vary depending on the input of the SA layer in the neural network shown in Fig. 16, where N = 2048 or 512, Nds = 512 or 128, D = 32 or 64, Dds = 64 or 128, and k = 16. In addition, the down sampling of the SA layer uses the above-mentioned farthest point sampling. In the FP layer, variables Nds, Nup, Dds, and Dup are determined by input A and input B, and therefore vary depending on the position of the FP layer shown in FIG. 16, with variable k=16.

[0161] FIG. 17 is a diagram showing an example of data storage in the learning point cloud DB. In the training point cloud DB 201 of the recording device 200, point cloud data is stored as files, and each point cloud data file is composed of 2560 or more points, each of which has six elements: position elements (X, Y, Z) and color elements (R, G, B). Each element is normalized to a range of 0 to 1 using the Min-Max algorithm. The number of point cloud data files is a multiple of 20.

[0162] FIG. 18 is a diagram illustrating an example of data storage in the color estimation DNN parameter DB. The color estimation DNN parameter DB 103 holds the weight parameters of the color estimation DNN shown in Figure 16 that are used in the color estimation DNN processing unit 102, and each weight parameter is complemented by a pair of Symbol and Value.

[0163] FIG. 19 is a flowchart illustrating an example of a processing operation by the learning operation unit of the computing terminal according to the first embodiment. First, in the process of S101-1, the learning operation unit 101 of the computing terminal 100 sequentially acquires P pieces of point cloud file data from the learning point cloud DB 201 of the recording device 200. P is the number of learnings performed at one time in the DNN mini-batch learning method, and in this embodiment, P=20.

[0164] Next, in the process of S101-2, the learning operation unit 101 generates a colored input point cloud, a colorless input point cloud, and a colored correct point cloud for each of the 20 point cloud data read in S101-1 based on the method shown in Figure 14.

[0165] The colored input point cloud is "Data1" shown in Figure 14, and in this embodiment, the learning operation unit 101 selects 2560 points by random sampling from the loaded point cloud data, and then randomly selects one point from the 2560 points, and the 512 points closest to this one point are removed, leaving 2048 points for use.

[0166] Next, the colored correct point cloud is "Data2" shown in Fig. 14, and is generated by the learning operation unit 101 selecting 2048 points by random sampling from the loaded point cloud data. Finally, the colorless input point cloud is generated by the learning operation unit 101 removing the color information (R, G, B) of the colored correct point cloud.

[0167] Next, in the process of S101-3, the learning operation unit 101 sends the 20 colored input point clouds and the colorless input point clouds generated in S101-2 to the color estimation DNN processing unit 102.

[0168] Next, in the process of S101-4, the learning operation unit 101 receives the 20 color output point clouds that are output from the color estimation DNN processing unit 102.

[0169] Next, in the process of S101-5, the learning operation unit 101 calculates a loss function from the 20 colored correct point clouds generated in S101-2 and the colored output point cloud received in S101-4. The colored correct point cloud is Y i and the colored output point cloud is defined as X i If we define i as (i = 1, 2, 3, ..., 20), the learning operation unit 101 can calculate the loss function value for all 20 data pairs using the above equation (3) as the following equation (6).

[0170]

number

[0171] Next, in the process of S101-6, the learning operation unit 101 uses the ADAM described above based on the result of the loss function calculated by the above formula (6) in S101-5 to update the color estimation DNN parameters stored in the color estimation DNN parameter DB 103. The learning rate lr, which is a parameter used in ADAM, is set to 0.01.

[0172] Finally, in S101-7, the learning operation unit 101 checks whether the 20 files acquired in the color estimation DNN parameter S101-1 have been acquired for all files in the learning point cloud DB 201. If all files have been acquired, the learning operation unit 101 counts the number of times learning has been performed. If the count in S101-7 is equal to or greater than the designated number, the learning operation unit 101 ends the process. In this embodiment, the designated number is 100, and if the count is equal to or greater than 100, the learning operation unit 101 ends the process. If the count is less than 100 or if all files have not been acquired, the learning operation unit 101 returns the process to S101-1.

[0173] FIG. 20 is a flowchart illustrating an example of processing operations in the color estimation DNN processing unit of the computing terminal according to the first embodiment. In S102-1, the color estimation DNN processing unit 102 of the computing terminal 100 receives a colored input point cloud and a colorless input point cloud, each consisting of 2048 points, from the learning operation unit 101 or the input operation unit 104.

[0174] Next, in S102-2, the color estimation DNN processing unit 102 of the computing terminal 100 refers to all weight parameters of the color estimation DNN shown in FIG. 16 from the color estimation DNN parameter DB 103.

[0175] Next, in S102-3, the color estimation DNN processing unit 102 substitutes the colored input point cloud and the colorless input point cloud received in S102-1 into the color estimation DNN shown in FIG. In addition, the color estimation DNN processing unit 102 assigns the weight parameters referenced in S102-2 to the color estimation DNN. Then, the color estimation DNN processing unit 102 sequentially performs processing based on the input data and outputs a color output point cloud consisting of 2048 points.

[0176] Finally, in S102-4, the color estimation DNN processing unit 102 transmits the colored output point cloud calculated in S102-3 to the learning operation unit 101 if the data received in S102-1 is from the learning operation unit 101, or to the input operation unit 104 if the data received in S102-1 is from the input operation unit 104.

[0177] FIG. 21 is a flowchart illustrating an example of a processing operation of the input operation unit of the computing terminal according to the first embodiment. In S104-1, the input operation unit 104 of the computing terminal 100 receives a colored input point cloud and a colorless input point cloud from the external input / output device 300. At this time, the colored input point cloud is a point cloud consisting of 2048 points and having position information and color information, and the colorless input point cloud is a point cloud also consisting of 2048 points and having only position information. The external input / output device 300 includes a separate program and a file I / O interface that can input and output data to and from the computing terminal 100.

[0178] Next, in S104-2, the input operation unit 104 transmits the colored input point cloud and the colorless input point cloud received in S104-1 to the color estimation DNN processing unit 102.

[0179] Next, in S104-3, the input operation unit 104 receives a colored output point cloud as a processing result from the color estimation DNN processing unit 102. Due to the structure of the color estimation DNN processing unit 102, this colored output point cloud is composed of 2048 points that have the same position information as the colorless input point cloud, with color information added to them.

[0180] Finally, in S104-4, the input operation unit 104 transmits the colored output point cloud received in S104-3 to the external input / output device 300, and the process ends.

[0181] (Second embodiment) Next, a second embodiment will be described. In this embodiment, an example of a DNN with a PTF layer, a GAN, and compatible with the data type disclosed in Non-Patent Document 1 will be described.

[0182] FIG. 22 is a block diagram showing an example of the overall configuration of an image processing system according to the second embodiment. The image processing system according to the second embodiment is configured from a computing terminal 100, a recording device 200, and an external input / output device 300, similar to the first embodiment. Also, inside the computing terminal 100, as in the first embodiment, there are a learning operation unit 101, a color estimation DNN processing unit 102, and an input operation unit 104 as processing programs, and compared to the first embodiment, a classification DNN processing unit 105 is further added.

[0183] As a variable storage area, the computing terminal 100 has a color estimation DNN parameter DB 103, similar to the first embodiment, and the computing terminal 100 further has a classification DNN parameter DB 106, unlike the first embodiment. Also, the recording device 200 has a training point cloud DB 201, similar to the first embodiment.

[0184] FIG. 23 is a diagram illustrating an example of a color estimation DNN used in the second embodiment. The color estimation DNN used in the second embodiment has the configuration shown in Fig. 23, which is based on the configuration shown in Fig. 6. The color estimation DNN shown in Fig. 23 receives a colored input point cloud of 2048 points and a colorless input point cloud of 512 points, and outputs a colored output point cloud of 512 points. First Embodiment and Similarly, in the example shown in FIG. 23, the colored input point cloud and the colorless input point cloud are not equivalent, while the colorless input point cloud and the colored output point cloud are equivalent point cloud data.

[0185] In the example shown in FIG. 23, the MLP layers of the neural network, i.e., MLP1, MLP2, and MLP3, are A, Figure 2B, Figure 2CThe SA layer has the same configuration as shown in Fig. 3, and the FP layer has the same configuration as shown in Fig. 5. In MLP1, D=32, in MLP2, D=128 and D'=256, and in MLP3, D=32 and D'=64.

[0186] In the SA layer, the variables N, Nds, D, and Dds excluding k vary depending on the input of the SA layer in the neural network shown in Fig. 23, where N = 2048, 512, or 128, Nds = 512, 128, or 32, D = 32 or 64, Dds = 64 or 128, and k = 16. In addition, the down sampling of the SA layer uses the above-mentioned farthest point sampling.

[0187] Similarly, in the FP layer, variables Nds, Nup, Dds, and Dup are determined by input A and input B, and therefore vary depending on the position of the FP layer shown in FIG. 23, with variable k=16.

[0188] The PTF layers used are the same as those divided and described in Figures 7 to 9. N and D of the PTF layers are determined by the input, and therefore vary depending on the position of the PTF layer shown in Figure 23. In the second embodiment, the variables of the PTF layers are k = 16 and H = 512.

[0189] FIG. 24 is a diagram illustrating an example of a classification DNN used in the second embodiment. The classification DNN used in the second embodiment has the configuration shown in Fig. 24, which is based on the configuration shown in Fig. 12. The classification DNN shown in Fig. 24 receives a colored point cloud of 2560 points as input and outputs an estimation result between 0 and 1.

[0190] In the example shown in FIG. 24, MLP1 in the MLP layer is A 11. In MLP1, D=32, and in MLP4, D=128, D'=64, and D''=32.

[0191] The SA layer uses the same configuration as that shown in Fig. 3. In this SA layer, the variables N, Nds, D, and Dds, excluding k, vary depending on the input to the SA layer shown in Fig. 12, with N = 2560 or 640, Nds = 640 or 160, D = 32 or 64, Dds = 64 or 128, and k = 16. In addition, the down sampling in the SA layer uses the above-mentioned farthest point sampling.

[0192] The PTF layer uses a configuration similar to that shown separately in Figures 7 to 9, and N and D of the PTF layer are determined by the input, so they vary depending on the position of the PTF layer shown in Figure 24. In the second embodiment, the variables of the PTF layer are k = 16 and H = 512. The Maxpool layer uses the above-mentioned average global pooling.

[0193] The configuration for holding data in the training point cloud DB 201 in the second embodiment is similar to the configuration shown in FIG. 17 in the first embodiment. Furthermore, the configuration for storing data in the color estimation DNN parameter DB 103 in the second embodiment is similar to the configuration shown in FIG. 18 in the first embodiment. This DNN parameter DB 103 holds the weight parameters of the color estimation DNN shown in Figure 23, which are used in the color estimation DNN processing unit 102, and each weight parameter is complemented by a pair of Symbol and Value.

[0194] Fig. 25 is a diagram showing an example of data storage of classification DNN parameters. The classification DNN parameter DB 106 stores weight parameters of the classification DNN shown in Fig. 24 that are used in the classification DNN processing unit 105, and each weight parameter is complemented by a pair of Symbol and Value.

[0195] FIG. 26 is a flowchart illustrating an example of a processing operation by the learning operation unit of the computing terminal according to the second embodiment. First, in the process of S101-1, the learning operation unit 101 of the computing terminal 100 acquires P pieces of point cloud file data from the learning point cloud DB 201 of the recording device 200. P is the number of learnings performed at one time in the DNN mini-batch learning method, and in this embodiment, P=20.

[0196] Next, in processing S101-2, the learning operation unit 101 generates a colored input point cloud, a colorless input point cloud, and a colored correct point cloud for each of the 20 point cloud data read in S101-1 based on the method shown in Figure 13.

[0197] The colored input point cloud is "Data1" shown in FIG. 13, and in this embodiment, 2560 points are selected by random sampling from the read point cloud data by the learning operation unit 101 (N pre = 2560), and then one point is randomly selected from these 2560 points, and the 512 points closest to this one are removed, leaving 2048 points (N pre -N´=2048).

[0198] Next, the colored correct point cloud is "Data2" shown in Fig. 13, and the 512 points removed in preprocessing are used (N' = 512). Finally, the colorless input point cloud is "Data3" shown in Fig. 13, and is generated by the learning operation unit 101 by removing the color information (R, G, B) of the colored correct point cloud.

[0199] Next, in the process of S101-3, the learning operation unit 101 sends the 20 colored input point clouds and the 20 colorless input point clouds generated in S101-2 to the color estimation DNN processing unit 102.

[0200] Next, in the process of S101-4, the learning operation unit 101 receives the 20 colored output point clouds that are output from the color estimation DNN processing unit 102.

[0201] Next, in the process of S101-5a, the learning operation unit 101 sends the 20 colored input point clouds, the colored correct point clouds, and the colored output data to the classification DNN processing unit 105.

[0202] Next, in the process of S101-5b, the classification DNN processing unit 105 receives the estimation results for the 20 correct point clouds and the estimation results for the 20 fake point clouds, each represented by a value of 0 to 1.

[0203] Next, in the processing of S101-5c, the learning operation unit 101 calculates the loss functions of the color estimation DNN and the classification DNN from the 20 colored correct point clouds generated in S101-2, the 20 colored output point clouds received in S101-4, the estimation results for the 20 correct point clouds and the estimation results for the 20 fake point clouds received in S101-5b.

[0204] For color estimation DNN, the colored correct point cloud is Y i and the colored output point cloud is defined as X i and the estimation result for the correct point group is defined as True. i (i = 1, 2, 3, …, 20), and the estimation results for the fake point cloud are called Fake i If we define i as (i = 1, 2, 3, ..., 20), the learning operation unit 101 can calculate the loss function values ​​for all 20 data pairs using the above equation (5) according to the following equation (7). The adjustment parameter variable λ in this equation (7) is defined as 0.5 in this embodiment.

[0205]

number

[0206] In addition, for the classification DNN, the estimation results for the correct point cloud are i (i = 1, 2, 3, …, 20), and the estimation results for the fake point cloud are called Fake i If we define i as (i = 1, 2, 3, ..., 20), the learning operation unit 101 can calculate the loss function values ​​for all 20 data pairs using the above equation (4) according to the following equation (8).

[0207]

number

[0208] Next, in the process of S101-6a, the learning operation unit 101 uses the ADAM described above based on the result of the loss function calculated by the above formula (7) in S101-5c to update the color estimation DNN parameters stored in the color estimation DNN parameter DB 103. The learning rate lr, which is a parameter used in ADAM, is set to 0.01.

[0209] Next, in the process of S101-6b, the learning operation unit 101 uses ADAM based on the result of the loss function calculated by the above formula (8) in S101-5c to update the classification DNN parameters stored in the classification DNN parameter DB 106. The learning rate lr, which is a parameter used in ADAM, is set to 0.01.

[0210] Finally, in S101-7, the learning operation unit 101 checks whether or not the 20 files acquired in S101-1 have been acquired for all files in the learning point cloud DB 201. When all files have been acquired, the learning operation unit 101 counts the number of learning times. If the number of times of this count in S101-7 is equal to or greater than the designated number, the learning operation unit 101 ends the process.

[0211] In this embodiment, the specified number of times is 100, and if the count is 100 or more, the learning operation unit 101 terminates the processing, and if the count is less than 100 or if all files have not been acquired, the learning operation unit 101 returns the processing to S101-1.

[0212] The procedure of the processing operation in the color estimation DNN processing unit 102 according to the second embodiment is similar to the example shown in FIG. 20 according to the first embodiment. However, there are two differences. The first difference is that the number of points in the colorless input point cloud input in S102-1 and the colored output point cloud calculated in S102-3 is 2048 in the first embodiment, but 512 in the second embodiment. The second difference is that the color estimation DNN processing unit 102 in the first embodiment uses the color estimation DNN shown in Fig. 16, but the color estimation DNN processing unit 102 in the second embodiment uses the color estimation DNN shown in Fig. 23.

[0213] The procedure of the processing operation of the input operation unit 104 according to the second embodiment is the same as the example shown in Fig. 21 according to the first embodiment, except that the number of points in the colorless input point cloud input in S104-1 and the colored output point cloud received in S104-3 is 2048 in the first embodiment, whereas it is 512 in the second embodiment.

[0214] FIG. 27 is a flowchart illustrating an example of a procedure of processing operations in the classification DNN processing unit according to the second embodiment. In S105-1, the discrimination DNN processing unit 105 receives the colored input point cloud, the colored correct point cloud, and the colored output point cloud from the learning operation unit 101. In S105-2, the classification DNN processing unit 105 refers to all weight parameters of the classification DNN shown in FIG. 24 from the classification DNN parameter DB 106.

[0215] In S105-3, the discrimination DNN processing unit 105 combines the colored input point cloud (2048 points, each point has a feature value of 6) received in S105-1 with the colored correct point cloud (512 points, each point has a feature value of 6) to create a correct point cloud (2560 points = 2048 + 512, each point has a feature value of 6). Similarly, the discrimination DNN processing unit 105 combines the colored input point cloud (2048 points, 6 features for each point) and the colored output point cloud (512 points, 6 features for each point) to create a fake point cloud (2560 points = 2048 + 512, 6 features for each point).

[0216] In S105-4, the classification DNN processing unit 105 first assigns the weight parameters referenced in S105-2 to the classification DNN shown in Fig. 24. Then, the classification DNN processing unit 105 inputs the correct point cloud created in S105-3 and calculates an estimation result of the correct point cloud indicated by a numerical value between 0 and 1. Next, the discrimination DNN processing unit 105 inputs the fake point cloud created in S105-3 and calculates the estimation result of the fake point cloud indicated by a numerical value between 0 and 1.

[0217] Finally, in S105-5, the classification DNN processing unit 105 transmits both the estimation result of the ground truth point cloud calculated in S105-4 and the estimation result of the fake point cloud to the learning operation unit 101. After completing the transmission, the classification DNN processing unit 105 terminates the processing.

[0218] FIG. 28 is a block diagram showing an example of the hardware configuration of a computing terminal of an image processing system according to an embodiment of the present invention. 28, the computing terminal 100 of the image processing system according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU. A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115. The same is true for the recording device 200 and the external input / output device 300 of the image processing system.

[0219] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

[0220] The input / output interface 113 is connected to an input device 500 and an output device 600 that are attached to the computing terminal 100 and used by users, etc. The input / output interface 113 takes in operation data input by a user or the like through an input device 500 such as a keyboard, a touch panel, a touchpad, or a mouse, and outputs output data to an output device 600 including a display device using a liquid crystal or an organic electroluminescence (EL) display, etc. Note that the input device 500 and the output device 600 may be devices built into the computing terminal 100, or may be input devices and output devices of other information terminals that can communicate with the computing terminal 100 via the network NW.

[0221] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a non-volatile memory such as a ROM, and stores programs necessary to execute various control processes, etc., according to one embodiment.

[0222] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM, and is used to store various data acquired and created during various processes.

[0223] A computing terminal 100 according to one embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 1 as software-based processing functional units.

[0224] Each information storage unit and each DB used as a working memory by each unit of the computing terminal 100 can be configured by using the data memory 112 shown in Fig. 28. However, these configured storage areas are not essential components within the computing terminal 100, and may be areas provided in a storage device such as an external storage medium such as a USB (Universal Serial Bus) memory, or a database server located in the cloud.

[0225] The processing function units in each part of the computing terminal 100 can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0226] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

[0227] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention. [Explanation of symbols]

[0228] 100...Computing terminal 101...Learning operation section 102...Color estimation DNN processing unit 103...Color estimation DNN parameter DB 104...input operation section 105...Discrimination DNN processing unit 106...Discrimination DNN parameter DB 200...Recording device 201...Learning point cloud database 300...External input / output device

Claims

1. an acquisition unit that acquires first point cloud data that does not have color information and second point cloud data that is located near the first point cloud data and has color information; an estimation processing unit that inputs the first and second point cloud data acquired by the acquisition unit into a neural network and outputs point cloud data in which color information of the first point cloud data is estimated; Equipped with The neural network adding a vector of the number of elements to a tensor representing the first point cloud data and compressing the number of point cloud data in the result of the addition, adding the vector of the number of elements to a tensor representing the second point cloud data and compressing the number of point cloud data in the result of the addition, combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, with the result of combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to a first number of tensors representing the first point cloud data with the result of adding a second number of tensors representing the first point cloud data that is greater than the first number, and outputting point cloud data with color information estimated for the first point cloud data. Image processing device.

2. an acquisition unit that acquires first point cloud data that does not have color information and second point cloud data that is located near the first point cloud data and has color information; an estimation processing unit that inputs the first and second point cloud data acquired by the acquisition unit into a neural network and outputs point cloud data in which color information of the first point cloud data is estimated; Equipped with The neural network adding a vector of the number of elements to a tensor representing the first point cloud data and compressing the number of point cloud data in the result of the addition, adding the vector of the number of elements to a tensor representing the second point cloud data and compressing the number of point cloud data in the result of the addition, combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, with the result of combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to a first number of tensors representing the first point cloud data with the result of adding a second number of tensors representing the first point cloud data that is greater than the first number, and outputting point cloud data with color information estimated for the first point cloud data a parameter update unit that acquires point cloud data having color information, generates ground truth data that is point cloud data having color information, point cloud data without color information, and point cloud data from which a portion of the point cloud data having color information has been removed based on the point cloud data, acquires the portion of data from which color information has been estimated by inputting the generated point cloud data without color information and the point cloud data from which a portion of the point cloud data having color information has been removed into the neural network, calculates a loss function for the neural network based on the acquired data and the ground truth data, and updates parameters of the neural network based on the calculated loss function; Image processing device.

3. The parameter update unit a second neural network that outputs a first scalar value when the input data is the correct answer data and a second scalar value when the input data is the part of the data whose color information is estimated, and calculates a loss function for the second neural network based on the output value; updating parameters of the second neural network based on the calculated loss function; The image processing device according to claim 2 .

4. A method performed by an image processing device, comprising: acquiring, by an acquisition unit of the image processing device, first point cloud data having no color information and second point cloud data located in the vicinity of the first point cloud data and having color information; an estimation processing unit of the image processing device inputs the first and second point cloud data acquired by the acquisition unit into a neural network, thereby outputting point cloud data in which color information is estimated for the first point cloud data; The neural network adding a vector of the number of elements to a tensor representing the first point cloud data and compressing the number of point cloud data in the result of the addition, adding the vector of the number of elements to a tensor representing the second point cloud data and compressing the number of point cloud data in the result of the addition, combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, with the result of combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to a first number of tensors representing the first point cloud data with the result of adding a second number of tensors representing the first point cloud data that is greater than the first number, and outputting point cloud data with color information estimated for the first point cloud data. Image processing methods.

5. A method performed by an image processing device, comprising: acquiring, by an acquisition unit of the image processing device, first point cloud data having no color information and second point cloud data located in the vicinity of the first point cloud data and having color information; an estimation processing unit of the image processing device inputs the first and second point cloud data acquired by the acquisition unit into a neural network, thereby outputting point cloud data in which color information is estimated for the first point cloud data; The neural network adding a vector of the number of elements to a tensor representing the first point cloud data and compressing the number of point cloud data in the result of the addition, adding the vector of the number of elements to a tensor representing the second point cloud data and compressing the number of point cloud data in the result of the addition, combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the first point cloud data with the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to the tensor representing the second point cloud data, with the result of combining the result of compressing the number of point cloud data as a result of adding the vector of the number of elements to a first number of tensors representing the first point cloud data with the result of adding a second number of tensors representing the first point cloud data that is greater than the first number, and outputting point cloud data with color information estimated for the first point cloud data a parameter update unit of the image processing device acquires point cloud data having color information, generates ground truth data which is point cloud data having color information, point cloud data without color information, and point cloud data from which a portion of the point cloud data having color information has been removed based on the point cloud data, acquires the portion of data from which color information has been estimated and is output by inputting the generated point cloud data without color information and the point cloud data from which a portion of the point cloud data having color information has been removed into the neural network, calculates a loss function for the neural network based on the acquired data and the ground truth data, and updates parameters of the neural network based on the calculated loss function; Image processing methods.

6. An image processing program that causes a processor to function as each unit of the image processing device according to any one of claims 1 to 3.

Images