Information processing apparatus, information processing method and program

Abstract

To provide an information processing apparatus, a method, and a program capable of realizing a three-dimensional shape completion technique with improved spatial representation ability.SOLUTION: An information processing apparatus comprises: an acquisition unit that acquires input data for a learning phase; a three-dimensional structure data generation unit that specifies three-dimensional coordinates of each pixel included in the input data by referring to depth data of each pixel in depth data, and generates three-dimensional structure data; a sampling unit that generates post-sampling three-dimensional structure data by sampling the three-dimensional structure data; a completion unit that estimates shielded or missing portions in the post-sampling three-dimensional structure data and applies completion processing; an estimation unit that executes shape estimation processing with reference to intermediate feature quantities in the completion processing; and a first learning unit that makes the completion means learn with reference to a first loss value, which is a loss value relating to a shape obtained through the shape estimation processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a known technology that scans the real world using a sensor and reconstructs it. In addition, since the 3D data obtained by the reconstruction may contain occluded or missing areas, a method for complementing these areas is also known (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Angela Dai et. al.,SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans,2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Non-Patent Document 1 is a voxel-based 3D shape interpolation method, but has a problem in that its spatial expression capability is limited.

[0005] The present disclosure has been made in view of the above problems, and an exemplary purpose thereof is to provide a three-dimensional shape completion method with improved spatial representation capabilities. [Means for solving the problem]

[0006] An information processing device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring input data, a three-dimensional structure data generation means for generating three-dimensional structure data from the input data, a sampling means for generating sampled three-dimensional structure data by sampling the three-dimensional structure data, a completion means for applying a completion process to the sampled three-dimensional structure data, an estimation means for executing a shape estimation process with reference to intermediate features in the completion process, and a first learning means for training the completion means with reference to a first loss value, which is a loss value related to the shape obtained by the shape estimation process.

[0007] An information processing device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring input data, a three-dimensional structure data generation means for generating three-dimensional structure data from the input data, a completion means for applying a completion process to the three-dimensional structure data, and an output data generation means for generating output data from the three-dimensional structure data to which the completion process has been applied, wherein the completion means is trained through a sampling process for generating sampled three-dimensional structure data by sampling three-dimensional structure data generated from learning data, a completion process by the completion means for the sampled three-dimensional structure data, a shape estimation process that references intermediate features in the completion process, and a first learning process that trains the completion means by reference to a first loss value that is a loss value related to the shape obtained by the shape estimation process.

[0008] An information processing method according to an exemplary aspect of the present disclosure includes acquiring input data, generating three-dimensional structure data from the input data, sampling the three-dimensional structure data to generate sampled three-dimensional structure data, applying a completion process to the sampled three-dimensional structure data by a completion means, performing a shape estimation process with reference to intermediate features in the completion process, and training the completion means with reference to a first loss value, which is a loss value related to the shape obtained by the shape estimation process.

[0009] An information processing method according to an exemplary aspect of the present disclosure includes acquiring input data, generating three-dimensional structure data from the input data, applying a completion process by a completion means to the three-dimensional structure data, and generating output data from the three-dimensional structure data to which the completion process has been applied, wherein the completion means is trained by sampling three-dimensional structure data generated from learning data to generate sampled three-dimensional structure data, applying the completion process by the completion means to the sampled three-dimensional structure data, performing a shape estimation process with reference to intermediate features in the completion process, and training the completion means with reference to a first loss value, which is a loss value related to the shape obtained by the shape estimation process. [Effects of the Invention]

[0010] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that a three-dimensional shape completion method with improved spatial representation capabilities can be provided. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 4] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 5] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 6] FIG. 2 is a diagram for explaining processing in an information processing device according to the present disclosure. [Figure 7] FIG. 2 is a diagram illustrating a flow of data in an information processing device according to the present disclosure. [Figure 8] FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 9] FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 10] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 11] FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 12] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 13] FIG. 1 is a block diagram illustrating a hardware configuration of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0013] [First embodiment] A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0014] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, a three-dimensional structure data generation unit 12, a sampling unit 14, a completion unit 13, an estimation unit 15, and a first learning unit 16.

[0015] (Acquisition part 11) The acquisition unit 11 acquires input data. Here, the input data is, for example, input data for the learning phase. Specific examples of the input data acquired by the acquisition unit 11 do not limit the present exemplary embodiment, but may include, for example: RGB data where each pixel (data point) represents an RGB value Depth data, where each pixel (data point) represents a depth value, and 3D point cloud data where each data point represents a 3D coordinate The three-dimensional point cloud data may be, for example, point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, but this example does not limit the present exemplary embodiment.

[0016] (3D structure data generation unit 12) The three-dimensional structure data generation unit 12 generates three-dimensional structure data from the input data acquired by the acquisition unit 11. As an example, the three-dimensional structure data generation unit 12 generates three-dimensional structure data from the input data acquired by the acquisition unit 11, which includes at least either RGB data or depth data. As an example, the three-dimensional structure data generation unit 12 generates three-dimensional structure data from the input data acquired by the acquisition unit 11, which includes at least either RGB data or depth data. The three-dimensional coordinates of each pixel included in the RGB data are determined by referencing the depth data of each pixel in the depth data; Generate 3D structure data with the specified 3D coordinates assigned to each pixel (each data point) Here, the three-dimensional structure data may be configured to include a feature amount of each data point in addition to the three-dimensional coordinates assigned to each data point. Here, each feature amount may be configured to include at least one of an RGB value and a normal value (normal vector) of each data point. Alternatively, the three-dimensional structure data generation unit 12 may be configured to generate attribute data including the feature amount of each data point (for example, at least one of an RGB value and a normal value (normal vector)) in association with the three-dimensional structure data including the three-dimensional coordinates of each data point.

[0017] Furthermore, the three-dimensional structure data generating unit 12 may generate the three-dimensional structure data using an algorithm such as Structure from Motion (SfM) or Simultaneous Localization and Mapping (SLAM).

[0018] The three-dimensional structure data generating unit 12, with the above-described configuration, can generate the three-dimensional structure data from one or more frames (one or more data sets) included in the input data acquired by the acquiring unit 11.

[0019] Furthermore, when the acquisition unit 11 acquires input data including three-dimensional point cloud data, the three-dimensional structure data generation unit 12 may be configured to output the three-dimensional point cloud data as three-dimensional structure data as is. Alternatively, the three-dimensional point cloud data may be configured to include the above-mentioned feature amounts of each data point (for example, at least one of RGB values ​​and normal values ​​(normal vectors)) and then output the result as three-dimensional structure data. Alternatively, the attribute data including the above-mentioned feature amounts of each data point may be output together with the three-dimensional structure data including the three-dimensional coordinates of each data point.

[0020] Note that, when the acquiring unit 11 acquires input data including three-dimensional point cloud data and the three-dimensional structure data generating unit 12 outputs the three-dimensional point cloud data as three-dimensional structure data without modification, the information processing device 1 may be configured without the three-dimensional structure data generating unit 12. Such a configuration is also included in this exemplary embodiment.

[0021] (Sampling section 14) The sampling unit 14 generates sampled three-dimensional structure data by sampling the three-dimensional structure data generated by the three-dimensional structure data generating unit 12. Here, the sampling process may include, for example, A process of generating the sampled three-dimensional structure data using only some of the frames included in the input data. Alternatively, the sampling process may include: A process of generating the sampled three-dimensional structure data using only some of the data points included in at least one of the input data and the three-dimensional structure data. The sampling process performed by the sampling unit 14 may be referred to as thinning process. Therefore, the three-dimensional structure data after sampling may also be referred to as thinned three-dimensional structure data.

[0022] (Complementary part 13) The completion unit 13 applies completion processing to the sampled 3D structure data generated by the sampling unit 14. Here, the completion processing may include processing for estimating (generating) a completed area (also referred to as a completed area or a completed structure) for occluded or missing areas in the sampled 3D structure data. The completion processing is performed using a machine learning-enabled completion model. More specifically, as an example, the sampled 3D structure data is input to a neural network, which is a completion model having multiple layers. The neural network to which the sampled 3D structure data is input outputs completed 3D structure data.

[0023] Furthermore, the intermediate feature values ​​generated (calculated) in the completion process executed by the completion unit 13 (for example, feature values ​​generated (calculated) in the intermediate layer of the neural network) are referenced by the estimation unit 15, which will be described later. The specific configuration of the completion unit 13 does not limit this exemplary embodiment, but as an example, an encoder that receives the sampled three-dimensional structure data and outputs the intermediate feature values; a decoder that receives the intermediate features and outputs the completed 3D structure data; The configuration may include:

[0024] (Estimation part 15) The estimation unit 15 performs a shape estimation process by referring to the intermediate feature values ​​obtained in the completion process by the completion unit 13. As an example, the estimation unit 15 performs the shape estimation process by calculating each value of a distance function from the intermediate feature values. In other words, the estimation unit 15 expresses the result of the shape estimation process as a distance function. Here, the distance function refers to, for example, a function defined by the distance from a point in space to each point on the object. For example, a signed distance function (SDF), which is an example of a distance function, expresses the shortest distance from a point in space to the surface of the object, assigning a negative value if the point is inside the object and a positive value if the point is outside the object. Therefore, in this case, the surface (contour) of the object is represented by a region (contour) where the SDF value is 0. A surface defined by SFD=0 in this way is sometimes called an implicit surface. Note that the distance function is not limited to the SDF described above, and an unsigned distance function may be used. As an example, the UDF value is positive, and the surface (contour) of the object is represented by a region (contour) where the UDF value takes a predetermined value.

[0025] Note that the specific processing by the estimation unit 15 does not limit the present exemplary embodiment, but as an example, the estimation unit 15 may perform the following: a feature transformation process for transforming the intermediate feature values ​​in the completion process into latent variables; The shape estimation process is performed by referring to the latent variables and the three-dimensional structure data after sampling. The above configuration may be adopted.

[0026] (First Study Section 16) The first learning unit 16 causes the completion unit 13 to learn (machine learn) by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process by the estimation unit 15. Here, the first loss value is, for example, Each value of the distance function obtained by the shape estimation process by the estimation unit 15, Each value of the distance function obtained by referring to the sampled three-dimensional structure data generated by the sampling unit 14 (i.e., each value of the distance function obtained from three-dimensional structure data that has not been processed by the completion unit 13 and the estimation unit 15), Then, the first learning unit 16 causes the complementing unit 13 to learn so that the first loss value becomes smaller, for example.

[0027] (Effects of information processing device 1) As described above, in the information processing device 1, Get the input data, Generate three-dimensional structure data from the input data; generating sampled three-dimensional structure data by sampling the three-dimensional structure data; Applying a complementation process to the sampled three-dimensional structure data by a complementation unit 13; A shape estimation process is performed by referring to the intermediate feature values ​​obtained in the completion process. The completion unit 13 is trained by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process. According to the above configuration, a shape estimation process is performed with reference to intermediate feature amounts in the completion process, and the completion unit 13 is trained with reference to a first loss value that is a loss value related to the shape obtained by the shape estimation process, so that the performance of the completion process by the completion unit 13 can be suitably improved. Therefore, according to the above configuration, it is possible to provide a 3D shape completion method with improved spatial representation performance.

[0028] (Additional notes regarding information processing device 1) As described above, when the acquisition unit 11 is configured to acquire three-dimensional point cloud data, the information processing device 1 may be configured without including the three-dimensional structure data generation unit 12. In other words, the information processing device 1 an acquisition unit 11 that acquires input data including three-dimensional point cloud data; a sampling unit 14 that generates sampled three-dimensional structure data by sampling the three-dimensional point cloud data (three-dimensional structure data); a complementation unit 13 that applies complementation processing to the sampled three-dimensional structure data; an estimation unit 15 that executes a shape estimation process by referring to intermediate feature amounts in the complementation process; a first learning unit 16 that causes the completion unit 13 to learn by referring to a first loss value that is a loss value related to the shape obtained by the shape estimation process; The information processing device 1 configured in this way can also achieve the above-mentioned effects.

[0029] (Flow of information processing method S1) Next, the flow of an information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1. As shown in Fig. 2, the information processing method S1 includes a step (process) S11 of acquiring input data, a step (process) S12 of generating three-dimensional point structure data, a step (process) S14 of generating three-dimensional structure data after sampling, a step (process) S15 of performing a completion process using a completion unit (completion model), a step (process) S16 of performing a shape estimation process, and a step (process) S17 of training the completion unit (completion model).

[0030] (Step S11) In step S11, the acquisition unit 11 acquires input data. The specific processing by the acquisition unit 11 has been described above, and therefore will not be described here.

[0031] (Step S12) Subsequently, in step S12, the three-dimensional structure data generation unit 12 generates three-dimensional structure data from the input data acquired by the acquisition unit 11 in step S11. The specific processing by the three-dimensional structure data generation unit 12 has been described above, and therefore will not be described here.

[0032] (Step S14) Subsequently, in step S14, the sampling unit 14 generates sampled three-dimensional structure data by sampling the three-dimensional structure data generated by the three-dimensional structure data generation unit 12 in step S13. The specific processing by the sampling unit 14 has been described above, and therefore will not be described here.

[0033] (Step S13) Subsequently, in step S13, the complementing unit 13 applies complementing processing to the sampled three-dimensional structure data generated by the sampling unit 14 in step S14. The specific processing by the complementing unit 13 has been described above, and therefore will not be described here.

[0034] (Step S15) Subsequently, in step S15, the estimation unit 15 executes a shape estimation process by referring to the intermediate feature amount in the completion process by the completion unit 13 in step S13. The specific process by the estimation unit 15 has been described above, and therefore will not be described here.

[0035] (Step S16) Subsequently, in step S16, the first learning unit 16 causes the completion unit 13 (complementation model) to learn by referring to the first loss value, which is a loss value related to the shape obtained by the shape estimation process by the estimation unit 15 in step S15. The specific process by the first learning unit 16 has been described above, and therefore will not be described here.

[0036] (Effect of information processing method S1) As described above, in the information processing method S1, Get the input data, Generate three-dimensional structure data from the input data; generating sampled three-dimensional structure data by sampling the three-dimensional structure data; Applying a completion process to the sampled three-dimensional structure data by a completion unit 13 (complementation model); A shape estimation process is performed by referring to the intermediate feature values ​​obtained in the completion process. The completion unit 13 (complementation model) is trained by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process. According to the above configuration, a shape estimation process is performed with reference to intermediate feature amounts in the completion process, and the completion unit 13 (completion model) is trained with reference to a first loss value, which is a loss value related to the shape obtained by the shape estimation process, so that the performance of the completion process by the completion unit 13 (completion model) can be suitably improved. Therefore, according to the above configuration, it is possible to provide a 3D shape completion method with improved spatial representation performance.

[0037] (Additional notes regarding information processing method S1) In the case where the acquiring unit 11 acquires three-dimensional point cloud data in step S11, the information processing method S1 may be configured not to include step S12. Step S11 of acquiring input data including three-dimensional point cloud data; Step S14 of generating sampled three-dimensional structure data by sampling the three-dimensional point cloud data (three-dimensional structure data); Step S13: applying a complementation process to the sampled three-dimensional structure data by a complementation unit 13 (complementation model); Step S15: executing a shape estimation process by referring to the intermediate feature amount in the complementation process; a step S16 of training the completion unit 13 (complementation model) by referring to a first loss value which is a loss value related to the shape obtained by the shape estimation process; The information processing method S1 configured in this manner can also achieve the above-mentioned effects.

[0038] (Configuration of information processing device 2) Next, the configuration of the information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2. As shown in Fig. 3, the information processing device 2 includes an acquisition unit 21, a three-dimensional structure data generation unit 22, a completion unit 23, and an output data generation unit 24.

[0039] (Acquisition part 21) The acquisition unit 11 acquires input data. Here, the input data is, for example, input data for the inference phase (estimation phase, test phase). Specific examples of the input data acquired by the acquisition unit 21 do not limit the present exemplary embodiment, but may include, for example: RGB data where each pixel (data point) represents an RGB value Depth data, where each pixel (data point) represents a depth value, and 3D point cloud data where each data point represents a 3D coordinate The three-dimensional point cloud data may be, for example, point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, but this example does not limit the present exemplary embodiment.

[0040] (3D structure data generation unit 22) The three-dimensional structure data generation unit 22 generates three-dimensional structure data from the input data acquired by the acquisition unit 21. As an example, the three-dimensional structure data generation unit 22 generates three-dimensional structure data from the input data acquired by the acquisition unit 21, which includes at least either RGB data or depth data. As an example, the three-dimensional structure data generation unit 22 generates three-dimensional structure data from the input data acquired by the acquisition unit 21. The three-dimensional coordinates of each pixel included in the RGB data are determined by referencing the depth data of each pixel in the depth data; Generate 3D structure data with the specified 3D coordinates assigned to each pixel (each data point) Here, the three-dimensional structure data may include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point in addition to the three-dimensional coordinates assigned to each data point. Alternatively, attribute data including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point may be generated in association with the three-dimensional structure data including the three-dimensional coordinates of each data point.

[0041] Furthermore, when the acquisition unit 21 acquires input data including three-dimensional point cloud data, the three-dimensional structure data generation unit 22 may be configured to output the three-dimensional point cloud data as three-dimensional structure data as is. Alternatively, the three-dimensional point cloud data may be configured to include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point described above, and then output the three-dimensional structure data. Alternatively, the three-dimensional structure data generation unit 22 may be configured to output attribute data including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point described above, together with the three-dimensional structure data including the three-dimensional coordinates of each data point.

[0042] Note that, when the acquiring unit 21 acquires input data including three-dimensional point cloud data and the three-dimensional structure data generating unit 22 outputs the three-dimensional point cloud data as is, the information processing device 2 may be configured without the three-dimensional structure data generating unit 22. Such a configuration is also included in this exemplary embodiment.

[0043] In this way, the three-dimensional structure data generation unit 22 can be configured to perform processing similar to that of the three-dimensional structure data generation unit 12 provided in the information processing device 1, as an example, but this does not limit this exemplary embodiment.

[0044] (Complementary part 23) The completion unit 23 applies a completion process to the three-dimensional structure data generated by the three-dimensional structure data generation unit 22. Here, the completion process may include a process of estimating (generating) a completed area (also called a completion area or a completion structure) for an occluded portion or a missing portion in the three-dimensional structure data. The completion process is performed using a machine-learned completion model (completion unit). More specifically, the completion model (completion unit) may be a sampling process for generating sampled 3D structure data by sampling the 3D structure data generated from the training data; - Complementing the sampled three-dimensional structure data using the complementary model (completion unit); A shape estimation process that references intermediate feature values ​​obtained in the completion process; a first learning process for learning the complement model (completion unit) by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; The above-described completion unit 13 trained by the first learning unit 16 in the information processing device 1 can be used as the completion unit (complementation model) trained in this manner, but this does not limit the present exemplary embodiment.

[0045] (Output data generation unit 24) The output data generation unit 24 generates output data from the three-dimensional structure data to which the complementation process by the complementation unit 23 has been applied (the complemented three-dimensional structure data). As an example, the output data generation unit 24 may generate output data by referring to each value of the distance function calculated for the complemented three-dimensional structure data. As an example, the output data generation unit 24 may be configured to restore, as a three-dimensional structure, a surface region in a voxel space containing each value of the distance function calculated for the complemented three-dimensional structure data, where the value of the distance function is 0, by the marching cubes method, and generate output data including the restored data. As an example, the output data generated by the output data generation unit 24 is presented to a user as a display image.

[0046] (Effects of information processing device 2) As described above, in the information processing device 2, Get the input data, Generate three-dimensional structure data from the input data; Applying a completion process to the three-dimensional structure data by a completion unit (complementation model); Generate output data from the 3D structure data to which the interpolation process has been applied. In this configuration, The complementation unit (complementation model) a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process by the interpolation unit (interpolation model) for the three-dimensional structure data after the sampling; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the completion unit (complementation model) by referring to a first loss value that is a loss value related to the shape obtained by the shape estimation process; It was learned by According to the above configuration, a completion unit (completion model) trained by a shape estimation process that references intermediate features in the completion process and a first learning process that trains the completion unit (completion model) by reference to a first loss value that is a loss value related to the shape obtained by the shape estimation process is used, so that output data is generated using a completion unit (completion model) with suitably improved completion process capabilities. Therefore, according to the above configuration, it is possible to provide a 3D shape completion method with improved spatial representation capabilities.

[0047] (Additional notes regarding information processing device 2) As described above, when the acquisition unit 21 is configured to acquire three-dimensional point cloud data, the information processing device 2 may not be configured to include the three-dimensional structure data generation unit 22. In other words, the information processing device 2 an acquisition unit 21 that acquires input data including three-dimensional point cloud data; a completion unit 23 that applies completion processing to the three-dimensional structure data using the above-mentioned completion unit (completion model); an output data generation unit 24 that generates output data from the three-dimensional structure data to which the interpolation processing has been applied; The information processing device 2 configured in this manner can also achieve the above-mentioned effects.

[0048] (Flow of information processing method S2) Next, the flow of information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of information processing method S2. As shown in Fig. 4, information processing method S2 includes step (process) S21 of acquiring input data, step (process) S22 of generating three-dimensional structure data, step (process) S23 of executing complementation processing, and step (process) S24 of generating output data.

[0049] (Step S21) In step S21, the acquisition unit 21 acquires input data. The specific processing by the acquisition unit 21 has been described above, and therefore will not be described here.

[0050] (Step S22) Subsequently, in step S22, the three-dimensional structure data generation unit 22 generates three-dimensional structure data from the input data acquired in step S21 by the acquisition unit 21. The specific processing by the three-dimensional structure data generation unit 22 has been described above, and therefore will not be described here.

[0051] (Step S23) Subsequently, in step S23, the complementing unit 23 applies complementing processing to the three-dimensional structure data generated in step S22 by the three-dimensional structure data generating unit 22. The specific processing by the complementing unit 23 has been described above, and therefore will not be described here.

[0052] (Step S24) Subsequently, in step S24, the output data generation unit 24 generates output data from the three-dimensional structure data (the three-dimensional structure data after completion) to which the completion process by the completion unit 23 in step S23 has been applied. The specific process by the output data generation unit 24 has been described above, and therefore will not be described here.

[0053] (Effect of information processing method S2) As described above, in the information processing method S2, Get the input data, Generate three-dimensional structure data from the input data; Applying a completion process to the three-dimensional structure data by a completion unit (complementation model); Generate output data from the 3D structure data to which the interpolation process has been applied. In this configuration, The complementation unit (complementation model) a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process by the interpolation unit (interpolation model) for the three-dimensional structure data after the sampling; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the completion unit (complementation model) by referring to a first loss value that is a loss value related to the shape obtained by the shape estimation process; It was learned by According to the above configuration, a completion unit (completion model) trained by a shape estimation process that references intermediate features in the completion process and a first learning process that trains the completion unit (completion model) by reference to a first loss value that is a loss value related to the shape obtained by the shape estimation process is used, so that output data is generated using a completion unit (completion model) with suitably improved completion process capabilities. Therefore, according to the above configuration, it is possible to provide a 3D shape completion method with improved spatial representation capabilities.

[0054] (Additional notes regarding information processing method S2) In the case where the three-dimensional point cloud data is acquired in step S21, the information processing method S2 may be configured not to include step S22. a step 21 of obtaining input data including three-dimensional point cloud data; Step S23 of applying a completion process by the above-mentioned completion unit (completion model) to the three-dimensional structure data; Step S24 of generating output data from the three-dimensional structure data to which the interpolation processing has been applied; The information processing method S2 configured in this manner can also achieve the above-mentioned effects.

[0055] Second Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0056] (Configuration of information processing device 100A) The configuration of an information processing device 100A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 100A. As shown in Fig. 5, the information processing device 100A includes a control unit 10A, a storage unit 20A, a communication unit 30, and an input / output unit 40.

[0057] (Communication unit 30) The communication unit 30 communicates with devices external to the information processing device 100A via a network. For example, the communication unit 30 transmits data supplied from the control unit 10A to the external device, and supplies data received from the external device to the control unit 10A. Note that the specific configuration of the network does not limit the present exemplary embodiment, and examples include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks.

[0058] (Input / output section 40) The input / output unit 40 is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input / output unit 40 may be configured to have input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel connected to it. In this configuration, the input / output unit 40 accepts various types of information input to the information processing device 100A from the connected input devices. Furthermore, the input / output unit 40 outputs various types of information to the connected output devices under the control of the control unit 10A. An example of the input / output unit 40 is an interface such as a USB (Universal Serial Bus).

[0059] (Storage unit 20A) The storage unit 20A stores various data referenced by the control unit 10A and various data generated by the control unit 10A. Input data IND 3D structure data SD -SSD 3D structure data after sampling Complemented 3D structure data CSD ·Shape estimation result FER First Loss Value LV1 Second Loss Value LV2 etc. are stored. Here, the input data IND is data acquired by the acquisition unit 11 (21) described later. Specific examples of the input data IND will be described later. The three-dimensional structure data SD is data generated by the three-dimensional structure data generation unit 12 (22) described later. Specific examples of the three-dimensional point cloud data SD will be described later. The sampled three-dimensional structure data SSD is data generated by the sampling unit 14 described later. Specific examples of the sampled three-dimensional structure data SSD will be described later. The interpolated three-dimensional structure data CSD is data generated by the interpolation unit 13 (23) described later. Specific examples of the interpolated three-dimensional structure data CSD will be described later. The shape estimation result FER is the result of a shape estimation process performed by the estimation unit 15 described later. Specific examples of the shape estimation result FER will be described later.

[0060] The first loss value LV1 is a loss value referenced by the first learning unit 16, which will be described later. Specific examples of the first loss value LV1 will be described later. The second loss value LV2 is a loss value referenced by the second learning unit 17, which will be described later. Specific examples of the second loss value LV2 will be described later. The first loss value LV1 and the second loss value LV2 may be collectively referred to simply as the loss value LV.

[0061] (Control unit 10A) As shown in FIG. 5 , the control unit 10A includes the acquisition unit 11, the three-dimensional structure data generation unit 12, the sampling unit 14, the completion unit 13, the estimation unit 15, and the first learning unit 16 described in the exemplary embodiment 1. Here, the acquisition unit 11 can be expressed as having the same configuration as the acquisition unit 21 described in the exemplary embodiment 1, and therefore the acquisition unit 11 may also be referred to as the acquisition unit 11 (21). The three-dimensional structure data generation unit 12 can be expressed as having the same configuration as the three-dimensional structure data generation unit 22 described in the exemplary embodiment 1, and therefore the three-dimensional structure data generation unit 12 may also be referred to as the three-dimensional structure data generation unit 12 (22). The completion unit 13 can be expressed as having the same configuration as the completion unit 23 described in the exemplary embodiment 1, and therefore the completion unit 13 may also be referred to as the completion unit 13 (23). The control unit 10A also includes a second learning unit 17, a distance function value calculation unit 18, and an output data generation unit 24.

[0062] (Acquisition part 11(21)) The acquisition unit 11 (21) acquires input data IND. Here, the acquisition unit 11 (21) acquires input data IND for learning in the learning phase, and acquires input data IND for inference in the inference phase (estimation phase, test phase). Although a specific example of the input data IND acquired by the acquisition unit 11 (21) does not limit this exemplary embodiment, as in the exemplary embodiment, as an example, RGB data where each pixel (data point) represents an RGB value Depth data, where each pixel (data point) represents a depth value, and 3D point cloud data where each data point represents a 3D coordinate The three-dimensional point cloud data may be, for example, point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, but this example does not limit the present exemplary embodiment.

[0063] The input data IND acquired by the acquisition unit 11 (21) is stored in the storage unit 20A, for example, and is referenced by the three-dimensional structure data generation unit 12 (22), the sampling unit 14, and the like.

[0064] (3D structure data generation unit 12(22)) The three-dimensional structure data generation unit 12 (22) generates three-dimensional structure data SD from the input data IND acquired by the acquisition unit 11 (12). As an example, the three-dimensional structure data generation unit 12 (22) generates three-dimensional structure data SD from the input data IND acquired by the acquisition unit 11 (21), which includes at least either RGB data or depth data. As an example, the three-dimensional structure data generation unit 12 (22) The three-dimensional coordinates of each pixel included in the RGB data are determined by referencing the depth data of each pixel in the depth data; Generate 3D structure data SD, where the specified 3D coordinates are assigned to each pixel (each data point). The three-dimensional structure data SD may be configured as follows. Here, similar to the first exemplary embodiment, the three-dimensional structure data SD may be configured to include feature amounts of each data point in addition to the three-dimensional coordinates assigned to each data point. Here, each feature amount may be configured to include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point. Alternatively, the three-dimensional structure data generation unit 12 (22) may be configured to generate attribute data including feature amounts of each data point (for example, at least one of the RGB values ​​and normal values ​​(normal vectors)) in association with the three-dimensional structure data SD including the three-dimensional coordinates of each data point.

[0065] Furthermore, the three-dimensional structure data generating unit 12 (22) may generate the three-dimensional structure data SD using an algorithm such as Structure from Motion (SfM) or Simultaneous Localization and Mapping (SLAM).

[0066] With the above-mentioned configuration, the three-dimensional structure data generation unit 12 (22) can generate the above-mentioned three-dimensional structure data SD from one or more frames (one or more data sets) included in the input data acquired by the acquisition unit 11 (21).

[0067] Furthermore, when the acquisition unit 11 (21) acquires input data IND including three-dimensional point cloud data, the three-dimensional structure data generation unit 12 (22) may be configured to output the three-dimensional point cloud data as three-dimensional structure data SD as is. Alternatively, the three-dimensional point cloud data SD may be configured to include the above-mentioned feature amounts of each data point (for example, at least one of RGB values ​​and normal values ​​(normal vectors)) and then output the three-dimensional structure data SD. Alternatively, the attribute data including the above-mentioned feature amounts of each data point may be output together with the three-dimensional structure data SD including the three-dimensional coordinates of each data point.

[0068] In addition, when the acquiring unit 11 (21) acquires input data IND including three-dimensional point cloud data and the three-dimensional structure data generating unit 12 (22) outputs the three-dimensional point cloud data as three-dimensional structure data SD as is, the information processing device 1 may be configured not to include the three-dimensional structure data generating unit 12 (22). Such a configuration is also included in this exemplary embodiment.

[0069] (Sampling section 14) The sampling unit 14 generates sampled three-dimensional structure data SSD by sampling the three-dimensional structure data SD generated by the three-dimensional structure data generating unit 12 (22). Here, the sampling process may include, for example, A process of generating the sampled three-dimensional structure data SSD using only some of the frames included in the input data IND. As an example, the sampling process includes: A process to generate 3D structure data SSD using only N% (N is a real number less than 100) of the frames included in the input data IND. Here, the sampling unit 14 may select the N% of frames by: Randomly select frames to remove Remove a series of consecutive frames in a way that a specific region is missing from the image or object represented by the input data IND. The following processing may be performed.

[0070] Alternatively, the sampling process may include: A process of generating the sampled three-dimensional structure data SSD using only some of the data points included in at least one of the input data IND and the three-dimensional structure data SD. As an example, the sampling process may include: - Processing to reduce the resolution of the depth image included in the input data IND A process of applying a thinning method to the 3D point cloud data included in the input data IND so that the density of the light receiving sensors of the LiDAR device is reduced. At least one of the above may be included.

[0071] The sampling process executed by the sampling unit 14 may also be called thinning process. Therefore, the three-dimensional structure data SSD after sampling may also be called thinned three-dimensional structure data.

[0072] (Supplementary Section 13(23)) In the learning phase, the completion unit 13 (23) applies completion processing to the sampled three-dimensional structure data SSD generated by the sampling unit 14, and in the inference phase, applies completion processing to the three-dimensional structure data SD generated by the three-dimensional structure data generation unit 12 (22).

[0073] Here, the completion process may include a process of estimating (generating) a completed area (also referred to as a completed area or a completed structure) for the sampled three-dimensional structure data SSD or for occluded or missing areas in the three-dimensional structure data SD. The completion process is performed using a machine learning-enabled completion model. More specifically, as an example, the completion process is performed by inputting the sampled three-dimensional structure data SSD or the three-dimensional structure data SD to a neural network, which is a completion model having multiple layers. The neural network to which the sampled three-dimensional structure data SSD or the three-dimensional structure data SD is input outputs completed three-dimensional structure data CSD.

[0074] Furthermore, the intermediate feature values ​​generated (calculated) in the completion process executed by the completion unit 13 (23) (for example, feature values ​​generated (calculated) in the intermediate layer of the neural network) are referenced by the estimation unit 15, which will be described later, in the learning phase. The specific configuration of the completion unit 13 (23) does not limit this exemplary embodiment, but as an example, an encoder to which the sampled three-dimensional structure data SSD or the sampled three-dimensional structure data SD is input and which outputs the intermediate features; a decoder that receives the intermediate features and outputs the completed 3D structure data CSD; A more specific configuration of the complementing unit 13 will be described later with reference to different drawings.

[0075] (Estimation part 15) The estimation unit 15 performs a shape estimation process by referring to the intermediate feature values ​​obtained in the completion process by the completion unit 13, and derives a shape estimation result FER as a result of the shape estimation process. As an example, the estimation unit 15 performs the shape estimation process by calculating each value of a distance function from the intermediate feature values. In other words, the estimation unit 15 expresses the result of the shape estimation process as a distance function. Here, the distance function refers to, for example, a function defined by the distance from a point in space to each point on the object. For example, a signed distance function (SDF), which is an example of a distance function, expresses the shortest distance from a point in space to the surface of the object, assigning a negative value if the point is inside the object and a positive value if the point is outside the object. Therefore, in this case, the surface (contour) of the object is represented by a region (contour) where the SDF value is 0. A surface defined by SFD=0 in this way is sometimes called an implicit surface.

[0076] 6 shows a distance function SDF calculated in the shape estimation process by the estimation unit 15, and an object representation example 1 using the distance function SDF. In this example, the SDF value inside the object OBJ is calculated as a negative value, and the SDF value outside the object OBJ is calculated as a positive value. The surface (contour) of the object OBJ is then represented (defined) by an implicit surface defined by SFD=0.

[0077] The lower part of Fig. 6 shows the distance function SDF calculated in the shape estimation process by the estimation unit 15, and a second example of object representation using the distance function SDF. In this example, a certain multidimensional function gives an SDF value in space, and a positive SDF value indicates the outside of the object, a negative SDF value indicates the inside of the object, and the cut surface when SDF = 0 (C1, C2, C3 in the lower part of Fig. 6) indicates the surface of the object.

[0078] As in the first exemplary embodiment, the distance function is not limited to the SDF described above, and an unsigned distance function may be used. As an example, the UDF value is positive, and the surface (contour) of the object is represented by a region (contour) where the UDF value takes a predetermined value.

[0079] In this way, the estimation unit 15 executes the shape estimation process using a distance function, and the shape estimation result FER is expressed by the distance function, so that it is possible to execute a three-dimensional shape completion process with improved spatial expression capability.

[0080] As an example, the estimation unit 15 calculates the following as shown in FIG. a feature transformation unit 151 that transforms the intermediate feature in the complementation process into a latent variable; a shape estimation unit 152 that executes the shape estimation process by referring to the latent variables and the sampled three-dimensional structure data; A more specific configuration of the estimation unit 15 will be described later with reference to different drawings.

[0081] (First Study Section 16) The first learning unit 16 causes the completion unit 13 (23) to learn by referring to a first loss value LV1, which is a loss value related to the shape (shape estimation result FER) obtained by the shape estimation process by the estimation unit 15. Here, the first loss value LV1 is, for example, Each value of the distance function obtained by the shape estimation process by the estimation unit 15, With reference to the sampled three-dimensional structure data SSD generated by the sampling unit 14, each value of the distance function calculated by the distance function value calculation unit 18 (that is, each value of the distance function obtained from the sampled three-dimensional structure data SSD that has not been processed by the completion unit 13 and the estimation unit 15) Then, the first learning unit 16 causes the complementing unit 13 to learn so that the first loss value LV1 becomes smaller, for example.

[0082] As an example, the first learning unit 16 performs the following as shown in FIG. a first loss calculation unit 161 that calculates the first loss value LV1; a first updating unit 162 that updates one or more parameters defining the complementing unit 13 (23) by referring to the first loss value LV1; Here, the one or more parameters that define the completion unit 13 (23) refer to one or more parameters that define a completion model that functions as the completion unit 13 (23). As an example, if the completion unit 13 (23) is configured with the above-mentioned encoder and decoder, the first update unit 162 updates the values ​​of one or more parameters that define the encoder so that the first loss value LV1 becomes smaller. As an example, the parameters updated by the first update unit 162 are stored in the storage unit 20A and are referred to in the learning process in the next step or the estimation process in the inference phase.

[0083] (Second Study Section 17) The second learning unit 17 trains the completion unit 13 (23) by referring to a second loss value LV2 indicating the difference between the completed three-dimensional structure data CSD output by the completion unit 13 (23) and the three-dimensional structure data SD generated by the three-dimensional structure data generation unit 12 (22). More specifically, the second learning unit 17 trains the encoder and decoder constituting the completion unit 13 (23) by referring to a second loss value LV2 indicating the difference between the completed three-dimensional structure data CSD output by the decoder constituting the completion unit 13 (23) and the three-dimensional structure data SD generated by the three-dimensional structure data generation unit 12 (22).

[0084] As an example, as shown in FIG. 5, the second learning unit 17 a second loss calculation unit 171 that calculates the second loss value LV2; a second updating unit 172 that updates one or more parameters defining the complementing unit 13 (23) by referring to the second loss value LV2; As an example, if the completion unit 13 (23) is configured by the encoder and decoder described above, the second update unit 172 updates the values ​​of one or more parameters that define the encoder and one or more parameters that define the decoder so that the second loss value LV2 becomes smaller. The parameters updated by the second update unit 172 are stored in the storage unit 20A, for example, and are referred to in the learning process in the next step or in the estimation process in the inference phase.

[0085] (Distance function value calculation unit 18) The distance function value calculation unit 18 calculates each value of the distance function by referring to the sampled three-dimensional structure data SSD generated by the sampling unit 14. In other words, the distance function value calculation unit 18 calculates each value of the distance function corresponding to the sampled three-dimensional structure data SSD that has not been processed by the completion unit 13 and the estimation unit 15. The distance function values ​​calculated by the distance function value calculation unit 18 are referred to by the first learning unit 16 described above.

[0086] (Output data generation unit 24) The output data generation unit 24 generates output data from the three-dimensional structure data (complemented three-dimensional structure data CSD) to which the complementation process by the complementation unit 23 has been applied. As an example, the output data generation unit 24 may generate output data by referring to each value of the distance function calculated for the complemented three-dimensional structure data CSD. As an example, the output data generation unit 24 may be configured to reconstruct, as a three-dimensional structure, a surface region in a voxel space containing each value of the distance function calculated for the complemented three-dimensional structure data CSD, where the distance function value is 0, using the marching cubes method, and generate output data including the reconstructed data. As an example, the output data generated by the output data generation unit 24 is presented to a user as a display image via a display included in the input / output unit 40. In other words, the output data generation unit 24 also functions as a display unit (display control unit) that displays the output data generated by the output data generation unit 24.

[0087] As described above, in the information processing device 100A, Get the input data IND, Generate three-dimensional structure data SD from the input data IND; generating sampled three-dimensional structure data SSD by sampling the three-dimensional structure data SD; A complementation process is applied to the sampled three-dimensional structure data SSD by a complementation unit 13, A shape estimation process is performed by referring to the intermediate feature values ​​obtained in the completion process. The completion unit 13 is trained by referring to a first loss value LV1, which is a loss value related to the shape obtained by the shape estimation process. According to the above configuration, a shape estimation process is performed with reference to intermediate features in the completion process, and the completion unit 13 is trained with reference to a first loss value LV1, which is a loss value related to the shape obtained by the shape estimation process, so that the performance of the completion process by the completion unit 13 can be suitably improved. Therefore, according to the above configuration, it is possible to provide a 3D shape completion method with improved spatial representation performance.

[0088] Furthermore, as described above, the information processing device 100A executes the shape estimation process using a distance function, which can further improve the spatial representation capability.

[0089] Furthermore, as described above, in the information processing device 100A, the second learning unit 17 trains the completion unit 13 (23) by referring to the second loss value LV2 indicating the difference between the completed three-dimensional structure data CSD output by the completion unit 13 (23) and the three-dimensional structure data SD generated by the three-dimensional structure data generation unit 12 (22), thereby providing a more suitable three-dimensional shape completion method.

[0090] (Data flow during the learning phase) Next, the flow of data in the learning phase of the information processing device 100A will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the configuration of networks that constitute the units included in the information processing device 100A, and the flow of data between these networks.

[0091] As shown in Fig. 7, the three-dimensional structure data SD generated by the three-dimensional structure data generation unit 12 (22) is converted into sampled three-dimensional structure data SSD by a sampling process (denoted as "Sampling" in Fig. 7) by the sampling unit 14. Then, in the learning phase, the sampled three-dimensional structure data SSD is input to the completion unit 13. Here, as shown in Fig. 7, the completion unit 13 an encoder 131 that receives the sampled 3D structure data SSD and outputs intermediate features; a decoder 132 that receives the intermediate features and outputs the completed three-dimensional structure data CSD; It is equipped with:

[0092] The three-dimensional structure data CSD after the complementation is input to a complementation loss calculation unit 171 (corresponding to the second loss calculation unit 171 described above). The complementation loss calculation unit (second loss calculation unit) 171 calculates a second loss value LV2 indicating the difference between the three-dimensional structure data SD and the three-dimensional structure data CSD after the complementation. The calculated second loss value LV2 is referred to by a second update unit 172 to update the parameters of the encoder 131 and the decoder 132.

[0093] On the other hand, the intermediate features output by the encoder 131 are input to a feature transformation network 151 (corresponding to the feature transformation unit 151 described above). The feature transformation network (feature transformation unit) 151 is configured to include, for example, a pooling layer and a fully connected layer, and derives latent variables from the intermediate features. The latent variables derived by the feature transformation network (feature transformation unit) 151 are input to a shape estimation unit 152. Here, the shape estimation unit 152 has, for example, a network configuration as a decoder. As shown in FIG. 7, the shape estimation unit 152 receives the following data together with the latent variables: - The point cloud near the surface in the sampled 3D structure data SSD (referred to as "Near Surface Point" in Figure 7), or -Surface point cloud with random noise added to the sampled 3D structure data SSD is input. Then, shape estimation unit 152 calculates the above-mentioned UDF value or SDF value as the shape estimation result FER. The UDF value or SDF value calculated by shape estimation unit 152 is input to shape loss estimation unit 161 (corresponding to first loss calculation unit 161 described above).

[0094] On the other hand, the above -Surface point cloud in the sampled 3D structure data SSD, or -Surface point cloud with random noise added to the sampled 3D structure data SSD are input to a distance function value calculation unit (distance function value calculation unit) 18, which refers to these point groups and calculates a UDF value or an SDF value corresponding to the sampled 3D structure data SSD. The UDF value or the SDF value calculated by the distance function value calculation unit 18 is input to a shape loss estimation unit 161 (corresponding to the above-mentioned first loss calculation unit 161).

[0095] The shape loss estimation unit (first loss calculation unit) 161 The UDF value or SDF value calculated by the shape estimation unit 152, The UDF value or SDF value calculated by the distance function value calculation unit 18 The first update unit 162 calculates a first loss function value LV1 that indicates the difference between the first loss function value LV1 and the second loss function value LV2. The first update unit 162 refers to the calculated first loss function value LV1 in order to update the parameters of the encoder 131.

[0096] (Process flow in the learning phase) Next, the flow of processing in the learning phase of information processing device 100A will be described with reference to Fig. 8. Fig. 8 is a diagram showing the flow of processing in the learning phase of information processing device 100A.

[0097] (Step S11) In step S11, the acquisition unit 11 executes a data acquisition process. As an example, the acquisition unit 11 acquires input data IND for learning. The specific process performed by the acquisition unit 11 has been described above, and therefore will not be described here.

[0098] (Step S12) In step S12, the three-dimensional structure data generation unit 12 generates three-dimensional structure data SD through a three-dimensional structure creation process. The specific process performed by the three-dimensional structure data generation unit 12 has been described above, and therefore will not be described here.

[0099] (Step S14) In step S14, the sampling unit 14 applies a three-dimensional structure sampling process to the three-dimensional structure data SD to generate sampled three-dimensional structure data SSD. The specific process performed by the sampling unit 14 has been described above, so a description thereof will be omitted here.

[0100] (Step S13) In step S13, the completion unit 13 applies region completion processing to the sampled three-dimensional structure data SSD to generate intermediate features and completed three-dimensional structure data CSD. The specific processing by the completion unit 13 has been described above, so a description thereof will be omitted here.

[0101] (Step S151) In step S151, the feature transform unit 151 derives latent variables by applying intermediate feature transform processing to the intermediate features generated by the completion unit 13. The specific processing by the feature transform unit 151 has been described above, and therefore will not be described here.

[0102] (Step S152) In step S152, shape estimation unit 152 executes shape estimation processing by referring to the latent variables derived by feature quantity conversion unit 151. The specific processing by shape estimation unit 152 has been described above, and therefore will not be described here.

[0103] (Step S18) On the other hand, in step S18, the distance function value calculation unit 18 calculates a distance function value corresponding to the sampled three-dimensional structure data SSD. The specific processing by the distance function value calculation unit 18 has been described above, so a description thereof will be omitted here.

[0104] (Step S161) In step S161, first loss calculation unit 161 executes a shape loss calculation process to calculate first loss value LV1 by referring to the distance function value calculated by distance function value calculation unit 18 and the distance function value indicated by the shape estimation result by shape estimation unit 152. The specific process by first loss calculation unit 161 has been described above, and therefore will not be described here.

[0105] (Step S162) In step S162, the first update unit 162 refers to the first loss value LV1 and executes a process (first update process) of updating the complement unit 13. The specific process by the first update unit 162 has been described above, and therefore will not be described here.

[0106] (Step S171) On the other hand, in step S171, the second loss calculation unit 171 executes a complementation loss calculation process by referring to the three-dimensional structure data SD and the complemented three-dimensional structure data SCD, and calculates a second loss value LV2. The specific process by the second loss calculation unit 171 has been described above, so a description thereof will be omitted here.

[0107] (Step S172) In step S172, the second update unit 172 refers to the second loss value LV2 and executes a process (second update process) of updating the complement unit 13. The specific process by the second update unit 172 has been described above, and therefore will not be described here.

[0108] For example, the series of processes from step S11 to step S162 or step S172 may be repeatedly executed until a predetermined convergence condition is satisfied. Furthermore, for example, the data frozen by at least one of the processes from step S11 to step S162 or step S172 may be presented to the user via the input / output unit 40.

[0109] (Processing flow in the inference phase) Next, the flow of processing in the inference phase (estimation phase, test phase) of information processing device 100A will be described with reference to Fig. 9. Fig. 9 is a diagram showing the flow of processing in the inference phase of information processing device 100A.

[0110] (Step S21) In step S21, the acquisition unit 21 executes a data acquisition process. As an example, the acquisition unit 21 acquires input data IND for inference. The specific process performed by the acquisition unit 21 has been described above, and therefore will not be described here.

[0111] (Step S22) Subsequently, in step S22, the three-dimensional structure data generation unit 22 generates three-dimensional structure data SD by referring to the input data IND for inference (three-dimensional structure creation process). The specific process by the three-dimensional structure data generation unit 22 has been described above, so a description thereof will be omitted here.

[0112] (Step S23) Subsequently, in step S23, the completion unit 23 refers to the three-dimensional structure data SD to generate the three-dimensional structure data CSD after completion (area completion process). Here, the completion unit 23 has learned through each process in the inference phase described above. The specific process by the completion unit 23 has been described above, so a description thereof will be omitted here.

[0113] (Step S24) Subsequently, in step S24, the output data generation unit 24 generates output data from the three-dimensional structure data CSD after completion (complemented three-dimensional structure creation process). The specific process by the output data generation unit 24 has been described above, and therefore will not be described here.

[0114] Third Embodiment A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0115] (Configuration of information processing device 100B) The configuration of an information processing device 100B according to this exemplary embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the information processing device 100B. As shown in Fig. 10, the information processing device 100B includes a control unit 10B, a storage unit 20B, a communication unit 30, and an input / output unit 40.

[0116] (Storage section 20B) The storage unit 20B stores a third loss value LV3 in addition to the various data stored in the storage unit 20A included in the information processing device 100A according to exemplary embodiment 2. The third loss value LV3 will be described later.

[0117] (control unit 10B) The control unit 10B includes a rendering unit 19 and a third learning unit 20 in addition to the components included in the information processing device 100A according to the second exemplary embodiment. The following description will focus on the differences from the information processing device 100A.

[0118] (Acquisition part 11(21)) The acquisition unit 11 (21) according to this exemplary embodiment acquires, in addition to each piece of data acquired by the acquisition unit 11 (21) according to exemplary embodiment 2, further acquires information regarding the position and orientation of the imaging device when the imaging device captured the RGB image included in the input data IND.

[0119] (Rendering Section 19) The rendering unit 19 The shape indicated by the shape estimation result FER by the estimation unit 15, Information about the position and orientation of the imaging device when the imaging device captured the RGB image included in the input data IND. Rendering is performed by referring to the

[0120] As an example, the rendering unit 19 performs volume rendering processing and ray tracing processing based on the shape indicated by the shape estimation result FER by the estimation unit 15, and uses information regarding the position and orientation of the imaging device to draw (generate) an image (rendered image) on a specified image plane.

[0121] (Third Study Section 20) The third learning unit 20 executes a learning process by referring to a third loss value LV3 that indicates the difference between the rendering image obtained by the rendering process and the RGB image included in the input data.

[0122] As an example, the third learning unit 20 may perform the following as shown in FIG. a third loss calculation unit 201 that calculates the third loss value LV3; a third updating unit 202 that updates one or more parameters defining the complementing unit 13 (23) by referring to the third loss value LV3; The third updating unit 202, for example, updates one or more parameters that define the complementing unit 13 (23) so that the third loss value LV3 becomes smaller.

[0123] According to the information processing device 100B according to this exemplary embodiment, as described above, The completion unit 13 (23) is trained by referring to the third loss value LV3, which indicates the difference between the rendered image obtained by the rendering process and the image included in the input data, thereby providing a 3D shape completion method with improved spatial representation capabilities.

[0124] (Processing flow by information processing device 100B) Next, the flow of processing by the information processing device 100B will be described with reference to Fig. 11. Fig. 11 is a diagram showing the flow of processing by the information processing device 100B in the learning phase. Note that the flow of processing by the information processing device 100B in the inference phase is the same as in the second exemplary embodiment, and therefore description thereof will be omitted.

[0125] As shown in FIG. 11, in the learning phase, the information processing device 100B executes the processes of steps S19, S201, and S202 in addition to the processes executed by the information processing device 100A.

[0126] (Step S19) In step S19, the rendering unit 19 performs rendering processing (planar image rendering processing) by referring to the result of the shape estimation processing (shape estimation result FER) by the estimation unit 15. The specific processing by the rendering unit 19 has been described above, and therefore will not be described here.

[0127] (Step S201) Next, in step S201, the third loss calculation unit 201 calculates a third loss value LV3 indicating the difference between the rendering image obtained by the rendering process in step S19 and the RGB image included in the input data IND (inter-image loss calculation process). The specific process by the third loss calculation unit 201 has been described above, so a description thereof will be omitted here.

[0128] (Step S202) Subsequently, in step S202, the third updating unit 202 refers to the third loss value LV3 and updates one or more parameters that define the complementing unit 13 (23). The specific processing by the third updating unit 202 has been described above, and therefore will not be described here.

[0129] [Fourth embodiment] A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0130] (Configuration of information processing device 100C) The configuration of an information processing device 100C according to this exemplary embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the information processing device 100C. As shown in Fig. 12, the information processing device 100C includes a control unit 10C, a storage unit 20C, a communication unit 30, and an input / output unit 40.

[0131] (Control unit 10C, memory unit 20C) The control unit 10C includes the components of the control unit 10A or 10B according to exemplary embodiment 2 or 3, namely, the acquisition unit 21, the three-dimensional structure data generation unit 22, the completion unit 23, and the output data generation unit 24, but does not include any other components.

[0132] Furthermore, the storage unit 20C stores the input data IND, the three-dimensional structure data SD, and the interpolated three-dimensional data CSD, among the data stored in the storage unit 20A or 20B according to exemplary embodiment 2 or 3, but does not store any other data. However, one or more parameters defining an interpolation model used in the interpolation process by the interpolation unit 23 are stored in the storage unit 20C. The interpolation model is, for example, learned (updated) by each process in the learning phase executed by the information processing device 100A or 100B according to exemplary embodiment 2 or 3.

[0133] The information processing device 100C having the above configuration is an acquisition unit 21 for acquiring input data IND; a three-dimensional structure data generating unit 33 for generating three-dimensional structure data SD from the input data IND; a completion unit 23 that applies a completion process to the three-dimensional structure data SD; an output data generation unit 24 that generates output data from the three-dimensional structure data to which the interpolation processing has been applied; Equipped with The completion unit 23 a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process by the interpolation means for the three-dimensional structure data after the sampling; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the complementing means by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; It was learned by

[0134] According to the above configuration, a completion unit (completion model) trained by a shape estimation process that references intermediate features in the completion process and a first learning process that references a first loss value, which is a loss value related to the shape obtained by the shape estimation process, is used, so that output data is generated using a completion unit (completion model) with suitably improved completion processing capabilities. Therefore, according to the above configuration, it is possible to provide a 3D shape completion method with improved spatial representation capabilities.

[0135] (Application example) In the following, application examples of the information processing devices 1, 100A, 100B, and 100C according to the above-described exemplary embodiments will be described.

[0136] The information processing devices 1, 100A, 100B, and 100C (hereinafter simply referred to as information processing devices 1, etc.) according to the above-described exemplary embodiments can also be applied to the medical and healthcare fields, for example. In these fields, the technology of the present application can be used to enable medical applications by performing three-dimensional shape completion processing based on medical images of patients.

[0137] When the information processing device 1 and the like are applied to the medical and healthcare fields, for example, processing may be performed in accordance with the following processing flow.

[0138] (Step S101: Scan step) A medical professional such as a doctor or medical staff uses an imaging device (endoscope, fMRI, etc.) to capture images of a patient's organs (stomach, intestines, etc.) and other relevant areas, and generates medical images. Then, the medical images are input as input data IND into an information processing device 1, etc.

[0139] (Step S102: 3D modeling step) Next, the three-dimensional structure data generation unit 12 (22) included in the information processing device 1 or the like refers to the input data IND and generates three-dimensional point structure data SD corresponding to the input data IND. Here, the three-dimensional structure data SD generated by the three-dimensional structure data generation unit 12 (22) may be configured to be presented to medical professionals, medical staff, patients, etc. via a display or the like included in the input / output unit 40. Note that this step can be applied to both the learning phase and the estimation phase.

[0140] (Step S103A: Decision-making step) Next, the completion unit 23 included in the information processing device 1 or the like performs a 3D shape completion process on the medical image using the machine-learned completion model described in each of the above-mentioned embodiments, thereby generating completed 3D structure data CSD. The output data generation unit 24 then generates output data by referencing the 3D structure data CSD and outputs the output data via the input / output unit 40. The completion unit 23 included in the information processing device 1 or the like may be configured to apply a segmentation process to the completed 3D structure data CSD and output the segmentation results. Furthermore, the semantic segmentation may be configured to perform segmentation (region classification) into lesions (inflammation, ulcers, polyps), normal areas, regions, etc., for example. In this way, by referring to the output data according to the completed 3D structure data CSD, medical professionals can, for example, formulate a treatment plan. Therefore, this device can assist medical professionals in making diagnostic decisions.

[0141] (Step S103B: Decision-making step) Instead of or in addition to the process of step S103A, the information processing device 1, etc. may perform the following process. That is, the complementing unit 23 of the information processing device 1, etc. may present the complemented three-dimensional structure data CSD to a medical professional via, for example, a display provided in the input / output unit 40. By referring to the complemented three-dimensional structure data CSD (3D model), the medical professional can understand, for example, the condition of a patient's organs. Therefore, the device can support the medical professional in making diagnostic decisions.

[0142] In this manner, in this application example, the acquisition unit 11 (21) of the information processing device 1 or the like acquires medical images as the input data, and the completion unit 23 or the output data generation unit 24 functions as a presentation means for displaying the output data to assist medical professionals in making decisions.

[0143] [Software implementation example] Some or all of the functions of the information processing devices 1, 100A, 100B, and 100C (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0144] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 13. Figure 13 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0145] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0146] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0147] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0148] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0149] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0150] (Appendix A1) An acquisition means for acquiring input data; a three-dimensional structure data generating means for generating three-dimensional structure data from the input data; a sampling means for generating sampled three-dimensional structure data by sampling the three-dimensional structure data; a complementation means for applying a complementation process to the sampled three-dimensional structure data; an estimation means for executing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; a first learning means for learning the interpolation means by referring to a first loss value which is a loss value related to the shape obtained by the shape estimation process; An information processing device comprising:

[0151] (Appendix A2) the estimation means calculates one or more distance function values ​​in the shape estimation process; The first loss value is a loss value for the one or more distance function values. 10. The information processing device according to claim 1,

[0152] (Appendix A3) The estimation means a feature conversion means for converting the intermediate feature in the complementation process into a latent variable; a shape estimation means for executing the shape estimation process by referring to the latent variables and the sampled three-dimensional structure data; Equipped with An information processing device according to appendix A1 or A2.

[0153] (Appendix A4) The complementing means comprises: an encoder that receives the sampled three-dimensional structure data and outputs the intermediate feature values; a decoder that receives the intermediate features and outputs the completed three-dimensional structure data; Equipped with The first learning means The encoder is trained by machine learning with reference to the first loss value. An information processing device according to any one of appendices A1 to A3.

[0154] (Appendix A5) a second learning means for performing machine learning on the encoder and the decoder by referring to a second loss value indicating a difference between the interpolated three-dimensional structure data output by the decoder and the three-dimensional structure data generated by the three-dimensional structure data generating means; The information processing device according to appended note A4,

[0155] (Appendix A6) the acquiring means further acquires information relating to a position and orientation of an imaging device when the imaging device captured the RGB image included in the input data; The information processing device includes: a rendering means for executing a rendering process by referring to the shape obtained by the shape estimation process and information about the position and orientation of the imaging device; 10. The information processing device according to claim 9, further comprising:

[0156] (Appendix A7) a third learning means for executing a learning process by referring to a third loss value indicating a difference between a rendering image obtained by the rendering process and an RGB image included in the input data; It also has 10. The information processing device according to claim 9, wherein the information processing device is an information processing device according to claim 8.

[0157] (Appendix A8) An acquisition means for acquiring input data; a three-dimensional structure data generating means for generating three-dimensional structure data from the input data; a complementation means for applying a complementation process to the three-dimensional structure data; an output data generating means for generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Equipped with The complementing means comprises: a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process by the interpolation means for the three-dimensional structure data after the sampling; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the complementing means by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing device.

[0158] (Appendix A9) a display means for displaying the output data generated by the output data generating means; 10. The information processing device according to claim 8,

[0159] (Appendix A10) the acquisition means acquires a medical image as the input data; The display means displays the output data to assist a medical professional in making a decision. 10. The information processing device according to claim 9.

[0160] (Appendix A12) Obtaining input data; generating three-dimensional structure data from the input data; applying a complementation process to the three-dimensional structure data by a complementation means; generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Including, The complementing means comprises: generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the training data; applying an interpolation process to the sampled three-dimensional structure data by the interpolation means; performing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; training the interpolation means by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing device.

[0161] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0162] (Appendix B1) an acquisition step of acquiring input data; a three-dimensional structure data generation step of generating three-dimensional structure data from the input data; a sampling step of generating sampled three-dimensional structure data by sampling the three-dimensional structure data; an interpolation step of applying an interpolation process to the sampled three-dimensional structure data; an estimation step of executing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; a first learning process for learning the interpolation process by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; An information processing method comprising:

[0163] (Appendix B2) the estimation step calculates one or more distance function values ​​in the shape estimation process; The first loss value is a loss value for the one or more distance function values. 1. The information processing method described in Appendix B1.

[0164] (Appendix B3) The estimation step includes: a feature transformation step of transforming the intermediate feature values ​​in the complementation process into latent variables; a shape estimation step of executing the shape estimation process by referring to the latent variables and the sampled three-dimensional structure data; Contains 1. An information processing method according to Appendix B1 or B2.

[0165] (Appendix B4) The complementing step includes: an encoder that receives the sampled three-dimensional structure data and outputs the intermediate feature values; a decoder that receives the intermediate features and outputs the completed three-dimensional structure data; Including, The first learning step includes: The encoder is trained by machine learning with reference to the first loss value. 1. An information processing method according to any one of appendices B1 to B3.

[0166] (Appendix B5) a second learning step of machine learning the encoder and the decoder by referring to a second loss value indicating a difference between the interpolated 3D structure data output by the decoder and the 3D structure data generated in the 3D structure data generating step; 2. The information processing method according to claim 1, further comprising:

[0167] (Appendix B6) the acquiring step further acquires information about a position and orientation of an imaging device when the imaging device captured the RGB image included in the input data; The information processing device includes: a rendering step of performing a rendering process by referring to the shape obtained by the shape estimation process and information about the position and orientation of the imaging device; An information processing method according to any one of appendices B1 to B5, comprising:

[0168] (Appendix B7) a third learning step of executing a learning process by referring to a third loss value indicating a difference between the rendered image obtained by the rendering process and the RGB image included in the input data; Further includes An information processing method as described in Appendix B6.

[0169] (Appendix B8) an acquisition step of acquiring input data; a three-dimensional structure data generation step of generating three-dimensional structure data from the input data; a completion step of applying an completion process to the three-dimensional structure data; an output data generation step of generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Including, The complementing step includes: a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process for the sampled three-dimensional structure data by the interpolation step; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the interpolation process by referring to a first loss value that is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing methods.

[0170] (Appendix B9) The output data generating step includes a display step of displaying the generated output data. The information processing method described in Appendix B8.

[0171] (Appendix B10) The acquiring step acquires a medical image as the input data, The display step displays the output data to assist a medical professional in making a decision. 1. The information processing method described in Appendix B9.

[0172] (Appendix B12) Obtaining input data; generating three-dimensional structure data from the input data; applying a completion process to the three-dimensional structure data by an completion step; generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Including, The complementing step includes: generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the training data; applying an interpolation process to the sampled three-dimensional structure data by the interpolation step; performing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; The interpolation step is trained by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process. It was learned by Information processing methods.

[0173] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0174] (Appendix C1) A program that causes a computer to function as an information processing device, The computer An acquisition means for acquiring input data; a three-dimensional structure data generating means for generating three-dimensional structure data from the input data; a sampling means for generating sampled three-dimensional structure data by sampling the three-dimensional structure data; a complementation means for applying a complementation process to the sampled three-dimensional structure data; an estimation means for executing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; a first learning means for learning the interpolation means by referring to a first loss value which is a loss value related to the shape obtained by the shape estimation process; An information processing program that functions as a

[0175] (Appendix C2) the estimation means calculates one or more distance function values ​​in the shape estimation process; The first loss value is a loss value for the one or more distance function values. An information processing program as described in Appendix C1.

[0176] (Appendix C3) The computer The estimation means a feature conversion means for converting the intermediate feature in the complementation process into a latent variable; a shape estimation process that executes the shape estimation process by referring to the latent variables and the sampled three-dimensional structure data; to function as An information processing program according to appendix C1 or C2.

[0177] (Appendix C4) The complementing means comprises: an encoder that receives the sampled three-dimensional structure data and outputs the intermediate feature values; a decoder that receives the intermediate features and outputs the completed three-dimensional structure data; Equipped with The first learning means The encoder is trained by machine learning with reference to the first loss value. An information processing program according to any one of appendices C1 to C3.

[0178] (Appendix C5) The computer a second learning process for machine learning the encoder and the decoder by referring to a second loss value indicating a difference between the interpolated three-dimensional structure data output by the decoder and the three-dimensional structure data generated by the three-dimensional structure data generating means; 10. The information processing program according to claim 9, wherein the information processing program functions as

[0179] (Appendix C6) The computer the acquiring means further acquires information relating to a position and orientation of an imaging device when the imaging device captured the RGB image included in the input data; The information processing device includes: a rendering process that performs a rendering process by referring to the shape obtained by the shape estimation process and information about the position and orientation of the imaging device; The information processing program according to any one of appendices C1 to C5,

[0180] (Appendix C7) The computer a third learning means for executing a learning process by referring to a third loss value indicating a difference between a rendering image obtained by the rendering process and an RGB image included in the input data; Further function as An information processing program as described in Appendix C6.

[0181] (Appendix C8) The computer An acquisition means for acquiring input data; a three-dimensional structure data generating means for generating three-dimensional structure data from the input data; a complementation means for applying a complementation process to the three-dimensional structure data; an output data generation process for generating output data from the three-dimensional structure data to which the interpolation process has been applied; It functions as The complementing means comprises: a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process by the interpolation means for the three-dimensional structure data after the sampling; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the complementing means by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing program.

[0182] (Appendix C9) The computer The output data generating means functions as a display process for displaying the output data generated by the output data generating means. An information processing program as described in Appendix C8.

[0183] (Appendix C10) the acquisition means acquires a medical image as the input data; The display means displays the output data to assist a medical professional in making a decision. An information processing program as described in Appendix C9.

[0184] (Appendix C12) Obtaining input data; generating three-dimensional structure data from the input data; applying a complementation process to the three-dimensional structure data by a complementation means; generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Including, The complementing means comprises: generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the training data; applying an interpolation process to the sampled three-dimensional structure data by the interpolation means; performing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; training the interpolation means by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing program.

[0185] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0186] (Appendix D1) at least one processor, an acquisition process for acquiring input data; a three-dimensional structure data generation process for generating three-dimensional structure data from the input data; a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data; an interpolation process for applying an interpolation process to the sampled three-dimensional structure data; an estimation process that executes a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; a first learning process for learning the interpolation process by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; An information processing device that executes the above.

[0187] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0188] (Appendix D2) In the estimation process, the at least one processor calculates one or more distance function values ​​in the shape estimation process; The first loss value is a loss value for the one or more distance function values. 10. The information processing device according to claim 9, wherein the information processing device is a device for processing information.

[0189] (Appendix D3) In the estimation process, the at least one processor a feature transformation process for transforming the intermediate feature values ​​in the complementation process into latent variables; a shape estimation process that executes the shape estimation process by referring to the latent variables and the sampled three-dimensional structure data; Run An information processing device according to appendix D1 or D2.

[0190] (Appendix D4) In the complementing process, the at least one processor an encoder that receives the sampled three-dimensional structure data and outputs the intermediate feature values; a decoder that receives the intermediate features and outputs the completed three-dimensional structure data; Equipped with The first learning process includes: The encoder is trained by machine learning with reference to the first loss value. An information processing device according to any one of appendices D1 to D3.

[0191] (Appendix D5) The at least one processor: a second learning process for machine learning the encoder and the decoder by referring to a second loss value indicating a difference between the interpolated 3D structure data output by the decoder and the 3D structure data generated by the 3D structure data generation process; and The information processing device according to appendix D4,

[0192] (Appendix D6) In the acquisition process, the at least one processor further acquires information regarding a position and orientation of an image capturing device when the image capturing device captured the RGB image included in the input data; The information processing device includes: a rendering process that performs a rendering process by referring to the shape obtained by the shape estimation process and information about the position and orientation of the imaging device; The information processing device according to any one of appendices D1 to D5,

[0193] (Appendix D7) The at least one processor: the at least one processor: a third learning process that executes a learning process by referring to a third loss value that indicates a difference between the rendering image obtained by the rendering process and the RGB image included in the input data; Run the following again: 10. The information processing device according to claim 9, wherein the information processing device is an information processing device according to claim 8.

[0194] (Appendix D8) an acquisition process for acquiring input data; a three-dimensional structure data generation process for generating three-dimensional structure data from the input data; a complementation process for applying complementation to the three-dimensional structure data; an output data generation process for generating output data from the three-dimensional structure data to which the interpolation process has been applied; Prepare, execute, In the complementing process, the at least one processor a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; a complementation process for the sampled three-dimensional structure data by the complementation process; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the interpolation process by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing device.

[0195] (Appendix D9) The at least one processor: A display process is executed to display the output data generated by the output data generation process. 10. The information processing device according to claim 8,

[0196] (Appendix D10) In the acquisition process, the at least one processor acquires a medical image as the input data; In the display process, the at least one processor displays the output data to assist a medical professional in making a decision. 10. The information processing device according to claim 9,

[0197] (Appendix D12) The at least one processor: Obtaining input data; generating three-dimensional structure data from the input data; applying a completion process to the three-dimensional structure data; generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Including, In the complementing process, the at least one processor generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the training data; applying the interpolation process to the sampled three-dimensional structure data; performing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; learning the interpolation process by referring to a first loss value that is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing device.

[0198] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0199] (Appendix E1) A program that causes a computer to function as an information processing device, The computer, an acquisition process for acquiring input data; a three-dimensional structure data generation process for generating three-dimensional structure data from the input data; a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data; an interpolation process for applying an interpolation process to the sampled three-dimensional structure data; an estimation process that executes a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; a first learning process for learning the interpolation process by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; A non-transitory recording medium on which an information processing program for executing the above is recorded. [Explanation of symbols]

[0200] 1, 2, 100A, 100B, 100C ···Information processing equipment 11,21 ... Acquisition unit (acquisition means) 12, 22 Three-dimensional structure data generation unit (three-dimensional structure data generation means) 13,23 Complementary part (complementary means) 14. Sampling unit (sampling means) 15... Estimation unit (estimation means) 16 First learning unit (first learning means) 24 Output data generation unit (output data generation means)

Claims

1. An acquisition means for acquiring input data; a three-dimensional structure data generating means for generating three-dimensional structure data from the input data; a sampling means for generating sampled three-dimensional structure data by sampling the three-dimensional structure data; a complementation means for applying a complementation process to the sampled three-dimensional structure data; an estimation means for executing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; a first learning means for learning the interpolation means by referring to a first loss value, which is a loss value related to the shape obtained by the shape estimation process; An information processing device comprising:

2. the estimation means calculates one or more distance function values ​​in the shape estimation process, The first loss value is a loss value for the one or more distance function values. The information processing device according to claim 1 .

3. The estimation means a feature conversion means for converting the intermediate feature in the complementation process into a latent variable; a shape estimation means for executing the shape estimation process by referring to the latent variables and the sampled three-dimensional structure data; Equipped with The information processing device according to claim 2 .

4. The complementing means comprises: an encoder that receives the sampled three-dimensional structure data and outputs the intermediate feature amount; a decoder that receives the intermediate feature and outputs the completed three-dimensional structure data; Equipped with The first learning means The encoder is trained by machine learning with reference to the first loss value. The information processing device according to claim 1 .

5. a second learning means for performing machine learning on the encoder and the decoder by referring to a second loss value indicating a difference between the interpolated three-dimensional structure data output by the decoder and the three-dimensional structure data generated by the three-dimensional structure data generating means; The information processing device according to claim 4, further comprising:

6. An acquisition means for acquiring input data; a three-dimensional structure data generating means for generating three-dimensional structure data from the input data; a complementation means for applying a complementation process to the three-dimensional structure data; an output data generating means for generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Equipped with The complementing means comprises: a sampling process for generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; an interpolation process by the interpolation means for the three-dimensional structure data after the sampling; a shape estimation process that refers to intermediate feature amounts obtained in the complementation process; a first learning process for learning the complementing means by referring to a first loss value that is a loss value related to the shape obtained by the shape estimation process; It was learned by Information processing device.

7. Obtaining input data; generating three-dimensional structure data from the input data; generating sampled three-dimensional structure data by sampling the three-dimensional structure data; applying a complementation process to the sampled three-dimensional structure data by a complementation means; performing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; and causing the complementing means to learn by referring to a first loss value which is a loss value related to the shape obtained by the shape estimation process. An information processing method comprising:

8. Obtaining input data; generating three-dimensional structure data from the input data; applying a complementation process to the three-dimensional structure data by a complementation means; generating output data from the three-dimensional structure data to which the interpolation processing has been applied; Including, The complementing means comprises: generating sampled three-dimensional structure data by sampling the three-dimensional structure data generated from the learning data; applying an interpolation process to the sampled three-dimensional structure data by the interpolation means; performing a shape estimation process by referring to intermediate feature amounts obtained in the complementation process; and causing the complementing means to learn by referring to a first loss value which is a loss value related to the shape obtained by the shape estimation process. It was learned by Information processing methods.

9. A program for causing a computer to function as the information processing device described in claim 1, the program causing a computer to function as the acquisition means, the three-dimensional structure data generation means, the sampling means, the complementation means, the estimation means, and the first learning means.

10. 7. A program for causing a computer to function as the information processing device according to claim 6, the program causing a computer to function as the acquiring means, the three-dimensional structure data generating means, the complementing means, and the output data generating means.

Images