Information processing device, information processing method, and program

Abstract

This information processing device comprises: an acquiring means that acquires input data; a point cloud data generating means that generates three-dimensional point cloud data from the input data; a thinned point cloud data generating means that generates three-dimensional thinned point cloud data by using thinning processing on the input data and / or the three-dimensional point cloud data; and a training means that refers to the three-dimensional point point cloud data and the three-dimensional thinned point cloud data to train an estimation model that uses point cloud data as input and uses completed point cloud data and a segmentation label pertaining to the completed point cloud data as output.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program.

[0002] A technique is known for segmenting target data, such as point cloud data, by attaching a label to each data point included in the target data, indicating to which of multiple candidate regions the data point belongs (see, for example, Non-Patent Document 1).

[0003] Angela Dai et. al., ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans, 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City,UT, USA, 2018 pp. 4578-4587.

[0004] The technology described in Non-Patent Document 1 aims to improve classification accuracy by using data with missing synthetic data to learn both missing or occluded regions and semantic segmentation.

[0005] However, the technology described in Non-Patent Document 1 requires the preparation of synthetic data in advance, which increases the time and effort required to create and pre-process training data (i.e., the cost of training).

[0006] The present disclosure has been made in consideration of the above problems, and its purpose is to provide a highly accurate segmentation technology while suppressing increases in costs.

[0007] An information processing device according to one aspect of the present disclosure includes an acquisition means for acquiring input data, a point cloud data generation means for generating three-dimensional point cloud data from the input data, a thinned point cloud data generation means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and a learning means for training an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0008] An information processing device according to one aspect of the present disclosure includes an acquisition means for acquiring input data, a point cloud data generation means for generating three-dimensional point cloud data from the input data, and an estimation means for estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and the segmentation labels related to the complemented point cloud data, using an estimation model that takes the point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0009] An information processing method according to one aspect of the present disclosure includes acquiring input data, generating three-dimensional point cloud data from the input data, generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and training an estimation model that takes point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0010] An information processing method according to one aspect of the present disclosure includes acquiring input data, generating three-dimensional point cloud data from the input data, and estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the generating step and the segmentation label for the complemented point cloud data using an estimation model that takes the point cloud data as input and outputs complemented point cloud data and segmentation labels for the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0011] A program according to one aspect of the present disclosure causes a computer to execute an acquisition process for acquiring input data, a point cloud data generation process for generating three-dimensional point cloud data from the input data, a thinned point cloud data generation process for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and a learning process for training an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0012] A program according to one aspect of the present disclosure causes a computer to execute an acquisition process for acquiring input data, a point cloud data generation process for generating three-dimensional point cloud data from the input data, and an estimation process for estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation labels related to the complemented point cloud data using an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0013] According to the present disclosure, it is possible to provide a highly accurate segmentation technique while suppressing increases in costs.

[0014] FIG. 1 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 2 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 3 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 4 is a flow diagram showing a flow of an information processing method according to the present disclosure. FIG. 4 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 5 is a diagram for explaining a flow of data in an information processing device according to the present disclosure. FIG. 6 is a diagram for explaining a flow of data in an information processing device according to the present disclosure. FIG. 7 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 8 is a diagram for explaining a flow of data in an information processing device according to the present disclosure. FIG. 9 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 10 is a diagram for explaining a process executed by an information processing device according to the present disclosure. FIG. 11 is a block diagram showing a configuration of an information processing device according to the present disclosure. FIG. 12 is a block diagram showing a hardware configuration of an information processing device according to the present disclosure.

[0015] The following are examples of embodiments of the present invention. However, the present invention is not limited to the following exemplary embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the following exemplary embodiments may also be included in the scope of the present invention. Alternatively, embodiments obtained by appropriately omitting some of the technical means employed in the following exemplary embodiments may also be included in the scope of the present invention. Furthermore, the effects described in the following exemplary embodiments are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects described in the following exemplary embodiments may also be included in the scope of the present invention.

[0016] [First Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for each of the exemplary embodiments described below. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0017] (Configuration of Information Processing Device 1) The configuration of the information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, a point cloud data generation unit 12, a thinned point cloud data generation unit 13, and a learning unit 14.

[0018] (Acquisition Unit 11) The acquisition unit 11 acquires input data. Here, the input data is, for example, input data for the learning phase. Specific examples of the input data acquired by the acquisition unit 11 do not limit the present exemplary embodiment, but may include, for example, at least one of: RGB data in which each pixel (data point) represents an RGB value; depth data in which each pixel (data point) represents a depth value; and three-dimensional point cloud data in which each data point represents three-dimensional coordinates. The three-dimensional point cloud data may be, for example, point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, but this example does not limit the present exemplary embodiment.

[0019] (Point Cloud Data Generator 12) The point cloud data generator 12 generates three-dimensional point cloud data from the input data acquired by the acquisition unit 11. As an example, the point cloud data generator 12 generates three-dimensional point cloud data from the input data acquired by the acquisition unit 11, the input data including at least one of RGB data and depth data. As an example, the point cloud data generator 12 may be configured to: identify the three-dimensional coordinates of each pixel included in the RGB data by referencing the depth data of each pixel in the depth data; and generate three-dimensional point cloud data in which the identified three-dimensional coordinates are assigned to each pixel (each data point). Here, the three-dimensional point cloud data may be configured to include feature amounts of each data point in addition to the three-dimensional coordinates assigned to each data point. Here, each feature amount may be configured to include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point. Alternatively, the point cloud data generation unit 12 may be configured to generate attribute data including feature quantities of each data point (e.g., at least one of RGB values ​​and normal values ​​(normal vectors)) in association with three-dimensional point cloud data including the three-dimensional coordinates of each data point.

[0020] Furthermore, the point cloud data generator 12 may generate the above-mentioned three-dimensional point cloud data using an algorithm such as Structure from Motion (SfM) or Simultaneous Localization and Mapping (SLAM).

[0021] With the above-described configuration, the point cloud data generator 12 can generate the above-described three-dimensional point cloud data from one or more frames (one or more data sets) included in the input data acquired by the acquisition unit 11.

[0022] Furthermore, when the acquisition unit 11 acquires input data including three-dimensional point cloud data, the point cloud data generation unit 12 may be configured to output the three-dimensional point cloud data as is. Alternatively, the point cloud data generation unit 12 may be configured to output the three-dimensional point cloud data after including the above-mentioned feature amounts of each data point (for example, at least one of RGB values ​​and normal values ​​(normal vectors)). Alternatively, the point cloud data generation unit 12 may be configured to output attribute data including the above-mentioned feature amounts of each data point together with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

[0023] Note that, when the acquisition unit 11 acquires input data including three-dimensional point cloud data and the point cloud data generation unit 12 outputs the three-dimensional point cloud data as is, the information processing device 1 may be configured without the point cloud data generation unit 12. Such a configuration is also included in this exemplary embodiment.

[0024] (Thinned Point Cloud Data Generator 13) The thinned point cloud data generator 13 generates three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data acquired by the acquisition unit 11 and the three-dimensional point cloud data generated by the point cloud data generator 12. Here, the thinning process may, for example, include: generating the three-dimensional thinned point cloud data using only some of the frames included in the input data. Alternatively, the thinning process may be configured to include: generating the three-dimensional thinned point cloud data using only some of the data points included in at least one of the input data and the three-dimensional point cloud data.

[0025] (Learning Unit 14) The learning unit 14 trains an estimation model that receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data. As an example, the learning unit 14 trains the estimation model by referring to the 3D point cloud data and the 3D thinned point cloud data. Note that the expression "completion" in this exemplary embodiment is derived from "point cloud completion," as an example, but this term does not limit this exemplary embodiment.

[0026] More specifically, as an example, the learning unit 14: - refers to training data including the three-dimensional point cloud data and the three-dimensional thinned point cloud data; - inputs the three-dimensional thinned point cloud data included in the training data into the estimation model, thereby causing the estimation model to generate three-dimensional point cloud data; and - performs machine learning on the estimation model so that the difference between the three-dimensional point cloud data generated by the estimation model and the three-dimensional point cloud data included in the training data is small.

[0027] Here, the 3D point cloud data included in the training data may include correct labels (also referred to as true labels) related to segmentation, or the correct labels related to segmentation may be associated with (associated with) the 3D point cloud data included in the training data. The learning unit 14 may be configured to train the estimation model so that the difference between the segmentation labels output (estimated) by the estimation model and the correct labels related to segmentation is small. Here, the correct labels may be included in the input data or associated with the input data, for example, and the acquiring unit 11 may acquire the correct labels.

[0028] The learning unit 14 configured as described above can preferably learn the estimation model that receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data. Note that the term "interpolation" in the above description includes the interpolation of so-called missing regions and the interpolation of occlusion regions, but this term does not limit this exemplary embodiment.

[0029] (Effects of Information Processing Device 1) As described above, the information processing device 1 is configured to: acquire input data; generate three-dimensional point cloud data from the input data; generate three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; and train an estimation model that uses point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. The information processing device 1 configured as described above generates three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and trains the estimation model by referring to the three-dimensional thinned point cloud data. This makes it possible to generate a highly accurate estimation model while suppressing increases in costs. Therefore, the above configuration makes it possible to provide a highly accurate segmentation technology while suppressing increases in costs.

[0030] (Additional Notes Regarding Information Processing Device 1) As described above, when the acquisition unit 11 is configured to acquire three-dimensional point cloud data, the information processing device 1 may not be configured to include the point cloud data generation unit 12. In other words, the information processing device 1 may be configured to include: an acquisition unit 11 that acquires input data including three-dimensional point cloud data; a thinned point cloud data generation unit 13 that generates three-dimensional thinned point cloud data by applying a thinning process to the input data; and a learning unit 14 that receives point cloud data as input, and trains an estimation model that outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. The information processing device 1 configured in this manner can also achieve the effects described above.

[0031] (Flow of Information Processing Method S1) Next, the flow of information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of information processing method S1. As shown in Fig. 2, information processing method S1 includes a step (process) S11 of acquiring input data, a step (process) S12 of generating three-dimensional point cloud data, a step (process) S13 of generating three-dimensional thinned point cloud data, and a step (process) S14 of training an estimation model.

[0032] (Step S11) In step S11, the acquiring unit 11 acquires input data. The specific processing by the acquiring unit 11 has been described above, and therefore will not be described here.

[0033] (Step S12) Subsequently, in step S12, the point cloud data generation unit 12 generates three-dimensional point cloud data from the input data acquired by the acquisition unit 11. The specific processing by the point cloud data generation unit 12 has been described above, and therefore will not be described here.

[0034] (Step S13) Subsequently, in step S13, the thinned point cloud data generator 13 generates three-dimensional thinned point cloud data by applying thinning processing to at least one of the input data acquired by the acquisition unit 11 and the three-dimensional point cloud data generated by the point cloud data generator 12. The specific processing by the thinned point cloud data generator 13 has been described above, and therefore will not be described here.

[0035] (Step S14) Subsequently, in step S14, the learning unit 14 trains an estimation model to which point cloud data is input, the estimation model outputting interpolated point cloud data and segmentation labels related to the interpolated point cloud data. As an example, the learning unit 14 trains the estimation model by referring to the 3D point cloud data and the 3D thinned point cloud data. Specific processing by the learning unit 14 has been described above, and therefore will not be described here.

[0036] (Effects of Information Processing Method S1) As described above, the information processing method S1 includes the following configuration: acquiring input data; generating three-dimensional point cloud data from the input data; applying a thinning process to at least one of the input data and the three-dimensional point cloud data to generate three-dimensional thinned point cloud data; and training an estimation model that uses point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data. According to the information processing method S1 configured as described above, the three-dimensional thinned point cloud data is generated by applying a thinning process to at least one of the input data and the three-dimensional point cloud data, and the estimation model is trained by referring to the three-dimensional thinned point cloud data. This makes it possible to generate a highly accurate estimation model while suppressing increases in costs. Therefore, the above configuration makes it possible to provide a highly accurate segmentation technology while suppressing increases in costs.

[0037] (Additional Notes Regarding Information Processing Method S1) In a configuration in which the acquisition unit 11 acquires three-dimensional point cloud data in step S11, the information processing method S1 may not include step S12. In other words, the information processing method S1 may include: an acquisition step S11 in which input data including three-dimensional point cloud data is acquired; a thinned point cloud data generation step S13 in which three-dimensional thinned point cloud data is generated by applying a thinning process to the input data; and a learning step S14 in which an estimation model is trained, with reference to the three-dimensional point cloud data and the three-dimensional thinned point cloud data, and the estimation model receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data. The information processing method S1 configured in this manner can also achieve the effects described above.

[0038] (Configuration of Information Processing Device 2) Next, the configuration of the information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2. As shown in Fig. 3, the information processing device 2 includes an acquisition unit 21, a point cloud data generation unit 22, and an estimation unit 23.

[0039] (Acquisition Unit 21) The acquisition unit 11 acquires input data. Here, the input data is, for example, input data for the inference phase (estimation phase, test phase). Specific examples of the input data acquired by the acquisition unit 21 do not limit the present exemplary embodiment, but may include, for example, at least one of: RGB data in which each pixel (data point) represents an RGB value; depth data in which each pixel (data point) represents a depth value; and three-dimensional point cloud data in which each data point represents three-dimensional coordinates. The three-dimensional point cloud data may be, for example, point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, but this example does not limit the present exemplary embodiment.

[0040] In this way, the acquisition unit 21 can be configured to acquire data in the same format as the input data acquired by the acquisition unit 11 provided in the information processing device 1 described above, but the acquisition unit 21 does not need to acquire the correct label regarding the segmentation described in the information processing device 1.

[0041] (Point Cloud Data Generator 22) The point cloud data generator 12 generates three-dimensional point cloud data from the input data acquired by the acquirer 21.

[0042] As an example, the point cloud data generation unit 22 generates three-dimensional point cloud data from input data acquired by the acquisition unit 21, the input data including at least one of RGB data and depth data. As an example, the point cloud data generation unit 22 may be configured to: identify the three-dimensional coordinates of each pixel included in the RGB data by referencing the depth data of each pixel in the depth data; and generate three-dimensional point cloud data in which the identified three-dimensional coordinates are assigned to each pixel (each data point). Here, the three-dimensional point cloud data may be configured to include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point in addition to the three-dimensional coordinates assigned to each data point. Alternatively, attribute data including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point may be generated in association with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

[0043] Furthermore, when the acquisition unit 21 acquires input data including three-dimensional point cloud data, the point cloud data generation unit 22 may be configured to output the three-dimensional point cloud data as is. Alternatively, the point cloud data generation unit 22 may be configured to output the three-dimensional point cloud data after including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point described above. Alternatively, the point cloud data generation unit 22 may be configured to output attribute data including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point described above, along with the three-dimensional point cloud data including the three-dimensional coordinates of each data point.

[0044] Note that, when the acquisition unit 21 acquires input data including three-dimensional point cloud data and the point cloud data generation unit 22 outputs the three-dimensional point cloud data as is, the information processing device 2 may be configured without including the point cloud data generation unit 22. Such a configuration is also included in this exemplary embodiment.

[0045] In this way, the point cloud data generation unit 22 can be configured to perform processing similar to that of the point cloud data generation unit 12 provided in the information processing device 1, as an example, but this does not limit this exemplary embodiment.

[0046] (Estimation unit 23) The estimation unit 23 uses a machine-learned estimation model to estimate interpolated point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation unit 22 and segmentation labels related to the interpolated point cloud data. Here, as one example, the machine-learned estimation model may be an estimation model trained by the learning unit 14 included in the information processing device 1. In other words, as one example, the machine-learned estimation model is an estimation model that receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, and is machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0047] (Effects of Information Processing Device 2) As described above, the information processing device 2 is configured to: acquire input data; generate three-dimensional point cloud data from the input data; and estimate the interpolated point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation labels related to the interpolated point cloud data using an estimation model that takes the point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, the estimation model being machine-learned with reference to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data. According to the information processing device 2 configured as described above, the interpolated point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation labels related to the interpolated point cloud data are estimated using the estimation model that was machine-learned with reference to the three-dimensional thinned point cloud data generated by applying a thinning process to the three-dimensional point cloud data, thereby making it possible to perform estimation processing using a highly accurate estimation model while suppressing increases in cost. Therefore, according to the above configuration, it is possible to provide a highly accurate segmentation technique while suppressing increases in costs.

[0048] (Additional Notes Regarding Information Processing Device 2) As described above, when the acquisition unit 21 is configured to acquire three-dimensional point cloud data, the information processing device 2 may not include the point cloud data generation unit 22. In other words, the information processing device 2 may include: an acquisition unit 21 that acquires input data including three-dimensional point cloud data; and an estimation unit 23 that estimates the interpolated point cloud data corresponding to the three-dimensional point cloud data acquired by the acquisition unit 21 and the segmentation labels related to the interpolated point cloud data, using an estimation model that receives the point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, the estimation model being machine-learned with reference to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data. The information processing device 2 configured in this manner can also achieve the above-described effects.

[0049] (Flow of Information Processing Method S2) Next, the flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S2. As shown in Fig. 4, the information processing method S2 includes a step (process) S21 of acquiring input data, a step (process) S22 of generating three-dimensional point cloud data, and a step (process) S23 of performing estimation using an estimation model.

[0050] (Step S21) In step S21, the acquiring unit 21 acquires input data. The specific processing by the acquiring unit 21 has been described above, and therefore will not be described here.

[0051] (Step S22) Subsequently, in step S22, the point cloud data generation unit 22 generates three-dimensional point cloud data from the input data acquired by the acquisition unit 21. The specific processing by the point cloud data generation unit 22 has been described above, and therefore will not be described here.

[0052] (Step S23) Subsequently, in step S23, the estimation unit 23 uses a machine-learned estimation model to estimate interpolated point cloud data corresponding to the 3D point cloud data generated in step S22 and segmentation labels related to the interpolated point cloud data. Here, as one example, the machine-learned estimation model can be the estimation model trained by the information processing method S1 described above. In other words, as one example, the machine-learned estimation model is an estimation model that receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, and is machine-learned by referring to the 3D point cloud data and 3D thinned point cloud data obtained by applying a thinning process to the 3D point cloud data.

[0053] (Effects of Information Processing Method S2) As described above, information processing method S2 employs the following configuration: acquire input data; generate three-dimensional point cloud data from the input data; and estimate the interpolated point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation labels related to the interpolated point cloud data using an estimation model that takes the point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, the estimation model being machine-learned with reference to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data. According to information processing method S2 configured as described above, the interpolated point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation labels related to the interpolated point cloud data are estimated using the estimation model machine-learned with reference to the three-dimensional thinned point cloud data generated by applying a thinning process to the three-dimensional point cloud data. This makes it possible to perform estimation processing using a highly accurate estimation model while suppressing increases in cost. Therefore, according to the above configuration, it is possible to provide a highly accurate segmentation technique while suppressing increases in costs.

[0054] (Additional Notes Regarding Information Processing Method S2) In the case where three-dimensional point cloud data is acquired in step S21, information processing method S2 may not include step S22. In other words, information processing method S2 may include: an acquisition step S21 of acquiring input data including three-dimensional point cloud data; and an estimation step S23 of estimating the interpolated point cloud data corresponding to the three-dimensional point cloud data acquired in the acquisition step S21 and the segmentation labels related to the interpolated point cloud data, using an estimation model that receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, the estimation model being machine-learned with reference to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data. The information processing method S2 configured in this manner can also achieve the effects described above.

[0055] [Second Embodiment] A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0056] (Configuration of Information Processing Device 1A) The configuration of the information processing device 1A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 1A. As shown in Fig. 5, the information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30, and an input / output unit 40.

[0057] (Communication Unit 30) The communication unit 30 communicates with devices external to the information processing device 1A via a network. As an example, the communication unit 30 transmits data supplied from the control unit 10A to the external device, and supplies data received from the external device to the control unit 10A. Note that the specific configuration of the network does not limit this exemplary embodiment, and examples include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks.

[0058] (Input / Output Unit 40) The input / output unit 40 is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input / output unit 40 may be configured to be connected to input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. In this configuration, the input / output unit 40 accepts various types of information input to the information processing device 1A from the connected input devices. Furthermore, the input / output unit 40 outputs various types of information to connected output devices under the control of the control unit 10A. An example of the input / output unit 40 is an interface such as a USB (Universal Serial Bus).

[0059] (Storage Unit 20A) The storage unit 20A stores various types of data referenced by the control unit 10A and various types of data generated by the control unit 10A. As an example, the storage unit 20A stores: Input data IND; Three-dimensional point cloud data PCD; Three-dimensional thinned point cloud data TPCD; First feature F1; Second feature F2; Estimation model PM; Segmentation data SG. Here, the input data IND is data acquired by an acquisition unit 11 (21) described later. Specific examples of the input data IND will be described later. The three-dimensional point cloud data PCD is data generated by a point cloud data generation unit 12 (22) described later. Specific examples of the three-dimensional point cloud data PCD will be described later. The three-dimensional thinned point cloud data TPCD is data generated by a thinned point cloud data generation unit 13 described later. Specific examples of the three-dimensional thinned point cloud data TPCD will be described later.

[0060] The first feature F1 is data referenced by a first feature selection unit 151, which will be described later. Specific examples of the first feature F1 will be described later. The second feature F2 is data referenced by a second feature selection unit 152, which will be described later. Specific examples of the second feature F2 will be described later.

[0061] The estimation model PM is a model for inference that is machine-learned by the learning unit 14 (described later) and referenced by the estimation unit 23. The storage unit 20A stores various programs and parameters that constitute the estimation model PM. Specific examples of the estimation model PM will be described later.

[0062] 5, the control unit 10A includes the acquisition unit 11, point cloud data generation unit 12, thinned point cloud data generation unit 13, learning unit 14, and estimation unit 23 described in exemplary embodiment 1. Here, the acquisition unit 11 can also be expressed as having the same configuration as the acquisition unit 21 described in exemplary embodiment 1, and therefore the acquisition unit 11 may also be referred to as the acquisition unit 11 (21). Furthermore, the point cloud data generation unit 12 can also be expressed as having the same configuration as the point cloud data generation unit 22 described in exemplary embodiment 1, and therefore the point cloud data generation unit 12 may also be referred to as the point cloud data generation unit 12 (22).

[0063] 5, the control unit 10A includes a feature quantity selection unit 15. Here, as shown in FIG. 5, the feature quantity selection unit 15 includes a first feature quantity selection unit 151 and a second feature quantity selection unit 152.

[0064] (Acquisition Unit 11 (21)) The acquisition unit 11 (21) acquires input data IND. Here, the acquisition unit 11 acquires input data for learning in the learning phase, and acquires input data for inference in the inference phase (estimation phase, test phase). Specific examples of the input data acquired by the acquisition unit 11 (21) do not limit this exemplary embodiment, but as an example, similar to the exemplary embodiment, the input data may be configured to include at least any of the following: RGB data in which each pixel (data point) represents an RGB value; depth data in which each pixel (data point) represents a depth value; and three-dimensional point cloud data in which each data point represents three-dimensional coordinates. Furthermore, the three-dimensional point cloud data may be point cloud data acquired by a LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) device, but this example does not limit this exemplary embodiment.

[0065] As an example, the acquisition unit 11 (21) can be configured to acquire input data including correct labels for segmentation (input data in which correct labels for segmentation are attached to each data point) in the learning phase, and to acquire input data that does not include the correct labels in the inference phase.

[0066] The input data IND acquired by the acquisition unit 11 (21) is stored in the memory unit 20A, for example, and is referenced by the point cloud data generation unit 12 (22), the thinned point cloud data generation unit 13, the learning unit 14, the estimation unit 23, etc.

[0067] (Point Cloud Data Generator 12 (22)) The point cloud data generator 12 (22) generates three-dimensional point cloud data PCD from the input data IND acquired by the acquisition unit 11 (21). As an example, the point cloud data generator 12 (22) generates three-dimensional point cloud data PCD from the input data IND acquired by the acquisition unit 11 (21), the input data IND including at least one of RGB data and depth data. As an example, the point cloud data generator 12 (22) can be configured to: - identify the three-dimensional coordinates of each pixel included in the RGB data by referencing the depth data of each pixel in the depth data; and - generate three-dimensional point cloud data PCD in which the identified three-dimensional coordinates are assigned to each pixel (each data point). Here, the three-dimensional point cloud data PCD may be configured to include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point in addition to the three-dimensional coordinates assigned to each data point. Alternatively, attribute data including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point may be generated in association with three-dimensional point cloud data including the three-dimensional coordinates of each data point.

[0068] Furthermore, when the acquisition unit 11 (21) acquires input data including three-dimensional point cloud data, the point cloud data generation unit 12 (22) may be configured to output the three-dimensional point cloud data as is as the three-dimensional point cloud data PCD. Alternatively, the acquisition unit 11 (21) may be configured to include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point in the three-dimensional point cloud data acquired and then output the three-dimensional point cloud data PCD. Alternatively, the acquisition unit 11 (21) may be configured to output attribute data including at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point, along with the three-dimensional point cloud data PCD including the three-dimensional coordinates of each data point.

[0069] Furthermore, when the acquisition unit 11 (21) acquires multiple frames of input data IND including at least one of RGB data, depth data RGB, and three-dimensional point cloud data, the point cloud data generation unit 12 (22) can be configured to, as an example, generate three-dimensional point cloud data PCD by referring to all frames included in the input data IND.

[0070] Furthermore, in a configuration in which the acquisition unit 11 (21) acquires input data including three-dimensional point cloud data and the point cloud data generation unit 12 (22) outputs the three-dimensional point cloud data included in the input data as three-dimensional point cloud data PCD as is, the information processing device 1A may be configured without including the point cloud data generation unit 12 (22). Such a configuration is also included in this exemplary embodiment.

[0071] (Thinned Point Cloud Data Generator 13) As in the first exemplary embodiment, the thinned point cloud data generator 13 generates three-dimensional thinned point cloud data TPCD by applying a thinning process to at least one of the input data IND acquired by the acquisition unit 11 (21) and the three-dimensional point cloud data PCD generated by the point cloud data generator 12 (22). Here, the thinning process may include, for example, the following: - A process of generating the three-dimensional thinned point cloud data TPCD using only a portion of the multiple frames included in the input data IND. More specifically, the thinning process may include, for example, a process of generating the three-dimensional thinned point cloud data TPCD using only N% (N is a real number less than 100) of the multiple frames included in the input data IND. Here, the thinned point cloud data generation unit 13 may perform the following processes to select the above N% of frames: - Randomly select frames to be removed - Remove consecutive frames together so that a specific area is missing from the image or object indicated by the input data IND.

[0072] Alternatively, the thinning process may include: a process of generating the three-dimensional thinned point cloud data TPCD using only some of the data points included in at least one of the input data IND and the three-dimensional point cloud data PCD. More specifically, the thinning process may include at least one of the following: a process of reducing the resolution of a depth image included in the input data IND; and a process of applying a thinning method to the three-dimensional point cloud data included in the input data IND such that the density of the light receiving sensors of the LiDAR device is reduced.

[0073] (Learning Unit 14) As in the exemplary embodiment 1, the learning unit 14 trains an estimation model PM to which the point cloud data IND is input, the estimation model outputting interpolated point cloud data and segmentation labels related to the interpolated point cloud data. As an example, the learning unit 14 trains the estimation model PM by referring to the three-dimensional point cloud data PCD and the three-dimensional thinned point cloud data TPCD.

[0074] More specifically, as an example, the learning unit 14: - refers to teacher data TD including the three-dimensional point cloud data PCD1 and three-dimensional thinned point cloud data TPCD1 obtained by thinning the three-dimensional point cloud data PCD1; - inputs the three-dimensional thinned point cloud data TPCD1 included in the teacher data TD into the estimation model PM, thereby causing the estimation model PM to generate three-dimensional point cloud data PCD2 (sometimes referred to as complemented three-dimensional point cloud data IPCD1); - performs machine learning on the estimation model PM so that the difference between the three-dimensional point cloud data PCD2 generated by the estimation model PM and the three-dimensional point cloud data PCD1 included in the teacher data TD is reduced.

[0075] Here, the 3D point cloud data PCD1 included in the training data TD may include a correct label GTL1 for segmentation, or the correct label GTL1 for segmentation may be associated with the 3D point cloud data PCD1 included in the training data TD. The learning unit 14 may then train the estimation model PM so that the difference between the segmentation label PL1 output (estimated) by the estimation model PM and the correct label GTL1 for segmentation is small. Here, the correct label GTL1 may be included in the input data IND or associated with the input data IND, and the acquisition unit 11 may acquire the correct label GTL1.

[0076] In addition, the learning unit 14 may be configured to learn the estimation model PM by further referring to a first feature F1 associated with at least one of the input data IND, the three-dimensional point cloud data PCD, and the three-dimensional thinned point cloud data TPCD.

[0077] Here, although a specific example of the first feature F1 does not limit this exemplary embodiment, as an example, the first feature F1 can be at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD. A specific method of training the estimation model PM with reference to the first feature F1 will be described later.

[0078] In this exemplary embodiment, the estimation model PM includes, as an example: a first estimation model PM1 that receives point cloud data as input and outputs interpolated point cloud data, and a second estimation model PM2 that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data. A more specific configuration example of the estimation model PM will be described later with reference to different drawings.

[0079] According to the learning unit 14 configured as described above, it is possible to suitably learn the estimation model PM, which takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data.

[0080] Furthermore, the learning unit 14 configured as described above learns the estimation model PM by referring to the three-dimensional thinned point cloud data TPCD generated by applying a thinning process to at least one of the input data IND and the three-dimensional point cloud data PCD, so that it is possible to generate a highly accurate estimation model while suppressing an increase in cost. Therefore, the above configuration makes it possible to provide a highly accurate segmentation technique while suppressing an increase in cost.

[0081] (Estimation Unit 23) As in the second exemplary embodiment, the estimation unit 23 uses a machine-learned estimation model to estimate interpolated point cloud data corresponding to the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22) and segmentation labels related to the interpolated point cloud data. Here, as an example, the machine-learned estimation model may be the estimation model PM trained by the learning unit 14. In other words, as an example, the machine-learned estimation model is an estimation model that receives point cloud data as input and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, and is the estimation model PM trained by machine learning with reference to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0082] In addition, the estimation unit 23 may be configured to further refer to a second feature F2 associated with at least one of the input data IND and the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22), and estimate complemented point cloud data corresponding to the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22) and segmentation labels related to the complemented point cloud data.

[0083] Here, although specific examples of the second feature F2 do not limit this exemplary embodiment, as an example, the second feature F2 can be at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD. The type of the second feature F2 (e.g., RGB values ​​or normal values) can be the same type as the type of the first feature F1 described above. A specific method of estimation with reference to the second feature F2 will be described later.

[0084] Furthermore, data including the "complemented point cloud data" and "segmentation labels related to the complemented point cloud data" estimated (generated) by the estimation unit 23 is stored in the storage unit 20A as segmentation data SD shown in Fig. 5, for example, and is provided to the outside of the information processing device 100A via the communication unit 30, the input / output unit 40, etc. In this exemplary embodiment, the segmentation data SD may also be expressed as a segmentation image, a segmentation result, etc.

[0085] As an example, the estimation unit 23 also functions as a presentation means that presents a segmentation image including objects having display colors and display textures corresponding to the segmentation labels estimated by the estimation unit 23 to the user via a display provided in the input / output unit 40.

[0086] The estimation unit 23 configured as described above performs estimation processing using the estimation model PM that has been trained with reference to the three-dimensional thinned point cloud data TPCD generated by applying thinning processing to at least one of the input data IND and the three-dimensional point cloud data PCD. Therefore, highly accurate estimation can be performed while suppressing increases in cost. Therefore, the above configuration makes it possible to provide a highly accurate segmentation technique while suppressing increases in cost.

[0087] 5, the feature quantity selection unit 15 includes a first feature quantity selection unit 151 and a second feature quantity selection unit 152. The first feature quantity selection unit 151, for example, selects a first feature quantity F1 to be referenced when the learning unit 14 learns the estimation model PM. On the other hand, the second feature quantity selection unit 152, for example, selects a second feature quantity F2 to be referenced when the estimation unit 23 performs estimation processing using the estimation model PM.

[0088] Specific examples of the first feature amount F1 and the second feature amount F2 include at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD. Specific examples of the processes performed by the first feature amount selection unit 151 and the second feature amount selection unit 152 will be described later with reference to different drawings.

[0089] (Data Flow in Learning Phase) Next, the data flow in the learning phase of the information processing device 1A will be described with reference to FIGS.

[0090] FIG. 6 is a diagram illustrating an example of the flow of data in the learning phase of the information processing device 1A. As shown in FIG. 6, first, the acquisition unit 11 acquires input data IND. Here, the input data IND is, for example, input data for learning, and each data point included in the input data is assigned, for example, a correct answer label GTL1 for segmentation. Then, the point cloud data generation unit 12 generates three-dimensional point cloud data PCD by referring to the input data IND. Here, the generated three-dimensional point cloud data PCD is supplied to the thinned point cloud data generation unit 13 and the first estimation model learning unit 141 provided in the learning unit 14.

[0091] On the other hand, the thinned point cloud data generation unit 13, for example, generates three-dimensional thinned point cloud data TPCD from the three-dimensional point cloud data PCD, and supplies the generated three-dimensional thinned point cloud data TPCD to the first estimation model learning unit 141. Here, the first estimation model learning unit 141 is configured to learn the first estimation model PM1 included in the estimation model PM.

[0092] In addition, the thinned point cloud data generation unit 13 associates at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point included in (or associated with) the input data IND or the three-dimensional point cloud data PCD, and the RGB values ​​and normal values ​​(normal vectors) corresponding to each data point included in the three-dimensional thinned point cloud data TPCD generated by the thinned point cloud data generation unit 13, with the coordinates of the data point, and stores them in the memory unit 20A as a first feature F1.

[0093] The first estimation model learning unit 141 inputs the three-dimensional thinned point cloud data TPCD1 generated by the thinned point cloud data generation unit 13 to the first estimation model PM1, acquires the interpolated three-dimensional point cloud data IPCD1 output by the first estimation model PM1, and supplies the interpolated three-dimensional point cloud data IPCD1 to the second estimation model learning unit 142.

[0094] In addition, the first estimation model learning unit 141 calculates a first loss value (first loss function) (also referred to as BCE_Loss) indicating the difference between the above-mentioned complemented three-dimensional point cloud data IPCD1 and the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12.

[0095] Furthermore, the first estimation model learning unit 141 supplies the coordinates of each data point included in the interpolated three-dimensional point cloud data IPCD1 to the first feature quantity selecting unit 151. Then, the first feature quantity selecting unit 151: compares the coordinates associated with the first feature quantities F1 stored in the memory unit 20A with the coordinates of each data point included in the interpolated three-dimensional point cloud data IPCD1, identifies the first feature quantities F1, among the first feature quantities F1 stored in the memory unit 20A, that correspond to each data point included in the interpolated three-dimensional point cloud data IPCD1, and supplies the identified first feature quantities F1 to the second estimation model learning unit 142.

[0096] The second estimation model learning unit 142 inputs the following into the second estimation model PM2, and obtains the estimated label PL1 related to the segmentation output by the second estimation model PM2: the complemented 3D point cloud data IPCD1 calculated by the first estimation model learning unit 141 using the first estimation model PM1; and the first feature F1 supplied from the first feature selection unit 151.

[0097] In addition, the second estimation model learning unit 142 calculates a second loss value (second loss function) (also referred to as CE_Loss) indicating the difference between the above-mentioned estimated label PL1 and, as an example, the correct label GTL1 for the segmentation acquired by the acquisition unit 11.

[0098] The learning unit 14 then calculates a total loss value Loss_total by linearly adding the first loss value BCE_Loss and the second loss value CE_Loss, Loss_total = CE_Loss + α × BCE_Loss, and updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the loss value Loss_total becomes smaller, where α is a weighting coefficient that can be set as appropriate.

[0099] In this way, the learning unit 14 refers to the integrated loss value Loss_total and updates (trains) the first estimation model PM1 and the second estimation model PM2, thereby suitably improving the accuracy of the complementation process and the accuracy of the segmentation process. Note that updating (training) the first estimation model PM1 and the second estimation model PM2 by referring to the integrated loss value Loss_total in this way can also be expressed as "simultaneously training the first estimation model PM1 and the second estimation model PM2."

[0100] In this way, the learning unit 14 trains a first estimation model PM that receives point cloud data as input and outputs complemented point cloud data, and a second estimation model PM2 that receives at least the complemented point cloud data as input and outputs segmentation labels related to the complemented point cloud data.

[0101] 7 is a diagram illustrating an example of the flow of data related to the network in the learning phase. As shown in Fig. 7, at least one of the acquisition unit 11 and the point cloud data generation unit 12 acquires training data TD including three-dimensional point cloud data PCD1 and ground truth labels GTL1 (also referred to as true value labels GTL1 in Fig. 7) assigned to each data point of the three-dimensional point cloud data PCD1.

[0102] Next, the three-dimensional point cloud data BPCD1, from which attribute data such as RGB values ​​and normal values ​​have been removed, is input to the thinned point cloud data generator 13, and three-dimensional thinned point cloud data BTPCD1 is generated by the thinned point cloud data generator 13. The three-dimensional thinned point cloud data BTPCD1 is then input to a first estimation model PM1 (also referred to as an interpolation network in FIG. 7).

[0103] The first estimation model PM1 then outputs the interpolated three-dimensional point cloud data IPCD1. The interpolated three-dimensional point cloud data IPCD1 is input to the second estimation model PM2 (also referred to as a semantic segmentation network in FIG. 7 ) together with the first feature F1 (referred to as first feature data F1 in FIG. 7 ). Here, the first feature F1 is the first feature F1 selected by the first feature selection unit 151 described above. The first feature F1 is input to the second estimation model PM2 in the form of a mask feature, for example. Here, the mask feature refers to a data group in which data points containing the first feature F1 selected by the first feature selection unit 151 have the value of the feature, and other data points are assigned a null feature (a predetermined value such as 0, 255, or −1 as the feature value).

[0104] Further, the difference between the interpolated three-dimensional point cloud data IPCD1 and the above-mentioned three-dimensional point cloud data BPCD1 is calculated as a first loss value (BCE_Loss).

[0105] The second estimation model PM2 then outputs an estimated label PL1 for the segmentation of each data point in the interpolated 3D point cloud data IPCD1. The difference between the estimated label PL1 and the ground truth label GTL1 is calculated as a second loss value (CE_Loss).

[0106] As described above, the learning unit 14 updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the total loss value represented by the first loss value and the second loss value becomes smaller.

[0107] (Data Flow in Estimation Phase) Next, the data flow in the estimation phase of the information processing device 1A will be described with reference to FIGS. 8 and 9. FIG.

[0108] FIG. 8 is a diagram illustrating an example of the flow of data in the estimation phase of the information processing device 1A. As shown in FIG. 8, first, the acquisition unit 21 acquires input data IND. Here, the input data IND is, as an example, input data for estimation. Unlike the input data for learning described above, each data point included in the input data is not assigned a correct answer label GTL1 for segmentation. Then, the point cloud data generation unit 22 generates three-dimensional point cloud data PCD by referring to the input data IND. Here, the generated three-dimensional point cloud data PCD is supplied to a first estimation model PM1 provided in the estimation unit 23.

[0109] In addition, the point cloud data generation unit 22 associates at least one of the RGB values ​​and normal values ​​(normal vectors) of each data point contained in (or associated with) the input data IND or the three-dimensional point cloud data PCD with the coordinates of the data point and stores them in the memory unit 20A as a second feature F2.

[0110] The estimation unit 23 inputs the three-dimensional point cloud data PCD generated by the point cloud data generation unit 22 to a first estimation model PM1. Here, the first estimation model PM1 is, for example, the first estimation model PM1 machine-learned by the first estimation model learning unit 141 described above in the learning phase.

[0111] The first estimation model PM1, to which the three-dimensional point cloud data PCD has been input, outputs interpolated three-dimensional point cloud data IPCD2. The estimation unit 23 also supplies the coordinates of each data point included in the interpolated three-dimensional point cloud data IPCD2 to the second feature quantity selection unit 152. The second feature quantity selection unit 152 then: compares the coordinates associated with the second feature quantities F2 stored in the storage unit 20A with the coordinates of each data point included in the interpolated three-dimensional point cloud data IPCD2; identifies, from the second feature quantities F2 stored in the storage unit 20A, second feature quantities F2 corresponding to each data point included in the interpolated three-dimensional point cloud data IPCD2; and supplies the identified second feature quantities F2 to the second estimation model PM2.

[0112] The estimation unit 23 inputs the following into the second estimation model PM2: - the complemented three-dimensional point cloud data IPCD2 calculated using the first estimation model PM1; and - the second feature F2 supplied from the second feature selection unit 152, and obtains the estimated label PL2 related to the segmentation output by the second estimation model PM2 (in other words, the segmentation label related to the complemented three-dimensional point cloud data IPCD2).

[0113] Then, data including the "complemented three-dimensional point cloud data IPCD2" and "segmentation labels related to the complemented three-dimensional point cloud data IPCD2" estimated (generated) by the estimation unit 23 is stored in the memory unit 20A as the segmentation data SD shown in Figure 5, as an example, and provided to the outside of the information processing device 100A via the communication unit 30 and the input / output unit 40, etc.

[0114] 9 is a diagram illustrating an example of a data flow related to a network in the estimation phase. As shown in FIG. 9, the three-dimensional point cloud data PCD2 is acquired by at least one of the acquisition unit 21 and the point cloud data generation unit 22.

[0115] Next, the three-dimensional point cloud data BPCD2 from which attribute data such as RGB values ​​and normal values ​​have been removed is input to the first estimation model PM1 (also referred to as the interpolation network in Figure 9).

[0116] The first estimation model PM1 then outputs the interpolated three-dimensional point cloud data IPCD2. The interpolated three-dimensional point cloud data IPCD2 is input to the second estimation model PM2 (also referred to as the semantic segmentation network in FIG. 7 ) together with the second feature F2 (referred to as second feature data F2 in FIG. 9 ). Here, the second feature F2 is the second feature F2 selected by the second feature selection unit 152 described above. The second feature F2 is input to the second estimation model PM2 in the form of a mask feature, for example. Here, the mask feature refers to a data group in which data points containing the second feature F2 selected by the second feature selection unit 152 have the value of the feature, and other data points are assigned a null feature (a predetermined value such as 0, 255, or −1 as the feature value).

[0117] The second estimation model PM2 then outputs estimated labels PL2 for the segmentation of each data point in the interpolated three-dimensional point cloud data IPCD2.

[0118] (Effects of information processing device 1A) As described above, the information processing device 1A employs the following configuration in the learning phase: - acquire input data; - generate three-dimensional point cloud data from the input data; - generate three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; - train an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0119] Furthermore, the information processing device 1A employs the following configuration in the estimation phase: - acquire input data; - generate three-dimensional point cloud data from the input data; - estimate the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation process and the segmentation labels related to the complemented point cloud data using an estimation model that takes point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0120] According to the information processing device 1A configured as described above, in the learning phase, the estimation model is trained by referring to 3D thinned point cloud data generated by applying a thinning process to 3D point cloud data. Then, in the estimation phase, the machine-learned estimation model is used to estimate interpolated point cloud data corresponding to the 3D point cloud data generated in the point cloud data generation process and segmentation labels related to the interpolated point cloud data. Therefore, with the above configuration, it is possible to perform estimation processing using a highly accurate estimation model while suppressing increases in cost.

[0121] Furthermore, with the information processing device 1A configured as described above, in the learning phase, the estimation model PM is trained by further referring to first feature quantities associated with at least one of the input data, the 3D point cloud data, and the 3D thinned point cloud data, and in the estimation phase, estimation processing is performed using the estimation model PM by further referring to second feature quantities associated with at least one of the input data and the 3D point cloud data. Therefore, with the above configuration, estimation processing using a more accurate estimation model can be performed while suppressing increases in cost.

[0122] (Additional Notes Regarding Exemplary Embodiment 2) In the above description, an example has been given in which the first estimation model PM1 and the second estimation model PM2 that constitute the estimation model PM are serially related, but this example does not limit the exemplary embodiment. For example, a configuration may be adopted in which the first estimation model PM1 and the second estimation model PM2 are parallel or partially parallel, and the first estimation model PM1 and the second estimation model PM2 are simultaneously trained. As an example, the estimation model PM may be configured to include: a first network; and a second network branched off from the first network. The loss value CE_Loss (or BCE_Loss) described above may be calculated by referring to the output of the first network, the loss value BCE_Loss (or CE_Loss) may be calculated by referring to the output of the second network, and the parameters of the first network and the second network may be updated by referring to a total loss value including these loss values. Such a configuration can also achieve various effects according to this exemplary embodiment. The same applies to other exemplary embodiments described in this specification.

[0123] [Third Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0124] (Configuration of Information Processing Device 1B) The configuration of the information processing device 1B according to this exemplary embodiment will be described with reference to Fig. 10. Fig. 10 is a block diagram showing the configuration of the information processing device 1B. As shown in Fig. 10, the information processing device 1B includes a projection unit 16 in addition to the units included in the information processing device 1A according to the second exemplary embodiment.

[0125] Furthermore, the storage unit 20B included in the information processing device 1B stores projected point cloud data PPCD (also referred to as post-projection point cloud data PPCD) in addition to the data stored in the storage unit 20A included in the information processing device 1A according to exemplary embodiment 2. The following description will focus on the differences from the information processing device 1A.

[0126] (Projection Unit 16) The projection unit 16 generates projected point cloud data PPCD by projecting the interpolated point cloud data generated by the estimation model PM onto a two-dimensional plane. Here, the two-dimensional plane is, for example, a two-dimensional plane corresponding to a projection onto a camera position (viewpoint) that was not used when the thinned point cloud data generation unit 13 generated the three-dimensional thinned point cloud data TPCD in the learning phase.

[0127] In other words, the projection unit 16 can be said to generate the projected point cloud data PPCD by projecting the complemented point cloud data generated by the estimation model PM from a camera position (viewpoint) that was not used when the thinned point cloud data generation unit 13 generated the three-dimensional thinned point cloud data TPCD during the learning phase.

[0128] However, the above example does not limit this exemplary embodiment, and the projection unit 16 may be expressed as being configured to generate projected point cloud data PPCD by projecting the interpolated point cloud data generated by the estimation model PM onto a position (viewpoint) different from the position (viewpoint) of an imaging device (including LiDAR) when the imaging device acquires at least one of the RGB values, depth values, and three-dimensional point cloud data included in the input data IND. Alternatively, the projection unit 16 may be expressed as being configured to generate projected point cloud data PPCD by projecting the interpolated point cloud data generated by the estimation model PM onto a viewpoint different from the viewpoint related to at least one of the RGB values, depth values, and three-dimensional point cloud data included in the input data IND that the point cloud data generation unit 12 (22) references when generating the three-dimensional point cloud data PCD.

[0129] (Learning Unit 14) The learning unit 14 according to this exemplary embodiment further refers to the projected point cloud data PPCD and the segmentation ground truth label GTL corresponding to the projected point cloud data, and trains the estimation model.

[0130] As an example, the learning unit 14 calculates a third loss value (third loss function) CE_2D_Loss indicating the difference between the estimated label estimated by the estimation model PM for the projected point cloud data PPCD and the segmentation ground truth label GTL acquired by the acquisition unit 11. Then, the learning unit 14 calculates a total loss value (Loss_total) by using a linear sum of the first loss value BCE_Loss, the second loss value CE_Loss, and the third loss value CE_2D_Loss described in the second exemplary embodiment, as follows: Loss_total = CE_Loss + α × BCE_Loss + β × CE_2D_Loss The learning unit 14 then updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the total loss value (Loss_total) becomes smaller. Here, α and β are weighting coefficients that can be set appropriately. Note that the third loss value CE_2D_Loss may be calculated by the projection unit 16.

[0131] (Data Flow in Learning Phase) Next, the data flow in the learning phase of the information processing device 1B will be described with reference to FIG.

[0132] 11 is a diagram illustrating an example of the data flow in the learning phase of the information processing device 1B. As shown in Fig. 11, in the learning phase of the information processing device 1B, in addition to the data flow in the learning phase of the information processing device 1A described with reference to Fig. 6, the following operations are performed: the segmentation correct label GTL is supplied from the acquisition unit 11 to the projection unit 16, the interpolated 3D point cloud data IPCD generated by the first estimation model PM1 is supplied from the first estimation model training unit 141, and the second estimation model training unit 142 supplies the estimated label PL, which is the estimated label estimated by the second estimation model PM2 and is associated with each data point of the 3D point cloud data PPCD after projection by the projection unit 16.

[0133] The projection unit 16 then calculates a third loss value CE_2D_Loss indicating the difference between the segmentation ground truth label GTL and the estimated label PL, and supplies the third loss value CE_2D_Loss to the learning unit 14. The learning unit 14 then calculates a total loss value (Loss_total) using the linear sum of the first loss value BCE_Loss, the second loss value CE_Loss, and the third loss value CE_2D_Loss, as described above. The learning unit 14 then updates the parameters of the first estimation model PM1 and the second estimation model PM2 so that the total loss value (Loss_total) becomes smaller.

[0134] (Effects of Information Processing Device 1B) The information processing device 1B configured as described above has the following configuration: - Projected point cloud data PPCD is generated by projecting the interpolated point cloud data IPCD generated by the estimation model PM onto a two-dimensional plane; - The learning unit 14 further refers to the projected point cloud data PPCD and the segmentation ground truth label GTL corresponding to the projected point cloud data PPCD to train the estimation model PM. With this configuration, by referring to the projected point cloud data PPCD, a more suitable loss function can be constructed, making it possible to generate an estimation model PM with higher estimation accuracy in the learning phase. Furthermore, it is possible to perform highly accurate estimation using such an estimation model PM in the estimation phase.

[0135] [Fourth Embodiment] A fourth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiments will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical obstacles arise.

[0136] (Configuration of Information Processing Device 1C) The configuration of the information processing device 1C according to this exemplary embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the information processing device 1C. As shown in Fig. 12, the control unit 10C included in the information processing device 1C has the same configuration as the control unit 10A included in the information processing device 1A. Meanwhile, the storage unit 20C included in the information processing device 1C stores thinning information TI in addition to the data stored in the storage unit 20A included in the information processing device 1A according to exemplary embodiment 2.

[0137] Here, the thinning information TI is information indicating what type of thinning processing was performed when the thinned point cloud data generation unit 13 performed the thinning processing. The following explanation will focus on the differences from the information processing device 1A.

[0138] (Thinning-out Information TI) The thinning-out information TI is information indicating what type of thinning-out processing was performed when the thinned point cloud data generation unit 13 performed the thinning-out processing. Fig. 13 is a diagram for explaining an example of the thinning-out information TI.

[0139] 13, a plurality of frames included in the input data IND acquired by the acquisition unit 11 in the learning phase are schematically shown. In the example shown in the upper part of Fig. 13, the frame numbers increase toward the right, for example.

[0140] The lower left side of Figure 13 shows thinning pattern 1 in the thinning process by the thinned point cloud data generation unit 13. Here, the shaded frames are frames that the thinned point cloud data generation unit 13 references when generating the three-dimensional thinned point cloud data TPCD, and the unshaded frames (white frames in the lower left side of Figure 13) are frames that are not referenced when generating the three-dimensional thinned point cloud data TPCD. In other words, the unshaded frames are frames that were thinned out when generating the three-dimensional thinned point cloud data TPCD. In thinning pattern 1, for a set of 10 consecutive frames, the following processing is performed: - referencing two consecutive frames, - thinning the next two consecutive frames, - referencing the next two consecutive frames, - thinning the next two consecutive frames, - referencing the next two consecutive frames.

[0141] 13 shows thinning pattern 2 in the thinning process by the thinned point cloud data generation unit 13. In thinning pattern 2, for a set of 10 consecutive frames, the following processing is performed: Reference three consecutive frames, thin out the next two consecutive frames, reference the next two consecutive frames, thin out the next two consecutive frames, and refer to the next one frame.

[0142] Then, the thinned point cloud data generator 13 stores, as an example, frame indices indicating the thinned frames as thinning information TI in the storage unit 20 C. By referring to such thinning information TI, the thinned point cloud data generator 13 generates multiple different three-dimensional thinned point cloud data TPCD using multiple thinning patterns for the same input data IND or the same three-dimensional point cloud data PCD.

[0143] In this way, the information processing device 1C according to this exemplary embodiment stores the thinned frame index when generating three-dimensional thinned point cloud data TPCD in the thinned point cloud data generation unit 13, thereby creating multiple three-dimensional thinned point cloud data TPCD with different thinning methods from a single data input, thereby strengthening the data used for learning in the learning phase.

[0144] (Effects of Information Processing Device 1C) According to the information processing device 1B configured as described above, the estimation model PM can be trained using a larger amount of data in the learning phase, thereby generating a more accurate estimation model PM. Furthermore, such an estimation model PM can be used to perform more accurate estimation.

[0145] Fifth Exemplary Embodiment A fifth exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiments will be denoted by the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technical means shown in each drawing referenced to describe this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0146] (Configuration of Information Processing Device 1D) The configuration of the information processing device 1D according to this exemplary embodiment will be described with reference to FIG. 14. FIG. 14 is a block diagram showing the configuration of the information processing device 1D. As shown in FIG. 14, the information processing device 1D has the following configuration (or data) among the units included in the information processing devices 1A, 1B, and 1C according to exemplary embodiments 2 to 4, and other configurations are not essential. In the control unit 10D, an acquisition unit 21, a point cloud data generation unit 22, an estimation unit 23, and a second feature amount selection unit 152, and in the storage unit 20D, input data IND, three-dimensional point cloud data PCD, second feature amounts F2, an estimation model PM, and segmentation data SG.

[0147] In other words, the information processing device 1D has a configuration for performing processing in the estimation phase using the estimation model PM, but does not have a configuration for learning the estimation model PM. As an example, the estimation model PM used by the information processing device 1D in the estimation processing may be the estimation model PM learned by the information processing devices 1A, 1B, and 1C according to the second to fourth exemplary embodiments.

[0148] According to the information processing device 1D configured as described above, the estimation model PM is machine-learned with reference to 3D thinned point cloud data generated by applying a thinning process to 3D point cloud data, and interpolated point cloud data corresponding to the 3D point cloud data generated in the point cloud data generation process and segmentation labels related to the interpolated point cloud data are estimated using the estimation model PM. This makes it possible to perform estimation processing using a highly accurate estimation model while suppressing increases in cost. Therefore, the above configuration makes it possible to provide a highly accurate segmentation technology while suppressing increases in cost.

[0149] (Application Examples) Hereinafter, application examples of the information processing devices 1, 1A, 1B, 1C, and 1D according to the above-described exemplary embodiments will be described.

[0150] The information processing devices 1, 1A, 1B, 1C, and 1D according to the above-described exemplary embodiments (hereinafter also simply referred to as the information processing device 1, etc.) can also be applied to the medical and healthcare fields, for example. In this field, the technology of the present application can be used to perform medical applications by performing segmentation based on medical images of patients.

[0151] When the information processing device 1 and the like are applied to the medical / healthcare field, for example, processing may be performed in accordance with the following processing flow.

[0152] (Step S101: Scanning Step) A medical professional such as a doctor or medical staff uses an imaging device (endoscope, fMRI, etc.) to capture images of a patient's organs (stomach, intestines, etc.) and other relevant areas to generate medical images, which are then input as input data IND into the information processing device 1, etc.

[0153] (Step S102: 3D Modeling Step) Subsequently, the point cloud data generation unit 12 (22) included in the information processing device 1 or the like refers to the input data IND and generates three-dimensional point cloud data PCD corresponding to the input data IND. Here, the three-dimensional point cloud data PCD generated by the point cloud data generation unit 12 (22) may be configured to be presented to medical professionals, medical staff, patients, etc. via a display or the like included in the input / output unit 40. Note that this step can be applied to both the learning phase and the estimation phase.

[0154] (Step S103A: Decision-Making Step) Next, the estimation unit 23 included in the information processing device 1 or the like performs semantic segmentation on the medical image using the machine-learned estimation model PM and outputs the segmentation results. Here, the semantic segmentation performs segmentation (area classification) into, for example, lesion areas (inflammation, ulcers, polyps), normal areas, and the like. This allows medical professionals to, for example, create a treatment plan. Therefore, this device can support medical professionals in making diagnostic decisions.

[0155] (Step S103B: Decision-Making Step) Instead of or in addition to the processing of step S103A, the information processing device 1 or the like may perform the following processing. That is, the estimation unit 23 of the information processing device 1 or the like outputs 3D point cloud data IPCD (or a 3D model constructed based on the 3D point cloud data IPCD) complemented by the first estimation model PM1 (for example, with defects complemented). For example, the output processing may be performed in the form of presentation to a medical professional via a display or the like provided in the input / output unit 40. By referring to the complemented 3D point cloud data (3D model), the medical professional can understand, for example, the condition of the patient's organs. Therefore, the present device can assist the medical professional in making diagnostic decisions.

[0156] In this manner, in this application example, the acquisition unit 11 (21) of the information processing device 1 or the like acquires medical images as the input data, and the estimation unit 23 functions as a presentation means for presenting the segmentation results to assist medical professionals in making decisions.

[0157] [Example of implementation by software] Some or all of the functions of the information processing devices 1, 1A, 1B, 1C, and 1D (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as an integrated circuit (IC chip), or by software.

[0158] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 15. Figure 15 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0159] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to function as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0160] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0161] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0162] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0163] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0164] (Appendix A1) An information processing device comprising: an acquisition means for acquiring input data; a point cloud data generation means for generating three-dimensional point cloud data from the input data; a thinned point cloud data generation means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; and a learning means for learning an estimation model that receives point cloud data as input, and outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0165] (Appendix A2) The information processing device according to Appendix A1, wherein the thinning process by the thinned point cloud data generating means includes a process of generating the three-dimensional thinned point cloud data using only a portion of the frames included in the input data.

[0166] (Appendix A3) The information processing device described in Appendix A1 or A2, wherein the thinning process by the thinned point cloud data generation means includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

[0167] (Supplementary Note A4) The information processing device according to Supplementary Note A1, wherein the learning means further refers to feature amounts associated with the three-dimensional thinned point cloud data to train the estimation model.

[0168] (Supplementary Note A5) The information processing device according to Supplementary Note A1 or A2, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0169] (Appendix A6) The information processing device according to any one of Appendices A1 to A5, further comprising a projection means for generating projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional plane, wherein the learning means further refers to the projected point cloud data and a segmentation correct label corresponding to the projected point cloud data to train the estimation model.

[0170] (Supplementary Note A7) The information processing device according to any one of Supplementary Notes A1 to A6, wherein the thinned point cloud data generating means generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

[0171] (Appendix A8) An information processing device comprising: an acquisition means for acquiring input data; a point cloud data generation means for generating three-dimensional point cloud data from the input data; and an estimation means for estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and the segmentation labels related to the complemented point cloud data, using an estimation model that receives point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0172] (Supplementary Note A9) The information processing device according to Supplementary Note A8, further comprising: a presentation unit that presents a segmentation result indicated by the segmentation label estimated by the estimation unit.

[0173] (Supplementary Note A10) The information processing device according to Supplementary Note A9, wherein the acquisition means acquires a medical image as the input data, and the presentation means presents the segmentation result to assist a medical professional in making a decision.

[0174] (Appendix A11) The information processing device according to any one of Appendices A8 to A10, wherein the estimation means further refers to feature amounts associated with the three-dimensional point cloud data generated by the point cloud data generation means, and estimates interpolated point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and segmentation labels related to the interpolated point cloud data.

[0175] (Supplementary Note A12) The information processing device according to any one of Supplementary Notes A8 to A11, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0176] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0177] (Appendix B1) An information processing method including: an acquisition step of acquiring input data; a point cloud data generation step of generating three-dimensional point cloud data from the input data; a thinned point cloud data generation step of generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; and a learning step of training an estimation model that takes point cloud data as input, and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0178] (Appendix B2) The information processing method according to Appendix B1, wherein the thinning process in the thinned point cloud data generation step includes a process of generating the three-dimensional thinned point cloud data using only a portion of the frames included in the input data.

[0179] (Appendix B3) The information processing method described in Appendix B1 or B2, wherein the thinning process by the thinned point cloud data generation step includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

[0180] (Supplementary Note B4) The information processing method according to Supplementary Note B1, wherein the learning step further refers to feature amounts associated with the three-dimensional thinned point cloud data to learn the estimation model.

[0181] (Supplementary Note B5) The information processing method according to Supplementary Note B1 or B2, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0182] (Appendix B6) The information processing method according to any one of Appendices B1 to B5, further comprising a projection step of generating projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional plane, wherein the learning step further refers to the projected point cloud data and a segmentation correct label corresponding to the projected point cloud data, thereby learning the estimation model.

[0183] (Supplementary Note B7) The information processing method according to any one of Supplementary Notes B1 to B6, wherein the thinned point cloud data generation step generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

[0184] (Appendix B8) An information processing method including: an acquisition step of acquiring input data; a point cloud data generation step of generating three-dimensional point cloud data from the input data; and an estimation step of estimating the complemented point cloud data corresponding to the three-dimensional point cloud data generated in the point cloud data generation step and the segmentation labels related to the complemented point cloud data, using an estimation model that takes the point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0185] (Supplementary Note B9) The information processing method according to Supplementary Note B8, further comprising a presentation step of presenting a segmentation result indicated by the segmentation label estimated by the estimation step.

[0186] (Supplementary Note B10) The information processing method according to Supplementary Note B9, wherein the obtaining step obtains a medical image as the input data, and the presenting step presents the segmentation result to assist a medical professional in making a decision.

[0187] (Appendix B11) The information processing method according to any one of Appendices B8 to B10, wherein the estimation step further refers to feature amounts associated with the three-dimensional point cloud data generated by the point cloud data generation step, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation step and segmentation labels related to the complemented point cloud data.

[0188] (Supplementary Note B12) The information processing method according to any one of Supplementary Notes B8 to B11, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0189] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0190] (Appendix C1) An information processing program that causes a computer to function as an information processing device, the information processing program causing the computer to function as: an acquisition means for acquiring input data; a point cloud data generation means for generating three-dimensional point cloud data from the input data; a thinned point cloud data generation means for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; and a learning means that receives point cloud data as input, and learns an estimation model that outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0191] (Appendix C2) The information processing program according to Appendix C1, wherein the thinning process by the thinned point cloud data generation means includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple frames included in the input data.

[0192] (Appendix C3) The information processing program described in Appendix C1 or C2, wherein the thinning process by the thinned point cloud data generation means includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

[0193] (Supplementary Note C4) The information processing program according to Supplementary Note C1, wherein the learning means further refers to feature amounts associated with the three-dimensional thinned point cloud data to learn the estimation model.

[0194] (Appendix C5) The information processing program according to Appendix C1 or C2, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0195] (Appendix C6) The information processing program according to any one of Appendices C1 to C5, further causing the computer to function as a projection means for generating projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional surface, and the learning means further referring to the projected point cloud data and a segmentation correct label corresponding to the projected point cloud data, to learn the estimation model.

[0196] (Supplementary Note C7) The information processing program according to any one of Supplementary Notes C1 to C6, wherein the thinned point cloud data generation means generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

[0197] (Appendix C8) An information processing program that causes the computer to function as: an acquisition means that acquires input data; a point cloud data generation means that generates three-dimensional point cloud data from the input data; and an estimation means that receives point cloud data as input, outputs interpolated point cloud data and segmentation labels related to the interpolated point cloud data, and estimates the interpolated point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means, and the segmentation labels related to the interpolated point cloud data, using an estimation model that is machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0198] (Supplementary Note C9) The information processing program according to Supplementary Note C8, which causes the computer to function as a presentation unit that presents a segmentation result indicated by the segmentation label estimated by the estimation unit.

[0199] (Supplementary Note C10) The information processing program according to Supplementary Note C9, wherein the acquisition means acquires a medical image as the input data, and the presentation means presents the segmentation result to assist a medical professional in making a decision.

[0200] (Appendix C11) The information processing program according to any one of Appendices C8 to C10, wherein the estimation means further refers to features associated with the three-dimensional point cloud data generated by the point cloud data generation means, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation means and segmentation labels related to the complemented point cloud data.

[0201] (Appendix C12) The information processing program according to any one of Appendices C8 to C11, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0202] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0203] (Appendix D1) An information processing device comprising at least one processor, the at least one processor executing: an acquisition process for acquiring input data; a point cloud data generation process for generating three-dimensional point cloud data from the input data; a thinned point cloud data generation process for generating three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; and a learning process for training an estimation model that receives point cloud data as input, and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0204] (Appendix D2) The information processing device according to appendix D1, wherein the thinning process by the thinned point cloud data generation process includes a process of generating the three-dimensional thinned point cloud data using only a portion of the frames included in the input data.

[0205] (Appendix D3) The information processing device described in Appendix D1 or D2, wherein the thinning process by the thinned point cloud data generation process includes a process of generating the three-dimensional thinned point cloud data using only a portion of the multiple data points included in at least one of the input data and the three-dimensional point cloud data.

[0206] (Supplementary Note D4) The information processing device according to Supplementary Note D1, wherein in the learning process, the at least one processor further refers to feature amounts associated with the three-dimensional thinned point cloud data to learn the estimation model.

[0207] (Appendix D5) The information processing device according to appendix D1 or D2, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0208] (Appendix D6) The information processing device described in any one of Appendices D1 to D5, wherein the at least one processor further executes a projection process to generate projected point cloud data by projecting the interpolated point cloud data generated by the estimation model onto a two-dimensional plane, and in the learning process, the at least one processor further refers to the projected point cloud data and a segmentation correct answer label corresponding to the projected point cloud data to learn the estimation model.

[0209] (Supplementary Note D7) The information processing device according to any one of Supplementary Notes D1 to D6, wherein the thinned point cloud data generation process generates a plurality of mutually different three-dimensional thinned point cloud data for the same input data or the same three-dimensional point cloud data.

[0210] (Appendix D8) The at least one processor executes: an acquisition process that acquires input data; a point cloud data generation process that generates three-dimensional point cloud data from the input data; and an estimation process that estimates the complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation process and the segmentation labels related to the complemented point cloud data, using an estimation model that takes the point cloud data as input and outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, the estimation model being machine-learned by referring to the three-dimensional point cloud data and three-dimensional thinned point cloud data obtained by applying a thinning process to the three-dimensional point cloud data.

[0211] (Supplementary Note D9) The information processing device according to Supplementary Note D8, wherein the at least one processor executes a presentation process of presenting a segmentation result indicated by the segmentation label estimated by the estimation process.

[0212] (Supplementary Note D10) The information processing device described in Supplementary Note D9, wherein in the acquisition process, the at least one processor acquires a medical image as the input data, and in the presentation process, the at least one processor presents the segmentation result to assist a medical professional in making a decision.

[0213] (Appendix D11) The information processing device described in any one of Appendices D8 to D10, wherein in the estimation process, the at least one processor further refers to features associated with the three-dimensional point cloud data generated by the point cloud data generation process, and estimates complemented point cloud data corresponding to the three-dimensional point cloud data generated by the point cloud data generation process and segmentation labels related to the complemented point cloud data.

[0214] (Appendix D12) The information processing device according to any one of Appendices D8 to D11, wherein the estimation models include: a first estimation model that receives point cloud data as input and outputs interpolated point cloud data; and a second estimation model that receives at least the interpolated point cloud data as input and outputs segmentation labels related to the interpolated point cloud data.

[0215] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0216] (Appendix E1) A non-transitory recording medium having recorded thereon an information processing program that causes a computer to function as an information processing device, the information processing program causing the computer to execute: an acquisition process that acquires input data; a point cloud data generation process that generates three-dimensional point cloud data from the input data; a thinned point cloud data generation process that generates three-dimensional thinned point cloud data by applying a thinning process to at least one of the input data and the three-dimensional point cloud data; and a learning process that uses point cloud data as input, and learns an estimation model that outputs complemented point cloud data and segmentation labels related to the complemented point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional thinned point cloud data.

[0217] REFERENCE SIGNS LIST 1, 2, 1A, 1B, 1C, 1D ... Information processing device 11, 21 ... Acquisition unit (acquisition means) 12, 22 ... Point cloud data generation unit (point cloud data generation means) 13 ... Thinned point cloud data generation unit (thinned point cloud data generation means) 14 ... Learning unit (learning means) 23 ... Estimation unit (estimation means) 16 ... Projection unit (projection means)

Claims

1. A means for acquiring input data, A point cloud data generation means for generating three-dimensional point cloud data from the aforementioned input data, A decimated point cloud data generation means that generates 3D decimated point cloud data by applying a decimation process to at least one of the input data and the 3D point cloud data, A learning means for training an estimation model that takes point cloud data as input and outputs interpolated point cloud data and segmentation labels related to said interpolated point cloud data, by referring to the three-dimensional point cloud data and the three-dimensional decimated point cloud data. An information processing device equipped with the following features.

2. The thinning process by the thinned point cloud data generation means includes: The process includes generating the three-dimensional decimated point cloud data using only a portion of the multiple frames included in the input data. The information processing apparatus according to claim 1.

3. The thinning process by the thinned point cloud data generation means includes: The process includes generating the three-dimensional decimated point cloud data using only some of the data points from among the multiple data points included in at least one of the input data and the three-dimensional point cloud data. The information processing apparatus according to claim 1.

4. The learning means further references the features associated with the three-dimensional decimated point cloud data to train the estimation model. The information processing apparatus according to claim 1.

5. The aforementioned estimation model is, A first estimation model takes point cloud data as input and outputs interpolated point cloud data, A second estimation model that takes at least the aforementioned interpolated point cloud data as input and outputs segmentation labels relating to the aforementioned interpolated point cloud data. Includes The information processing apparatus according to claim 1.

6. The system further comprises projection means for generating projected point cloud data by projecting the complementary point cloud data generated by the estimation model onto a two-dimensional plane, The learning means further references the projected point cloud data and the segmentation ground truth labels corresponding to the projected point cloud data to train the estimation model. The information processing apparatus according to claim 1.

7. The aforementioned decimated point cloud data generation means generates a plurality of different 3D decimated point cloud data sets from the same input data or the same 3D point cloud data. An information processing apparatus according to any one of claims 1 to 6.

8. A means for acquiring input data, A point cloud data generation means for generating three-dimensional point cloud data from the aforementioned input data, An estimation model that takes point cloud data as input and outputs interpolated point cloud data and segmentation labels relating to said interpolated point cloud data, wherein an estimation means estimates interpolated point cloud data corresponding to the 3D point cloud data generated by the point cloud data generation means and segmentation labels relating to said interpolated point cloud data, using an estimation model that has been trained by referencing 3D point cloud data and 3D decimated point cloud data obtained by applying a decimation process to said 3D point cloud data. An information processing device equipped with the following features.

9. Obtaining input data, To generate 3D point cloud data from the aforementioned input data, The process involves generating 3D decimated point cloud data by applying a decimation process to at least one of the input data and the 3D point cloud data. An estimation model that takes point cloud data as input and outputs interpolated point cloud data and segmentation labels related to said interpolated point cloud data is trained by referring to the three-dimensional point cloud data and the three-dimensional decimated point cloud data. An information processing method that includes this.

10. On the computer, The data acquisition process for obtaining input data, A point cloud data generation process that generates 3D point cloud data from the aforementioned input data, A decimated point cloud data generation process that generates 3D decimated point cloud data by applying a decimation process to at least one of the input data and the 3D point cloud data, A learning process that trains an estimation model, which takes point cloud data as input and outputs interpolated point cloud data and segmentation labels related to said interpolated point cloud data, by referring to the 3D point cloud data and the 3D decimated point cloud data. A program that executes the command.