Information processing apparatus, information processing method, and program

Abstract

To suitably reconstruct both local details and a global three dimensional shape of an object in an image.SOLUTION: The information processing apparatus includes an acquisition unit configured to acquire input data including a plurality of images under a plurality of illumination conditions, an estimation unit configured to execute normal estimation processing and depth estimation processing with reference to the input data, a priority determination unit configured to determine a priority of each of the normal estimation processing and the depth estimation processing, and a generation unit configured to generate output data with reference to a result of the normal estimation processing, a result of the depth estimation processing, and the priority.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is known a technique called Photometric Stereo, which refers to a plurality of images captured under a plurality of lighting conditions and grasps the shape of an object in the images (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] S. Ikehara, Universal Photometric Stereo Network using Global Lighting Contexts,arXiv:2206.02452v1, Jun 2022 Summary of the Invention [Problem to be solved by the invention]

[0004] Although the technology described in Non-Patent Document 1 excels at reconstructing local detailed shapes such as the surface texture of an object, it has issues in terms of reconstructing the global three-dimensional shape of the object.

[0005] The present disclosure has been made in consideration of the above problems, and an exemplary purpose thereof is to provide a technique that can preferably reconstruct both local details and the global three-dimensional shape of an object in an image. [Means for solving the problem]

[0006] An information processing device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions, an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data, a priority determination means for determining the priority of each of the normal estimation process and the depth estimation process, and a generation means for generating output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority.

[0007] An information processing device according to an exemplary aspect of the present disclosure includes an acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions, an estimation means for performing a normal estimation process and a depth estimation process using an estimation model by referring to the input data, a priority determination means for determining the priority of each of the normal estimation process and the depth estimation process, and a learning means for training the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority.

[0008] An information processing method according to an exemplary aspect of the present disclosure includes acquiring input data including a plurality of images under a plurality of lighting conditions, performing a normal estimation process and a depth estimation process by referring to the input data, determining a priority for each of the normal estimation process and the depth estimation process, and generating output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority.

[0009] An information processing method according to an exemplary aspect of the present disclosure includes acquiring input data including a plurality of images under a plurality of lighting conditions, performing a normal estimation process and a depth estimation process using an estimation model by referring to the input data, determining a priority of each of the normal estimation process and the depth estimation process, and training the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority.

[0010] The information processing device according to each aspect of the present invention may be realized by a computer. In this case, the program that realizes the information processing device on a computer by causing the computer to operate as each part (software element) of the information processing device, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention. [Effects of the Invention]

[0011] According to an exemplary aspect of the present disclosure, an exemplary effect is achieved in that both local details and the global three-dimensional shape of an object can be suitably reconstructed. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 2] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 3] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 4] FIG. 1 is a flow diagram showing the flow of an information processing method according to the present disclosure. [Figure 5] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 6] FIG. 2 is a diagram for explaining processing in an information processing device according to the present disclosure. [Figure 7] FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 8] FIG. 2 is a diagram for explaining processing in an information processing device according to the present disclosure. [Figure 9] FIG. 1 is a diagram illustrating an example of a network configuration in an information processing device according to the present disclosure. [Figure 10] FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 11] FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 12]FIG. 2 is a diagram illustrating a processing flow in an information processing device according to the present disclosure. [Figure 13] 1 is a block diagram illustrating a configuration of an information processing device according to the present disclosure. [Figure 14] FIG. 1 is a block diagram illustrating a hardware configuration of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technical means employed in the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, the effects mentioned in the exemplary embodiments shown below are examples of effects expected in the exemplary embodiments, and do not define the scope of the present invention. In other words, embodiments that do not exhibit the effects mentioned in the exemplary embodiments shown below may also be included in the scope of the present invention.

[0014] [First embodiment] A first exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form of each exemplary embodiment described later. Note that the scope of application of each technical means employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technical means employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise. Furthermore, each technical means shown in the drawings referred to in describing this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure to the extent that no particular technical obstacles arise.

[0015] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the information processing device 1. The information processing device 1 may also be called an image processing device or a learning device. As shown in Fig. 1, the information processing device 1 includes an acquisition unit 11, an estimation unit 12, a priority determination unit 13, and a learning unit 14.

[0016] (Acquisition part 11) The acquisition unit 11 acquires input data including a plurality of images under a plurality of lighting conditions. Here, the input data is, for example, input data for a learning phase. The plurality of images included in the input data are, for example, The image may be a plurality of captured images (actual images) captured by an imaging device under a plurality of different lighting conditions, and may be captured images of one or a plurality of objects. The images may be multiple images (CG, etc.) generated by an image generating device under multiple different lighting conditions, and may include one or multiple objects.

[0017] For example, the above-mentioned multiple images are taken in an environment where light sources 1 to 3 are placed. Image 1 obtained by capturing an object under lighting condition 1 (light source 1 is on) Image 2 obtained by capturing an object under lighting condition 2 (light source 2 is on) Image 3 obtained by capturing an object under lighting condition 3 (light source 3 is on) Here, the "object" may be a living body or a non-living body, and this does not limit the present exemplary embodiment in any way. The plurality of images included in the input data are, for example, RGB data (RGB images) in which each pixel (data point) represents an RGB value, but are not limited to this. In addition to the plurality of images, the input data may also include, as data relating to the object, Depth data (depth image) where each pixel (data point) represents a depth value 3D point cloud data where each data point represents a 3D coordinate The three-dimensional point cloud data may include at least one of the following: As an example, the three-dimensional point cloud data may be point cloud data acquired by a line laser scanner, but this example does not limit the present exemplary embodiment.

[0018] (Estimation part 12) The estimation unit 12 refers to the input data and executes normal estimation processing and depth estimation processing using an estimation model. Here, the estimation model is a learning target in the learning processing executed by the information processing device 1. Furthermore, although the specific configuration of the estimation unit 12 does not limit this exemplary embodiment, as an example, using the estimation model, a first extraction process for extracting one or more first feature amounts from the plurality of images included in the input data; a second extraction process for extracting one or more second feature amounts from depth information obtained from the input data; The normal estimation process and the depth estimation process are performed by referring to the first feature amount and the second feature amount. Furthermore, when the input data includes the depth image or three-dimensional point cloud data, the estimation unit 12 may be configured to execute the following: A process of extracting the one or more second feature amounts from the depth image or the three-dimensional point cloud data as the depth information. Alternatively, if the input data does not include the depth image or three-dimensional point cloud data, the estimation unit 12 may execute: A depth information generation process for generating the depth information from the plurality of images included in the input data. The method may be further configured to perform the following.

[0019] The format of the data indicating the results of the normal estimation process and the depth estimation process is not particularly limited, but may be, for example, a set of two-dimensional data points corresponding to the image included in the input data. As an example, the result of the normal estimation process may be expressed in the form of a normal estimation map, and the result of the depth estimation process may be expressed in the form of a depth estimation map.

[0020] (Priority determination unit 13) The priority determination unit 13 determines the priority of each of the normal estimation process and the depth estimation process. Although the details of the priority determination process by the priority determination unit 13 do not limit this exemplary embodiment, as an example, the priority may be determined by referring to at least one of the multiple images included in the input data, the result of the normal estimation process, and the result of the depth estimation process.

[0021] As an example, the priority determination unit 13 determines the following as the respective priorities: The weighting coefficient W is multiplied by the result of the normal estimation process NR. NR A weighting coefficient W multiplied by the result DR of the depth estimation process DR Here, as an example, the priority determination unit 13 may assign a larger weight to a process having a higher priority out of the normal estimation process and the depth estimation process. As an example, the priority determination unit 13 may be configured to perform a determination process as to which of the results of the normal estimation process and the depth estimation process is more reliable, and assign a higher priority to the process determined to be more reliable. However, this example does not limit the present exemplary embodiment.

[0022] Furthermore, the priority determination unit 13 may be configured to calculate local priorities as the above priorities. As an example, the priority determination unit 13 may be configured to determine the above priorities for each partial region or each data point in the normal estimation map and depth estimation map. In this configuration, the weighting coefficients calculated by the priority determination unit 13 are calculated as follows: (W NR ) i,j , (W DR ) i,j It can also be written as:

[0023] (Study Section 14) The learning unit 14 trains the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority. A first loss value L representing the difference between the result NR of the normal estimation process and the ground truth data regarding the normal. NR and, A second loss value L representing the difference between the result DR of the depth estimation process and the ground truth data regarding depth. DR and, The priority and and updates a plurality of parameters that define the estimation model so that the value of the loss function LF becomes smaller.

[0024] More specifically, as an example, the learning unit 14 uses a weighting factor W NR , W DR Using LF=W NR ×L NR +W DR ×L DR and updating a plurality of parameters defining the estimation model so that the value of the loss function LF becomes smaller.

[0025] The learning unit 14 may also be configured to calculate a local loss function as the loss function. As an example, the priority determination unit 13 may be configured to calculate the loss function for each partial region or each data point in the normal estimation map and depth estimation map. In this configuration, the weighting coefficients calculated by the learning unit 14 are calculated using a two-dimensional index (i, j) that specifies each partial region or each data point in the normal estimation map or depth estimation map, as follows: LF i,j =(W NR ) i,j ×(L NR ) i,j +(W DR ) i,j ×(L DR )i,j The sum of the local loss functions can be expressed as LF=ΣLF i,j Here, Σ represents the sum over the two-dimensional index (i,j).

[0026] In addition, when the plurality of images included in the input data are a plurality of images (CG, etc.) generated by an image generating device as described above, the correct answer data is A normal map and a depth map derived from the image data generated by the image generation device Furthermore, when the plurality of images included in the input data are a plurality of captured images (real-life images) captured by an imaging device as described above, the following may be used as the correct answer data: Normal and depth maps obtained by referencing data obtained by measuring the captured object However, these examples are not intended to limit the present exemplary embodiment.

[0027] (Effects of information processing device 1) As described above, in the information processing device 1, Obtain input data containing multiple images under multiple lighting conditions, Performing normal estimation and depth estimation using an estimation model with reference to the input data; determining the priority of each of the normal estimation process and the depth estimation process; The estimation model is trained using a loss function according to the results of the normal estimation process, the results of the depth estimation process, and the priority. According to the above configuration, the priority of each of the normal estimation process and the depth estimation process is determined, and the estimation model is trained using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority. Reconstruction of local details of the object based on said normal estimation process; Reconstructing the global 3D shape of the object based on the depth estimation process; In other words, the above configuration makes it possible to reconstruct both the local details and the global three-dimensional shape of the object in an appropriate manner.

[0028] (Flow of information processing method S1) Next, the flow of information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of information processing method S1. As shown in Fig. 2, information processing method S1 includes step (process) S11 of acquiring input data, step (process) S12 of executing normal estimation processing and depth estimation processing, step (process) S13 of determining the priorities of each of the normal estimation processing and the depth estimation processing, and step (process) S14 of learning an estimation model using a loss function.

[0029] (Step S11) In step S11, the acquisition unit 11 acquires input data including a plurality of images under a plurality of lighting conditions. The specific processing by the acquisition unit 11 has been described above, and therefore will not be described here.

[0030] (Step S12) Next, in step S12, the estimation unit 12 refers to the input data and performs normal estimation processing and depth estimation processing using the estimation model. The specific processing by the estimation unit 12 has been described above, so a description thereof will be omitted here.

[0031] (Step S13) Subsequently, in step S14, the priority determination unit 13 determines the priority of each of the normal estimation process and the depth estimation process. The specific process by the priority determination unit 13 has been described above, and therefore will not be described here.

[0032] (Step S14) Next, in step S13, the learning unit 14 trains the estimation model using a loss function according to the results of the normal estimation process, the results of the depth estimation process, and the priority. The specific processing by the learning unit 14 has been described above, so a description thereof will be omitted here.

[0033] The processes from step S12 to step S14 may be repeated multiple times until the value of the loss function satisfies a predetermined convergence condition, but this example does not limit the present exemplary embodiment.

[0034] (Effect of information processing method S1) As described above, in the information processing method S1, Obtain input data containing multiple images under multiple lighting conditions, Performing normal estimation and depth estimation using an estimation model with reference to the input data; determining the priority of each of the normal estimation process and the depth estimation process; The estimation model is trained using a loss function according to the results of the normal estimation process, the results of the depth estimation process, and the priority. According to the above configuration, the same effects as those of the information processing device 1 are achieved.

[0035] (Configuration of information processing device 2) Next, the configuration of the information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing device 2. The information processing device 2 may also be referred to as an image generation device or an inference device. As shown in Fig. 3, the information processing device 2 includes an acquisition unit 21, an estimation unit 21, a priority determination unit 23, and a generation unit 24.

[0036] (Acquisition part 21) The acquisition unit 21 acquires input data including a plurality of images under a plurality of lighting conditions. Here, the input data is, for example, input data for the inference phase. The plurality of images included in the input data may, for example, be The image may be a plurality of captured images (actual images) captured by an imaging device under a plurality of different lighting conditions, and may be captured images of one or a plurality of objects. The images may be multiple images (CG, etc.) generated by an image generating device under multiple different lighting conditions, and may include one or multiple objects.

[0037] For example, the above-mentioned multiple images are taken in an environment where light sources 1 to 3 are placed. Image 1 obtained by capturing an object under lighting condition 1 (light source 1 is on) Image 2 obtained by capturing an object under lighting condition 2 (light source 2 is on) Image 3 obtained by capturing an object under lighting condition 3 (light source 3 is on) Here, the "object" may be a living body or a non-living body, and this does not limit the present exemplary embodiment in any way.

[0038] The plurality of images included in the input data are, for example, RGB data (RGB images) in which each pixel (data point) represents an RGB value, but are not limited to this. In addition to the plurality of images, the input data may also include, as data relating to the object, Depth data (depth image) where each pixel (data point) represents a depth value 3D point cloud data where each data point represents a 3D coordinate The three-dimensional point cloud data may include at least one of the following: As an example, the three-dimensional point cloud data may be point cloud data acquired by a line laser scanner, but this example does not limit the present exemplary embodiment.

[0039] (Estimation part 22) The estimation unit 22 refers to the input data and executes normal estimation processing and depth estimation processing. Here, the normal estimation processing and the depth estimation processing may be configured to be executed by an estimation model that has been trained (with updated parameters) by the learning unit 14 included in the information processing device 1, as an example. However, this does not limit the present exemplary embodiment. Furthermore, although the specific configuration of the estimation unit 22 does not limit the present exemplary embodiment, as an example, the following may be executed by using the above estimation model: a first extraction process for extracting one or more first feature amounts from the plurality of images included in the input data; a second extraction process for extracting one or more second feature amounts from depth information obtained from the input data; The normal estimation process and the depth estimation process are performed by referring to the first feature amount and the second feature amount. Furthermore, when the input data includes the depth image or three-dimensional point cloud data, the estimation unit 22 may be configured to execute the following: A process of extracting the one or more second feature amounts from the depth image or the three-dimensional point cloud data as the depth information. Alternatively, if the input data does not include the depth image or three-dimensional point cloud data, the estimation unit 22 may execute: A depth information generation process for generating the depth information from the plurality of images included in the input data. The method may be further configured to perform the following.

[0040] The format of the data indicating the results of the normal estimation process and the depth estimation process is not particularly limited, but may be, for example, a set of two-dimensional data points corresponding to the image included in the input data. As an example, the result of the normal estimation process may be expressed in the form of a normal estimation map, and the result of the depth estimation process may be expressed in the form of a depth estimation map.

[0041] (Priority determination unit 23) The priority determination unit 23 determines the priority of each of the normal estimation process and the depth estimation process. Although the details of the priority determination process by the priority determination unit 23 do not limit this exemplary embodiment, as an example, the priority may be determined by referring to at least one of the multiple images included in the input data, the result of the normal estimation process, and the result of the depth estimation process.

[0042] As an example, the priority determination unit 23 determines the respective priorities as follows: The weighting coefficient W is multiplied by the result of the normal estimation process NR. NR A weighting coefficient W multiplied by the result DR of the depth estimation process DR Here, as an example, the priority determination unit 13 may assign a larger weight to a process having a higher priority out of the normal estimation process and the depth estimation process. As an example, the priority determination unit 23 may be configured to perform a determination process as to which of the results of the normal estimation process and the depth estimation process is more reliable, and assign a higher priority to the process determined to be more reliable. However, this example does not limit the present exemplary embodiment.

[0043] Furthermore, the priority determination unit 23 may be configured to calculate local priorities as the above priorities. As an example, the priority determination unit 23 may be configured to determine the above priorities for each partial region or each data point in the normal estimation map and depth estimation map. In this configuration, the weighting coefficients calculated by the priority determination unit 23 are calculated as follows: (W NR ) i,j , (W DR ) i,j It can also be written as:

[0044] (Generation unit 24) The generation unit 24 generates output data by referring to the results of the normal estimation process, the results of the depth estimation process, and the priorities. The output data is, for example, three-dimensional data (also referred to as three-dimensional reconstruction data) related to the object included in the input data.

[0045] As an example, the generation unit 24 generates three-dimensional data as output data by integrating three-dimensional data obtained by referring to the result of the normal estimation process and three-dimensional data obtained by referring to the result of the depth estimation process according to their respective priorities.

[0046] As an example, the generation unit 24 While performing a process of reconstructing three-dimensional data by referring to the result of the depth estimation process, In an area where the priority of the normal estimation process is equal to or higher than a predetermined threshold (in other words, an area where the priority determination unit 23 has determined that the normal estimation process should be prioritized), the three-dimensional data in that area is replaced with the integral value of the result of the normal estimation process.

[0047] Furthermore, the generation unit 24 may output a normal map and a depth map in addition to the above three-dimensional data. In this case, the generation unit 24 may be configured to refer to the priorities determined by the priority determination unit 23, and perform replacement processing using a differential value of depth in areas where the priority for normal estimation processing is equal to or less than a predetermined threshold (or smaller), and perform replacement processing using an integral value of normal in areas where the priority for depth estimation processing is equal to or greater than the predetermined threshold (or larger).

[0048] (Effects of information processing device 2) As described above, in the information processing device 2, Obtain input data containing multiple images under multiple lighting conditions, Execute normal estimation processing and depth estimation processing by referring to the input data; determining the priority of each of the normal estimation process and the depth estimation process; Generating output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. According to the above configuration, the priority of each of the normal estimation process and the depth estimation process is determined, and output data is generated by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. Reconstruction of local details of the object based on said normal estimation process; Reconstructing the global 3D shape of the object based on the depth estimation process; In other words, the above configuration makes it possible to reconstruct both the local details and the global three-dimensional shape of the object in an appropriate manner.

[0049] (Flow of information processing method S2) Next, the flow of an information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a flow diagram showing the flow of the information processing method S2. As shown in Fig. 4, the information processing method S2 includes a step (process) S21 of acquiring input data, a step (process) S22 of executing normal estimation processing and depth estimation processing, a step (process) S23 of determining the priority of each of the normal estimation processing and the depth estimation processing, and a step (process) S24 of generating output data.

[0050] (Step S21) In step S21, the acquisition unit 21 acquires input data including a plurality of images taken under a plurality of lighting conditions. The specific processing performed by the acquisition unit 21 has been described above, and therefore will not be described here.

[0051] (Step S22) Subsequently, in step S22, the estimation unit 22 executes normal estimation processing and depth estimation processing with reference to the input data. The specific processing by the estimation unit 22 has been described above, and therefore will not be described here.

[0052] (Step S23) Subsequently, in step S23, the priority determination unit 23 determines the priority of each of the normal estimation process and the depth estimation process. The specific process by the priority determination unit 23 has been described above, and therefore will not be described here.

[0053] (Step S24) Next, in step S24, the generation unit 24 generates output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. The specific processing by the generation unit 24 has been described above, so a description thereof will be omitted here.

[0054] (Effect of information processing method S2) As described above, in the information processing method S2, Obtain input data containing multiple images under multiple lighting conditions, Execute normal estimation processing and depth estimation processing by referring to the input data; determining the priority of each of the normal estimation process and the depth estimation process; Generating output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. According to the above configuration, the same effects as those of the information processing device 2 are achieved.

[0055] Second Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0056] (Configuration of information processing system 1A) The configuration of an information processing system 1A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 3 is a block diagram showing the configuration of the information processing system 1A. As shown in Fig. 5, the information processing system 1A includes an information processing device 100A and an imaging device 50 connected to the information processing device 100A via a network N. Here, the specific configuration of the network N does not limit this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0057] (imaging device 50) The imaging device 50 acquires at least one of input data used in a learning phase (to be described later) and input data used in an inference phase (to be described later). The imaging device 50 may be an imaging device that captures an image of an object in the real world, or may have the function of an image generation device that generates three-dimensional data in a virtual space.

[0058] An image captured or generated by the imaging device 50 includes, for example, a plurality of images captured under a plurality of lighting conditions. Fig. 6 is a diagram for explaining an image captured or generated by the imaging device 50. As shown in the upper part of Fig. 6, an object OBJ is placed in an environment where, for example, a plurality of light sources (60-1 to 60-3) are placed. In the example shown in Fig. 6, the imaging device 50 captures images of the object OBJ under a plurality of lighting conditions.

[0059] The multiple images captured in this way include the following: Image 1 (IMG1 in Figure 6) obtained by capturing an image of object OBJ under lighting condition 1 (light source 60-1 is on) Image 2 (IMG2 in Figure 6) obtained by capturing the object OBJ under lighting condition 2 (light source 60-2 is on) Image 3 (IMG3 in Figure 6) obtained by capturing the object OBJ under lighting condition 3 (light source 60-3 is on) etc. may be included.

[0060] The image captured or generated by the imaging device 50 is acquired by the acquisition unit 11 (21) described later.

[0061] (Configuration of information processing device 100A) Next, the configuration of the information processing device 100A according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the information processing device 100A. As shown in Fig. 5, the information processing device 100A includes a control unit 10A, a storage unit 20A, a communication unit 30, and an input / output unit 40.

[0062] (Communication unit 30) The communication unit 30 communicates with devices external to the information processing device 100A via a network. For example, the communication unit 30 transmits data supplied from the control unit 10A to the external device, and supplies data received from the external device to the control unit 10A. Note that the specific configuration of the network does not limit the present exemplary embodiment, and examples include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks.

[0063] (Input / output section 40) The input / output unit 40 is configured to include at least one of input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel. Alternatively, the input / output unit 40 may be configured to have input / output devices such as a keyboard, a mouse, a display, a printer, and a touch panel connected to it. In this configuration, the input / output unit 40 accepts various types of information input to the information processing device 100A from the connected input devices. Furthermore, the input / output unit 40 outputs various types of information to the connected output devices under the control of the control unit 10A. An example of the input / output unit 40 is an interface such as a USB (Universal Serial Bus).

[0064] (Storage unit 20A) The storage unit 20A stores various data referenced by the control unit 10A and various data generated by the control unit 10A. Input data IND including image group IMG First feature F1 Second feature F2 Normal estimation result NR ·Depth estimation result DR ·Priority PI Loss function LF Output data OUT Specific examples of this data will be described later.

[0065] (Control unit 10A) As shown in Fig. 5, the control unit 10A includes an acquisition unit 11, an estimation unit 12, a priority determination unit 13, a learning unit 14, and a generation unit 24, all of which are described in the exemplary embodiment 1. Here, the acquisition unit 11 may be expressed as having the same configuration as the acquisition unit 21 described in the exemplary embodiment 1, and therefore the acquisition unit 11 may also be referred to as the acquisition unit 11 (21). Furthermore, the estimation unit 12 may be expressed as having the same configuration as the estimation unit 22 described in the exemplary embodiment 1, and therefore the estimation unit 12 may also be referred to as the estimation unit 12 (22). Furthermore, the priority determination unit 13 may be expressed as having the same configuration as the priority determination unit 23 described in the exemplary embodiment 1, and therefore the priority determination unit 13 may also be referred to as the priority determination unit 13 (23).

[0066] (Acquisition part 11(21)) The acquisition unit 11 (21) acquires input data IND including a plurality of images IMG under a plurality of lighting conditions. Here, the acquisition unit 11 (21) acquires input data IND for learning in the learning phase, and acquires input data IND for inference in the inference phase. Furthermore, the plurality of images IMG included in the input data IND may be, for example, as in the first exemplary embodiment: The image may be a plurality of captured images (actual images) captured by an imaging device under a plurality of different lighting conditions, and may be captured images of one or a plurality of objects. The images may be multiple images (CG, etc.) generated by an image generating device under multiple different lighting conditions, and may include one or multiple objects.

[0067] For example, the plurality of images IMG are as shown in the lower part of FIG. Image 1 (IMG1 in Figure 6) obtained by capturing an image of object OBJ under lighting condition 1 (light source 60-1 is on) Image 2 (IMG2 in Figure 6) obtained by capturing the object OBJ under lighting condition 2 (light source 60-2 is on) Image 3 (IMG3 in Figure 6) obtained by capturing the object OBJ under lighting condition 3 (light source 60-3 is on) Here, the "object" may be a living body or a non-living body, and this does not limit the present exemplary embodiment in any way.

[0068] The plurality of image IMG included in the input data is, for example, RGB data (RGB image) in which each pixel (data point) represents an RGB value, but is not limited to this. In addition to the plurality of image IMG, the input data may also include, as data related to the object, Depth data (depth image) where each pixel (data point) represents a depth value 3D point cloud data where each data point represents a 3D coordinate The three-dimensional point cloud data may include at least one of the following. The three-dimensional point cloud data may be point cloud data acquired by a line laser scanner, for example, but this example does not limit this exemplary embodiment. Note that the processing by the acquisition unit 11 (21) is, for example, similar to that of the acquisition unit 11 and the acquisition unit 21 according to the exemplary embodiment 1, and therefore, redundant explanations may be omitted.

[0069] (Estimation part 12(22)) The estimation unit 12 (22) refers to the input data IND and performs normal estimation processing and depth estimation processing using an estimation model MD. Here, in the learning phase, the estimation unit 12 (22) performs the above estimation processing using the learned inference model MD, and in the inference phase, performs the above estimation processing using the trained inference model MD.

[0070] 5, the estimation unit 12 (22) includes, for example, a first extraction unit 121, a second extraction unit 122, and a normal / depth estimation unit 123. The processing by each of these units corresponds to more specific processing related to the normal estimation processing and the depth estimation processing, and is executed, for example, by the estimation model MD.

[0071] (First extraction unit 121) The first extraction unit 121 extracts one or more first feature amounts from the plurality of images included in the input data IND. Specific processing by the first extraction unit 121 will be described later.

[0072] (Second extraction unit 122) The second extraction unit 122 extracts one or more second feature amounts from the depth information obtained from the input data IND. Specific processing by the second extraction unit 122 will be described later.

[0073] (Normal / depth estimation unit 123) The normal / depth estimation unit 123 performs the normal estimation process and the depth estimation process by referring to the first feature amount and the second feature amount. Specific processes performed by the normal / depth estimation unit 123 will be described later.

[0074] (Priority determination unit 13(23)) The priority determination unit 13 (23) determines the priority of each of the normal estimation process and the depth estimation process. Although the details of the priority determination process by the priority determination unit 13 (23) do not limit this exemplary embodiment, as an example, the priority may be determined by referring to at least one of the multiple images included in the input data, the result of the normal estimation process, and the result of the depth estimation process. Specific processing by the priority determination unit 13 (23) will be described later.

[0075] (Study Section 14) The learning unit 14 trains the estimation model MD using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority. A first loss value L representing the difference between the result NR of the normal estimation process and the ground truth data regarding the normal. NR and, A second loss value L representing the difference between the result DR of the depth estimation process and the ground truth data regarding depth. DR and, The priority and The learning unit 14 calculates a loss function LF according to the above, and updates a plurality of parameters that define the estimation model MD so that the value of the loss function LF becomes smaller. Specific processing by the learning unit 14 will be described later.

[0076] (Generation unit 24) The generation unit 24 generates output data by referring to the results of the normal estimation process, the results of the depth estimation process, and the priorities. The output data is, for example, three-dimensional data (also referred to as three-dimensional reconstruction data) related to the object included in the input data.

[0077] As an example, the generation unit 24 generates three-dimensional data as output data by integrating three-dimensional data obtained by referring to the result of the normal estimation process and three-dimensional data obtained by referring to the result of the depth estimation process according to their respective priorities.

[0078] As an example, the generation unit 24 While performing a process of reconstructing three-dimensional data by referring to the result of the depth estimation process, In an area where the priority of the normal estimation process is equal to or higher than a predetermined threshold (in other words, an area where the priority determination unit 23 has determined that the normal estimation process should be prioritized), the three-dimensional data in that area is replaced with the integral value of the result of the normal estimation process.

[0079] Furthermore, the generation unit 24 may output a normal map and a depth map in addition to the above three-dimensional data. In this case, the generation unit 24 may be configured to refer to the priorities determined by the priority determination unit 13 (23) and perform replacement processing using a differential value of depth in areas where the priority for normal estimation processing is equal to or less than a predetermined threshold (or smaller), and perform replacement processing using an integral value of normal in areas where the priority for depth estimation processing is equal to or greater than the predetermined threshold (or larger).

[0080] (Example 1 of the process flow in the learning phase) Next, an example of the flow of processing by the information processing device 100A will be described with reference to a specific configuration example of the information processing device 100A according to this exemplary embodiment, with reference to Fig. 7. Fig. 7 is a diagram showing example 1 of the flow of processing by the information processing device 100A in the learning phase.

[0081] First, input data IND including an image group IMG_T is acquired by an acquisition unit 11 (input data acquisition unit 11 in FIG. 7). Here, the notation IMG_T is a notation to indicate that the above-mentioned multiple images IMG are images in the learning phase, but this does not limit this exemplary embodiment. The input data IND is input to a first extraction unit 121 and a second extraction unit 122.

[0082] (First extraction unit 121) As described above, the first extraction unit 121 extracts one or more first feature amounts from the plurality of images IMG_T included in the input data IND. As shown in FIG. 7 , the first extraction unit 121 includes an image feature extraction unit 1211 and a light source image feature aggregation unit 1212.

[0083] The image feature extraction unit 1211 is realized as an encoder, for example, and executes an encoding process to extract one or more first feature values ​​from each of the plurality of images IMG_T included in the input data IND. Here, the encoding process may include a process to extract the one or more first feature values ​​for each of a plurality of regions or a plurality of data points included in each of the plurality of images IMG_T. Note that the encoding process is an example of a process executed by the above-mentioned estimation model MD.

[0084] The light source image feature aggregating unit 1212 generates aggregated first feature amounts by aggregating the first feature amounts extracted by the image feature extracting unit 1211 from each of the plurality of images IMG_T. Here, the aggregating process may be performed for each of a plurality of regions or a plurality of data points included in each of the plurality of images IMG_T. As an example, the light source image feature aggregating unit 1212 A first feature FR11 extracted from a region R1 in an image IMG_T1 obtained by capturing an image of an object OBJ under illumination condition 1; A first feature FR21 extracted from the region R1 in the image IMG_T2 obtained by capturing an image of the object OBJ under illumination condition 2; A first feature FR31 extracted from the region R1 in the image IMG_T3 obtained by capturing the object OBJ under illumination condition 3. may be configured to generate the first feature value FR1 in the region R1 by aggregating the above.

[0085] The light source image feature aggregation unit performs the aggregation process as follows: Process 1: adding up components of each dimension in each of the plurality of first feature quantities; and Process 2: A process of finding the maximum value of each dimension component in each of the multiple first feature quantities. For example, the process 1 may include the following: The first feature FR11 extracted from the above image IMG_T1 = (0.1, 0.2, ) The first feature FR21 extracted from the above image IMG_T2 = (0.2, 0.4, ) The first feature FR31 extracted from the above image IMG_T3 = (0.3, 0.1, ) By adding up the components of each dimension, First feature FR1=(0.6,0.7,) The one or more first feature amounts aggregated by the light source image feature aggregation unit 1212 are supplied to a multiple attention mechanism 1231, which will be described later.

[0086] (Second extraction unit 122) As described above, the second extraction unit 122 extracts one or more second feature amounts from the depth information obtained from the input data IND. The second extraction unit 122 includes a coarse depth estimation unit 1221 and a depth feature extraction unit 1222, as shown in FIG.

[0087] The coarse depth estimation unit 1221 generates the depth information from the multiple images included in the input data. As an example, the coarse depth estimation unit 1221 performs monocular depth estimation processing with reference to multiple images IMG_T included in the input data IND. As an example, the coarse depth estimation unit 1221 applies the monocular depth estimation processing to an image obtained by taking the average or median of each pixel of the multiple images IMG_T. Alternatively, the coarse depth estimation unit 1221 may be configured to apply the monocular depth estimation processing to each of the images included in the multiple images IMG_T. Here, as an example, the monocular depth estimation processing is performed for each of multiple regions or multiple data points included in each of the multiple images IMG_T. More specifically, as an example, the coarse depth estimation unit 1221 performs the following processing. Each pixel value of the above image IMG_T1 - Each pixel value of the above image IMG_T2 - Each pixel value of the above image IMG_T3 for each pixel to generate an averaged image IMG_T, estimate the depth for each of the multiple regions or multiple data points in the averaged image IMG_T, and supply an image (also referred to as a depth image or depth information) showing the estimation result to the depth feature extraction unit 1222. Note that the depth estimation process is an example of a process executed by the above-mentioned estimation model MD.

[0088] The depth feature extraction unit 1222 extracts one or more second feature amounts from the depth image estimated by the coarse depth estimation unit 1221. The depth feature extraction unit 1222 is realized as an encoder, for example, and executes an encoding process to extract one or more second feature amounts from the depth image. The depth feature extraction unit 1222 supplies the extracted one or more second feature amounts to a multiple attention mechanism 1231, which will be described later. Note that the encoding process is an example of a process executed by the estimation model MD described above.

[0089] (Normal / depth estimation unit 123) As described above, the normal / depth estimation unit 123 performs normal estimation processing and fine depth estimation processing by referring to the first feature amount and the second feature amount. As shown in FIG. 7 , the normal / depth estimation unit 123 includes a multiple attention mechanism 1231, a normal estimation unit 1232, and a fine depth estimation unit 1233.

[0090] The multi-attention mechanism 1231 performs multi-attention processing (for example, multi-head attention processing) by referring to the one or more first feature amounts after aggregation supplied from the light source image feature aggregation unit 1212 and the one or more second feature amounts supplied from the depth feature extraction unit 1222. The multi-attention processing performed by the multi-attention mechanism 1231 can also be expressed as processing that causes the one or more first feature amounts after aggregation supplied from the light source image feature aggregation unit 1212 to interact with the one or more second feature amounts supplied from the depth feature extraction unit 1222. As an example of the multi-attention processing, the multi-attention mechanism 1231 performs assigning the one or more first features after the aggregation and the one or more second features to one or more elements of a query Q, a key K, and a value V; For each query Q, key K, and value V, a weight W is assigned. Q i , W K i , and W V i Perform a linear transformation using QW after linear transformation Q i , K.W. K i , and VW V i Apply Scaled Dot-Product Attention to Concatenate the results of the scaled dot-product attention. The weight W of the above concatenated result O The result of the calculation is output as the result of the multi-attention processing. The multi-attention mechanism 1231 may execute, as the multi-attention process, a cross-attention process in which the one or more first features after the aggregation interact with the one or more second features.

[0091] The normal estimation unit 1232 performs normal estimation processing by referring to the result of the multi-attention processing. The normal estimation unit 1232 is realized as a decoder, for example, and performs normal estimation by a decoding process by referring to the result of the multi-attention processing. The normal estimation result by the normal estimation unit 1232 is expressed in the form of a normal estimation map, for example. The normal estimation result by the normal estimation unit 1232 is supplied to the normal / depth loss calculation unit 141 and the normal / depth priority determination unit 13.

[0092] The fine depth estimation unit 1233 performs fine depth estimation processing by referring to the result of the multi-attention processing. The fine depth estimation unit 1233 is realized as a decoder, for example, and performs fine depth estimation by a decoding process by referring to the result of the multi-attention processing. Here, the depth estimation processing by the fine depth estimation unit 1233 is more accurate than the depth estimation processing performed by the coarse depth estimation unit 1221. For example, the depth estimation result by the fine depth estimation unit 1233 is expressed in the form of a depth estimation map. The depth estimation result by the fine depth estimation unit 1233 is supplied to the normal / depth loss calculation unit 141 and the normal / depth priority determination unit 13. The multi-attention processing, normal estimation processing, and depth estimation processing are also included in the processing performed by the estimation model MD.

[0093] (Normal / depth priority determination unit 13) The normal / depth priority determination unit 13 (corresponding to the priority determination unit 13) determines the priority of each of the normal estimation process by the normal estimation unit 1232 and the depth estimation process by the fine depth estimation unit 1233. As an example, the normal / depth priority determination unit 13 determines whether to prioritize a normal or a depth for each of a plurality of regions or a plurality of data points included in the normal estimation map and the depth estimation map.

[0094] As an example, as described in the first exemplary embodiment, the normal / depth priority determination unit 13 determines the following as the respective priorities: The weighting coefficient W is multiplied by the result of the normal estimation process NR. NR A weighting coefficient W multiplied by the result DR of the depth estimation process DR Here, the normal / depth priority determination unit 13 may, for example, assign a larger weight to a process having a higher priority out of the normal estimation process and the depth estimation process.

[0095] The normal / depth priority determination unit 13 may be configured to determine the priority for each area or each data point in the normal estimation map and depth estimation map. In this configuration, the weighting coefficients calculated by the normal / depth priority determination unit 13 are calculated as follows: (W NR ) i,j , (W DR ) i,j It can also be written as:

[0096] The normal / depth priority determination unit 13 executes, as an example, any one of the examples described below or a combination thereof.

[0097] (Example 1) For example, the normal / depth priority determination unit 13 may refer to multiple images IMG_T included in the input data IND, and prioritize the weight of normal estimation in areas of the image that are sufficiently illuminated, and prioritize the weight of depth estimation in areas that are not sufficiently illuminated.

[0098] More specifically, the normal / depth priority determination unit 13 performs the following, for example: By referring to a plurality of images IMG_T included in the input data IND, the degree of light reception under each of a plurality of illumination conditions is determined for each of a plurality of regions or a plurality of data points included in the images; A higher priority is set in the normal estimation process for areas or data points that are determined to have received light at a predetermined level or more in a greater number of images among the plurality of images; A lower priority is set in the normal estimation process for areas or data points that are determined to have received light at a predetermined level or more in fewer images among the plurality of images. The following process may be performed.

[0099] The above process will be described below with reference to Fig. 8. In the example shown in Fig. 8, among the plurality of images IMG_T included in the input data IND, a region R1 and a region R2 are shown in the first image IMG1 to the third image IMG3.

[0100] 8, region R1 has a good degree of light reception in all of the first to third images IMG1 to IMG3. On the other hand, region R2 has a good degree of light reception in the first image IMG1, but the degree of light reception is reduced in the second image IMG2, and is further reduced in the third image IMG3.

[0101] In such a case, the normal / depth priority determination unit 13 may, for example, The region R1 is determined to have a predetermined level or more of light reception in all of the first image IMG1 to the third image IMG3, In the region R2, it is determined that the degree of light reception is equal to or greater than a predetermined level in the first image IMG1, but is less than the predetermined level in the second image IMG2 and the third image IMG3; Based on the above determination results, a higher priority is set for normal estimation processing in region R1, and a higher priority is set for depth estimation processing in region R2 (in other words, a lower priority is set for normal estimation processing in region R2). The following process is performed.

[0102] (Example 2) The normal / depth priority determination unit 13 may also perform a process of determining the priority for each region of multiple images IMG_T included in the input data IND so that the value of a loss function corresponding to the result of the normal estimation process, the result of the depth estimation process, and the priority is minimized. This process may also be expressed as a process of dynamically changing weights so as to minimize the loss indicated by the loss function in the loss calculation. By performing this process, the normal / depth priority determination unit 13 prioritizes loss due to normal estimation (normal loss) in regions or data points where normal estimation is performed appropriately, and prioritizes loss due to depth estimation (depth loss) in regions or data points where depth estimation is performed appropriately.

[0103] (Example 3) Furthermore, the above-described fine depth estimation unit 1233 may be configured to calculate the reliability of depth estimation in the depth estimation process, and the normal / depth priority determination unit 13 may determine the priority by referring to the reliability in the fine depth estimation unit 1233. As an example, the normal / depth priority determination unit 13 may determining a higher priority for depth estimation processing for regions or data points where the confidence level is higher; In areas or data points where the confidence is lower, a lower priority is given to the depth estimation process (in other words, a higher priority is given to the normal estimation process). The following processing may be performed.

[0104] (Example 4) The normal / depth priority determination unit 13 may be configured to present the priority of each region determined by the above process to the user via the input / output unit 40. As an example, the normal / depth priority determination unit 13 may Generate a priority map that indicates the priority of each area, The priority map is presented to the user via the input / output unit 40 together with the normal estimation map and depth estimation map described above. Alternatively, an instruction from the user to whom the priority is presented may be received, and the priority for each area may be changed in accordance with the instruction.

[0105] (Study Section 14) The learning unit 14 learns the estimation model MD using a loss function according to the results of the normal estimation process, the results of the depth estimation process, and the priority. As an example, the learning unit 14 includes a normal / depth loss calculation unit 141 and a parameter update unit 142, as shown in FIG.

[0106] The normal and depth loss calculation unit 141 A first loss value L representing the difference between the result NR of the normal estimation process and the ground truth data regarding the normal. NR and, A second loss value L representing the difference between the result DR of the depth estimation process and the ground truth data regarding depth. DR and, The priority and Calculate the loss function LF according to

[0107] More specifically, as an example, the learning unit 14 uses a weighting factor W NR , W DR Using LF=W NR ×L NR +W DR ×L DR and updating a plurality of parameters defining the estimation model so that the value of the loss function LF becomes smaller.

[0108] Furthermore, the learning unit 14 may be configured to calculate a local loss function (a loss function dependent on a region) as the loss function. In this configuration, the weighting coefficients calculated by the learning unit 14 are calculated using a two-dimensional index (i, j) that specifies each region or each data point in the normal estimation map or depth estimation map, as follows: LF i,j =(W NR ) i,j ×(L NR ) i,j +(W DR ) i,j ×(L DR ) i,j The sum of the local loss functions can be expressed as LF=ΣLF i,j Here, Σ represents the sum over the two-dimensional index (i,j).

[0109] In addition, when the plurality of images included in the input data are a plurality of images (CG, etc.) generated by an image generating device as described above, the correct answer data is A normal map and a depth map derived from the image data generated by the image generation device Furthermore, when the plurality of images included in the input data are a plurality of captured images (real-life images) captured by the imaging device 50 as described above, the following may be used as the correct answer data: Normal and depth maps obtained by referencing data obtained by measuring the captured object However, these examples are not intended to limit the present exemplary embodiment.

[0110] The parameter update unit 142 updates a plurality of parameters that define the estimation model MD so that the value of the loss function LF calculated as above becomes smaller. In other words, the parameter update unit 142 updates the parameters that define the estimation model MD so that the value of the loss function LF calculated as above becomes smaller. The process of extracting the above-mentioned first feature amounts The process of extracting the above-mentioned multiple second feature amounts - Multi-attention processing as described above The normal estimation process described above -Depth estimation process as described above The above-mentioned priority determination process At least one of one or more parameters of the estimation model MD that executes at least one of the above is updated so that the value of the loss function LF becomes smaller.

[0111] The updated parameters of the estimation model MD are stored in the storage unit 20 A. Note that the above-mentioned processes by the first extraction unit 121, the second extraction unit 122, the normal / depth estimation unit 123, the priority determination unit 13, and the learning unit 14 may be configured to be repeatedly executed multiple times until the value of the loss function LF satisfies a predetermined convergence condition.

[0112] As described above, in the information processing device 100A, in the learning phase, Obtain input data containing multiple images under multiple lighting conditions, Performing normal estimation and depth estimation using an estimation model with reference to the input data; determining the priority of each of the normal estimation process and the depth estimation process; The estimation model is trained using a loss function according to the results of the normal estimation process, the results of the depth estimation process, and the priority. In this way, the information processing device 100A determines the priority of each of the normal estimation process and the depth estimation process, and trains the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority. Therefore, with the above configuration, Reconstruction of local details of the object based on said normal estimation process; Reconstructing the global 3D shape of the object based on the depth estimation process; In other words, the above configuration makes it possible to reconstruct both the local details and the global three-dimensional shape of the object in an appropriate manner.

[0113] Note that the processing in this example is not limited to the above example. For example, a configuration may be adopted in which camera parameters are input to the coarse depth estimation unit 1221. Here, the camera parameters include, as an example, the focal length and optical center of the camera when capturing the image group IMG_T described above. Here, the camera parameters may be parameters common to multiple images included in the image group IMG_T, but this does not limit this exemplary embodiment. Furthermore, when generating the depth information from the multiple images included in the image group IMG_T, the coarse depth estimation unit 1221 may be configured to generate the depth information by monocular depth estimation assuming orthogonal projection. By performing processing in the coarse depth estimation unit 1221 that references the camera parameters, the accuracy of learning using projective projection can be improved. (Network Configuration Example in Information Processing Device 100A) Next, a configuration example of a network (a neural network is an example, the same applies below) in the information processing device 100A will be described with reference to Fig. 9. As shown in Fig. 9, each of a plurality of images (IMG1, IMG2 in Fig. 9) included in the input data IND is input to each of a plurality of networks (reference numerals 12111, 12112 in Fig. 9) constituting the image feature extraction unit 1211. Then, first feature amounts, which are outputs of these plurality of networks 12111, 12112, are aggregated by the light source image feature aggregation unit 1212.

[0114] On the other hand, each of the multiple images (IMG1, IMG2 in FIG. 9) included in the input data IND is referenced by the coarse depth estimation unit 1221 to generate a depth image (DEPTH_IMG in FIG. 9). Then, the depth image is input to a network constituting the depth feature extraction unit 1222, and one or more second feature amounts are extracted.

[0115] The aggregated first feature amount and second feature amount are input to a multi-attention mechanism 1231, where multi-attention processing is applied. The result of the multi-attention processing is then input to a network constituting a normal estimation unit 1232 and a network constituting a fine depth estimation unit 1233. The normal estimation result NR by the normal estimation unit 1232 and the depth estimation result DR by the fine depth estimation unit 1233 are then input to a network constituting a normal / depth priority determination unit 13. The normal / depth priority determination unit 13 then determines the priority of the normal estimation processing and the priority of the depth estimation processing by referring to these results. The result of the normal estimation processing, the result of the depth estimation processing, and the priority are then used in loss calculation by the learning unit 14.

[0116] (Example 1 of the processing flow in the inference phase) Next, another example of the flow of processing by the information processing device 100A will be described with reference to a specific configuration example of the information processing device 100A according to this exemplary embodiment, with reference to Fig. 10. Fig. 10 is a diagram showing example 1 of the flow of processing by the information processing device 100A in the inference phase.

[0117] First, input data IND including an image group IMG_T is acquired by an acquisition unit 21 (input data acquisition unit 21 in FIG. 10 ). Here, the notation IMG_I is a notation to indicate that the above-mentioned multiple images IMG are images in the inference phase, but this does not limit this exemplary embodiment. The input data IND is input to a first extraction unit 121 and a second extraction unit 122.

[0118] (First extraction unit 121) As described above, the first extraction unit 121 extracts one or more first feature amounts from the plurality of images IMG_T included in the input data IND. An example of the configuration and processing of the first extraction unit 121 is similar to the example described using Fig. 7, and therefore description thereof will be omitted here. However, in this example, each process by the first extraction unit 121 is executed by the inference model MD that has been trained in the above-mentioned learning phase.

[0119] (Second extraction unit 122) As described above, the second extraction unit 122 extracts one or more second features from the depth information obtained from the input data IND. An example of the configuration and processing of the second extraction unit 122 is the same as the example described using Fig. 7, and therefore a description thereof will be omitted here. However, in this example, each process by the second extraction unit 122 is executed by the inference model MD that has been trained in the above-described learning phase.

[0120] (Normal / depth estimation unit 123) As described above, the normal / depth estimation unit 123 performs normal estimation processing and fine depth estimation processing by referring to the first feature amount and the second feature amount. An example of the configuration and processing of the normal / depth estimation unit 123 is the same as the example described using Fig. 7, so description thereof will be omitted here. However, in this example, each processing by the normal / depth estimation unit 123 is performed by the inference model MD that has been trained in the above-mentioned learning phase.

[0121] (Normal / depth priority determination unit 23) The normal / depth priority determination unit 23 (corresponding to the priority determination unit 23) determines the priority of each of the normal estimation process by the normal estimation unit 1232 and the depth estimation process by the fine depth estimation unit 1233. An example configuration and processing of the normal / depth priority determination unit 23 are similar to the example configuration and processing of the normal / depth priority determination unit 13 described using Fig. 7, and therefore description thereof will be omitted here. However, in this example, each processing by the normal / depth priority determination unit 23 can be executed by the inference model MD that has been trained in the above-mentioned learning phase.

[0122] (Output data generation unit 24) The output data generation unit 24 (generation unit 24) generates output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. The output data is, for example, three-dimensional data (also referred to as three-dimensional reconstruction data) related to the object included in the input data.

[0123] The output data generation unit 24, for example, generates three-dimensional data as output data by integrating three-dimensional data obtained by referring to the result of the normal estimation process and three-dimensional data obtained by referring to the result of the depth estimation process according to the priority of each.

[0124] As an example, the output data generation unit 24 While performing a process of reconstructing three-dimensional data by referring to the result of the depth estimation process, In an area where the priority of the normal estimation process is equal to or higher than a predetermined threshold (in other words, an area where the normal / depth priority determination unit 23 has determined that the normal estimation process should be prioritized), the 3D data in that area is replaced with the integral value of the result of the normal estimation process.

[0125] Furthermore, the output data generation unit 24 may output a normal map and a depth map in addition to the above three-dimensional data. In this case, the output data generation unit 24 may be configured to refer to the priorities determined by the priority determination unit 13 (23) and perform replacement processing using a depth differential value in areas where the priority for normal estimation processing is equal to or less than a predetermined threshold (or smaller), and to perform replacement processing using a normal integral value in areas where the priority for depth estimation processing is equal to or greater than the predetermined threshold (or larger).

[0126] As described above, in the information processing device 100A, in the inference phase, Obtain input data containing multiple images under multiple lighting conditions, Execute normal estimation processing and depth estimation processing by referring to the input data; determining the priority of each of the normal estimation process and the depth estimation process; Generating output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. According to the above configuration, the priority of each of the normal estimation process and the depth estimation process is determined, and output data is generated by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority. Reconstruction of local details of the object based on said normal estimation process; Reconstructing the global 3D shape of the object based on the depth estimation process; In other words, the above configuration makes it possible to reconstruct both the local details and the global three-dimensional shape of the object in an appropriate manner.

[0127] (Example 2 of the process flow in the learning phase) Next, another example of the flow of processing by the information processing device 100A will be described with reference to a specific configuration example of the information processing device 100A according to this exemplary embodiment, with reference to Fig. 11. Fig. 11 is a diagram showing example 2 of the flow of processing by the information processing device 100A in the learning phase.

[0128] 11, in this example, the second extraction unit 122 does not include a coarse depth estimation unit 1221. In this example, a depth image is included in the input data IND acquired by the input data acquisition unit 11, and the second unit 122 is configured to extract the one or more second feature amounts from the depth image as depth information using a depth feature extraction unit 1222.

[0129] As an example, in this exemplary embodiment, the imaging device 50 includes a depth camera, which Depth data (depth image) in which each pixel (data point) represents a depth value, and represents the object OBJ Alternatively, the imaging device 50 may be provided with a line laser scanner, and the line laser scanner may be used to obtain 3D point cloud data where each data point represents a 3D coordinate and represents an object OBJ and include the data in the input data IND. Other configurations and processes according to this example are the same as those described with reference to Fig. 7, and therefore will not be described here. Even with such a configuration, the above-mentioned effects can be achieved.

[0130] (Example 2 of the processing flow in the inference phase) Next, another example of the flow of processing by the information processing device 100A will be described with reference to a specific configuration example of the information processing device 100A according to this exemplary embodiment, with reference to Fig. 12. Fig. 12 is a diagram showing example 2 of the flow of processing by the information processing device 100A in the inference phase.

[0131] As shown in Fig. 12, in this example, the second extraction unit 122 does not include a coarse depth estimation unit 1221. In this example, a depth image is included in the input data IND acquired by the input data acquisition unit 11, and the second unit 122 employs a configuration in which the depth feature extraction unit 1222 extracts the one or more second feature amounts from the depth image as depth information. Other configurations and processes according to this example are similar to those described using Fig. 10, and therefore will not be described here. Even with this configuration, the above-described effects can be achieved.

[0132] Third Embodiment A second exemplary embodiment, which is one example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same functions as those described in the above exemplary embodiment will be assigned the same reference numerals, and their description will be omitted as appropriate. The scope of application of each technology employed in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology employed in this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs. Furthermore, each technology shown in each drawing referenced to explain this exemplary embodiment can also be employed in other exemplary embodiments included in the present disclosure, to the extent that no particular technical hindrance occurs.

[0133] (Configuration of information processing system 1B) The configuration of an information processing system 1B according to this exemplary embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the configuration of the information processing system 1B. As shown in Fig. 13, the information processing system 1B differs from the configuration described in exemplary embodiment 2 in that the information processing device 100B does not include a learning unit 14, but is otherwise similar to exemplary embodiment 2.

[0134] The estimation unit 12 (22) and the priority determination unit 13 (23) according to this exemplary embodiment execute each process related to the inference phase, for example, using the inference model MD learned by the information processing device 100A described above. The content of each process related to the inference phase is similar to the process in the inference phase executed by the information processing device 100A described above, and therefore description thereof will be omitted here.

[0135] [Software implementation example] Some or all of the functions of the information processing devices 1, 2, 100A, and 100B (hereinafter also referred to as "each of the above devices") may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0136] In the latter case, each of the above devices is realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 14. Figure 14 is a block diagram showing the hardware configuration of computer C that functions as each of the above devices.

[0137] The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as each of the above-mentioned devices. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing the functions of each of the above-mentioned devices.

[0138] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0139] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0140] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0141] [Appendix A] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0142] (Appendix A1) acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data; a priority determination means for determining a priority of each of the normal estimation process and the depth estimation process; a generating means for generating output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority; An information processing device comprising:

[0143] (Appendix A2) The priority determination means The priority is determined by referring to at least one of the plurality of images included in the input data, the result of the normal estimation process, and the result of the depth estimation process. 10. The information processing device according to claim 1,

[0144] (Appendix A3) The estimation means a first extraction means for extracting one or more first feature amounts from the plurality of images included in the input data; a second extraction means for extracting one or more second feature amounts from depth information obtained from the input data; a normal depth estimation means for performing the normal estimation process and the depth estimation process by referring to the first feature amount and the second feature amount; The information processing device according to appendix A2,

[0145] (Appendix A4) The second extraction means a depth information generating means for generating the depth information from the plurality of images included in the input data; 10. The information processing device according to claim 9, wherein the information processing device is a

[0146] (Appendix A5) the input data includes a depth image; The second extraction means extracts the one or more second feature amounts from the depth image as the depth information. 10. The information processing device according to claim 9, wherein the information processing device is a

[0147] (Appendix A6) The normal depth estimation means a multi-attention processing means for performing multi-attention processing by referring to the first feature amount and the second feature amount; a normal estimation means for executing the normal estimation process by referring to a result of the multi-attention process; a depth estimation means for executing the depth estimation process by referring to a result of the multi-attention process; 10. The information processing device according to claim 9, further comprising:

[0148] (Appendix A7) The priority determination means With reference to the plurality of images included in the input data, a degree of light reception in each of a plurality of regions included in the image is determined under each of the plurality of illumination conditions; A higher priority is set in the normal estimation process for an area in which it is determined that a greater number of images among the plurality of images have received light at a predetermined level or more; A lower priority is set in the normal estimation process for an area where it is determined that light is received at a predetermined level or more in fewer images among the plurality of images. An information processing device according to any one of appendices A1 to A6.

[0149] (Appendix A8) The priority determination means The priority is determined so that a value of a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority is reduced. An information processing device according to any one of appendices A1 to A7.

[0150] (Appendix A9) the estimation means executes the normal estimation process and the depth estimation process using an estimation model; The information processing device includes: a learning means for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; The information processing device according to any one of appendices A1 to A8, further comprising:

[0151] (Appendix A10) acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data and using an estimation model; a priority determination means for determining a priority of each of the normal estimation process and the depth estimation process; a learning means for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; An information processing device comprising:

[0152] [Appendix B] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0153] (Appendix B1) an acquisition process in which at least one processor acquires input data comprising a plurality of images under a plurality of lighting conditions; an estimation process in which the at least one processor performs a normal estimation process and a depth estimation process by referring to the input data; a priority determination process in which the at least one processor determines a priority of each of the normal estimation process and the depth estimation process; a generation process in which the at least one processor generates output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority; An information processing method comprising:

[0154] (Appendix B2) In the priority determination process, the at least one processor The priority is determined by referring to at least one of the plurality of images included in the input data, the result of the normal estimation process, and the result of the depth estimation process. 1. The information processing method described in Appendix B1.

[0155] (Appendix B3) The estimation process includes: a first extraction process in which the at least one processor extracts one or more first feature amounts from the plurality of images included in the input data; a second extraction process in which the at least one processor extracts one or more second feature amounts from depth information obtained from the input data; a normal depth estimation process in which the at least one processor executes the normal estimation process and the depth estimation process by referring to the first feature amount and the second feature amount; 2. An information processing method according to claim 1, comprising:

[0156] (Appendix B4) The second extraction process includes: The at least one processor includes a depth information generation process for generating the depth information from the plurality of images included in the input data. The information processing method described in Appendix B3.

[0157] (Appendix B5) the input data includes a depth image; The second extraction process extracts the one or more second feature amounts from the depth image as the depth information. The information processing method described in Appendix B3.

[0158] (Appendix B6) The normal depth estimation process includes: a multi-attention process performed by the at least one processor with reference to the first feature amount and the second feature amount; a normal estimation process in which the at least one processor executes the normal estimation process by referring to a result of the multi-attention process; a depth estimation process in which the at least one processor executes the depth estimation process by referring to a result of the multi-attention process; An information processing method according to any one of appendices B3 to B5, comprising:

[0159] (Appendix B7) In the priority determination process, the at least one processor With reference to the plurality of images included in the input data, a degree of light reception in each of a plurality of regions included in the image is determined under each of the plurality of illumination conditions; A higher priority is set in the normal estimation process for an area in which it is determined that a greater number of images among the plurality of images have received light at a predetermined level or more; A lower priority is set in the normal estimation process for an area where it is determined that light is received at a predetermined level or more in fewer images among the plurality of images. 10. An information processing method according to any one of appendices B1 to B6.

[0160] (Appendix B8) In the priority determination process, the at least one processor The priority is determined so that a value of a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority is reduced. 10. An information processing method according to any one of appendices B1 to B7.

[0161] (Appendix B9) the estimation process performs the normal estimation process and the depth estimation process using an estimation model; The information processing device includes: a learning process in which the at least one processor learns the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; The information processing method according to any one of appendices B1 to B8, further comprising:

[0162] (Appendix B10) an acquisition process in which the at least one processor acquires input data comprising a plurality of images under a plurality of lighting conditions; an estimation process in which the at least one processor performs a normal estimation process and a depth estimation process by referring to the input data and using an estimation model; a priority determination process in which the at least one processor determines a priority of each of the normal estimation process and the depth estimation process; a learning process in which the at least one processor learns the estimation model using a loss function according to a result of the normal estimation process, a result of the depth estimation process, and the priority; An information processing method comprising:

[0163] [Appendix C] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0164] (Appendix C1) A program that causes a computer to function as an information processing device, The computer acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data; a priority determination means for determining a priority of each of the normal estimation process and the depth estimation process; a generating means for generating output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority; An information processing program that functions as a

[0165] (Appendix C2) The priority determination means The priority is determined by referring to at least one of the plurality of images included in the input data, the result of the normal estimation process, and the result of the depth estimation process. An information processing program as described in Appendix C1.

[0166] (Appendix C3) The estimation means causes the computer to a first extraction means for extracting one or more first feature amounts from the plurality of images included in the input data; a second extraction means for extracting one or more second feature amounts from depth information obtained from the input data; a normal depth estimation process that performs the normal estimation process and the depth estimation process by referring to the first feature amount and the second feature amount; 2. The information processing program according to claim 1, wherein the information processing program functions as

[0167] (Appendix C4) The second extraction means includes: a depth information generating process for generating the depth information from the plurality of images included in the input data; An information processing program as described in Appendix C3.

[0168] (Appendix C5) the input data includes a depth image; The second extraction means extracts the one or more second feature amounts from the depth image as the depth information. An information processing program as described in Appendix C3.

[0169] (Appendix C6) The normal depth estimation means causes the computer to a multi-attention processing means for performing multi-attention processing by referring to the first feature amount and the second feature amount; a normal estimation means for executing the normal estimation process by referring to a result of the multi-attention process; a depth estimation process that performs the depth estimation process by referring to a result of the multi-attention process; The information processing program according to any one of appendices C3 to C5,

[0170] (Appendix C7) The priority determination means With reference to the plurality of images included in the input data, a degree of light reception in each of a plurality of regions included in the image is determined under each of the plurality of illumination conditions; A higher priority is set in the normal estimation process for an area in which it is determined that a greater number of images among the plurality of images have received light at a predetermined level or more; A lower priority is set in the normal estimation process for an area where it is determined that light is received at a predetermined level or more in fewer images among the plurality of images. An information processing program according to any one of appendices C1 to C6.

[0171] (Appendix C8) The priority determination means The priority is determined so that a value of a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority is reduced. An information processing program according to any one of appendices C1 to C7.

[0172] (Appendix C9) the estimation means executes the normal estimation process and the depth estimation process using an estimation model; The information processing device includes: The computer a learning means for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; The information processing program according to any one of appendices C1 to C8, further functioning as:

[0173] (Appendix C10) The computer acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data and using an estimation model; a priority determination means for determining a priority of each of the normal estimation process and the depth estimation process; a learning process for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; An information processing program that functions as a

[0174] [Appendix D] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0175] (Appendix D1) at least one processor, an acquisition process for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation process that performs a normal estimation process and a depth estimation process by referring to the input data; a priority determination process for determining priorities of the normal estimation process and the depth estimation process; a generation process for generating output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority; An information processing device that executes the above.

[0176] The information processing device may further include a memory, and the memory may store a program for causing the at least one processor to execute each of the processes.

[0177] (Appendix D2) In the priority determination process, the at least one processor The priority is determined by referring to at least one of the plurality of images included in the input data, the result of the normal estimation process, and the result of the depth estimation process. 10. The information processing device according to claim 9, wherein the information processing device is a device for processing information.

[0178] (Appendix D3) In the estimation process, the at least one processor a first extraction process for extracting one or more first feature amounts from the plurality of images included in the input data; a second extraction process for extracting one or more second feature amounts from depth information obtained from the input data; a normal depth estimation process that performs the normal estimation process and the depth estimation process by referring to the first feature amount and the second feature amount; The information processing device according to appendix D2,

[0179] (Appendix D4) The second extraction process includes: performing a depth information generation process for generating the depth information from the plurality of images included in the input data; 10. The information processing device according to claim 9, wherein the information processing device is an information processing device according to claim 1, wherein

[0180] (Appendix D5) the input data includes a depth image; The second extraction process extracts the one or more second feature amounts from the depth image as the depth information. 10. The information processing device according to claim 9, wherein the information processing device is an information processing device according to claim 1, wherein

[0181] (Appendix D6) In the normal depth estimation process, the at least one processor: multi-attention processing with reference to the first feature amount and the second feature amount; a normal estimation process that executes the normal estimation process by referring to a result of the multi-attention process; a depth estimation process that performs the depth estimation process by referring to a result of the multi-attention process; The information processing device according to any one of appendices D3 to D5,

[0182] (Appendix D7) In the priority determination process, the at least one processor With reference to the plurality of images included in the input data, a degree of light reception in each of a plurality of regions included in the image is determined under each of the plurality of illumination conditions; A higher priority is set in the normal estimation process for an area in which it is determined that a greater number of images among the plurality of images have received light at a predetermined level or more; A lower priority is set in the normal estimation process for an area where it is determined that light is received at a predetermined level or more in fewer images among the plurality of images. An information processing device according to any one of appendices D1 to D6.

[0183] (Appendix D8) In the priority determination process, the at least one processor The priority is determined so that a value of a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority is reduced. An information processing device according to any one of appendices D1 to D7.

[0184] (Appendix D9) In the estimation process, the at least one processor performs the normal estimation process and the depth estimation process using an estimation model; The information processing device, by the at least one processor, a learning process for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; The information processing device according to any one of appendices D1 to D8, further executing the following.

[0185] (Appendix D10) The at least one processor: an acquisition process for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation process that refers to the input data and performs a normal estimation process and a depth estimation process using an estimation model; a priority determination process for determining priorities of the normal estimation process and the depth estimation process; a learning process for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; An information processing device that executes the above.

[0186] [Appendix E] This disclosure includes the techniques described in the following appendices. However, the present invention is not limited to the techniques described in the following appendices, and various modifications are possible within the scope of the claims.

[0187] (Appendix E1) A program that causes a computer to function as an information processing device, The computer, an acquisition process for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation process that performs a normal estimation process and a depth estimation process by referring to the input data; a priority determination process for determining priorities of the normal estimation process and the depth estimation process; a generation process for generating output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority; A non-transitory recording medium on which an information processing program for executing the above is recorded. [Explanation of symbols]

[0188] 1, 2, 100A, 100B ···Information processing equipment 11,21 ···Acquisition unit (acquisition means) 12,22... Estimation unit (estimation means) 13,23...Priority determination unit (priority determination means) 14 Learning section (learning means) 24...Generation unit (generation means)

Claims

1. acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data; a priority determination means for determining a priority of each of the normal estimation process and the depth estimation process; a generating means for generating output data by referring to a result of the normal estimation process, a result of the depth estimation process, and the priority; An information processing device comprising:

2. The priority determination means The priority is determined by referring to at least one of the plurality of images included in the input data, the result of the normal estimation process, and the result of the depth estimation process. The information processing device according to claim 1 .

3. The estimation means a first extraction means for extracting one or more first feature amounts from the plurality of images included in the input data; a second extraction means for extracting one or more second feature amounts from depth information obtained from the input data; a normal depth estimation means for performing the normal estimation process and the depth estimation process by referring to the first feature amount and the second feature amount; The information processing device according to claim 2 , further comprising:

4. The second extraction means a depth information generating means for generating the depth information from the plurality of images included in the input data; The information processing device according to claim 3 .

5. the input data includes a depth image; The second extraction means extracts the one or more second feature amounts from the depth image as the depth information. The information processing device according to claim 3 .

6. The normal depth estimation means a multi-attention processing means for performing a multi-attention process by referring to the first feature amount and the second feature amount; a normal estimation means for executing the normal estimation process by referring to a result of the multi-attention process; a depth estimation means for executing the depth estimation process by referring to a result of the multi-attention process; The information processing device according to claim 3 , further comprising:

7. the estimation means executes the normal estimation process and the depth estimation process using an estimation model; The information processing device includes: a learning means for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; The information processing device according to claim 1 , further comprising:

8. acquisition means for acquiring input data including a plurality of images under a plurality of lighting conditions; an estimation means for performing a normal estimation process and a depth estimation process by referring to the input data and using an estimation model; a priority determination means for determining a priority of each of the normal estimation process and the depth estimation process; a learning means for learning the estimation model using a loss function according to the result of the normal estimation process, the result of the depth estimation process, and the priority; An information processing device comprising:

9. acquiring input data comprising a plurality of images under a plurality of lighting conditions; performing a normal estimation process and a depth estimation process by referring to the input data; determining a priority of each of the normal estimation process and the depth estimation process; generating output data by referring to the result of the normal estimation process, the result of the depth estimation process, and the priority; An information processing method comprising:

10. 2. A program for causing a computer to function as the information processing device according to claim 1, the program causing a computer to function as the acquiring means, the estimating means, the priority determining means, and the generating means.

Images