Control device, control method, and recording medium

Abstract

Provided is a control device comprising: a shape identification unit that identifies the shape of an object; a determination unit that determines a viewpoint according to the identified shape; a control unit that controls the imaging device so as to capture an image of the object at the determined viewpoint; and a modeling unit that models the object using NeRF on the basis of the image captured by the imaging device.

Description

Control device, control method, and recording medium

[0001] The present disclosure relates to a control device and the like.

[0002] There are techniques for modeling objects in real space. For example, NeRF (Neural Radiance Fields) is known. NeRF is a technique for generating a free viewpoint image from multiple images captured from various viewpoints. For example, Patent Literature 1 describes that NeRF learns a multilayer perceptron (MLP) function that inputs images from multiple viewpoints and the camera posture at that time and outputs the brightness and density of each point in the captured space.

[0003] International Publication No. 2023 / 281802

[0004] To model an object using NeRF, multiple images of the object are required. Because it is difficult to understand technologies such as NeRF and capture images of the object, it can be difficult to recreate the object.

[0005] An example of an object of the present disclosure is to provide a control device or the like that improves the reproducibility of an object.

[0006] A control device according to one aspect of the present disclosure includes: a shape identification means for identifying the shape of an object; a determination means for determining a viewpoint according to the identified shape; a control means for controlling an imaging device to image the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on an image captured by the imaging device.

[0007] A control device according to one aspect of the present disclosure includes: a detection means for detecting a change in gloss from an image representing an object; a determination means for determining a viewpoint from which the detected change in gloss can be captured; a control means for controlling the imaging device to capture an image of the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on the image captured by the imaging device.

[0008] A control device according to one aspect of the present disclosure includes: a material identification means for identifying a material of an object; a determination means for determining a viewpoint based on the identified material; a control means for controlling an imaging device to capture an image of the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on an image captured by the imaging device.

[0009] In one aspect of the present disclosure, a control method includes a computer performing the following processes: identifying the shape of an object; determining a viewpoint according to the identified shape; controlling an imaging device to capture an image of the object from the determined viewpoint; and modeling the object using NeRF based on an image captured by the imaging device.

[0010] In one aspect of the present disclosure, a control method includes a computer executing the following processes: detecting a change in gloss from an image representing an object; determining a viewpoint from which the detected change in gloss can be captured; controlling the imaging device to capture an image of the object from the determined viewpoint; and modeling the object using NeRF based on the image captured by the imaging device.

[0011] In one aspect of the present disclosure, a control method includes a computer performing the following processes: identifying the material of an object; determining a viewpoint based on the identified material; controlling an imaging device to capture an image of the object from the determined viewpoint; and modeling the object using NeRF based on an image captured by the imaging device.

[0012] A program in one aspect of the present disclosure causes a computer to perform the following processes: identify the shape of an object; determine a viewpoint according to the identified shape; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

[0013] A program in one aspect of the present disclosure causes a computer to perform the following processes: detect changes in gloss from an image representing an object; determine a viewpoint from which the detected change in gloss can be captured; control the imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on the image captured by the imaging device.

[0014] A program in one aspect of the present disclosure causes a computer to perform the following processes: identify the material of an object; determine a viewpoint based on the identified material; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

[0015] Each program may be stored in a non-transitory computer-readable recording medium.

[0016] According to the present disclosure, it is possible to improve the degree of reproduction of an object.

[0017] 1 is an explanatory diagram showing an example of the configuration of a control system including a control device. FIG. 1 is a block diagram showing an example of the configuration of a control device. FIG. 2 is an explanatory diagram showing an example of a table correlating shapes with the number of viewpoints and the positions of the viewpoints. FIG. 3 is a flowchart showing an example of the operation of a control device. FIG. 4 is a block diagram showing an example of the configuration of a control device. FIG. 5 is a flowchart showing an example of the operation of a control device. FIG. 6 is an explanatory diagram showing an example of the configuration of a control system including a control device. FIG. 7 is a block diagram showing an example of the configuration of a control device. FIG. 8 is an explanatory diagram showing the positional relationship between a light source and a viewpoint. FIG. 9 is an explanatory diagram showing an example of a table correlating materials with the positional relationship between an imaging device and a light source and the amount of light. FIG. 9 is a flowchart showing an example of the operation of a control device. FIG. 10 is an explanatory diagram showing an example of the configuration of a control system including a control device. FIG. 11 is a block diagram showing an example of the configuration of a control device. FIG. 12 is an explanatory diagram showing an example of imaging by an imaging device provided in a drone. FIG. 13 is a flowchart showing an example of the operation of a control device. FIG. 14 is a flowchart showing an example of the operation of a control device.

[0018] Hereinafter, embodiments of a control device, a control method, a program, and a non-transitory recording medium for recording the program according to the present disclosure will be described in detail with reference to the accompanying drawings. The disclosed technology is not limited to these embodiments.

[0019] Here, we briefly explain NeRF. For example, NeRF is a method for representing 3D (three-dimensional) objects using deep learning. When an object is modeled using NeRF, the object's shape, surface gloss, and other characteristics are expressed mathematically. Unlike typical 3D mesh models, NeRF can reproduce different appearances depending on the viewpoint. For example, in modeling that creates 3D mesh data, objects of the same color are modeled identically. In contrast, NeRF can reproduce differences in the appearance of an object due to how light hits the object and the material. Differences in appearance include, for example, gloss. For this reason, it is desirable to include images of the object that serve as the basis for modeling, captured from various viewpoints. When using NeRF, the greater the number of viewpoints, the higher the reproduction quality of the model. However, simply increasing the number of viewpoints is impractical. On the other hand, randomly determining the viewpoint of the imaging device may result in a decrease in the reproduction quality of the object. In each embodiment, the control device can improve the reproduction quality of the object by determining the viewpoint for imaging the object used in NeRF.

[0020] First Embodiment A first embodiment will be described in detail with reference to the drawings. In the first embodiment, a control device identifies the shape of an object and controls an imaging device to capture an image of the object from a viewpoint corresponding to the shape.

[0021] 1 is an explanatory diagram showing an example of the configuration of a control system including a control device. The control system 1 includes a control device 10 and an imaging device 11.

[0022] For example, the control device 10 is connected to an imaging device 11 via a communication network. The imaging device 11 captures an image of an object. For example, the imaging device 11 may be fixedly disposed so as to be able to capture an image of the object, or may be able to change its position and orientation so as to be able to capture an image of the object. For example, the imaging device 11 may be movably mounted on a rail. Furthermore, for example, the imaging device 11 may be mounted on a mobile body so as to be able to move. The mobile body may be, for example, an automated guided vehicle (AGV), a drone, or the like. Note that in the fourth embodiment, an example in which the imaging device 11 is mounted on a drone will be described.

[0023] 2 is a block diagram showing an example of the configuration of the control device 10. In FIG. 2, the control device 10 includes a shape specifying unit 101, a determining unit 103, a control unit 105, and a modeling unit 107.

[0024] The shape specifying unit 101 specifies the shape of the object. The shape specifying unit 101 is an example of a shape specifying means.

[0025] As a method for the shape identification unit 101 to identify the shape of an object, the shape identification unit 101 may identify the shape of the object from a plurality of different shapes based on an image representing the object. The image representing the object may be an image of the object captured, or a reproduced image of the object obtained by modeling, and is not particularly limited. The reproduced image of the object obtained by modeling is an image of an object represented by three-dimensional model data obtained by modeling. Here, an image of the object captured will be used as an example for explanation. For example, the shape identification unit 101 recognizes the object from the image of the object captured. More specifically, for example, the shape identification unit 101 identifies shape features of the object from the image of the object captured, and identifies a shape corresponding to the shape features of the object from a plurality of shapes stored in advance in a database or the like.

[0026] The plurality of shapes may be predetermined depending on the application. For example, in a solution that reproduces household items, the plurality of shapes may be predetermined, such as a refrigerator, a washing machine, a bed, a shelf, etc.

[0027] The shape specification unit 101 may also specify the complexity of the shape of the object. Examples of complex shapes include shapes with concaves and convexes, shapes with blind spots, shapes with shadows, non-flat shapes, and shapes with sophisticated designs. Taking a refrigerator as an example, the handle of the refrigerator door is a blind spot or a shadowed part. For example, a non-flat shape is a shape that cannot be expressed with straight lines and planes. For example, designer furniture has a sophisticated shape and is likely to have a complex shape. For example, the shape specification unit 101 may specify whether the object has concaves and convexes, blind spots, or shadows. Examples of non-complex shapes include flat shapes. Indicators such as the degree of complexity may be used to indicate the complexity of the shape of the object. For example, the shape specification unit 101 may specify whether the object has an concave-convex shape. If the object has an concave-convex shape, the shape specification unit 101 may specify that the object has a high degree of shape complexity.

[0028] As another method for the shape identification unit 101 to identify the shape of an object, the shape identification unit 101 may recognize the shape of the object based on distance data obtained by measuring the distance to the object using a distance measurement sensor. The distance data is an example of information indicating the distance from a predetermined position to the object. The type of distance measurement sensor is not particularly limited. Here, a time-of-flight (TOF) camera will be used as an example of the distance measurement sensor. For example, if the distance measurement sensor is a TOF camera, the distance data may be an image of the object captured by the TOF camera. For example, the shape identification unit 101 may identify a portion of the object with a difference in elevation based on the distance data. For example, a portion with a difference in elevation is a portion with a difference of a predetermined distance or more. The shape identification unit 101 may also identify a portion with unevenness. The image may be captured using a TOF camera in response to a user operation. An example of capturing an image by a user using a TOF camera will be described in the fourth embodiment.

[0029] As another method for the shape specifying unit 101 to specify the shape of an object, the shape specifying unit 101 may specify the shape of the object from digitized drawing data of the object. In the case of commercially available furniture or home appliances, drawing data of the object may be obtained. The drawing data may be two-dimensional data such as a floor plan, or may be three-dimensional data.

[0030] The determination unit 103 determines viewpoints according to the identified shape. The determination unit 103 is an example of a determination means. For example, the determination unit 103 determines at least one of the number of viewpoints and the viewpoint positions according to the identified shape. For example, a table in which shapes are associated with the number of viewpoints and the viewpoint positions according to the shape may be prepared in advance, such as a rule base. The determination unit 103 may determine the number of viewpoints and the viewpoint positions corresponding to the shape identified by the shape identification unit 101 from the table.

[0031] 3 is an explanatory diagram showing an example of a table that associates shapes with the number of viewpoints and the positions of the viewpoints. The table associates shapes with the number of viewpoints and the positions of the viewpoints, for example. In the table, examples of shapes include a refrigerator and a curtain.

[0032] Taking a refrigerator as an example, if there is an image of the front of the refrigerator, it may be possible to reproduce the back of the refrigerator even if there is no image of the back of the refrigerator. The front of the refrigerator is the side where the refrigerator door is located. However, the shape of the front of the refrigerator changes, for example, because the door can be loosely fastened. On the other hand, the back of the refrigerator often has a flat shape, and may be inferred from the front of the refrigerator. For this reason, the determination unit 103 determines the viewpoints so that there are many viewpoints for capturing images of the front of the refrigerator and few viewpoints for capturing images of the back of the refrigerator.

[0033] When the shape specification unit 101 specifies the complexity of the shape, the determination unit 103 determines the viewpoints so that the number of viewpoints increases as the specified shape becomes more complex. For example, the determination unit 103 may determine the viewpoints so that the number of viewpoints increases as the degree of complexity increases. Furthermore, for example, the determination unit 103 may determine the viewpoints so that the number of viewpoints decreases as the degree of complexity decreases.

[0034] Furthermore, for example, if the shape of the object is flat, the determination unit 103 may determine the viewpoints so as to circle the object at equal intervals. For example, if the shape of the object is not flat, the determination unit 103 determines the viewpoints so as to be able to concentrate on capturing images of portions where continuity is broken. If the shape of the object includes flat portions and non-flat portions, the determination unit 103 determines the viewpoints so that the number of viewpoints in the flat portions is small and the number of viewpoints in the non-flat portions is large.

[0035] For example, the determination unit 103 determines a position at which a part of the object that has a difference in elevation can be imaged as the viewpoint.

[0036] The control unit 105 controls the imaging device 11 so as to capture an image of the object from the determined viewpoint. The control unit 105 is an example of a control means. The control unit 105 controls at least one of the position and orientation of the imaging device 11 so as to capture an image of the object from the determined viewpoint. The specific method by which the control unit 105 controls the position and orientation of the imaging device 11 is not particularly limited, and any existing technology may be used. For example, if the imaging device 11 is movably mounted on a rail, the control unit 105 may move the imaging device 11 so as to capture an image of the object from the determined viewpoint. For example, if the imaging device 11 is mounted on a moving object, the control unit 105 may move the moving object so as to capture an image of the object from the determined viewpoint.

[0037] Then, for example, after at least one of the position and the orientation has been controlled, the control unit 105 causes the imaging device 11 to capture an image of the object.

[0038] When multiple viewpoints are determined, the control unit 105 controls at least one of the position and the orientation of the imaging device 11 for each of the multiple viewpoints, causing the imaging device 11 to capture an image of the target object.

[0039] The modeling unit 107 models an object using NeRF based on an image captured by the imaging device 11. The modeling unit 107 is an example of a modeling means. Existing technology may be used as a method for modeling an object using NeRF. For example, the modeling unit 107 uses as input, in addition to the image, the viewpoint position of the imaging device 11 when capturing the image and the orientation of the imaging device 11. For example, the modeling unit 107 generates a model that represents the object using a mathematical formula. The modeling unit 107 may then generate three-dimensional model data based on the model. For example, the modeling unit 107 can generate three-dimensional model data of the object when viewed from a specific viewpoint. There may be multiple specific viewpoints. Therefore, the modeling unit 107 may generate three-dimensional model data of the object when viewed from each position in 360 degrees.

[0040] 4 is a flowchart showing an example of the operation of the control device 10. The shape specifying unit 101 specifies the shape of the object (step S101). Next, the determining unit 103 determines a viewpoint according to the shape (step S102).

[0041] The control unit 105 controls the image capturing device 11 to capture an image of the object from the determined viewpoint (step S103). The modeling unit 107 then models the object using NeRF based on the captured image (step S104). The control device 10 then ends the series of processes shown in FIG.

[0042] As described above, when NeRF is used, the greater the number of viewpoints, the higher the reproducibility of a newly generated free-viewpoint image of an object. However, in reality, capturing an object from a large number of different viewpoints can be difficult. For example, when using a drone, battery issues can arise. Therefore, in order to actually prepare images for NeRF, it is desirable to limit the number of viewpoints. However, it is difficult for a typical user to understand the NeRF technology and capture an object. For example, a large number of unnecessary images may be captured, or necessary images may not be captured. As such, it may be difficult to reproduce an object using NeRF. Therefore, in the first embodiment, the control device 10 determines a viewpoint according to the shape of the object and controls the imaging device 11 to capture the object from the determined viewpoint. Then, the control device 10 models the object using NeRF based on the captured image. This allows the viewpoints to be narrowed down depending on the object. Therefore, the reproducibility of the object can be improved.

[0043] Second Embodiment A second embodiment will be described in detail with reference to the drawings. In the second embodiment, a control device detects changes in glossiness from a reproduced image of an object obtained by modeling, and controls an imaging device to capture an image of a portion where the glossiness changes from a viewpoint that can capture the image. Below, descriptions that overlap with the above description will be omitted to the extent that the description of the second embodiment is not unclear.

[0044] An example of a control system including the control device may be the same as the example described in Fig. 1, and therefore detailed description thereof will be omitted. Note that the reference numeral for the imaging device is the imaging device 11 described in Fig. 1.

[0045] 5 is a block diagram showing an example of the configuration of the control device 20. The control device 20 includes a determination unit 203, a control unit 205, a modeling unit 207, and a detection unit 209.

[0046] The determination unit 203 may have a function similar to that of the determination unit 103 shown in Fig. 2. The control unit 205 may have a function similar to that of the control unit 105 shown in Fig. 2. The modeling unit 207 may have a function similar to that of the modeling unit 107 shown in Fig. 2.

[0047] The detection unit 209 detects a change in glossiness from an image representing an object. The detection unit 209 is an example of a detection means. The image representing an object may be an image of the object captured in advance, an image of the object captured by the imaging device 11, or a reproduced image of the object obtained by modeling using NeRF. Here, a reproduced image of an object obtained by modeling using NeRF will be described as an example. As described above, NeRF can reproduce the glossiness, etc., of an object. Therefore, the detection unit 209 detects a change in glossiness from the reproduced image of the object obtained by modeling. For example, a portion where glossiness changes may be a portion with a complex shape, such as unevenness or a different material. Therefore, it is expected that a portion where glossiness changes needs to be reproduced more accurately. Here, the reproduced image may be a reproduced image from a viewpoint specified by the user, and is not particularly limited. Furthermore, a portion where glossiness ends is likely to be a change in shape. Therefore, the detection unit 209 detects a portion where glossiness ends as a portion where glossiness changes from the image representing the object.

[0048] The determination unit 203 determines a viewpoint from which an image of the detected gloss change portion of the object can be captured. The determination unit 203 is an example of a determination means. The determination unit 203 identifies the detected gloss change portion of the object and determines a viewpoint from which an image of the identified portion can be captured.

[0049] Next, as described in the first embodiment, the control unit 205 controls the image capturing device 11 so as to capture an image of the object from the determined viewpoint. The control unit 205 is an example of a control unit.

[0050] As described in the first embodiment, the modeling unit 207 then models the object using NeRF based on the image captured by the imaging device 11. The modeling unit 207 is an example of a modeling means. For example, the modeling unit 207 adds a newly captured image and models the object using NeRF.

[0051] 6 is a flowchart showing an example of the operation of the control device 20. The modeling unit 207 uses NeRF to model an object based on an image captured by the imaging device 11 (step S201).

[0052] Next, the detection unit 209 detects a change in glossiness from the reproduced image of the object obtained by modeling (step S202).

[0053] The determination unit 203 determines a viewpoint from which the detected gloss change portion of the object can be captured (step S203). The control unit 205 controls the imaging device 11 to capture an image of the object from the determined viewpoint (step S204). The modeling unit 207 models the object using NeRF based on the image captured by the imaging device 11 (step S205). In step S205, the modeling unit 207 adds the newly captured image and models the object using NeRF. Then, the control device 20 ends the series of processes shown in FIG. 6.

[0054] Next, as described in the first embodiment, the control unit 205 controls the imaging device so that the object is imaged from the viewpoint determined by the determination unit 203 .

[0055] As explained in the first embodiment, it is desirable to limit the number of viewpoints in order to actually prepare images for NeRF. However, it is difficult to understand technologies such as NeRF and capture images of an object. For this reason, it may be difficult to reproduce an object using NeRF. Therefore, in the second embodiment, the control device 20 detects changes in glossiness from the reproduced image of the object obtained by modeling, and determines a viewpoint that can capture the detected glossiness change in the object. Then, the control device 20 models the object using NeRF based on the captured image. This improves the reproduction quality of the object.

[0056] Although gloss has been used as an example, the detection unit 209 may detect changes in texture elements such as gloss, color, texture, and combinations thereof from an image representing an object, and the determination unit 203 may determine a viewpoint from which the part where a change in texture element is detected can be imaged.

[0057] (Third Embodiment) A third embodiment will be described in detail with reference to the drawings. In the third embodiment, in order to capture an image for NeRF, a control device determines a viewpoint of an imaging device based on the material of an object, and controls the imaging device to capture an image of the object from the viewpoint. Below, to the extent that the description of the third embodiment is not unclear, description of content that overlaps with the above description will be omitted.

[0058] 7 is an explanatory diagram showing an example of the configuration of a control system including a control device 3. The control system 3 includes, for example, a control device 30, a light source 32, and an imaging device 31.

[0059] For example, the control device 30 is connected to an imaging device 31 via a communication network. The imaging device 31 captures an image of an object. For example, the imaging device 31 may be fixedly disposed so as to be able to capture an image of the object, or may be able to change its position and orientation so as to be able to capture an image of the object. For example, the imaging device 31 may be movably attached on a rail. Furthermore, for example, the imaging device 31 may be mounted on a mobile object so as to be able to move.

[0060] For example, the control device 30 is connected to the light source 32 via a communication network. The light source 32 is the light source 32 used when the imaging device 31 captures an image of an object. For example, the light source 32 may be able to change the amount and color of light. Furthermore, the light source 32 may be fixedly disposed so as to illuminate the object, or may be able to change its position and orientation so as to illuminate the object. For example, the light source 32 may be movably attached on a rail. Furthermore, for example, the light source 32 may be mounted on a moving body so as to be movably attached. Furthermore, the light source 32 may be attached to the imaging device 31.

[0061] 8 is a block diagram showing an example of the configuration of the control device 30. The control device 30 includes a material specifying unit 302, a determining unit 303, a control unit 305, and a modeling unit 307.

[0062] The determination unit 303 may have a function similar to that of the determination unit 103 shown in Fig. 2. The control unit 305 may have a function similar to that of the control unit 105 shown in Fig. 2. The modeling unit 307 may have a function similar to that of the modeling unit 107 shown in Fig. 2.

[0063] The material identification unit 302 identifies the material of the object. The material identification unit 302 is an example of a material identification means.

[0064] As a specific example of how the material identification unit 302 identifies the material of an object, the material identification unit 302 may identify the material of the object from a plurality of different materials based on an image of the object. Specifically, for example, the material identification unit 302 recognizes the characteristics of the material of the object from the image of the object, and identifies a characteristic corresponding to the material characteristic of the object from a plurality of different materials stored in advance in a database or the like. An example of the material characteristic is gloss.

[0065] The plurality of materials may be predetermined depending on the purpose. For example, in a solution that reproduces items in a house, the plurality of materials may be predetermined, such as a refrigerator material, a washing machine material, a bed material, a shelf material, and a wall material.

[0066] As another specific example of how the material identification unit 302 identifies the material of an object, the material identification unit 302 may identify the material of the object based on the type of object identified from a captured image of the object. If the object is a refrigerator, the material identification unit 302 may identify the material of the object as steel. For example, the method of identifying the type of object based on a captured image of the object may be object recognition based on the captured image of the object, and is not particularly limited.

[0067] The methods by which the material identification unit 302 identifies the material of an object may be combined. For example, the material identification unit 302 may narrow down the material of the object from multiple materials based on the type of object identified by object recognition, recognize characteristics of the material of the object from an image of the object, and identify a material that corresponds to the identified characteristics from the narrowed down materials.

[0068] The determining unit 303 determines the viewpoint of the image capturing device 31 based on the material identified by the identifying unit. The determining unit 303 is an example of a determining means.

[0069] For example, when the position of the light source 32 is determined, the determination unit 303 determines the position of the viewpoint based on the identified material and the position of the light source 32. When the position of the light source 32 is determined, the case where the light source 32 is fixedly disposed or the position of the light source 32 is determined in advance may occur.

[0070] FIG. 9 is an explanatory diagram showing the positional relationship between the light source 32 and the viewpoint. For example, light from the light source 32 is reflected by an object. In this case, as shown in FIG. 9 , it is expected that the gloss of the object can be captured better by positioning the viewpoint in the direction of travel of the reflected light from the light source 32. However, if the viewpoint is located beyond the diffused light, it is expected that the gloss of the object cannot be captured properly. In order to better reproduce gloss, etc., it is necessary to capture the image from a better viewpoint. Furthermore, reflectance varies depending on the material. Therefore, the determination unit 303 determines the position of the viewpoint based on the identified material and the position of the light source 32. The determination unit 303 determines the position of the viewpoint based on the reflectance according to the material and the position of the light source 32 so that the viewpoint is set in the direction of travel of the reflected light from the light source 32.

[0071] For example, a table may be prepared in advance, such as a rule base, in which materials are associated with the positional relationship between the light source 32 corresponding to the material and the viewpoint of the image capture device 31. The determination unit 303 may identify, from the table, the positional relationship corresponding to the material identified by the material identification unit 302, and determine the position of the viewpoint so as to achieve the identified positional relationship.

[0072] Furthermore, if the position of the light source 32 has not been determined, the determination unit 303 determines the position of the viewpoint and the position of the light source 32 based on the identified material. For example, the determination unit 303 may determine the position of the viewpoint and the position of the light source 32 based on the reflectance of the identified material so that the viewpoint is set at the location where light from the light source 32 is reflected. For example, a table in which materials are associated with the positional relationship between the light source 32 and the viewpoint according to the material may be prepared in advance. The determination unit 303 may identify a positional relationship corresponding to the material identified by the material identification unit 302 from the table, and determine the position of the light source 32 and the viewpoint so as to achieve the identified positional relationship.

[0073] Depending on the amount of light, the object may not be captured properly. Therefore, the determination unit 303 may determine the amount of light from the light source 32 depending on the material of the object.

[0074] FIG. 10 is an explanatory diagram showing an example of a table that associates materials with the positional relationship between the imaging device 31 and the light source 32 and the light intensity. The table associates materials with, for example, the positional relationship between the viewpoint of the imaging device 31 and the light source 32, and the light intensity of the light source 32. The determination unit 303 may identify, from the table, the positional relationship between the viewpoint of the imaging device 31 and the light source 32 and the light intensity of the light source 32 that corresponds to the material of the object. The determination unit 303 then determines the position of the viewpoint based on the identified positional relationship between the viewpoint of the imaging device 31 and the light source 32. The determination unit 303 may also determine the position of the viewpoint and the position of the light source 32 based on the identified positional relationship between the viewpoint of the imaging device 31 and the light source 32. The determination unit 303 then determines the identified light intensity as the light intensity of the light source 32.

[0075] Furthermore, for example, an object with little gloss may not require a large number of viewpoints for imaging. On the other hand, an object with a high gloss may require a large number of viewpoints for imaging. Therefore, the determination unit 303 may determine the number of viewpoints based on the material. For example, a rule-based table that associates materials with the number of viewpoints may be prepared in advance. The determination unit 303 may then specify the number of viewpoints corresponding to the material from the table. In this way, narrowing down the number of viewpoints makes it possible to reduce the effort required for imaging. Alternatively, it makes it possible to shorten the time required for imaging.

[0076] For example, there may be cases where the object is made of a plurality of different materials. The determination unit 303 determines the viewpoints of the object so that the number of viewpoints for the glossy material portion of the object is greater than the number of viewpoints for the less glossy material portion of the object.

[0077] Next, as explained in the first embodiment, the control unit 305 controls the image capturing device 31 so as to capture an image of the object from the determined viewpoint.

[0078] Furthermore, when the determination unit 303 determines the position of the light source 32, the control unit 305 controls the light source 32 so that the light source 32 is at the determined position.

[0079] As described in the first embodiment, the modeling unit 307 models the object using NeRF based on the image captured by the imaging device 31 .

[0080] 11 is a flowchart showing an example of the operation of the control device 30. The material identification unit 302 identifies the material of the object (step S301). The determination unit 303 determines the viewpoint based on the material (step S302).

[0081] The control unit 305 controls the imaging device 31 to capture an image of the object from the determined viewpoint (step S303). The modeling unit 307 models the object using NeRF based on the image captured by the imaging device 31 (step S304). Then, the control device 30 ends the series of processes shown in FIG. 11.

[0082] As described above, the reflectance of light varies depending on the material. Therefore, in the third embodiment, the control device 30 identifies the material of the object, determines the viewpoint of the imaging device 31 that will capture the object based on the material, and controls the imaging device 31 to capture the object from the viewpoint. The control device 30 then models the object using NeRF based on the captured image. This improves the degree of reproduction of the object.

[0083] (Fourth embodiment) The fourth embodiment will be described in detail with reference to the drawings. In the fourth embodiment, an example in which an imaging device is mounted on a moving body will be described. In addition, in the fourth embodiment, an example in which the contents described in the first to third embodiments are combined will be described. Below, to the extent that the description of the fourth embodiment is not unclear, description of contents that overlap with the above description will be omitted.

[0084] 12 is an explanatory diagram showing an example of the configuration of a control system including a control device 4. The control system 4 includes, for example, a control device 40, a light source 42, and a moving object 43.

[0085] For example, the control device 40 is connected to a mobile object 43 via a communication network. The mobile object 43 includes an imaging device 41. The imaging device 41 captures an image of an object. For example, the mobile object 43 is capable of changing its position and orientation so that the imaging device 41 can capture an image of the object. The imaging device 41 may also be capable of changing its position and orientation on the mobile object 43. In FIG. 12 , a drone is given as an example of the mobile object 43, but the mobile object 43 is not particularly limited to an AGV or the like.

[0086] For example, the control device 40 is connected to the light source 42 via a communication network. The light source 42 is a light source 42 used when the imaging device 41 captures an image of an object. The light source 42 may have the same function as the light source 42 described with reference to FIG. 7 .

[0087] For example, the control device 40 is connected to a terminal device 44 used by a user via a communication network. The type of the terminal device 44 is not particularly limited, and may be a personal computer (PC), a smartphone, a tablet device, or the like. Note that the terminal device 44 may be pre-installed with an application program that is capable of transmitting information to the control device 40 and outputting information from the control device 40. Alternatively, the terminal device 44 may be able to access a website connected to the control device 40 and send and receive information to and from the control device 40 via the website.

[0088] 13 is a block diagram showing an example of the configuration of the control device 40. The control device 40 includes a shape specifying unit 401, a material specifying unit 402, a determining unit 403, a control unit 405, a modeling unit 407, a detecting unit 409, an output unit 411, a receiving unit 413, and an acquiring unit 415.

[0089] The shape identification unit 401 may have the function of the shape identification unit 101 shown in FIG. 2. The material identification unit 402 may have the function of the material identification unit 302 shown in FIG. 8 as a basic function. The determination unit 403 may have at least one of the function of the determination unit 103 shown in FIG. 2, the function of the determination unit 203 shown in FIG. 5, and the function of the determination unit 303 shown in FIG. 8. The control unit 405 may have at least one of the function of the control unit 105 shown in FIG. 2, the function of the control unit 205 shown in FIG. 5, and the function of the control unit 305 shown in FIG. 8. The modeling unit 407 may have at least one of the function of the modeling unit 107 shown in FIG. 2, the function of the modeling unit 207 shown in FIG. 5, and the function of the modeling unit 307 shown in FIG. 8. The detection unit 409 may have the function of the detection unit 209 shown in FIG. 5.

[0090] <Example of Drone Control> The control unit 405 may move the drone so as to capture an image of the target object from the viewpoint determined by the determination unit 403. The control unit 405 may also change the orientation of the imaging device 41. Then, the control unit 405 causes the imaging device 41 to capture an image of the target object.

[0091] 14 is an explanatory diagram showing an example in which an image is captured by an imaging device 41 provided on a drone. For example, if the target object is a refrigerator, the control unit 405 controls the drone to fly around the refrigerator. In FIG. 14, the target object is placed indoors, but the target object may also be located outdoors.

[0092] <Output Example> The output unit 411 has a function of outputting various information. The output unit 411 is an example of output means. Here, the output format by the output unit 411 may be output to a storage device or output to an output device. The output device is not particularly limited to a display device, etc. The type of display device is not particularly limited. For example, if the display device is provided in the terminal device 44, the output unit 411 displays various information on the terminal device 44. Furthermore, the display device may be an HMD (Head Mounted Display). For example, the output unit 411 displays various information on the HMD. That is, the output unit 411 may display various information in a virtual reality space or may display various information in real space.

[0093] The output unit 411 may output an image of the object captured by the imaging device 41. The output unit 411 may also output an image of the object captured by a TOF camera.

[0094] Furthermore, the output unit 411 may output a reproduced image of the object modeled by the modeling unit 407. The output unit 411 may output the reproduced image to an HMD. That is, the output unit 411 may virtually reproduce the subject in a virtual reality space, or may virtually reproduce the subject in real space. In this way, the control device 40 may be used in VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), etc.

[0095] Furthermore, for example, the output unit 411 may output information about the shape of the object identified by the shape identification unit 401. Furthermore, for example, the output unit 411 may output information about the material of the object identified by the material identification unit 402.

[0096] The output unit 411 may output a reproduced image of the object obtained by modeling and an image captured by the imaging device 41 so that they can be compared.

[0097] The reproduced image of the object obtained by modeling may be a reproduced image as seen from a viewpoint specified by the user. Therefore, the receiving unit 413 may receive input of a viewpoint for reproducing the object. Then, the output unit 411 outputs the reproduced image of the object obtained by modeling as seen from the specified viewpoint.

[0098] <Example of Repeated Image Capture by TOF Camera> In the first embodiment, an example has been described in which the shape identification unit 401 recognizes the shape of an object from an image of the object captured by a TOF camera. Here, an example will be described in which image capture by a TOF camera is performed repeatedly. In particular, the description will be given taking an example in which the TOF camera is provided in the terminal device 44.

[0099] First, the acquisition unit 415 acquires, from the terminal device 44, an image of at least a part of an object captured by a TOF camera provided in the terminal device 44. The acquisition unit 415 is an example of an acquisition means.

[0100] Next, the shape specifying unit 401 specifies the shape of the object based on the image of the object captured by the TOF camera.

[0101] The output unit 411 then outputs the shape of the object identified by the shape identification unit 401 to the terminal device 44. For example, the output unit 411 may highlight the portion of the object captured by the TOF camera where the elevation difference is identified by the shape identification unit 401. The output unit 411 may also highlight the portion of the object captured by the imaging device 41 where the elevation difference is identified by the shape identification unit 401. The acquisition unit 415 then acquires a new image of the object captured by the TOF camera in response to a user operation on the terminal device 44. This allows the user to capture the object using the TOF camera while checking the portion of the object where the elevation difference is located. Note that the user's operation on the terminal device 44 refers to the user's operation on an input device provided on the terminal device 44. For example, the user's operation on the terminal device 44 refers to the user's touch operation on a touch panel display provided on the terminal device 44.

[0102] The determining unit 403 determines a viewpoint from which a part of the object having a difference in elevation can be imaged. The control unit 405 then controls at least one of the imaging device 41 and the moving body 43 so that the imaging device 41 can image the object from the viewpoint determined by the determining unit 403.

[0103] The modeling unit 407 performs modeling based on the captured image. The output unit 411 may output the object obtained by modeling and the image captured by the imaging device 41 in which parts with height differences are highlighted so that they can be compared.

[0104] <Example of Gloss Designation> In the second embodiment, an example was described in which the detection unit 409 detects a portion where gloss changes from an image representing an object. Here, an example will be described in which the portion where gloss changes is designated by the user.

[0105] The output unit 411 outputs an image representing the object and an instruction to specify a portion of the image representing the object where gloss changes. For example, the output unit 411 displays these on the terminal device 44.

[0106] Then, the accepting unit 413 accepts designation of a portion where glossiness changes in the image representing the object. The accepting unit 413 is an example of a accepting means. The accepting unit 413 accepts designation of a portion where glossiness changes in the image representing the object, for example, via an input device. The input device may be provided in, for example, the terminal device 44. For example, the accepting unit 413 may accept designation of a portion where glossiness changes in the image representing the object by a touch operation on a touch panel display provided in the terminal device 44.

[0107] Next, the determination unit 403 determines a viewpoint from which the specified portion having a change in gloss can be imaged. The control unit 405 controls at least one of the imaging device 41 and the moving body 43 so that the imaging device 41 can capture an image from the viewpoint determined by the determination unit 403. The control unit 405 then models the object based on the image captured by the imaging device 41.

[0108] <Another Example of Determining the Viewpoint Based on the Light Source 42> In the first embodiment, an example was described in which the determination unit 403 determines the viewpoint based on the shape of the object. Furthermore, in the third embodiment, an example was described in which the position of the viewpoint is determined based on the position of the light source 42 and the material, since reflectance varies depending on the material. Also, an example was described in which the position of the light source 42 and the position of the viewpoint are determined based on the material. For example, there are cases in which the positional relationship between the light source 42 and the viewpoint of the imaging device 41 is predetermined, and the light source 42 is fixedly disposed, or the position of the light source 42 is determined before the position of the viewpoint. For example, the determination unit 403 determines the position of the viewpoint based on the position of the light source 42 so that the positional relationship between the light source 42 and the viewpoint is the predetermined positional relationship.

[0109] <Other Control Examples of the Light Source 42> In the third embodiment, the determination unit 403 determines the position of the light source 42 based on the material of the object, and the control unit 405 controls the light source 42 to be at the determined position. For example, the positional relationship between the light source 42 and the viewpoint of the imaging device 41 may be predetermined, and the viewpoint may be fixed, or the position of the viewpoint may be determined before the position of the light source 42. For example, the determination unit 403 determines the position of the viewpoint based on the position of the light source 42 so that the positional relationship between the light source 42 and the viewpoint of the imaging device 41 is the predetermined positional relationship. The control unit 405 controls the light source 42 to be at the determined position of the light source 42. For example, the determination unit 403 may determine the color of the light source 42 according to the location where the object is located.

[0110] Furthermore, for example, there may be cases where the user wants to check how an object looks in the morning sun. Therefore, the determination unit 403 may determine the color of the light source 42 according to the location where the object is placed. The control unit 405 changes the color of the light source 42 to the color determined by the determination unit 403. This allows the user to check how the object looks different depending on the location where the object is placed.

[0111] <Example of Designation of Shape of Object> In the first embodiment, an example has been described in which the shape specification unit 401 specifies the shape of the object. The receiving unit 413 may also receive a designation of the shape of the object.

[0112] For example, the output unit 411 outputs an image representing the object to the terminal device 44. The image representing the object may be a captured image of the object, drawing data of the object, or a reproduced image of the object obtained by modeling. The receiving unit 413 may receive input of the shape of the object. The determining unit 403 may determine the viewpoint based on the received shape of the object.

[0113] Alternatively, the output unit 411 may output the shape identified by the shape identification unit 401 as a shape candidate. Then, the receiving unit 413 may receive a shape selection from the shape candidates. The determining unit 403 may determine the viewpoint based on the shape of the selected object.

[0114] <Example of Designation of Viewpoint> In the first to third embodiments, an example has been described in which the control unit 405 controls the imaging device 41 so that imaging can be performed at the viewpoint determined by the determination unit 403. For example, the user may select at least some of the viewpoints from the determined viewpoints.

[0115] The output unit 411 outputs the viewpoint determined by the determination unit 403 as a viewpoint candidate. For example, the output unit 411 may output, together with the viewpoint candidate, a portion of the object that may be captured when imaging is performed from the viewpoint candidate to the image of the object. For example, the reception unit 413 receives a selection of a viewpoint from the viewpoint candidate. The control unit 405 controls the imaging device 41 to capture an image of the object from a viewpoint selected from the viewpoint candidate.

[0116] 15 and 16 are flowcharts showing an example of the operation of the control device 40. Here, an example will be described in which a user combines an example of capturing an image of an object using a TOF camera, an example of identifying the shape of the object from an image captured by the TOF camera and determining a viewpoint based on the shape, and an example of accepting a designation of gloss of a reproduced image. Note that the combination method is merely an example and is not particularly limited.

[0117] The identification unit identifies the shape of the object based on the image captured by the TOF camera (step S401). The output unit 411 outputs the shape identified by the identification unit to the terminal device 44 (step S402). Then, the terminal device 44 captures an image of the object using the TOF camera provided in the terminal device 44 in response to a user operation.

[0118] The acquisition unit 415 acquires a new image of the object captured by the TOF camera provided in the terminal device 44 (step S403). The shape identification unit 401 identifies the shape of the object based on the new image (step S404). The determination unit 403 determines the viewpoint of the imaging device 41 based on the shape of the object identified by the identification unit (step S405).

[0119] The control unit 405 controls the moving body 43 equipped with the imaging device 41 so that the imaging device 41 can capture an image of the target object from the viewpoint determined by the determination unit 403 (step S406).

[0120] The modeling unit 407 models the object using NeRF based on the image captured by the imaging device 41 (step S407). The output unit 411 displays a reproduced image of the object obtained by modeling on the terminal device 44 (step S408). In step S408, for example, the output unit 411 may display reproduced images of the object as viewed from a plurality of viewpoints. Next, the receiving unit 413 receives designation of a portion in the image representing the object where gloss changes (step S409).

[0121] The determination unit 403 determines a viewpoint from which the specified gloss change portion can be captured (step S410). The control unit 405 controls the mobile object 43 equipped with the imaging device 41 so that the imaging device 41 can be captured from the viewpoint determined by the determination unit 403 (step S411). The modeling unit 407 models the object based on the image captured by the imaging device 41 (step S412). The output unit 411 then displays a reproduced image of the object obtained by the modeling on the terminal device 44 (step S413). In step S413, for example, the output unit 411 may display reproduced images of the object as viewed from multiple viewpoints. The control device 40 then terminates the series of processes shown in FIGS. 15 and 16.

[0122] As described above, in the fourth embodiment, the terminal device 44 is equipped with a TOF camera, and the control device 40 may display the shape of the object on the terminal device 44 and acquire a new image of the object captured by the TOF camera in response to a user operation on the terminal device 44. The control device 40 then identifies the shape of the object from the acquired image. In this way, the control device 40 repeatedly captures images of the subject using the TOF camera in response to a user operation. This allows the shape of the object to be identified with greater accuracy. This allows for improved reproduction of the object.

[0123] Furthermore, the reflection of light from the light source 42 makes the object appear glossy. For this reason, the positional relationship between the light source 42 and the viewpoint may be determined in advance. For this reason, the control device 40 determines the position of the viewpoint based on the position of the light source 42. This makes it possible to more effectively capture the surface properties of the object, such as the gloss of the object. Alternatively, the control device 40 may determine the position of the light source 42 based on the position of the viewpoint determined by the determination unit 403. This makes it possible to more effectively capture the surface properties of the object, such as the gloss of the object.

[0124] Furthermore, the determination unit 403 may determine the color of the light source 42 according to the location where the object is placed, thereby making it possible to reproduce the difference in appearance of the object depending on the location where the object is placed.

[0125] For example, making the imaging device 41 movable may be time-consuming. Therefore, the imaging device 41 may be mounted on a mobile body 43, and the control device 40 may control the mobile body 43 so that the object is imaged from a determined viewpoint. This makes it easier to image various objects.

[0126] The appearance of an object changes depending on the light intensity of the light source 42. Therefore, the control device 40 identifies the material of the object from multiple types of materials and determines the light intensity of the light source 42 according to the identified material. The control device 40 then controls the light source 42 to achieve the determined light intensity.

[0127] The above is the description of each embodiment. Each embodiment may be modified and used.

[0128] <Modification> In the fourth embodiment, an example has been described in which the imaging device 41 is provided in the moving object 43. For example, the imaging device 41 may be provided in a terminal device operated by a user. Then, the output unit 411 may output to the terminal device 44 an instruction to cause the user to operate the imaging device 41 provided in the terminal device 44 from the viewpoint determined by the determination unit 403. The user may capture an image of the object using the imaging device 41 provided in the terminal device 44 based on the instruction. In this way, the control unit 405 does not have to be configured to control the imaging device 41. For example, the control device 40 may be configured to include a shape identification unit 401, a determination unit 403, an output unit 411, and a modeling unit 407.

[0129] This concludes the description of the modified examples. The embodiments and modified examples may be combined as appropriate. There are no particular limitations on how they may be combined.

[0130] Furthermore, the control devices 10, 20, 30, and 40 may each include a portion of each functional unit and information. For example, the control device 40 may include a shape identification unit 401, a determination unit 403, and an output unit 411. For example, the control device 40 may include a material identification unit 402, a determination unit 403, and an output unit 411.

[0131] Furthermore, each embodiment is not limited to the above-described examples and can be modified in various ways. Furthermore, the configuration of the control devices 10, 20, 30, and 40 is not particularly limited. For example, the functional units of the control devices 10, 20, 30, and 40 may be implemented by a single device. Alternatively, for example, each functional unit or database of the control devices 10, 20, 30, and 40 may be implemented by a different device and configured as a system. For example, each functional unit of the control devices 10, 20, 30, and 40 may be implemented by multiple servers and configured as a system. For example, each functional unit may be implemented by a database server including each database and a server having each functional unit.

[0132] A system may be realized that includes a server having some of the functional units of the control device and another server having some of the functional units of the control device. The number of servers is not particularly limited. For example, in the first embodiment, a system may be realized that includes a first server having a shape identification unit, a determination unit, and a control unit, and a second server having a modeling unit. In such a case, the first server may transmit a captured image to the second server. The second server may then transmit a reproduced image of the subject obtained by modeling to the first server.

[0133] In addition, in the embodiment, each database may include part of the information described above. Also, each piece of information may include information other than the information described above.

[0134] Furthermore, the process of generating information to be displayed on the terminal device 44 may be performed by a functional unit included in the control device 40, such as the output unit 411. This process may also be performed by the terminal device 44. That is, the terminal device 44 may generate information for a screen to be displayed on the terminal device 44 based on data received from the control device 40, and display the screen.

[0135] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration in which each device such as the control devices 10, 20, 30, and 40 and the terminal device 44 is realized by a computer will be described.

[0136] 17 is an explanatory diagram showing an example of the hardware configuration of a computer. For example, some or all of the devices can be realized using any combination of a computer 80 and a program as shown in FIG.

[0137] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0138] The processor 801 controls the entire computer 80. The processor 801 may be, for example, a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, or a combination thereof, and is not particularly limited.

[0139] The computer 80 also includes a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include semiconductor memory such as flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801.

[0140] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0141] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0142] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0143] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 17 . The computer 80 may have components other than those shown in FIG. 17 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0144] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0145] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0146] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits such as FPGAs (Field Programmable Gate Arrays). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. These circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0147] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0148] The control methods described in the respective embodiments may be implemented by a computer such as the control devices 10, 20, 30, and 40.

[0149] Each program described in each embodiment is recorded on a computer-readable recording medium such as a HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. Each program is executed by being read from the recording medium by a computer. Each program may also be distributed via a communication network NT.

[0150] The functions of the components of the control device 10 and the control device 20 described above may be realized by dedicated hardware, such as a computer. Alternatively, the components may be realized by software. Alternatively, the components may be realized by a combination of hardware and software.

[0151] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may include embodiments in which the features described herein are appropriately combined or substituted as necessary. For example, features described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of description does not limit the order in which the multiple operations are performed. Therefore, when implementing the embodiments, the order of the multiple operations may be changed as long as the content is not affected.

[0152] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0153] (Supplementary Note 1) A control device comprising: a shape identification means for identifying a shape of an object; a determination means for determining a viewpoint according to the identified shape; a control means for controlling an imaging device to image the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on an image captured by the imaging device.

[0154] (Supplementary Note 2) The control device according to Supplementary Note 1, wherein the determining means determines the viewpoints such that the number of viewpoints increases as the specified shape becomes more complex.

[0155] (Supplementary Note 3) The control device according to Supplementary Note 1 or 2, wherein the shape identification means identifies the shape of the object based on an image of the object.

[0156] (Supplementary Note 4) The control device described in any one of Supplementary Notes 1 to 3, wherein the shape identification means identifies a part of the object that has a difference in elevation based on distance data obtained by measuring the distance to the object by a distance measuring sensor, and the determination means determines the viewpoint from which the part with the difference in elevation can be imaged.

[0157] (Supplementary Note 5) A control device as described in Supplementary Note 4, comprising: an output means for displaying the identified shape on a terminal device equipped with the distance measurement sensor; and an acquisition means for acquiring distance data obtained by newly measuring the distance to the object by the distance measurement sensor in response to a user operation on the terminal device, wherein the shape identification means identifies the part of the object where there is a difference in elevation based on the newly acquired distance data, and the determination means determines the viewpoint from which the part with the difference in elevation can be imaged.

[0158] (Supplementary Note 6) The control device according to any one of Supplementary Notes 1 to 4, wherein the determining means determines the position of the viewpoint based on a position of a light source.

[0159] (Supplementary Note 7) The control device according to any one of Supplementary Notes 1 to 6, wherein the determining means determines a color of the light source according to a location where the object is placed.

[0160] (Supplementary Note 8) The control device according to any one of Supplementary Notes 1 to 7, wherein the imaging device is mounted on a moving body, and the control means controls the moving body so as to capture an image of the target object at the determined viewpoint.

[0161] (Supplementary Note 9) A control device according to any one of Supplementary Notes 1 to 8, further comprising: a material identification means for identifying a material of the object; wherein the determination means determines a light intensity of a light source according to the identified material; and the control means controls the light source to achieve the determined light intensity.

[0162] (Supplementary Note 10) The control device according to any one of Supplementary Notes 1 to 9, wherein the determining means determines a position of a light source based on the determined position of the viewpoint, and the control means controls the light source so that the light source is at the determined position.

[0163] (Supplementary Note 11) The control device according to any one of Supplementary Notes 1 to 9, further comprising: a material identification unit that identifies a material of the object; and the determination unit that determines the viewpoint based on the identified material.

[0164] (Supplementary Note 12) The control device according to Supplementary Note 11, wherein the determining means determines the number of viewpoints based on the identified material.

[0165] (Supplementary Note 13) The control device according to Supplementary Note 11 or 12, wherein the determining means determines the position of the viewpoint based on the identified positions of the material and light source.

[0166] (Supplementary Note 14) The control device according to Supplementary Note 11 or 12, wherein the determining means determines a position of the viewpoint and a position of a light source based on the identified material, and the control means controls the light source so that the light source is positioned as determined.

[0167] (Supplementary Note 15) The control device according to any one of Supplementary Notes 1 to 14, further comprising: a detection unit that detects a change in gloss from an image representing the object; and the determination unit that determines a viewpoint from which an image of the detected change in gloss can be captured.

[0168] (Supplementary Note 16) The control device according to any one of Supplementary Notes 1 to 14, further comprising: a receiving means for receiving a designation of a portion of gloss change from an image representing the object; and the determining means for determining a viewpoint from which the designated portion of gloss change can be imaged.

[0169] (Supplementary Note 17) A control device comprising: a detection means for detecting a change in gloss from an image representing an object; a determination means for determining a viewpoint from which the detected change in gloss can be captured; a control means for controlling an imaging device to capture an image of the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on the image captured by the imaging device.

[0170] (Supplementary Note 18) The control device according to any one of Supplementary Notes 15 to 17, wherein the image representing the object is at least one of a reproduced image of the object obtained by the modeling and an image of the object.

[0171] (Supplementary Note 19) A control device comprising: a material identification means for identifying a material of an object; a determination means for determining a viewpoint based on the identified material; a control means for controlling an imaging device to image the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on an image captured by the imaging device.

[0172] (Supplementary Note 20) The control device according to Supplementary Note 19, wherein the material identification means identifies the material of the object from a plurality of types of materials.

[0173] (Supplementary Note 21) The control device according to Supplementary Note 19 or 20, wherein the determining means determines the position of the viewpoint based on the identified positions of the material and light source.

[0174] (Supplementary Note 22) The control device according to Supplementary Note 19 or 20, wherein the determining means determines a position of the viewpoint and a position of a light source based on the identified material, and the control means controls the light source so that the light source is positioned as determined.

[0175] (Supplementary Note 23) The control device according to any one of Supplementary Notes 1 to 22, further comprising: an output unit that outputs a reproduced image of the object obtained by the modeling.

[0176] (Supplementary Note 24) The control device according to any one of Supplementary Notes 1 to 23, further comprising: an output means for outputting the determined viewpoint as a candidate viewpoint; and the control means controls the imaging device to image the object from a viewpoint selected from the candidate viewpoints.

[0177] (Supplementary Note 25) A control method in which a computer executes the following processes: identifies the shape of an object; determines a viewpoint according to the identified shape; controls an imaging device to capture an image of the object from the determined viewpoint; and models the object using NeRF based on the image captured by the imaging device.

[0178] (Supplementary Note 26) A control method in which a computer executes the following processes: detects a change in gloss from an image representing an object; determines a viewpoint from which the detected change in gloss can be captured; controls an imaging device to capture an image of the object from the determined viewpoint; and models the object using NeRF based on the image captured by the imaging device.

[0179] (Supplementary Note 27) A control method in which a computer executes the following processes: identifies the material of an object; determines a viewpoint based on the identified material; controls an imaging device to capture an image of the object from the determined viewpoint; and models the object using NeRF based on an image captured by the imaging device.

[0180] (Supplementary Note 28) A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: identify the shape of an object; determine a viewpoint according to the identified shape; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

[0181] (Supplementary Note 29) A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: detect a change in gloss from an image representing an object; determine a viewpoint from which the detected change in gloss can be captured; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on the image captured by the imaging device.

[0182] (Supplementary Note 30) A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: identify a material of an object; determine a viewpoint based on the identified material; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

[0183] (Supplementary Note 31) A program that causes a computer to execute the following processes: identify the shape of an object; determine a viewpoint according to the identified shape; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

[0184] (Supplementary Note 32) A program that causes a computer to execute the following processes: detect a change in gloss from an image representing an object; determine a viewpoint from which the detected change in gloss can be captured; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on the image captured by the imaging device.

[0185] (Supplementary Note 33) A program that causes a computer to execute the following processes: identify the material of an object; determine a viewpoint based on the identified material; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

[0186] Furthermore, some or all of the configurations described in Supplementary Notes 2 to 16, Supplementary Notes 18, Supplementary Notes 23, and Supplementary Notes 24, which are dependent on Supplementary Notes 1 described above, may also be dependent on Supplementary Notes 25, Supplementary Notes 28, and Supplementary Notes 31 in the same dependent relationship as Supplementary Notes 2 to 16, Supplementary Notes 18, Supplementary Notes 23, and Supplementary Notes 24. Furthermore, some or all of the configurations described in Supplementary Notes 18, Supplementary Notes 23, and Supplementary Notes 24, which are dependent on Supplementary Notes 17 described above, may also be dependent on Supplementary Notes 26, Supplementary Notes 29, and Supplementary Notes 32 in the same dependent relationship as Supplementary Notes 18, Supplementary Notes 23, and Supplementary Notes 24. Furthermore, some or all of the configurations described in Supplementary Notes 20 to 24, which are dependent on Supplementary Notes 19, may also be dependent on Supplementary Notes 27, Supplementary Notes 30, and Supplementary Notes 33 in the same dependent relationship as Supplementary Notes 20 to 24. Furthermore, not limited to Appendix 1, Appendix 17, Appendix 19, Appendix 25 to Appendix 33, but also to various hardware, software, various recording means for recording software, or systems, some or all of the configurations described as appendices may be subordinated to the scope of each of the above-mentioned embodiments.

[0187] 1, 3, 4 Control system 10, 20, 30, 40 Control device 11, 21, 31, 41 Imaging device 32, 42 Light source 43 Mobile body 44 Terminal device 80 Computer 101, 401 Shape identification unit 103, 203, 303, 403 Determination unit 105, 205, 305, 405 Control unit 107, 207, 307, 407 Modeling unit 209, 409 Detection unit 302, 402 Material identification unit 411 Output unit 413 Reception unit 415 Acquisition unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus NT Communication network

Claims

1. A control device comprising: a shape identification means for identifying the shape of an object; a determination means for determining a viewpoint corresponding to the identified shape; a control means for controlling an imaging device to image the object from the determined viewpoint; and a modeling means for modeling the object using Neural Radiance Fields (NeRF) based on an image captured by the imaging device.

2. The control device according to claim 1, wherein said determining means determines said viewpoints so that the more complex the specified shape, the greater the number of said viewpoints.

3. The control device according to claim 1 or 2, wherein the shape identification means identifies the shape of the object based on an image of the object.

4. A control device as described in any one of claims 1 to 3, wherein the shape identification means identifies parts of the object that have differences in elevation based on distance data measured by a distance sensor to the object, and the determination means determines the viewpoint from which the parts with differences in elevation can be imaged.

5. A control device as described in claim 4, comprising: an output means for displaying the identified shape on a terminal device equipped with the distance measurement sensor; and an acquisition means for acquiring distance data obtained by newly measuring the distance to the object by the distance measurement sensor in response to a user's operation on the terminal device, wherein the shape identification means identifies the part of the object where there is a difference in elevation based on the newly acquired distance data, and the determination means determines the viewpoint from which the part where there is a difference in elevation can be imaged.

6. The control device according to any one of claims 1 to 4, wherein the determining means determines the color of the light source according to the location where the object is placed.

7. A control device according to any one of claims 1 to 6, wherein the imaging device is mounted on a moving body, and the control means controls the moving body so as to capture an image of the object from the determined viewpoint.

8. A control device according to any one of claims 1 to 7, further comprising: a material identification means for identifying the material of the object; wherein the determination means determines the light intensity of the light source according to the identified material; and the control means controls the light source to achieve the determined light intensity.

9. A control device comprising: a detection means for detecting changes in gloss from an image representing an object; a determination means for determining a viewpoint from which the detected change in gloss can be captured; a control means for controlling an imaging device to capture an image of the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on the image captured by the imaging device.

10. The control device according to claim 9, wherein the image representing the object is at least one of a reproduced image of the object obtained by the modeling and an image of the object.

11. A control device comprising: a material identification means for identifying a material of an object; a determination means for determining a viewpoint based on the identified material; a control means for controlling an imaging device to image the object from the determined viewpoint; and a modeling means for modeling the object using NeRF based on an image captured by the imaging device.

12. The control device according to claim 11, wherein the material identification means identifies the material of the object from a plurality of types of materials.

13. The control device according to claim 11 or 12, wherein the determining means determines the position of the viewpoint based on the identified positions of the material and light source.

14. A control device according to claim 11 or 12, wherein the determining means determines the position of the viewpoint and the position of the light source based on the identified material, and the control means controls the light source so that the light source is positioned as determined.

15. A control method in which a computer executes the following processes: identifies the shape of an object; determines a viewpoint according to the identified shape; controls an imaging device to capture an image of the object from the determined viewpoint; and models the object using NeRF based on the image captured by the imaging device.

16. A control method in which a computer executes the following processes: detects changes in gloss from an image representing an object; determines a viewpoint from which the detected change in gloss can be captured; controls an imaging device to capture an image of the object from the determined viewpoint; and models the object using NeRF based on the image captured by the imaging device.

17. A control method in which a computer executes the following processes: identifies the material of an object; determines a viewpoint based on the identified material; controls an imaging device to capture an image of the object from the determined viewpoint; and models the object using NeRF based on the image captured by the imaging device.

18. A non-transitory computer-readable recording medium having recorded thereon a program that causes a computer to execute the following processes: identify the shape of an object; determine a viewpoint according to the identified shape; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on the image captured by the imaging device.

19. A non-transitory computer-readable recording medium having recorded thereon a program for causing a computer to execute the following processes: detect a change in gloss from an image representing an object; determine a viewpoint from which the detected change in gloss can be captured; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on the image captured by the imaging device.

20. A non-transitory computer-readable recording medium having recorded thereon a program that causes a computer to execute the following processes: identify the material of an object; determine a viewpoint based on the identified material; control an imaging device to capture an image of the object from the determined viewpoint; and model the object using NeRF based on an image captured by the imaging device.

Images