Information processing system, method, and program

Abstract

Provided are an information processing system, a method, and a program that simplify the generation of a three-dimensional model of an existing object. The information processing system comprises at least one processor. The processor acquires first information which includes at least one of information regarding the object, information regarding an environment for imaging or measuring the object, information regarding a condition for imaging or measuring the object, and information regarding an importance degree of imaging or measurement, and generates, on the basis of the first information, second information regarding the generation of the three-dimensional model of the object.

Description

Information processing system, method and program

[0001] The present invention relates to an information processing system, method, and program, and more particularly to an information processing system, method, and program for processing information relating to the generation of a three-dimensional model.

[0002] Patent Document 1 describes a technology for generating a three-dimensional image of a physical object, and describes determining surface attribute data to be applied to a three-dimensional model based on shape data obtained from the physical object.

[0003] Patent Document 2 describes a technique for measuring the position and orientation of a measurement object, and describes switching measurement modes based on the shape of model data that imitates the measurement object.

[0004] JP 2003-168129 A JP 2012-021958 A

[0005] One embodiment of the technology of the present disclosure provides an information processing system, method, and program that facilitates the generation of a three-dimensional model of a real object.

[0006] (1) An information processing system comprising at least one processor, the processor acquiring first information including at least one of information regarding an object, information regarding the environment in which the object is photographed or measured, information regarding the conditions for photographing or measuring the object, and information regarding the importance of the photographing or measurement, and generating second information regarding the generation of three-dimensional data of the object based on the first information.

[0007] (2) The information processing system according to (1), wherein the processor receives input of the first information from a user and acquires the first information.

[0008] (3) An information processing system described in (1) or (2), in which the information about the object includes information about transparent areas, the information about the environment in which the object is photographed or measured includes information about lighting conditions, and the information about the conditions for photographing or measuring the object includes at least one of information about whether the photographing or measurement is done by hand and information about the scanning speed.

[0009] (4) An information processing system described in any one of (1) to (3), wherein the second information is information on a method used to generate three-dimensional data, and the processor determines the method to be used to generate the three-dimensional data based on the first information.

[0010] (5) An information processing system according to (4), wherein the first information includes at least information on transparent areas of the object, and the processor determines a method to be used to generate the three-dimensional data based on the information on the transparent areas.

[0011] (6) An information processing system described in (5), wherein the first information includes, as information on the transparent area, at least one of information on the presence or absence of a transparent area and information on the proportion of the object occupied by the transparent area, and the processor determines a method to be used to generate the three-dimensional data based on at least one of information on the presence or absence of a transparent area and information on the proportion of the object occupied by the transparent area.

[0012] (7) The information processing system according to (6), wherein the processor determines whether to use photogrammetry or volume intersection as the method to be used for generating the three-dimensional data.

[0013] (8) An information processing system according to any one of (5) to (7), wherein the processor determines a method to be used for generating three-dimensional data for each region of the object based on information about the transparent region.

[0014] (9) An information processing system described in any one of (1) to (8), wherein the second information is information on a photographing technique for an image used to generate three-dimensional data, the first information includes at least information on lighting conditions, and the processor determines a photographing technique for an image used to generate three-dimensional data based on the information on lighting conditions.

[0015] (10) The information processing system according to (9), wherein the processor determines whether or not to employ polarized light photography as the photography technique.

[0016] (11) An information processing system described in (9) or (10), in which the information on the lighting condition includes at least one of information on whether or not specular reflection occurs and information on the proportion of the object occupied by the area where specular reflection occurs.

[0017] (12) The information processing system according to any one of (9) to (11), wherein the information on the lighting condition includes information on whether the lighting condition causes specular reflection on the object.

[0018] (13) An information processing system according to any one of (9) to (12), wherein the information on the lighting status includes information on whether the lighting can be adjusted.

[0019] (14) An information processing system described in any one of (1) to (13), wherein the first information includes at least information on whether the photograph was taken handheld, and the processor determines the photographing conditions of the object based on the information on whether the photograph was taken handheld, and generates the second information.

[0020] (15) The information processing system according to (14), wherein the processor determines, as the photographing condition, a shutter speed setting when photographing the object.

[0021] (16) An information processing system described in any one of (1) to (15), wherein the second information is information on the photographing conditions of the object, the first information includes at least information on the scanning speed, and the processor determines the photographing conditions based on the information on the scanning speed.

[0022] (17) The information processing system according to (16), wherein the processor determines, as the photographing condition, a shutter speed setting when photographing the object.

[0023] (18) An information processing system described in any one of (1) to (17), wherein the second information is information on processing conditions when generating three-dimensional data, the first information includes at least information on the importance of photographing or measurement, and the processor determines the processing conditions based on the information on the importance of photographing or measurement.

[0024] (19) The information processing system according to (18), wherein the processor determines a processing mode for generating three-dimensional data as a processing condition.

[0025] (20) An information processing method including a step of acquiring first information including at least one of information about an object, information about the environment in which the object is photographed or measured, information about the conditions for photographing or measuring the object, and information about the importance of the photographing or measurement, and a step of generating second information regarding the generation of three-dimensional data of the object based on the first information.

[0026] (21) An information processing program that causes a computer to realize a function of acquiring first information including at least one of information about an object, information about the environment in which the object is photographed or measured, information about the conditions for photographing or measuring the object, and information about the importance of the photographing or measurement, and a function of generating second information regarding the generation of three-dimensional data of the object based on the first information.

[0027] FIG. 1 shows a schematic configuration of a three-dimensional model generation system; Block diagram showing an example of the hardware configuration of a three-dimensional model generation device; Block diagram of main functions of a three-dimensional model generation device; Flowchart showing the processing procedure for determining a three-dimensional model generation method; Block diagram of main functions of a three-dimensional model generation device; Flowchart showing the processing procedure for generating a three-dimensional model; FIG. 2 shows a schematic configuration of a three-dimensional model generation system; Block diagram of main functions of a three-dimensional model generation device; Flowchart showing the processing procedure for providing photography support information; FIG. 3 shows a schematic configuration of a three-dimensional model generation system; Block diagram of main functions of a three-dimensional model generation device; Flowchart showing the processing procedure for providing photography support information; FIG. 4 shows a schematic configuration of a three-dimensional model generation system; Block diagram of main functions of a three-dimensional model generation device; Flowchart showing the processing procedure for providing photography support information;

[0028] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] First Embodiment In recent years, three-dimensional content (content that can be viewed stereoscopically) has been increasingly utilized in fields such as games, metaverse-type communication spaces, architecture, and video production.

[0030] Some three-dimensional content production involves scanning real-world objects in three dimensions using cameras or specialized equipment, and then converting the object's shape into three-dimensional data (a technique known as 3D scanning).

[0031] There are various methods for 3D scanning, including photogrammetry, volumetric analysis, LiDAR (Light Detection and Ranging), ToF (Time of Flight), and structured light, but each has its own advantages and disadvantages, making it difficult to select the optimal method.

[0032] In this embodiment, a system (3D model generation system) is presented that allows even a user with little knowledge of 3D scanning to generate a high-quality 3D model (a geometric model drawn in 3D).

[0033] [System Configuration] FIG. 1 is a diagram showing the schematic configuration of a three-dimensional model generation system.

[0034] The three-dimensional model generation system 1 of this embodiment is configured as a system that generates a three-dimensional model M of an existing object O. In particular, the three-dimensional model generation system 1 of this embodiment is configured as a system that captures images of the object O from multiple viewpoints (multi-viewpoints) and generates the three-dimensional model M from the obtained group of two-dimensional images (multi-viewpoint images).

[0035] Photogrammetry or volume intersection is used to generate the three-dimensional model M. Both are techniques for generating a three-dimensional model of an object from multi-viewpoint images (three-dimensional shape reconstruction techniques).

[0036] In photogrammetry, the position and orientation of the camera are estimated from images using SfM (Structure from Motion), and a 3D model is generated using MVS (Multi-View Stereo). Images taken with a general digital camera can be used.

[0037] The volume intersection method, also known as the silhouette method, is a technique for generating 3D models based on the silhouette of an object. In volume intersection, silhouettes of an object photographed from multiple viewpoints are used to create a cone (volume) with each viewpoint as a vertex and the silhouette as a cross section. The silhouettes are then back-projected into 3D space, and their intersections (common areas) are found to generate a 3D model. As with photogrammetry, images taken with a standard digital camera can be used, and silhouettes are extracted using background subtraction and other methods.

[0038] Photogrammetry involves matching feature points between images and generating a 3D model using the results of that processing. For this reason, objects that do not have a sufficient number of feature points available from the images, such as transparent or translucent objects, are considered unsuitable for photogrammetry.

[0039] On the other hand, the volume intersection method uses silhouettes to generate 3D models, so even if the object has transparent or semi-transparent parts, a 3D model can be generated as long as the silhouette can be obtained. However, the accuracy of the 3D shape that can be reconstructed is inferior to that of photogrammetry.

[0040] The 3D model generation system 1 of this embodiment acquires information about an object from a user, and automatically switches the 3D model generation method based on the acquired information to generate a 3D model of the object. A 3D model is an example of 3D data.

[0041] 1, a three-dimensional model generation system 1 according to the present embodiment includes an imaging device 10 and a three-dimensional model generation device 100. The three-dimensional model generation system 1 according to the present embodiment is an example of an information processing system.

[0042] [Photographing Device] The photographing device 10 is a device that photographs an object O. The photographing device 10 is configured as a general digital camera. That is, the photographing device 10 is configured as a camera that receives light that has passed through a lens with an imaging element (for example, a CMOS image sensor (CMOS: Complementary Metal-Oxide-Semiconductor)), converts the received light into a digital signal, and records the digital signal in memory. Digital cameras also include those that are incorporated into other devices. For example, they also include those that are incorporated into smartphones, tablet terminals, etc.

[0043] Note that the image capturing may be performed using one image capturing device 10 or multiple image capturing devices 10. Furthermore, the image capturing may be performed in the form of a moving image instead of a still image.

[0044] [Three-dimensional model generating device] FIG. 2 is a block diagram showing an example of the hardware configuration of a three-dimensional model generating device.

[0045] The three-dimensional model generating device 100 is configured by a computer such as a PC (Personal Computer), and includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, an operation unit 104, a display unit 105, an interface unit 106, and the like.

[0046] The processor 101 executes programs and functions as various processing units. As an example, in this embodiment, the processor 101 is configured as a CPU (Central Processing Unit). Various programs and data executed by the processor 101 are stored in at least one of the main memory unit 102 and the auxiliary memory unit 103. The term "program" is synonymous with "software."

[0047] The main memory unit 102 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). The RAM is used as a work area for the processor 101. The ROM stores a basic input / output program and the like.

[0048] The auxiliary storage unit 103 is configured by, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0049] The operation unit 104 is composed of, for example, a keyboard, a mouse, a touch panel, and the like.

[0050] The display unit 105 is configured by, for example, a liquid crystal display (LCD), an organic electroluminescence diode display (OLED display), or the like.

[0051] The interface unit 106 includes an interface for connecting to external devices such as the image capturing device 10, and an interface for connecting the 3D model generating device 100 to a network such as the Internet.

[0052] FIG. 3 is a block diagram of the main functions of the three-dimensional model generating device.

[0053] 3, the 3D model generating device 100 has functions such as an image acquisition unit 100A, a 3D model generation unit 100B, an object information acquisition unit 100C, a generation method determination unit 100D, and a notification unit 100E. The functions of each unit are realized by a processor 101 executing a predetermined program.

[0054] The image acquisition unit 100A acquires a multi-viewpoint image of an object. A multi-viewpoint image is an image of an object captured from multiple viewpoints. For example, the multi-viewpoint image is captured by the image capture device 10 and stored in the auxiliary storage unit 103, and is acquired by reading it from the auxiliary storage unit 103. Alternatively, the image may be acquired directly from the image capture device 10.

[0055] The 3D model generation unit 100B generates a 3D model from the multi-viewpoint images. The 3D model generation unit 100B has a first processing unit 100B1 and a second processing unit 100B2. The first processing unit 100B1 generates the 3D model by photogrammetry. The second processing unit 100B2 generates the 3D model by a volume intersection method. The generation of a 3D model by photogrammetry and the volume intersection method is a well-known technique, so a detailed description thereof will be omitted.

[0056] The 3D model generation unit 100B generates a 3D model using the generation method determined by the generation method determination unit 100D. Therefore, if the generation method determination unit 100D determines to generate a 3D model using photogrammetry, the 3D model generation unit 100B generates the 3D model using photogrammetry. Therefore, in this case, the 3D model generation unit 100B processes the multi-viewpoint image using the first processing unit 100B1 to generate a 3D model. On the other hand, if the generation method determination unit 100D determines to generate a 3D model using the volume intersection method, the 3D model generation unit 100B generates the 3D model using the volume intersection method. Therefore, in this case, the 3D model generation unit 100B processes the multi-viewpoint image using the second processing unit 100B2 to generate a 3D model.

[0057] The object information acquisition unit 100C acquires information (object information) about the object O for generating a three-dimensional model M. As an example, in this embodiment, information about the presence or absence of a transparent region of the object is acquired as the object information. The object information acquisition unit 100C, for example, displays a predetermined input screen on the display unit 105, accepts input of information about the presence or absence of a transparent region from the user, and acquires the information. For example, a screen that allows the user to select "yes" or "no" for a transparent region is displayed on the display unit 105, accepts the selection of "yes" or "no" from the user, and acquires information about the presence or absence of a transparent region. The selection operation is performed via the operation unit 104. In this embodiment, information about the presence or absence of a transparent region is an example of information about a transparent region of the object. Furthermore, the object information is an example of first information.

[0058] The generation method determination unit 100D determines a generation method for the 3D model based on the object information acquired by the object information acquisition unit 100C. More specifically, it determines whether to use photogrammetry or volume intersection as the generation method for the 3D model. As described above, in the present embodiment, information on the presence or absence of transparent regions is acquired as object information. Therefore, the generation method determination unit 100D determines a generation method for the 3D model based on the information on the presence or absence of transparent regions. Specifically, it determines a generation method for the 3D model according to the following criteria. That is, if the object has transparent regions, the generation method determination unit 100D determines to generate a 3D model using volume intersection. On the other hand, if the object does not have transparent regions, the generation method determination unit 100D determines to generate a 3D model using photogrammetry. In the present embodiment, the information on the generation method for the 3D model determined by the generation method determination unit 100D is an example of second information.

[0059] Information on the 3D model generation method determined by generation method determination unit 100D is added to 3D model generation unit 100B. 3D model generation unit 100B generates a 3D model from the multi-viewpoint images using the generation method determined by generation method determination unit 100D.

[0060] Furthermore, information on the generation method of the three-dimensional model determined by the generation method determination unit 100D is sent to the notification unit 100E. The notification unit 100E notifies the user of the information on the determined generation method of the three-dimensional model. As an example, in this embodiment, the information on the determined generation method of the three-dimensional model is displayed on the display unit 105 to notify the user.

[0061] [Generation of 3D Model] Next, a method for generating a 3D model M of an existing object O using the 3D model generation system 1 of this embodiment will be described.

[0062] (1) Determining the method for generating a 3D model First, determine the method for generating a 3D model from multi-viewpoint images. That is, determine whether to use photogrammetry or volume intersection to generate the 3D model.

[0063] FIG. 4 is a flowchart showing the procedure of a process for determining a generation method for a three-dimensional model.

[0064] First, object information is acquired from the user (step S1). In this embodiment, the object information is information on the presence or absence of a transparent region in the object. The 3D model generation device 100 accepts input of the information on the presence or absence of a transparent region from the user and acquires the object information. The user checks whether or not a transparent region exists in the object for which a 3D model is to be generated, and inputs the confirmation result to the 3D model generation device 100. That is, if the object has a transparent region, information indicating "transparent region exists" is input to the 3D model generation device 100. On the other hand, if the object does not have a transparent region, information indicating "no transparent region exists" is input to the 3D model generation device 100.

[0065] The 3D model generation device 100 determines a method for generating a 3D model based on the acquired object information (information on the presence or absence of transparent areas). First, it determines whether or not there are transparent areas (step S2). If there are transparent areas, the 3D model generation device 100 determines to generate a 3D model by the volume intersection method (step S3). On the other hand, if there are no transparent areas, the 3D model generation device 100 determines to generate a 3D model by photogrammetry (step S4).

[0066] In this way, the three-dimensional model generating device 100 determines the method for generating a three-dimensional model based on object information (information on the presence or absence of transparent regions) obtained by user input.

[0067] Information about the determined generation method of the three-dimensional model is notified to the user. In this embodiment, the information about the determined generation method of the three-dimensional model is displayed on the display unit 105. The user checks the display on the display unit 105 to confirm the generation method of the three-dimensional model.

[0068] (2) Generation of a 3D Model The 3D model generation system 1 of this embodiment generates a 3D model from multi-viewpoint images.

[0069] First, the object O is photographed from multiple viewpoints using the imaging device 10. As described above, the photographing may be performed using one imaging device 10 or multiple imaging devices 10.

[0070] Multi-viewpoint images obtained by photographing are input to the 3D model generation device 100. The 3D model generation device 100 processes the input multi-viewpoint images to generate a 3D model. At this time, the 3D model is generated using a generation method determined in advance. That is, if it is determined that photogrammetry will be used for generation, the 3D model is generated using photogrammetry. On the other hand, if it is determined that volume intersection will be used for generation, the 3D model is generated using volume intersection.

[0071] As described above, the 3D model generation system 1 of this embodiment acquires information about the object from the user, and generates a 3D model by automatically switching the 3D model generation method based on the acquired information. This allows even a user with little knowledge about 3D scanning to easily generate a high-quality 3D model.

[0072] [Modifications] [Transparent Region Information] In the above embodiment, information on the presence or absence of transparent regions is acquired as information on transparent regions of an object. However, instead of or in addition to this, information on the proportion of transparent regions in the object may be acquired. This information on the proportion of transparent regions does not necessarily need to be an exact value, and an approximate value may be input. For example, a visually estimated value may be input. When information on the proportion of transparent regions is acquired as information on transparent regions, a threshold is set, and a method for generating a 3D model is determined by comparing with the threshold. For example, if the proportion of transparent regions is equal to or less than the threshold, photogrammetry is selected, and if it exceeds the threshold, visual volume intersection is selected.

[0073] Alternatively, for example, a user may subjectively determine the number of transparent areas, and the result of that determination may be acquired as information about the transparent areas. In this case, the method for generating the three-dimensional model is determined based on the information about the number of transparent areas. Specifically, if information indicating a small number of transparent areas is input, photogrammetry is selected, and if information indicating a large number of transparent areas is input, volume intersection is selected.

[0074] [Object Information] In the above embodiment, information about transparent regions (particularly, information about the presence or absence of transparent regions) is acquired as information about the object (object information), but the information acquired as information about the object is not limited to this. Other information may be acquired instead of or in addition to the information about transparent regions.

[0075] For example, when generating a three-dimensional model using photogrammetry or volume intersection, the method to be used can be determined by acquiring information on the texture, gloss, etc. of the object. When acquiring information on the texture of the object and determining the method to be used, the method to be used is determined, for example, based on the presence or absence or strength of the texture. Specifically, when there is no texture or the texture is weak, volume intersection is selected, and in other cases, photogrammetry is selected. Furthermore, when gloss information is acquired and determining the method to be used, the method to be used is determined based on the presence or absence or strength of the gloss. Specifically, when there is gloss or the gloss is strong, volume intersection is selected, and in other cases, photogrammetry is selected.

[0076] [Method for Acquiring Object Information] In the above embodiment, information about the object (object information) is manually input by the user, but it may also be acquired automatically. For example, when acquiring information about the presence or absence of a transparent area as object information, it may be configured to determine the presence or absence of a transparent area from an image of the object. A multi-view image may be used as the image of the object (at least one of the multiple images constituting the multi-view image is used to determine the presence or absence of a transparent area). Furthermore, a known image recognition method may be used to detect a transparent area. For example, a trained machine learning model that has been trained to recognize a transparent area from an image may be used to detect a transparent area from an image.

[0077] Alternatively, the object information may be input by the user as the name of the object (article name), and information on the presence or absence of a transparent area may be acquired. In this case, for example, a table may be prepared that associates the name of the object with information on the presence or absence of a transparent area, and the information on the presence or absence of a transparent area may be acquired from the name of the object by referring to the table.

[0078] [Method of Generating a 3D Model] In the above embodiment, an example has been described in which a 3D model of an object is generated from multi-viewpoint images using photogrammetry or a volume intersection method, but the method of generating a 3D model of an object is not limited to this. Furthermore, a configuration may be adopted in which multiple methods can be selected. Furthermore, a configuration may be adopted in which other 3D scanning methods can be selected. For example, a configuration may be adopted in which a 3D model can be generated using a LiDAR method, a ToF method, a structured light method, or the like.

[0079] [Manual Selection] In the above embodiment, the method to be used for generating the 3D model is automatically selected according to the decision, but manual selection is also possible. In the above embodiment, information on the 3D model generation method decided by the generation method decision unit 100D is notified to the user via the notification unit 100E. The user decides on the 3D model generation method with reference to the notified information. In this case, the 3D model generation device 100 functions as a device that supports the generation of the 3D model.

[0080] [Second embodiment] In this embodiment, when an object has transparent regions, a 3D model of the object is generated by changing the 3D model generation method for each region. Here, an example will be described in which a 3D model of the object is generated from multi-viewpoint images. In particular, an example will be described in which a 3D model is generated using photogrammetry and a volume intersection method.

[0081] As mentioned above, photogrammetry has difficulty with transparent or semi-transparent objects. On the other hand, visual hull intersection can generate a 3D model even if the object has transparent or semi-transparent parts, as long as the silhouette can be obtained.

[0082] In this embodiment, if the object contains transparent areas (including semi-transparent areas), the shape of the transparent areas is converted into 3D data using the volume intersection method, and the shape of the non-transparent areas is converted into 3D data using photogrammetry.

[0083] The hardware configuration of the device used to generate the 3D model (3D model generation device) is the same as that of the 3D model generation device 100 in the first embodiment. Therefore, only the functions of the 3D model generation device 100 will be described here.

[0084] FIG. 5 is a block diagram of the main functions of the three-dimensional model generating device.

[0085] 5, the 3D model generating device 100 has functions such as an image acquisition unit 100A, a 3D model generation unit 100B, a transparent region extraction unit 100F, and a generation method determination unit 100D. The functions of each unit are realized by a processor 101 executing a predetermined program.

[0086] The image acquisition unit 100A acquires multi-viewpoint images of an object.

[0087] The 3D model generation unit 100B generates a 3D model from the multi-viewpoint images. The 3D model generation unit 100B has functions such as a first processing unit 100B1, a second processing unit 100B2, and a synthesis processing unit 100B3. The first processing unit 100B1 generates a 3D model from the multi-viewpoint images using photogrammetry. The second processing unit 100B2 generates a 3D model from the multi-viewpoint images using a volume intersection method. The synthesis processing unit 100B3 synthesizes the 3D models generated for each region to generate a single 3D model that constitutes the whole.

[0088] The three-dimensional model generating unit 100B generates a three-dimensional model for each region of the object in accordance with the generation method (three-dimensional shape restoration method) for each region determined by the generation method determining unit 100D.

[0089] The transparent region extraction unit 100F extracts transparent regions (including semi-transparent regions) of an object and identifies their positions and ranges. The transparent region extraction unit 100F extracts transparent regions of an object from a multi-viewpoint image and identifies their positions and ranges. A known image recognition method can be used to extract transparent regions. Information about the transparent regions (positions and ranges) is added to the generation method determination unit 100D.

[0090] The generation method determination unit 100D determines the method for generating a 3D model for each region. Specifically, it selects volume intersection for transparent regions and photogrammetry for non-transparent regions. If the entire region is transparent, it determines that the entire region will be generated using volume intersection. If the entire region is non-transparent, it determines that the entire region will be generated using photogrammetry.

[0091] FIG. 6 is a flowchart showing the procedure of the three-dimensional model generation process.

[0092] First, a multi-viewpoint image of an object is acquired (step S11). The multi-viewpoint image is acquired by, for example, capturing an image using the image capturing device 10, storing it in the auxiliary storage unit 103, and reading it out from the auxiliary storage unit 103. The multi-viewpoint image may be captured using one image capturing device 10, or may be captured using multiple image capturing devices 10.

[0093] Next, a transparent area of ​​the object is extracted based on the acquired multi-viewpoint image, and its position and range are identified (step S12).

[0094] Next, a method for generating a 3D model for each region is determined based on the information about the transparent regions of the object (step S13). Specifically, the visual volume intersection method is selected for transparent regions, and photogrammetry is selected for non-transparent regions.

[0095] Next, a 3D model of the object is generated from the multi-viewpoint images using the determined generation method (step S14). If the entire object is transparent, the 3D model of the entire object is generated using volume intersection. If the entire object is opaque, the 3D model of the entire object is generated using photogrammetry. If the object includes a transparent area, the 3D model of the transparent area is generated using volume intersection, and the 3D model of the non-transparent area is generated using photogrammetry. The data for each area is then combined to generate a single 3D model as a whole.

[0096] As described above, according to this embodiment, by changing the method of generating a three-dimensional model for each region, a high-quality three-dimensional model can be easily generated.

[0097] In the above embodiment, an example has been described in which a three-dimensional model is automatically generated, but a three-dimensional model suitable for each area may be presented to the user.

[0098] Furthermore, in the above embodiment, an example has been described in which a three-dimensional model is generated from a two-dimensional image, but the three-dimensional scanning method used is not limited to this, and other methods may also be used.

[0099] [Third embodiment] When generating a three-dimensional model from an image of an object, if the object in the image has specular reflection (gloss), there is a problem in that the original state of the object (a state in which the effects of light are eliminated) cannot be reproduced.

[0100] One way to reduce specular reflection is to use a polarizing filter, but this requires setting the polarizing filter at an appropriate angle before shooting, which is less convenient than regular photography.

[0101] In this embodiment, a system is presented that enables even a user with little knowledge about photography to take high-quality images suitable for generating a three-dimensional model.

[0102] [Three-dimensional Model Generation System] FIG. 7 is a diagram showing a schematic configuration of a three-dimensional model generation system.

[0103] The three-dimensional model generation system 1 of this embodiment is configured as a system that generates a three-dimensional model M of an object O from multi-viewpoint images.

[0104] As shown in FIG. 7, the three-dimensional model generation system 1 includes a photographing device 10 and a three-dimensional model generation device 100 .

[0105] [Photographing Device] The photographing device 10 is a device that photographs an object O. The photographing device 10 is configured as a general digital camera. Note that photographing may be performed using one photographing device 10 or multiple photographing devices 10.

[0106] [3D Model Generation Device] The hardware configuration of the 3D model generation device 100 is the same as that of the 3D model generation device 100 of the first embodiment. That is, it is configured by a computer such as a PC, and includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, an operation unit 104, a display unit 105, an interface unit 106, etc. (see FIG. 2 ).

[0107] FIG. 8 is a block diagram of the main functions of the three-dimensional model generating device.

[0108] The 3D model generating device 100 has a function of generating a 3D model of an object from multi-viewpoint images (3D model generating function) and a function of supporting the capturing of multi-viewpoint images (capturing support function).

[0109] [3D Model Generation Function] The 3D model generation function is a function for generating a 3D model of an object from multi-viewpoint images. The 3D model generation device 100 has the functions of an image acquisition unit 100A, a 3D model generation unit 100B, etc. as the 3D model generation function.

[0110] The image acquisition unit 100A acquires multi-viewpoint images of an object. The multi-viewpoint images are acquired by, for example, capturing images by the image capturing device 10, storing them in the auxiliary storage unit 103, and reading them out from the auxiliary storage unit 103. Alternatively, the image acquisition unit 100A may be configured to acquire the images directly from the image capturing device 10.

[0111] The 3D model generating unit 100B generates a 3D model from the multi-viewpoint images. As an example, in this embodiment, the 3D model of the object is generated from the multi-viewpoint images by photogrammetry.

[0112] [Photography Support Function] In this embodiment, the photography support function provides a function of presenting information (photography support information) that supports the photographing of multi-viewpoint images to the user. Specifically, the photography support function determines the optimal photography technique (recommended photography technique) based on the environment in which the object is photographed, particularly the lighting conditions, and presents the optimal photography technique to the user. As shown in FIG. 7 , the 3D model generation device 100 has, as the photography support function, functions such as an illumination information acquisition unit 100G, a photography technique determination unit 100H, and a notification unit 100E.

[0113] The illumination information acquisition unit 100G acquires information on the illumination state (illumination information). The illumination information acquisition unit 100G receives input of the illumination state information from a user and acquires the illumination state information. In this embodiment, information on the presence or absence of specular reflection is acquired as the illumination information. That is, information on whether or not the illumination state is such that specular reflection occurs is acquired as the illumination information. The illumination information acquisition unit 100G, for example, displays a predetermined input screen on the display unit 105 to accept input of the information on the presence or absence of specular reflection from the user. For example, the illumination information acquisition unit 100G displays a screen on the display unit 105 that allows the user to select "yes" or "no" for specular reflection, and receives the user's selection of "yes" or "no" to acquire the information on the presence or absence of specular reflection. The selection operation is performed via the operation unit 104. In this embodiment, the information on the presence or absence of specular reflection is an example of information on the illumination state. Furthermore, the information on the illumination state is an example of information on the environment in which the object is photographed (or measured), and is also an example of first information.

[0114] The imaging technique determination unit 100H determines the optimal imaging technique (recommended imaging technique) based on information about the lighting conditions obtained from the user (in this embodiment, information about the presence or absence of specular reflection). Here, imaging using a polarizing filter is defined as "polarized imaging" and imaging without a polarizing filter is defined as "normal imaging," and it determines which one to adopt (whether to adopt polarized imaging or not). Specifically, if "specular reflection is present," "polarized imaging" is selected. On the other hand, if "specular reflection is absent," "normal imaging" is selected. In this embodiment, the imaging technique information determined by the imaging technique determination unit 100H (information about the imaging technique for images used to generate three-dimensional data) is an example of second information.

[0115] The notification unit 100E notifies the user of information about the shooting technique determined by the shooting technique determination unit 100H as shooting support information. As an example, in this embodiment, the information about the determined shooting technique (shooting support information) is displayed on the display unit 105 to notify the user.

[0116] [Generation of 3D Model] Next, a method for generating a 3D model M of an object O from multi-viewpoint images using the 3D model generation system 1 of this embodiment will be described.

[0117] (1) Provision of Photographing Support Information First, photographing support information is provided from the three-dimensional model generating device 100.

[0118] FIG. 9 is a flowchart showing the procedure of a process for providing photographing support information in the three-dimensional model generating device.

[0119] First, information on the illumination state is acquired from the user (step S21). In this embodiment, the illumination state information is acquired by accepting input of the illumination state information from the user. The illumination state information is information on the presence or absence of specular reflection.

[0120] The three-dimensional model generation device 100 determines the photography technique based on the acquired information on the lighting conditions (information on the presence or absence of specular reflection). First, it is determined whether specular reflection occurs (step S22). If specular reflection is present, the three-dimensional model generation device 100 selects polarized photography as the photography technique (step S23). On the other hand, if specular reflection is absent, the three-dimensional model generation device 100 selects normal photography as the photography technique (step S24).

[0121] The three-dimensional model generating device 100 notifies the user of information on the determined photography technique as photography support information (step S25). In this embodiment, the information on the determined photography technique (photography support information) is displayed on the display unit 105 to notify the user. The photography support information is provided to the user as information on the optimal photography technique based on the lighting conditions.

[0122] (2) Generation of 3D Models The user is presented with the photography support information, determines the photography technique, and photographs multi-viewpoint images of the object. For example, if normal photography is presented as the optimal photography technique, the user photographs the object using the normal photography technique (a photography technique that does not use a polarizing filter). On the other hand, if polarized photography is presented as the optimal photography technique, the user photographs the object using a polarizing filter. Specifically, the user attaches a polarizing filter to the lens and sets it at an appropriate angle (an angle that can suppress specular reflection) to photograph the object.

[0123] Multi-viewpoint images obtained by photographing are input to the 3D model generation device 100. The 3D model generation device 100 processes the input multi-viewpoint images to generate a 3D model.

[0124] As described above, according to this embodiment, it is possible to photograph an object while suppressing the occurrence of specular reflection. This makes it possible to generate a high-quality 3D model. Furthermore, it is possible to select an appropriate photographing technique depending on the lighting conditions, thereby optimizing the balance between the quality of the generated 3D model and work efficiency.

[0125] [Modifications] [Illumination Condition Information] In the above embodiment, information on the presence or absence of specular reflection is acquired as illumination condition information. However, instead of or in addition to this, information on the proportion of the area where specular reflection occurs to the object may be acquired. The proportion does not necessarily have to be an exact value, and an approximate value may be input. For example, a visually estimated value may be input. When this information is acquired as illumination condition information, a threshold is set, and the imaging method is determined by comparing with the threshold. For example, if the proportion of the area where specular reflection occurs to the object is equal to or less than the threshold, normal imaging is selected, and if it exceeds the threshold, polarized imaging is selected.

[0126] Furthermore, the information on the lighting condition may be information on the degree (strength) of specular reflection. The information on the degree of specular reflection may be information visually confirmed by the user and input. In this case, for example, polarized photography may be selected in the case of strong specular reflection, and normal photography may be selected in the case of weak specular reflection.

[0127] Alternatively, for example, the user may subjectively determine the number of areas where specular reflection occurs, and the result of this determination may be acquired as information on the lighting condition. In this case, the shooting method is determined based on the information on whether the proportion of areas where specular reflection occurs is high or low. Specifically, if information indicating a low proportion is input, normal shooting is selected, and if information indicating a high proportion is input, polarized shooting is selected.

[0128] Furthermore, information on the setting status of lighting devices may be acquired as information on the lighting conditions. For example, information such as the number of lighting devices, whether a diffuser is used, and brightness settings may be acquired as information on the lighting conditions. By acquiring this information, it is possible to estimate whether specular reflection occurs. Therefore, it is possible to estimate the optimal shooting method (recommended shooting method). For example, by using multiple lighting devices to uniformly irradiate the object with illumination light, it is possible to suppress the occurrence of specular reflection. Therefore, it is possible to select the optimal shooting method depending on the number of lighting devices. For example, when multiple lighting devices (above a threshold) are used, it is assumed that the lighting conditions suppress the occurrence of specular reflection, and normal shooting is selected. Furthermore, specular reflection can also be suppressed by using a diffuser. Therefore, it is possible to select the optimal shooting method depending on whether a diffuser is used. Furthermore, specular reflection can also be suppressed by setting the brightness. Therefore, it is possible to select the optimal shooting method depending on the brightness setting.

[0129] Furthermore, information on the lighting environment may be acquired as information on the shooting environment, such as information on whether the lighting is adjustable (information on whether the lighting is adjustable), and the optimal shooting method (recommended shooting method) may be determined based on the acquired information. For example, polarized light shooting is preferable in environments where lighting cannot be adjusted (such as outdoors). Therefore, information on whether the lighting is adjustable is acquired, and the shooting method is determined based on that information. Specifically, if the lighting is adjustable, normal shooting is selected, and if the lighting is not adjustable, polarized light shooting is selected.

[0130] Furthermore, information on the lighting environment may be acquired, such as whether the lighting generates specular reflection on the object, and the optimal shooting method (recommended shooting method) may be determined based on the acquired information. For example, polarized shooting may be selected when the lighting generates specular reflection on the object, and normal shooting may be selected otherwise.

[0131] In this way, the information on the lighting conditions or the information on the environment in which the object is photographed may be any information that can be used to determine the presence or absence of specular reflection or the degree of its occurrence.

[0132] [Method of Acquiring Information on Illumination Conditions] In the above embodiment, the user manually inputs information on illumination conditions, but it may also be configured to acquire the information automatically. For example, it is possible to determine the presence or absence and / or degree of specular reflection from an image of an object, and automatically acquire information on illumination conditions. A multi-view image can be used for the image of the object (at least one of the multiple images that make up the multi-view image is used to determine the presence or absence of specular reflection).

[0133] [Method for generating a three-dimensional model] In the above embodiment, an example has been described in which a three-dimensional model of an object is generated from multi-viewpoint images using photogrammetry, but the present invention can also be applied to generating a three-dimensional model using other methods (for example, volume intersection method, etc.).

[0134] The present invention can also be applied to the generation of 3D models using other 3D scanning methods. For example, when generating a 3D model using LiDAR, information about the environment in which the object is measured can be acquired, and based on the acquired information, the optimal measurement method can be determined and presented to the user.

[0135] [Presenting Photographing Support Information] In the above embodiment, the photographing support information (information on the optimal photographing technique) is displayed on the display unit 105 of the three-dimensional model generation device 100 to notify the user, but the method of notifying the user of the photographing support information is not limited to this. For example, the photographing support information may be transmitted to the photographing device 10 and displayed on the display unit of the photographing device 10.

[0136] In the above embodiment, the 3D model generation device 100 is provided with a photography support function. However, the photography support function may be provided independently in another device. For example, the photography support function may be provided in the photography device 10. Alternatively, the photography support function may be provided in a mobile device such as a smartphone or a tablet terminal.

[0137] [Photography Support] When taking polarized photographs, information about the polarizing filter settings may also be provided to the user. For example, when photographing by fixing the photographing device 10 and rotating the subject, the optimal polarizing filter setting angle may be calculated and provided to the user.

[0138] Fourth Embodiment When generating a three-dimensional model of an object from photographed images of the object, it is necessary to photograph the object from multiple viewpoints.

[0139] As a method for capturing a multi-viewpoint image, there is a method for capturing an image by moving the object side, and a method for capturing an image by moving the image capturing device side.

[0140] A common method of photographing a moving object is to use a turntable, in which the object is placed on the turntable and rotated, and photographed from a fixed position using a photographing device.

[0141] On the other hand, handheld photography is a common form of photography in which the camera is moved, in which the photographer (user) holds the camera in his or her hand and photographs the subject from various angles.

[0142] While handheld photography has the advantage of being simple and easy to use, it has the disadvantage of being prone to image blur due to camera shake, which can have a negative impact on the generation of a 3D model.

[0143] On the other hand, the influence of camera shake can be reduced by changing the shooting conditions, for example, by increasing the shutter speed.

[0144] In this embodiment, a system is presented that enables even a user with little knowledge about photography to take high-quality images suitable for generating a three-dimensional model.

[0145] [Three-dimensional Model Generation System] FIG. 10 is a diagram showing a schematic configuration of a three-dimensional model generation system.

[0146] The three-dimensional model generation system 1 of this embodiment is configured as a system that generates a three-dimensional model M of an object O from multi-viewpoint images.

[0147] As shown in FIG. 10, the three-dimensional model generation system 1 includes an imaging device 10 and a three-dimensional model generation device 100 .

[0148] [Photographing Device] The photographing device 10 is a device that photographs an object O. The photographing device 10 is configured as a general digital camera.

[0149] [3D Model Generation Device] The hardware configuration of the 3D model generation device 100 is the same as that of the 3D model generation device 100 of the first embodiment. That is, it is configured by a computer such as a PC, and includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, an operation unit 104, a display unit 105, an interface unit 106, etc. (see FIG. 2 ).

[0150] FIG. 11 is a block diagram of the main functions of the three-dimensional model generating device.

[0151] The 3D model generating device 100 has a function of generating a 3D model of an object from multi-viewpoint images (3D model generating function) and a function of supporting the capturing of multi-viewpoint images (capturing support function).

[0152] [3D Model Generation Function] The 3D model generation function is the same as the 3D model generation function in the 3D model generation device 100 of the third embodiment, and therefore a description thereof will be omitted.

[0153] [Photography Support Function] In this embodiment, the photography support function acquires information on the photography form from the user, determines the photography conditions based on the acquired information, and presents the determined photography conditions to the user. As shown in Fig. 11, the three-dimensional model generation device 100 has, as the photography support function, functions such as a photography form information acquisition unit 100J, a photography condition determination unit 100K, and a notification unit 100E.

[0154] The shooting mode information acquisition unit 100J acquires information on the shooting mode (shooting mode information). The shooting mode information acquisition unit 100J accepts input of shooting mode information from a user and acquires the shooting mode information. In this embodiment, information on whether shooting is being performed by handheld shooting is acquired as the shooting mode information. The shooting mode information acquisition unit 100J, for example, displays a predetermined input screen on the display unit 105 to accept input of information on whether shooting is being performed by handheld shooting from the user. For example, the shooting mode information acquisition unit 100J displays a screen on the display unit 105 that prompts the user to select "Yes" or "No" regarding whether shooting is being performed by handheld shooting, and acquires the information on whether shooting is being performed by handheld shooting by accepting the user's selection of "Yes" or "No." The selection operation is performed via the operation unit 104. In this embodiment, the shooting mode information acquired by the shooting mode information acquisition unit 100J (information on whether shooting is being performed by handheld shooting) is an example of first information.

[0155] The photographing condition determination unit 100K determines the photographing conditions based on information about the photographing mode acquired from the user. In this embodiment, when photographing is performed by hand, it is determined that photographing is performed at a high shutter speed as the photographing condition. "Photographing at a high shutter speed" means that photographing is performed with a high (fast) shutter speed. On the other hand, when photographing is not performed by hand, it is determined that photographing is performed at a low shutter speed as the photographing condition. "Photographing at a low shutter speed" means that photographing is performed with a low (slow) shutter speed.

[0156] "High shutter speed" and "low shutter speed" are determined from the perspective of suppressing the effects of camera shake. That is, a "high shutter speed" is a shutter speed that can suppress the effects of camera shake when shooting handheld. Generally, it is believed that the effects of camera shake can be suppressed by setting a shutter speed faster than 1 / focal length [seconds]. Therefore, "shooting at a high shutter speed" means shooting at a shutter speed faster than 1 / focal length [seconds]. On the other hand, "low shutter speed" means shooting at a shutter speed slower than 1 / focal length [seconds]. Alternatively, with 1 / 60 [seconds] as the reference, a shutter speed faster than 1 / 60 [seconds] can be defined as a "high shutter speed," and a shutter speed slower than 1 / 60 [seconds] can be defined as a "low shutter speed." Note that the shutter speed that can suppress the effects of camera shake varies depending on whether or not a camera shake correction function is provided. Therefore, it is preferable to set the shutter speed used as the reference for high and low shutter speeds depending on whether or not a camera shake correction function is provided.

[0157] In the present embodiment, the information on the shooting conditions determined by the shooting condition determination unit 100K is an example of the second information.

[0158] The notification unit 100E notifies the user of information on the shooting conditions determined by the shooting condition determination unit 100K as shooting support information. As an example, in this embodiment, the information on the determined shooting conditions (shooting support information) is displayed on the display unit 105 to notify the user.

[0159] [Generation of 3D Model] Next, a method for generating a 3D model M of an object O from multi-viewpoint images using the 3D model generation system 1 of this embodiment will be described.

[0160] (1) Provision of Photographing Support Information First, photographing support information is provided from the three-dimensional model generating device 100.

[0161] FIG. 12 is a flowchart showing the procedure of a process for providing photography support information in the three-dimensional model generating device.

[0162] First, information on the shooting mode is acquired from the user (step S31). In this embodiment, the information on the shooting mode is acquired by accepting input of the information on the shooting mode from the user. The information on the shooting mode is information on whether or not shooting is performed by handholding the camera.

[0163] The three-dimensional model generation device 100 determines the shooting conditions based on the acquired shooting mode information (information on whether shooting was done handheld). First, it is determined whether shooting was done handheld (step S32). If shooting was done handheld, the three-dimensional model generation device 100 determines that shooting should be done with a high shutter speed as the shooting condition (step S33). On the other hand, if shooting was not done handheld, the three-dimensional model generation device 100 determines that shooting should be done with a low shutter speed as the shooting condition (step S34).

[0164] The three-dimensional model generating device 100 notifies the user of the information on the determined shooting conditions as shooting support information (step S35). In this embodiment, the information on the determined shooting conditions (shooting support information) is notified to the user by displaying it on the display unit 105. The shooting support information is provided to the user as optimal shooting setting information (recommended shooting setting information) based on the shooting mode.

[0165] (2) Generation of a 3D model The user receives the shooting support information, sets the exposure, and captures multi-viewpoint images of the object. Therefore, for example, when shooting handheld, the user sets a high shutter speed to capture the object. This reduces image blurring and allows images suitable for generating a 3D model to be captured. On the other hand, when shooting not handheld, the user sets a low shutter speed to capture the object.

[0166] Multi-viewpoint images obtained by photographing are input to the 3D model generation device 100. The 3D model generation device 100 processes the input multi-viewpoint images to generate a 3D model.

[0167] As described above, according to this embodiment, even a user with little knowledge about photography can take high-quality images suitable for generating a three-dimensional model, thereby making it possible to easily generate a high-quality three-dimensional model.

[0168] [Modifications] [Photographing Conditions] In the above embodiment, the shutter speed setting is determined as a photographing condition, but the items determined as photographing conditions are not limited to this. For example, the shutter speed and sensitivity (ISO sensitivity (ISO: International Organization for Standardization)) can also be determined according to the photographing mode. In this case, for example, high sensitivity and high shutter speed are set for handheld photography, and low sensitivity and low shutter speed are set for non-handheld photography. Also, for example, the shutter speed, aperture value (F-number), and sensitivity can also be determined according to the photographing mode. In this case, for example, for handheld photography, the aperture value is fixed and high sensitivity and high shutter speed are set. Also, for non-handheld photography, the aperture value is fixed and low sensitivity and low shutter speed are set.

[0169] [Presenting Photographing Support Information] In the above embodiment, the photographing support information (information on photographing conditions) is displayed on the display unit 105 of the three-dimensional model generation device 100 to notify the user, but the method of notifying the user of the photographing support information is not limited to this. For example, the photographing support information may be transmitted to the photographing device 10 and displayed on the display unit of the photographing device 10.

[0170] In the above embodiment, the 3D model generation device 100 is provided with a photography support function. However, the photography support function may be provided independently in another device. For example, the photography support function may be provided in the photography device 10. Alternatively, the photography support function may be provided in a mobile device such as a smartphone or a tablet terminal.

[0171] If the photographing device 10 is equipped with a photographing support function, it may be configured to automatically set the exposure. For example, when photographing by hand, the exposure is automatically set to a high shutter speed and high sensitivity. When photographing is not by hand, the exposure is automatically set to a low shutter speed and low sensitivity.

[0172] Furthermore, the imaging device 10 and the three-dimensional model generating device 100 may be connected to each other so that they can communicate with each other, and the imaging conditions determined by the three-dimensional model generating device 100 may be automatically reflected in the imaging device 10 .

[0173] [Method for generating a three-dimensional model] In the above embodiment, an example has been described in which a three-dimensional model of an object is generated from multi-viewpoint images using photogrammetry, but the present invention can also be applied to generating a three-dimensional model using other methods (for example, volume intersection method, etc.).

[0174] Fifth Embodiment As described above, one way of capturing multi-viewpoint images is to use a turntable. When capturing images using a turntable, the subject moves, and so-called subject blur (motion blur) occurs unless an appropriate shutter speed is set for capture. Images with subject blur adversely affect the generation of a 3D model.

[0175] In this embodiment, a system is presented that enables even a user with little knowledge about photography to take high-quality images suitable for generating a three-dimensional model.

[0176] [Three-dimensional Model Generation System] FIG. 13 is a diagram showing a schematic configuration of a three-dimensional model generation system.

[0177] The three-dimensional model generation system 1 of this embodiment is configured as a system that generates a three-dimensional model M of an object O from multi-viewpoint images.

[0178] As shown in FIG. 13, the three-dimensional model generation system 1 includes an imaging device 10 and a three-dimensional model generation device 100 .

[0179] [Photographing Device] The photographing device 10 is a device that photographs an object O. The photographing device 10 is configured as a general digital camera.

[0180] The image capturing device 10 is fixed on a tripod or the like and captures an image of an object O rotating at a fixed position from the fixed position. As an example, in this embodiment, the object O is placed on a turntable 20 and rotated. The turntable 20 rotates at a constant rotational speed N [rpm].

[0181] It is also possible to adopt a configuration in which photographing is performed using a plurality of photographing devices 10. In this case, for example, the plurality of photographing devices 10 are used to simultaneously photograph the object O from different positions or directions.

[0182] [3D Model Generation Device] The hardware configuration of the 3D model generation device 100 is the same as that of the 3D model generation device 100 of the first embodiment. That is, it is configured by a computer such as a PC, and includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, an operation unit 104, a display unit 105, an interface unit 106, etc. (see FIG. 2 ).

[0183] FIG. 14 is a block diagram of the main functions of the three-dimensional model generating device.

[0184] The 3D model generating device 100 has a function of generating a 3D model of an object from multi-viewpoint images (3D model generating function) and a function of supporting the capturing of multi-viewpoint images (capturing support function).

[0185] [3D Model Generation Function] The 3D model generation function is the same as the 3D model generation function in the 3D model generation device 100 of the third embodiment, and therefore a description thereof will be omitted.

[0186] [Photography Support Function] In this embodiment, the photography support function acquires information on the photography mode from the user, determines the photography conditions based on the acquired information, and presents them to the user. As shown in Fig. 14, the three-dimensional model generation device 100 has, as its photography support function, functions such as a scanning speed information acquisition unit 100L, a photography condition determination unit 100K, and a notification unit 100E.

[0187] The scanning speed information acquisition unit 100L acquires scanning speed information (scanning speed information). The scanning speed is the speed at which the object O is scanned. As described above, in this embodiment, the rotating object O is photographed from a fixed position and the object O is scanned. Therefore, the rotation speed of the object O is the scanning speed. The rotation speed of the object O is the speed at which the turntable 20 rotates (rotation speed N). Therefore, the scanning speed information acquisition unit 100L acquires information on the rotation speed N of the turntable 20 as scanning speed information. The scanning speed information acquisition unit 100L accepts input of scanning speed information (rotation speed of the turntable 20) from the user and acquires the scanning speed information. For example, the scanning speed information acquisition unit 100L displays a predetermined input screen on the display unit 105 to accept input of scanning speed information from the user. The input operation is performed via the operation unit 104. In the present embodiment, the scanning speed information (imaging condition information) acquired by the scanning speed information acquisition unit 100L is an example of the first information.

[0188] The photographing condition determination unit 100K determines photographing conditions based on the scanning speed (rotation speed of the turntable 20) information acquired from the user. Specifically, it determines a shutter speed. This shutter speed is a shutter speed that can suppress subject blur.

[0189] Here, the higher the shutter speed, the more effectively it can suppress subject blur. On the other hand, in order to increase the shutter speed, it is necessary to increase the sensitivity (assuming the aperture value is fixed). Increasing the sensitivity increases noise components (decreasing the signal-to-noise ratio). Therefore, it is preferable to set the shutter speed to the minimum necessary to suppress subject blur.

[0190] In the present embodiment, the information on the photographing conditions (information on the shutter speed) determined by the photographing condition determination unit 100K is an example of the second information.

[0191] The notification unit 100E notifies the user of information about the shooting conditions (shutter speed information) determined by the shooting condition determination unit 100K as shooting support information. As an example, in this embodiment, the information about the determined shooting conditions (shooting support information) is displayed on the display unit 105 to notify the user.

[0192] [Generation of 3D Model] Next, a method for generating a 3D model M of an object O from multi-viewpoint images using the 3D model generation system 1 of this embodiment will be described.

[0193] (1) Provision of Photographing Support Information First, photographing support information is provided from the three-dimensional model generating device 100.

[0194] FIG. 15 is a flowchart showing the procedure of a process for providing photographing support information in the three-dimensional model generating device.

[0195] First, scanning speed information is acquired from the user (step S41). In this embodiment, the scanning speed information is acquired by accepting input of information on the rotation speed N of the turntable 20 from the user.

[0196] The three-dimensional model generating device 100 determines the photographing conditions based on the acquired scanning speed information (step S42). In this embodiment, a shutter speed that can suppress subject blur is determined.

[0197] The three-dimensional model generating device 100 notifies the user of the determined information on the shooting conditions (shutter speed) as shooting support information (step S43). In this embodiment, the determined information on the shooting conditions (shutter speed) is notified to the user by displaying it on the display unit 105. The shooting support information is provided to the user as setting information for an optimal shutter speed based on the scanning speed (setting information for a recommended shutter speed).

[0198] (2) Generation of a 3D model The user receives the photographing support information, sets the exposure, and photographs the object. More specifically, the user sets the shutter speed presented by the 3D model generation device 100, and also sets the aperture value and sensitivity to obtain the appropriate exposure, and then photographs the object.

[0199] As described above, the photographing is performed using the turntable 20. That is, the object O is placed on the turntable 20 and photographed while being rotated at a constant speed. The photographing device 10 photographs the rotating object O from a fixed position.

[0200] Multi-viewpoint images obtained by photographing are input to the 3D model generation device 100. The 3D model generation device 100 processes the input multi-viewpoint images to generate a 3D model.

[0201] As described above, according to this embodiment, even a user with little knowledge about photography can take high-quality images suitable for generating a three-dimensional model, thereby making it possible to easily generate a high-quality three-dimensional model.

[0202] [Modifications] [Photographing Conditions] In the above embodiment, the shutter speed is determined as a photographing condition, but the items determined as photographing conditions are not limited to this. For example, the photographing interval may be determined in addition to or instead of the shutter speed. By adjusting the photographing interval, it is possible to adjust the overlap rate between adjacent images. Furthermore, by setting the photographing interval appropriately, it is possible to reduce missed shots.

[0203] [Photographing Mode] In the above embodiment, a rotating object is photographed from a fixed position, but the photographing may be performed by rotating the photographing device itself. For example, a stationary object may be photographed by a photographing device that rotates (revolves) around it at a constant speed. In this case, the rotational speed (revolution speed) of the photographing device becomes the scanning speed. Alternatively, an object that rotates (spins) at a constant speed may be photographed by a photographing device that rotates (revolves) around it at a constant speed. In this case, the difference between the rotational speed (revolution speed) of the photographing device and the rotational speed (rotation speed) of the object becomes the scanning speed.

[0204] [Presenting Photographing Support Information] In the above embodiment, the photographing support information (information on photographing conditions) is displayed on the display unit 105 of the three-dimensional model generation device 100 to notify the user, but the method of notifying the user of the photographing support information is not limited to this. For example, the photographing support information may be transmitted to the photographing device 10 and displayed on the display unit of the photographing device 10.

[0205] In the above embodiment, the 3D model generation device 100 is provided with a photography support function. However, the photography support function may be provided independently in another device. For example, the photography support function may be provided in the photography device 10. Alternatively, the photography support function may be provided in a mobile device such as a smartphone or a tablet terminal.

[0206] If the photographing device 10 is equipped with a photographing support function, the exposure may be set automatically. That is, the exposure is set automatically by setting a determined shutter speed.

[0207] Furthermore, the imaging device 10 and the three-dimensional model generating device 100 may be connected to each other so that they can communicate with each other, and the imaging conditions determined by the three-dimensional model generating device 100 may be automatically reflected in the imaging device 10 .

[0208] [Method for generating a three-dimensional model] In the above embodiment, an example has been described in which a three-dimensional model of an object is generated from multi-viewpoint images using photogrammetry, but the present invention can also be applied to generating a three-dimensional model using other methods (for example, volume intersection method, etc.).

[0209] The present invention can also be applied to the generation of three-dimensional models using other methods. For example, the present invention can be applied to the generation of three-dimensional models using LiDAR, ToF, structured light, etc. For example, when generating a three-dimensional model using LiDAR and ToF, optimal measurement conditions are determined based on scanning speed information and presented to the user or automatically set.

[0210] Sixth Embodiment The quality of the generated 3D model is preferably determined according to the importance of the 3D model. That is, the more important the 3D model, the higher the quality (reproducibility, image quality) of the generated 3D model. Generally, the higher the quality of the 3D model, the greater the number of images or measurement data required for processing. Furthermore, the higher the quality, the greater the processing man-hours and the longer the processing time.

[0211] It is difficult for a user with little knowledge about the generation of three-dimensional models to appropriately set the processing according to the importance of the three-dimensional model.

[0212] In this embodiment, a system is presented that allows even a user with little knowledge about generating three-dimensional models to appropriately set the necessary processing depending on the importance of the three-dimensional model to be generated, and to generate a three-dimensional model of quality according to the importance.

[0213] [Three-dimensional Model Generation System] FIG. 16 is a diagram showing a schematic configuration of a three-dimensional model generation system.

[0214] The three-dimensional model generation system 1 of this embodiment is configured as a system that generates a three-dimensional model M of an object O from multi-viewpoint images.

[0215] As shown in FIG. 16, the three-dimensional model generation system 1 includes an imaging device 10 and a three-dimensional model generation device 100 .

[0216] [Photographing Device] The photographing device 10 is a device that photographs an object O. The photographing device 10 is configured as a general digital camera. Note that photographing may be performed using one photographing device 10 or multiple photographing devices 10.

[0217] [3D Model Generation Device] The hardware configuration of the 3D model generation device 100 is the same as that of the 3D model generation device 100 of the first embodiment. That is, it is configured by a computer such as a PC, and includes a processor 101, a main memory unit 102, an auxiliary memory unit 103, an operation unit 104, a display unit 105, an interface unit 106, etc. (see FIG. 2 ).

[0218] FIG. 17 is a block diagram of the main functions of the three-dimensional model generating device.

[0219] 17, the 3D model generating device 100 has functions such as an image acquisition unit 100A, a 3D model generation unit 100B, an importance information acquisition unit 100M, and a processing mode determination unit 100N. The functions of each unit are realized by a processor 101 executing a predetermined program.

[0220] The image acquisition unit 100A acquires multi-viewpoint images of an object. The multi-viewpoint images are acquired by, for example, capturing images by the image capturing device 10, storing them in the auxiliary storage unit 103, and reading them out from the auxiliary storage unit 103. Alternatively, the image acquisition unit 100A may be configured to acquire the images directly from the image capturing device 10.

[0221] The 3D model generation unit 100B generates a 3D model from the multi-viewpoint images. As an example, in this embodiment, a 3D model of an object is generated from the multi-viewpoint images by photogrammetry. The 3D model generation unit 100B processes the multi-viewpoint images in a processing mode determined by the processing mode determination unit 100N to generate a 3D model of the object.

[0222] Here, the term "processing mode" refers to a processing mode for generating a 3D model of an object from multi-viewpoint images. Changing the processing mode changes the quality of the generated 3D model. As an example, this embodiment has two processing modes: a "high quality mode" and a "low quality mode." The "high quality mode" is a mode that enables the generation of a high-quality 3D model. The "low quality mode" is a mode that generates a lower-quality 3D model compared to the high quality mode. In each mode, the quality of the generated 3D model is adjusted by changing the settings of specific parameters. As an example, this embodiment adjusts the quality of the generated 3D model by changing the settings of the number of images used to generate the 3D model and the number of iterations of the recursive estimation process of the 3D shape. In the high quality mode, a large number of images are used, and a large number of iterations of the recursive estimation process of the 3D shape are set. In the low quality mode, the number of images used and the number of iterations of the recursive estimation process of the 3D shape are set to the minimum necessary. By setting a large number of images to be used and a large number of iterations of the recursive estimation process of the 3D shape, it becomes possible to generate a high-quality 3D model, but this increases the number of processing steps and requires a long time for generation.On the other hand, by setting the number of images to be used and the number of iterations of the recursive estimation process of the 3D shape to the minimum necessary, the time required for generation can be shortened.

[0223] The importance information acquisition unit 100M acquires importance information (importance information) about the 3D model to be generated. Here, "importance" is subjectively determined by the user. The higher the required quality, the higher the importance is set. The importance information acquisition unit 100M accepts input of importance information from the user and acquires the importance information. As an example, in this embodiment, importance is divided into two levels, "high" and "low," and the importance information is acquired. The importance information acquisition unit 100M displays, for example, a predetermined selection screen (a screen that allows the user to select "high" or "low" for importance) on the display unit 105 and accepts input (selection) of importance information from the user. The selection operation is performed via the operation unit 104. In this embodiment, the importance information acquired by the importance information acquisition unit 100M is an example of first information.

[0224] The processing mode determination unit 100N determines the processing mode based on importance information acquired from the user. In this embodiment, the processing mode is determined by selecting either the high-quality mode or the low-quality mode. Specifically, if the importance is "high," the high-quality mode is selected, and if the importance is "low," the low-quality mode is selected. In this embodiment, the information on the processing mode of the 3D model determined by the processing mode determination unit 100N is an example of second information.

[0225] Information on the processing mode determined by processing mode determination unit 100N is added to 3D model generation unit 100B. 3D model generation unit 100B processes the multi-viewpoint images in the processing mode determined by generation method determination unit 100D, and generates a 3D model of the object.

[0226] [Generation of 3D Model] Next, a method for generating a 3D model M of an existing object O using the 3D model generation system 1 of this embodiment will be described.

[0227] First, the object O is photographed from multiple viewpoints using the imaging device 10. The photographing may be performed using one imaging device 10 or multiple imaging devices 10.

[0228] Multi-viewpoint images obtained by photographing are input to a three-dimensional model generating device 100, and a three-dimensional model is generated.

[0229] FIG. 18 is a flowchart showing the procedure for generating a three-dimensional model from multi-viewpoint images.

[0230] First, a multi-viewpoint image of the object is acquired (step S51).

[0231] Next, importance information for the 3D model to be generated is acquired (step S52). In this embodiment, the importance information is information of "high" or "low". The 3D model generating device 100 accepts an input (selection) of importance information from the user and acquires the importance information for the 3D model to be generated.

[0232] Next, a processing mode for generating a 3D model (processing mode) is determined based on the acquired importance information (step S53). In this embodiment, either high-quality mode or low-quality mode is selected to determine the processing mode. Specifically, if the importance is "high," the high-quality mode is selected, and if the importance is "low," the low-quality mode is selected.

[0233] Next, the multi-view images are processed in the determined processing mode to generate a 3D model (step S54). For example, if the high-quality mode is selected, the multi-view images are processed in the high-quality mode to generate a 3D model. On the other hand, if the low-quality mode is selected, the multi-view images are processed in the low-quality mode to generate a 3D model. In the high-quality mode, a high-quality 3D model is generated. On the other hand, in the low-quality mode, a low-quality 3D model is generated compared to the high-quality mode.

[0234] As described above, according to the 3D model generation system 1 of this embodiment, processing conditions (processing modes) are automatically set according to the importance of the 3D model to be generated, and a 3D model is generated. This allows even a user with little knowledge of 3D scanning to appropriately generate a 3D model of quality according to the importance. In other words, a high-quality 3D model can be generated for items of high importance, and a 3D model of low importance can be generated in a short time.

[0235] [Variations] [Importance Information] In the above embodiment, importance is divided into two levels, "high" and "low," and the information is acquired, but it may also be divided into multiple levels and acquired.

[0236] [Processing Conditions for Generating a 3D Model] In the above embodiment, the processing mode is determined as the processing condition, but the content of the determined processing condition is not limited to this. Specific processing parameters (such as the number of images used to generate a 3D model and the number of iterations of the recursive estimation process for the 3D shape) may be determined individually based on the importance information.

[0237] [Method for Generating a 3D Model] In the above embodiment, an example has been described in which a 3D model of an object is generated from multi-viewpoint images using photogrammetry, but the method for generating a 3D model of an object is not limited to this. Other generation methods can also be adopted. Furthermore, the present invention can also be applied to cases in which a 3D model is generated using image measurement data using a LiDAR method, a ToF method, or the like.

[0238] [Manual Selection] In the above embodiment, an example has been described in which a 3D model is automatically generated in a determined processing mode, but a configuration may also be adopted in which a processing mode is manually selected according to the determination, and a 3D model is generated in the selected processing mode. In this case, information about the determined processing mode is notified to the user. The user selects a processing mode and generates a 3D model based on the notified information.

[0239] [Presentation of Other Information] In the above embodiment, the processing conditions for generating a 3D model are determined based on information about the importance of the 3D model to be generated. However, the imaging conditions, measurement conditions, etc. may be determined based on the importance information and presented to the user. For example, the number of images to be captured or the number of data to be measured may be determined based on the information about the importance of the 3D model to be generated and presented to the user. Furthermore, the overlap rate of adjacent images or the overlap rate of measurement ranges may be determined based on the information about the importance of the 3D model to be generated and presented to the user.

[0240] Other Embodiments The hardware for realizing the information processing system of the present invention may be configured with various processors. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and functions as various processing units; a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture; and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration specifically designed to execute specific processes. A processing unit constituting the damage diagram creation support device may be configured with one of the various processors described above, or may be configured with two or more processors of the same or different types. For example, a processing unit may be configured with multiple FPGAs or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be configured with a single processor. A first example of configuring multiple processing units with a single processor is a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units. Second, there is a form in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, various processing units are configured as a hardware structure using one or more of the above-mentioned various processors. Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0241] 1...3D model generation system 10...Photographing device 20...Turntable 100...3D model generation device 100A...Image acquisition unit 100B...3D model generation unit 100B1...First processing unit 100B2...Second processing unit 100B3...Synthesis processing unit 100C...Object information acquisition unit 100D...Generation method determination unit 100E...Notification unit 100F...Transparent area extraction unit 100G...Illumination information acquisition unit 100H...Photographing method determination unit 100J...Photographing form information acquisition unit 100K...Photographing condition determination unit 100L...Scanning speed information acquisition unit 100M...Importance information acquisition unit 100N...Processing mode determination unit 101...Processor 102...Main memory unit 103...Auxiliary memory unit 104...Operation unit 105...Display unit 106...Interface unit M...3D model O...Object S1 to S4...Processing procedures for determining a method for generating a 3D model S11 to S14...Processing procedures for generating a 3D model S21 to S25...Processing procedures for providing photographing support information S31 to S35...Processing procedures for providing photographing support information S41 to S43...Processing procedures for providing photographing support information S51 to S54...Processing procedures for generating a 3D model

Claims

1. An information processing system comprising at least one processor, which acquires first information including at least one of information relating to an object, information relating to the environment in which the object is photographed or measured, information relating to the conditions for photographing or measuring the object, and information relating to the importance of the photographing or measurement, and generates second information relating to the generation of three-dimensional data of the object based on the first information.

2. The information processing system according to claim 1, wherein the processor acquires the first information by accepting an input of the first information from a user.

3. The information processing system of claim 1, wherein the information about the object includes information about transparent areas, the information about the environment in which the object is photographed or measured includes information about lighting conditions, and the information about the conditions for photographing or measuring the object includes at least one of information about whether the photographing or measurement is performed by hand and information about the scanning speed.

4. The information processing system according to claim 1, wherein the second information is information on a method to be used for generating the three-dimensional data, and the processor determines the method to be used for generating the three-dimensional data based on the first information.

5. The information processing system according to claim 4, wherein the first information includes at least information of transparent areas of the object, and the processor determines a method to be used for generating the three-dimensional data based on the information of the transparent areas.

6. The information processing system of claim 5, wherein the first information includes, as the information on the transparent region, at least one of information on the presence or absence of the transparent region and information on the proportion of the object that the transparent region occupies, and the processor determines a method to be used to generate the three-dimensional data based on at least one of information on the presence or absence of the transparent region and information on the proportion of the object that the transparent region occupies.

7. The information processing system according to claim 6, wherein the processor determines whether to use photogrammetry or a volume intersection method as a technique to be used for generating the three-dimensional data.

8. The information processing system according to claim 5, wherein the processor determines a method to be used for generating the three-dimensional data for each region of the object based on the information on the transparent regions.

9. An information processing system described in any one of claims 1 to 8, wherein the second information is information on a shooting method for images used to generate the three-dimensional data, the first information includes at least information on lighting conditions, and the processor determines a shooting method for images to be used to generate the three-dimensional data based on the information on lighting conditions.

10. The information processing system according to claim 9, wherein the processor determines whether or not to employ polarized light photography as the photography technique.

11. The information processing system according to claim 9, wherein the information on the lighting conditions includes at least one of information on the presence or absence of specular reflection and information on the proportion of the area where the specular reflection occurs to the object.

12. The information processing system according to claim 9, wherein the information on the lighting condition includes information on whether the lighting condition causes specular reflection on the object.

13. The information processing system according to claim 9, wherein the information on the lighting state includes information on whether the lighting can be adjusted.

14. An information processing system according to any one of claims 1 to 8, wherein the first information includes at least information on whether the photograph was taken by hand-holding, and the processor determines photographing conditions for the object based on the information on whether the photograph was taken by hand-holding, and generates the second information.

15. The information processing system according to claim 14, wherein the processor determines, as the photographing condition, a shutter speed setting for photographing the object.

16. An information processing system according to any one of claims 1 to 8, wherein the second information is information on the shooting conditions of the object, the first information includes at least information on a scanning speed, and the processor determines the shooting conditions based on the information on the scanning speed.

17. The information processing system according to claim 16, wherein the processor determines, as the photographing condition, a shutter speed setting for photographing the object.

18. An information processing system described in any one of claims 1 to 8, wherein the second information is information on processing conditions when generating the three-dimensional data, the first information includes at least information on the importance of photographing or measurement, and the processor determines the processing conditions based on the information on the importance of photographing or measurement.

19. The information processing system according to claim 18, wherein the processor determines, as the processing condition, a processing mode for generating the three-dimensional data.

20. An information processing method comprising: a step of acquiring first information including at least one of information on an object, information on the environment in which the object is photographed or measured, information on the conditions for photographing or measuring the object, and information on the importance of the photographing or measurement; and a step of generating second information regarding the generation of three-dimensional data of the object based on the first information.

21. An information processing program that causes a computer to realize a function of acquiring first information including at least one of information on an object, information on the environment in which the object is photographed or measured, information on the conditions for photographing or measuring the object, and information on the importance of photographing or measuring, and a function of generating second information regarding the generation of three-dimensional data of the object based on the first information.

22. A non-transitory computer-readable recording medium having the program according to claim 21 recorded thereon.

Images