Information processing system, method, and program

Abstract

Provided are an information processing system, method, and program with which it is easy to generate a three-dimensional model of an actually existing object. The information processing system includes at least one processor. The processor performs: a process for acquiring first information including at least one of information relating to an object, information relating to an environment in which the object is imaged or measured, information relating to conditions in which the object is imaged or measured, and information relating to the degree of importance of imaging or measurement; and generating second information relating to the generation of the three-dimensional data of the object on the basis of the first information.

Description

Technical Field

[0001] This invention relates to an information processing system, method, and program, and more particularly to an information processing system, method, and program for processing information related to the generation of three-dimensional models. Background Technology

[0002] Patent Document 1 describes a technique for generating three-dimensional images of physical objects, and describes a method for determining surface attribute data applicable to a three-dimensional model based on shape data obtained from the physical object.

[0003] Patent document 2 describes a technique for measuring the position and posture of a measurement object, and describes a method for switching measurement modes based on the shape of a model data simulating the measurement object.

[0004] Previous technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2003-168129

[0007] Patent Document 2: Japanese Patent Application Publication No. 2012-021958 Summary of the Invention

[0008] One embodiment of the present invention provides an information processing system, method, and program that can easily generate three-dimensional models of real-world objects.

[0009] -Means used to solve technical problems-

[0010] (1) An information processing system comprising at least one processor, the processor performing the following processing: acquiring first information including at least one of information related to an object, information related to the environment of photographing or measuring the object, information related to the conditions of photographing or measuring the object, and information on the importance of photographing or measuring; and generating second information related to the generation of three-dimensional data of the object based on the first information.

[0011] (2) The information processing system according to (1), wherein,

[0012] The processor obtains the first information from the user's input of the first information.

[0013] (3) The information processing system according to (1) or (2), wherein,

[0014] Information related to the object includes information about the transparent area; information related to the environment in which the object is being photographed or measured includes information about the lighting conditions; and information related to the conditions in which the object is being photographed or measured includes at least one of whether the photograph or measurement is handheld and the scanning speed.

[0015] (4) The information processing system according to any one of (1) to (3), wherein,

[0016] The second piece of information is the method used in the generation of the 3D data. The processor determines the method used in the generation of the 3D data based on the first piece of information.

[0017] (5) The information processing system according to (4), wherein,

[0018] The first piece of information includes at least information about the transparent areas of the object, and the processor determines the method used to generate the 3D data based on the information about the transparent areas.

[0019] (6) The information processing system according to (5), wherein,

[0020] The first piece of information includes at least one of the information on whether there is a transparent area and the information on the proportion of the transparent area in the object. The processor determines the method used to generate the 3D data based on at least one of the information on whether there is a transparent area and the information on the proportion of the transparent area in the object.

[0021] (7) The information processing system according to (6), wherein,

[0022] The processor decides whether to use photogrammetry or visual hull cross-validation as the method for generating 3D data.

[0023] (8) The information processing system according to any one of (5) to (7), wherein,

[0024] The processor determines the method used to generate 3D data for each region of the object based on information from the transparent areas.

[0025] (9) The information processing system according to any one of (1) to (8), wherein,

[0026] The second piece of information is the photographic method used to generate the images in the 3D data generation. The first piece of information includes at least the lighting state information. The processor determines the photographic method used to generate the images in the 3D data generation based on the lighting state information.

[0027] (10) The information processing system according to (9), wherein,

[0028] The processor determines whether to use polarized photography as the photographic method.

[0029] (11) The information processing system according to (9) or (10), wherein,

[0030] The lighting status information includes at least one of the following: whether specular reflection occurs or not, and the proportion of the area in the object where specular reflection occurs.

[0031] (12) The information processing system according to any one of (9) to (11), wherein,

[0032] The lighting status information includes whether the lighting is designed to produce specular reflections on objects.

[0033] (13) The information processing system according to any one of (9) to (12), wherein,

[0034] The lighting status information includes whether the lighting can be adjusted.

[0035] (14) The information processing system according to any one of (1) to (13), wherein,

[0036] The first piece of information includes at least whether the shot was taken handheld. The processor determines the photographic conditions of the object based on whether the shot was taken handheld and generates the second piece of information.

[0037] (15) The information processing system according to (14), wherein,

[0038] The processor determines the shutter speed setting when photographing a subject, which serves as the photographic condition.

[0039] (16) The information processing system according to any one of (1) to (15), wherein,

[0040] The second piece of information is the photographic conditions of the object. The first piece of information includes at least the scanning speed information. The processor determines the photographic conditions based on the scanning speed information.

[0041] (17) The information processing system according to (16), wherein,

[0042] The processor determines the shutter speed setting when photographing a subject, which serves as the photographic condition.

[0043] (18) The information processing system according to any one of (1) to (17), wherein,

[0044] The second piece of information is the processing conditions when generating 3D data. The first piece of information includes at least the importance of the image or measurement. The processor determines the processing conditions based on the importance of the image or measurement.

[0045] (19) The information processing system according to (18), wherein,

[0046] The processor determines the processing mode for generating 3D data as the processing condition.

[0047] (20) An information processing method comprising the steps of: acquiring first information containing at least one of information related to an object, information related to the environment in which the object is photographed or measured, information related to the conditions in which the object is photographed or measured, and information on the importance of the photographing or measurement; and generating second information related to the generation of three-dimensional data of the object based on the first information.

[0048] (21) An information processing program that enables a computer to perform the following functions: acquiring first information containing at least one of information related to an object, information related to the environment in which the object is photographed or measured, information related to the conditions in which the object is photographed or measured, and information on the importance of the photograph or measurement; and generating second information related to the generation of three-dimensional data of the object based on the first information. Attached Figure Description

[0049] Figure 1 This is a diagram showing the general structure of a 3D model generation system.

[0050] Figure 2 This is a block diagram illustrating an example of the hardware structure of a 3D model generation device.

[0051] Figure 3 This is a block diagram of the main functions of a 3D model generation device.

[0052] Figure 4 This is a flowchart illustrating the order of processes that determine the method for generating a 3D model.

[0053] Figure 5 This is a block diagram of the main functions of a 3D model generation device.

[0054] Figure 6 This is a flowchart showing the sequence of processes involved in generating a 3D model.

[0055] Figure 7 This is a diagram showing the general structure of a 3D model generation system.

[0056] Figure 8 This is a block diagram of the main functions of a 3D model generation device.

[0057] Figure 9 This is a flowchart illustrating the sequence of processes that provide photographic assistance information.

[0058] Figure 10 This is a diagram showing the general structure of a 3D model generation system.

[0059] Figure 11 This is a block diagram of the main functions of a 3D model generation device.

[0060] Figure 12 This is a flowchart illustrating the sequence of processes that provide photographic assistance information.

[0061] Figure 13 This is a diagram showing the general structure of a 3D model generation system.

[0062] Figure 14 This is a block diagram of the main functions of a 3D model generation device.

[0063] Figure 15 This is a flowchart illustrating the sequence of processes that provide photographic assistance information.

[0064] Figure 16 This is a diagram showing the general structure of a 3D model generation system.

[0065] Figure 17 This is a block diagram of the main functions of a 3D model generation device.

[0066] Figure 18 This is a flowchart illustrating the sequence of processes for generating a 3D model from multi-view images. Detailed Implementation

[0067] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0068] [First Implementation]

[0069] In recent years, the application of three-dimensional content (content that can be viewed in three dimensions) has been advancing in fields such as games or metaverse-type communication spaces, architecture, and video production.

[0070] In part of the creation of 3D content, a method is used to scan an actual object in three dimensions using a camera or special equipment and then digitize the shape of that object into 3D data (so-called 3D scanning).

[0071] Various methods exist for 3D scanning, including photogrammetry, apparent volume cross-referencing, LiDAR (Light Detection and Ranging), Time of Flight (ToF), and structured light. However, due to their respective advantages and disadvantages, it is difficult to select the optimal method.

[0072] In this embodiment, a system (3D model generation system) is proposed that enables even users lacking knowledge about 3D scanning to generate high-quality 3D models (3D depicted shape models).

[0073] [System Architecture]

[0074] Figure 1 This is a diagram showing the general structure of a 3D model generation system.

[0075] The three-dimensional model generation system 1 of this embodiment is configured to generate a three-dimensional model M of an actual object O. In particular, the three-dimensional model generation system 1 of this embodiment is configured to: photograph the object O from multiple viewpoints (multi-viewpoints) and generate a three-dimensional model M based on the obtained two-dimensional image set (multi-viewpoint images).

[0076] In the generation of the 3D model M, photogrammetry or the apparent volume cross method is used. Both are techniques for generating 3D models of objects based on multi-viewpoint images (3D shape restoration techniques).

[0077] In photogrammetry, the position and pose of the camera are estimated from the image using SfM (Structure from Motion), and a 3D model is generated using MVS (Multi-View Stereo). The image can be taken using a typical digital camera.

[0078] The apparent volume intersection method, also known as the contour method, is a method for generating a 3D model based on the object's contour. In the apparent volume intersection method, the object's contour, captured from multiple viewpoints, is used to create a cone (apparent volume) with each viewpoint as a vertex and the contour as a cross section. The contour is then back-projected into 3D space, and the intersecting portions (common portions) are determined, thus generating the 3D model. Similar to photogrammetry, images can be taken with a typical digital camera, and the contour can be extracted through background subtraction and other methods.

[0079] Photogrammetry matches feature points between images and uses the processing results to generate a 3D model. Therefore, objects for which a sufficient number of feature points cannot be obtained from images, such as transparent or translucent objects, are not suitable for photogrammetry.

[0080] On the other hand, since the apparent volume cross method is a method for generating 3D models using contours, it can generate a 3D model even if the object has transparent or semi-transparent parts, as long as the contour can be obtained. However, the accuracy of the reconstructed 3D shape is worse than that of photogrammetry.

[0081] The 3D model generation system 1 of this embodiment obtains information related to the object from the user and automatically switches the 3D model generation method to generate a 3D model of the object based on the obtained information. A 3D model is an example of 3D data.

[0082] like Figure 1As shown, the 3D model generation system 1 of this embodiment includes a photography device 10 and a 3D model generation device 100. The 3D model generation system 1 of this embodiment is an example of an information processing system.

[0083] [Photographic installation]

[0084] The photographic device 10 is a device for photographing the object O. The photographic device 10 is composed of a conventional digital camera. That is, it is a camera that uses an imaging element (e.g., a CMOS image sensor (CMOS: Complementary Metal-Oxide-Semiconductor)) to receive light transmitted through a lens, convert it into a digital signal, and record it in memory. Digital cameras also include those assembled into other devices. For example, they include digital cameras assembled into smartphones, tablet terminals, etc.

[0085] Alternatively, the camera can be configured to be photographed using a single camera device 10, or it can be configured to be photographed using multiple camera devices 10. Furthermore, it can be configured to capture images in the form of moving images rather than still images.

[0086] [3D Model Generation Device]

[0087] Figure 2 This is a block diagram illustrating an example of the hardware structure of a 3D model generation device.

[0088] The 3D model generation device 100 is composed of a computer such as a PC (Personal Computer) and includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, an operation unit 104, a display unit 105, and an interface unit 106.

[0089] The processor 101 executes programs and functions as various processing units. As an example, in this embodiment, the processor 101 is composed of a CPU (Central Processing Unit). Various programs and data executed by the processor 101 are stored in at least one of the main storage unit 102 and the auxiliary storage unit 103. The terms "program" and "software" have the same meaning.

[0090] The main storage unit 102 includes RAM (Random Access Memory) and ROM (Read Only Memory). The RAM is used as the working area of ​​the processor 101. The ROM stores basic input / output programs, etc.

[0091] The auxiliary storage unit 103 may be composed of, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0092] The operation unit 104 may consist of, for example, a keyboard, mouse, touch panel, etc.

[0093] The display unit 105 may be composed of, for example, a liquid crystal display (LCD) or an organic EL display (OLED display).

[0094] The interface unit 106 includes an interface for connecting to external devices such as the camera device 10 and an interface for connecting the 3D model generation device 100 to a network such as the Internet.

[0095] Figure 3 This is a block diagram of the main functions of a 3D model generation device.

[0096] like Figure 3 As shown, the 3D model generation apparatus 100 includes functions such as an image acquisition unit 100A, a 3D model generation unit 100B, an object information acquisition unit 100C, a generation method determination unit 100D, and a notification unit 100E. The functions of each unit are implemented by the processor 101 executing a predetermined program.

[0097] The image acquisition unit 100A acquires multi-view images of an object. A multi-view image is an image of the object captured from multiple viewpoints. For example, the multi-view image is stored in the auxiliary storage unit 103, and then read from and retrieved from the auxiliary storage unit 103. Alternatively, it can be configured to acquire the image directly from the photographic device 10.

[0098] The 3D model generation unit 100B generates a 3D model based on multi-viewpoint images. The 3D model generation unit 100B includes a first processing unit 100B1 and a second processing unit 100B2. The first processing unit 100B1 generates the 3D model using photogrammetry. The second processing unit 100B2 generates the 3D model using the view volume cross method. The generation of 3D models based on photogrammetry and the view volume cross method are well-known techniques; therefore, detailed descriptions are omitted.

[0099] The 3D model generation unit 100B generates a 3D model using the generation method determined by the generation method determination unit 100D. Therefore, when the generation method determination unit 100D determines that the 3D model will be generated using photogrammetry, the 3D model generation unit 100B generates the 3D model using photogrammetry. In this case, the 3D model generation unit 100B generates the 3D model by processing multi-view images using the first processing unit 100B1. On the other hand, when the generation method determination unit 100D determines that the 3D model will be generated using the view volume cross method, the 3D model generation unit 100B generates the 3D model using the view volume cross method. In this case, the 3D model generation unit 100B generates the 3D model by processing multi-view images using the second processing unit 100B2.

[0100] The object information acquisition unit 100C acquires information (object information) about the object O used to generate the 3D model M. For example, in this embodiment, information about whether or not the object has a transparent area is acquired as object information. The object information acquisition unit 100C acquires this information, for example, by displaying a predetermined input screen on the display unit 105 and receiving input from the user regarding the presence or absence of a transparent area. For example, a screen displaying a selection of "yes" or "no" regarding the transparent area is displayed on the display unit 105, and the user receives a selection of "yes" or "no" to acquire the information regarding the presence or absence of a transparent area. The selection operation is performed via the operation unit 104. In this embodiment, the information regarding the presence or absence of a transparent area is an example of information about the transparent area of ​​the object. Furthermore, the object information is an example of first information.

[0101] The generation method determination unit 100D determines the generation method of the 3D model based on the object information acquired by the object information acquisition unit 100C. More specifically, it determines whether to use photogrammetry or the apparent volume cross method as the generation method of the 3D model. As described above, in this embodiment, information on the presence or absence of transparent areas is acquired as object information. Therefore, the generation method determination unit 100D determines the generation method of the 3D model based on the presence or absence of transparent areas. Specifically, the generation method of the 3D model is determined according to the following criteria: If the object has transparent areas, the generation method determination unit 100D determines to generate the 3D model using the apparent volume cross method. On the other hand, if there are no transparent areas, the generation method determination unit 100D determines to generate the 3D model using photogrammetry. In this embodiment, the information on the generation method of the 3D model determined by the generation method determination unit 100D is an example of the second information.

[0102] The information on the generation method of the 3D model, determined by the generation method determination unit 100D, is provided to the 3D model generation unit 100B. The 3D model generation unit 100B uses the generation method determined by the generation method determination unit 100D to generate a 3D model based on the multi-view images.

[0103] Furthermore, information regarding the generation method of the 3D model, determined by the generation method determination unit 100D, is provided to the notification unit 100E. The notification unit 100E then notifies the user of the determined 3D model generation method. As an example, in this embodiment, the user is notified by displaying the determined 3D model generation method information on the display unit 105.

[0104] [Generation of 3D Models]

[0105] Next, the method for generating a three-dimensional model M of an actual object O using the three-dimensional model generation system 1 of this embodiment will be described.

[0106] (1) Determination of the method for generating 3D models

[0107] First, determine the method for generating the 3D model from the multi-viewpoint images. That is, decide whether to use photogrammetry or the cross-view volume method to generate the 3D model.

[0108] Figure 4 This is a flowchart illustrating the order of processes that determine the method for generating a 3D model.

[0109] First, object information is obtained from the user (step S1). In this embodiment, object information refers to whether an object has transparent areas. The 3D model generation apparatus 100 obtains object information by receiving input from the user regarding the presence or absence of transparent areas. The user confirms whether the object in the generated 3D model has transparent areas and inputs the confirmation result into the 3D model generation apparatus 100. That is, if there are transparent areas in the object, the information "there are transparent areas" is input into the 3D model generation apparatus 100. On the other hand, if there are no transparent areas in the object, the information "there are no transparent areas" is input into the 3D model generation apparatus 100.

[0110] The 3D model generation apparatus 100 determines the method for generating the 3D model based on the acquired object information (information on whether there are transparent areas). First, it determines whether there are transparent areas (step S2). If there are transparent areas, the 3D model generation apparatus 100 decides to generate the 3D model using the apparent volume cross method (step S3). On the other hand, if there are no transparent areas, the 3D model generation apparatus 100 decides to generate the 3D model using photogrammetry (step S4).

[0111] Thus, the 3D model generation device 100 determines the 3D model generation method based on the object information (information on whether there are transparent areas) obtained through user input.

[0112] The user is notified of the determined method for generating the 3D model. In this embodiment, the information about the determined method for generating the 3D model is displayed on the display unit 105. The user confirms the method for generating the 3D model by checking the display unit 105.

[0113] (2) Generation of three-dimensional models

[0114] In the three-dimensional model generation system 1 of this embodiment, a three-dimensional model is generated based on multi-viewpoint images.

[0115] First, the object O is photographed from multiple viewpoints using the photographic device 10. As described above, photography can be performed using one photographic device 10 or multiple photographic devices 10.

[0116] Multi-view images obtained through photography are input into the 3D model generation apparatus 100. The 3D model generation apparatus 100 processes the input multi-view images to generate a 3D model. At this time, the 3D model is generated using a predetermined generation method. That is, if it is decided to generate the model using photogrammetry, the 3D model is generated using photogrammetry. On the other hand, if it is decided to generate the model using the view volume cross method, the 3D model is generated using the view volume cross method.

[0117] As explained above, the 3D model generation system 1 according to this embodiment obtains information related to the object from the user and automatically switches the 3D model generation method to generate a 3D model based on the obtained information. Therefore, even users lacking knowledge of 3D scanning can easily generate high-quality 3D models.

[0118] [Variation Example]

[0119] [Information about the transparent area]

[0120] In the above embodiments, the structure is set to obtain information about the presence or absence of transparent areas as information about the transparent areas of an object. However, it can also be set to obtain information about the proportion of transparent areas in the object, or other alternative methods. This proportion of transparent areas does not necessarily need to be an exact value; it can be set to an approximate value. For example, it can be set to an input value obtained by visual inspection. When obtaining information about the proportion of transparent areas as the transparent area information, a threshold is set, and the method for generating the 3D model is determined by comparing it with the threshold. For example, the structure could be set as follows: if the proportion of transparent areas is below the threshold, photogrammetry is selected; if it exceeds the threshold, the apparent volume cross method is selected.

[0121] Alternatively, the structure could be set as follows: the user subjectively judges the amount of transparent area, and this judgment is used as information about the transparent area. In this case, the method for generating the 3D model is determined based on the proportion of transparent area. Specifically, if the input information indicates a small proportion of transparent area, photogrammetry is selected; if the input information indicates a large proportion of transparent area, the apparent volume cross method is selected.

[0122] [Object Information]

[0123] In the above embodiments, the structure is designed to acquire information about transparent areas (especially whether or not there are transparent areas) as information related to the object (object information), but the information acquired as information related to the object is not limited to this. It may also be designed to acquire other information besides the transparent area information, or information that replaces the transparent area information.

[0124] For example, when generating a 3D model using photogrammetry or apparent volume cross-referencing, information such as the object's texture and gloss can be obtained to determine the method to be used. When determining the method based on texture information, the method can be chosen based on the presence or absence of texture or its intensity. Specifically, apparent volume cross-referencing is chosen when there is no texture or a weak texture; otherwise, photogrammetry is chosen. Similarly, when determining the method based on gloss information, the method is chosen based on the presence or absence of gloss or its intensity. Specifically, apparent volume cross-referencing is chosen when there is gloss or a pronounced gloss; otherwise, photogrammetry is chosen.

[0125] [Methods for obtaining object information]

[0126] In the above embodiments, a structure is provided where the user manually inputs information related to the object (object information), but it can also be provided where it is automatically acquired. For example, the structure can be as follows: when information about the presence or absence of transparent areas is acquired as object information, the presence or absence of transparent areas is determined based on an image of the object. The image of the object can be obtained using a multi-view image (the presence or absence of transparent areas is determined using at least one of the multiple images constituting the multi-view image). Furthermore, known image recognition methods can be used in the detection of transparent areas. For example, a learned machine learning model that has been machine learned in a way that identifies transparent areas from an image can be used to detect transparent areas from the image.

[0127] Furthermore, it can also be structured as follows: as object information, information about the presence or absence of transparent areas is obtained from the user's input of the object's name (item name). In this case, for example, a table is prepared that associates the object's name with information about the presence or absence of transparent areas, and the reference table obtains information about the presence or absence of transparent areas based on the object's name.

[0128] Methods for generating 3D models

[0129] In the above embodiments, the example described is the generation of a 3D model of an object from multi-viewpoint images using photogrammetry or the view volume cross method; however, the method for generating a 3D model of an object is not limited to this. Furthermore, the structure can be configured to allow selection of multiple methods. It can also be configured to allow selection of other 3D scanning methods. For example, it can be configured to generate a 3D model using LiDAR, ToF, or structured light methods.

[0130] [Manual Selection]

[0131] In the above embodiment, the method used in generating the 3D model is automatically selected based on the determination, but it can also be configured for manual selection. In the above embodiment, information about the 3D model generation method determined by the generation method determination unit 100D is notified to the user via the notification unit 100E. The user refers to the notified information to determine the 3D model generation method. In this case, the 3D model generation apparatus 100 functions as an auxiliary device for generating the 3D model.

[0132] [Second Implementation]

[0133] In this embodiment, when the object has transparent areas, the method for generating the 3D model of the object is changed for each area to generate the 3D model. Here, the case of generating the 3D model of the object based on multi-viewpoint images will be described as an example. In particular, the case of generating the 3D model using photogrammetry and the apparent volume cross method will be described as an example.

[0134] As mentioned above, photogrammetry is not suitable for transparent or translucent objects. On the other hand, even if an object has transparent or translucent parts, the apparent volume cross method can generate a 3D model as long as the outline can be obtained.

[0135] In this embodiment, when the object contains transparent areas (including semi-transparent areas), the shape of the transparent areas is digitized into three dimensions using the apparent volume cross method, and the shape of the non-transparent areas is digitized into three dimensions using photogrammetry.

[0136] Furthermore, the hardware structure of the apparatus used to generate the 3D model (3D model generation apparatus) is the same as that of the 3D model generation apparatus 100 in the first embodiment described above. Therefore, only the functions of the 3D model generation apparatus 100 will be described here.

[0137] Figure 5 This is a block diagram of the main functions of a 3D model generation device.

[0138] like Figure 5 As shown, the 3D model generation apparatus 100 has the functions of an image acquisition unit 100A, a 3D model generation unit 100B, a transparent area extraction unit 100F, and a generation method determination unit 100D. The functions of each unit are implemented by the processor 101 executing a predetermined program.

[0139] The image acquisition unit 100A acquires multi-view images of the object.

[0140] The 3D model generation unit 100B generates a 3D model based on multi-viewpoint images. The 3D model generation unit 100B has the functions of a first processing unit 100B1, a second processing unit 100B2, and a compositing processing unit 100B3. The first processing unit 100B1 generates the 3D model based on multi-viewpoint images through photogrammetry. The second processing unit 100B2 generates the 3D model based on multi-viewpoint images using the view volume cross method. The compositing processing unit 100B3 combines the 3D models generated for each region to generate a single 3D model constituting the whole.

[0141] The 3D model generation unit 100B generates a 3D model for each region of the object according to the generation method (3D shape restoration method) determined by the generation method determination unit 100D for each region.

[0142] The transparent region extraction unit 100F extracts the transparent regions (including semi-transparent regions) of an object and determines their positions and extents. The transparent region extraction unit 100F extracts the transparent regions of the object from a multi-view image and determines their positions and extents. Known image recognition methods can be used in the extraction of transparent regions. The information (position and extent) of the transparent regions is provided to the generation method determination unit 100D.

[0143] The generation method determination unit 100D determines the generation method for the 3D model for each region. Specifically, the apparent volume cross method is selected for transparent areas, and photogrammetry is selected for non-transparent areas. Furthermore, when the entire area is transparent, the generation method is determined to be used for the entire area. And when the entire area is non-transparent, the generation method is determined to be used for the entire area.

[0144] Figure 6 This is a flowchart showing the sequence of processes involved in generating a 3D model.

[0145] First, a multi-view image of the object is acquired (step S11). The multi-view image is, for example, an image captured by the camera device 10 stored in the auxiliary storage unit 103, and then read from and retrieved from the auxiliary storage unit 103. The multi-view image can be captured by one camera device 10 or by multiple camera devices 10.

[0146] Next, the transparent areas of the objects are extracted based on the acquired multi-view images, and their positions and ranges are determined (step S12).

[0147] Next, based on the information of the transparent areas of the object, the method for generating the 3D model is determined for each area (step S13). Specifically, the apparent volume cross method is selected for transparent areas, and photogrammetry is selected for non-transparent areas.

[0148] Next, using the determined generation method, a 3D model of the object is generated based on the multi-viewpoint images (step S14). Here, if the object is entirely transparent, a 3D model of the whole is generated using the view volume cross method. If the object is entirely opaque, a 3D model of the whole is generated using photogrammetry. Furthermore, if a part of the object contains a transparent area, the 3D model of the transparent area is generated using the view volume cross method, while the 3D model of the opaque area is generated using photogrammetry. Then, the data from each area are combined to generate a single 3D model.

[0149] Thus, according to this embodiment, by changing the generation method of the 3D model according to each region, a high-quality 3D model can be easily generated.

[0150] Furthermore, in the above implementation, the case of automatically generating a 3D model was used as an example, but it can also be set up to present a 3D model suitable for each region to the user.

[0151] Furthermore, while the above embodiment uses the example of generating a three-dimensional model from a two-dimensional image, the three-dimensional scanning method used is not limited to this. Other methods can also be used.

[0152] [Third Implementation]

[0153] When generating a 3D model based on an image of an object, if the object reflected in the image exhibits specular reflection (gloss), there is a problem that the original state of the object (the state excluding the influence of light) cannot be reproduced.

[0154] One method to suppress specular reflection is to use polarizing filters during photography. However, this method is less practical than regular photography, as it requires setting the polarizing filter to the appropriate angle for each shot.

[0155] In this embodiment, a system is proposed that enables even users lacking knowledge of photography to capture high-quality images suitable for the generation of 3D models.

[0156] [3D Model Generation System]

[0157] Figure 7 This is a diagram showing the general structure of a 3D model generation system.

[0158] The three-dimensional model generation system 1 of this embodiment is configured as a system for generating a three-dimensional model M of an object O based on multi-viewpoint images.

[0159] like Figure 7 As shown, the 3D model generation system 1 includes a photography device 10 and a 3D model generation device 100.

[0160] [Photographic installation]

[0161] The photographic device 10 is a device for photographing the object O. The photographic device 10 is composed of a conventional digital camera. Alternatively, it can be configured such that photography is performed by a single photographic device 10, or it can be configured such that multiple photographic devices 10 are used.

[0162] [3D Model Generation Device]

[0163] The hardware structure of the 3D model generation device 100 is the same as that of the 3D model generation device 100 in the first embodiment described above. That is, it is composed of a computer such as a PC, and includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, an operation unit 104, a display unit 105, and an interface unit 106, etc. (see reference). Figure 2 ).

[0164] Figure 8 This is a block diagram of the main functions of a 3D model generation device.

[0165] The 3D model generation device 100 has the function of generating a 3D model of an object based on multi-viewpoint images (3D model generation function) and the function of assisting in the photography of multi-viewpoint images (photography assistance function).

[0166] [3D Model Generation Function]

[0167] The 3D model generation function is the function of generating a 3D model of an object based on multi-viewpoint images. The 3D model generation device 100 has the functions of an image acquisition unit 100A and a 3D model generation unit 100B, etc., as the 3D model generation function.

[0168] The image acquisition unit 100A acquires multi-view images of the object. For example, the multi-view images are stored in the auxiliary storage unit 103, and then read from and acquired from the auxiliary storage unit 103. Alternatively, it can be configured to acquire the images directly from the photographic device 10.

[0169] The 3D model generation unit 100B generates a 3D model based on multi-viewpoint images. As an example, in this embodiment, a 3D model of an object is generated by photogrammetry based on multi-viewpoint images.

[0170] [Photography Assistance Functions]

[0171] In this embodiment, as a photography assistance function, a feature is provided to prompt the user with information (photography assistance information) that assists in capturing multi-view images. Specifically, the optimal photography method (recommended photography method) is determined based on the environment of the subject, especially the lighting conditions, and then prompted to the user. Figure 7 As shown, the 3D model generation device 100 has functions such as a lighting information acquisition unit 100G, a photography method determination unit 100H, and a notification unit 100E as a photography assistance function.

[0172] The lighting information acquisition unit 100G acquires lighting status information (lighting information). The lighting information acquisition unit 100G acquires lighting status information from the user's input of lighting status information. In this embodiment, as lighting information, information on whether specular reflection occurs is acquired. That is, information on whether a lighting state produces specular reflection is acquired as lighting information. The lighting information acquisition unit 100G receives input from the user regarding whether specular reflection occurs, for example, by displaying a predetermined input screen on the display unit 105. For example, a screen displaying a "yes" or "no" selection regarding specular reflection is displayed on the display unit 105, and the user receives a "yes" or "no" selection to acquire information on whether specular reflection occurs. The selection operation is performed via the operation unit 104. In this embodiment, the information on whether specular reflection occurs is an example of lighting status information. Furthermore, the lighting status information is an example of information related to the environment of the object being photographed (or measured), and is an example of first information.

[0173] The photography method determination unit 100H determines the optimal photography method (recommended photography method) based on the lighting conditions information obtained from the user (in this embodiment, information on the presence or absence of specular reflection). Here, photography using a polarizing filter is referred to as "polarized photography," and photography without a polarizing filter is referred to as "normal photography," and a decision is made on which one to use (determining whether to use polarized photography). Specifically, in the case of "specular reflection," "polarized photography" is selected. On the other hand, in the case of "no specular reflection," "normal photography" is selected. In this embodiment, the photography method information determined by the photography method determination unit 100H (information on the photography method of the image used in the generation of 3D data) is an example of the second information.

[0174] The notification unit 100E notifies the user of the photography method information determined by the photography method determination unit 100H as photography assistance information. As an example, in this embodiment, the user is notified by displaying the determined photography method information (photography assistance information) on the display unit 105.

[0175] [Generation of 3D Models]

[0176] Next, the method for generating a three-dimensional model M of an object O based on multi-view images using the three-dimensional model generation system 1 of this embodiment will be described.

[0177] (1) Provision of photographic auxiliary information

[0178] First, the 3D model generation device 100 receives photographic aid information.

[0179] Figure 9 This is a flowchart illustrating the sequence of processes that provide photographic aid information in a 3D model generation apparatus.

[0180] First, lighting status information is obtained from the user (step S21). In this embodiment, the lighting status information is obtained from the user's input of lighting status information. The lighting status information includes whether or not there is specular reflection.

[0181] The 3D model generation apparatus 100 determines the photography method based on the acquired lighting state information (information on the presence or absence of specular reflection). First, it determines whether specular reflection occurs (step S22). If specular reflection occurs, the 3D model generation apparatus 100 selects polarized photography as the photography method (step S23). On the other hand, if there is no specular reflection, the 3D model generation apparatus 100 selects normal photography as the photography method (step S24).

[0182] The 3D model generation apparatus 100 notifies the user of the determined photographic method information as photographic assistance information (step S25). In this embodiment, the user is notified by displaying the determined photographic method information (photographic assistance information) on the display unit 105. The photographic assistance information is provided to the user as information on the optimal photographic method based on the lighting conditions.

[0183] (2) Generation of three-dimensional models

[0184] The user receives prompts from photographic assistance information to determine the photographic method and captures multi-viewpoint images of the subject. For example, if the prompt indicates that normal photography is the best method, the user photographs the subject using the usual method (without using a polarizing filter). On the other hand, if the prompt indicates that polarized photography is the best method, the user photographs the subject using a polarizing filter. Specifically, a polarizing filter is mounted on the lens and set to an appropriate angle (an angle that suppresses specular reflections) to photograph the subject.

[0185] Multi-view images obtained through photography are input into the 3D model generation apparatus 100. The 3D model generation apparatus 100 processes the input multi-view images to generate a 3D model.

[0186] As explained above, according to this embodiment, specular reflection can be suppressed when photographing the object. This allows for the generation of high-quality 3D models. Furthermore, by selecting an appropriate photographic method based on the lighting conditions, a balance between the quality of the generated 3D model and operational efficiency can be optimized.

[0187] [Variation Example]

[0188] [Lighting status information]

[0189] In the above embodiment, the structure is set to obtain information on the presence or absence of specular reflection as information on the lighting state. However, it can also be set to obtain information on the proportion of the specular reflection-producing area in the object, or other than this. The proportion does not necessarily need to be an exact value; it can also be set to an approximate value. For example, it can be set to an input value obtained by visual inspection. When this information is obtained as information on the lighting state, a threshold is set, and the photographic method is determined by comparing it with the threshold. For example, the structure could be set as follows: if the proportion of the specular reflection-producing area in the object is below the threshold, normal photography is selected; if it exceeds the threshold, polarized photography is selected.

[0190] Furthermore, it can be configured to acquire information about the degree (intensity) of specular reflection as information about the lighting state. It can also be configured to input information about the degree of specular reflection into information for visual confirmation by the user. In this case, for example, polarized photography is selected in the case of strong specular reflection, and normal photography is selected in the case of weak specular reflection.

[0191] Alternatively, the structure could be set as follows: the user subjectively judges the amount of specular reflection generated, and this judgment is used as information about the lighting state. In this case, the photographic method is determined based on the proportion of specular reflection generated. Specifically, if the proportion is low, normal photography is selected; if the proportion is high, polarized photography is selected.

[0192] Furthermore, the structure can be configured to acquire information about the lighting equipment's settings as lighting state information. For example, information such as the number of lighting equipment, whether a diffuser is used, and the brightness setting can be acquired as lighting state information. By acquiring this information, it is possible to estimate whether specular reflection occurs. Therefore, the optimal photographic method (recommended photographic method) can be estimated. For example, multiple lighting equipment can be used to uniformly illuminate the object to suppress specular reflection. Therefore, the optimal photographic method can be selected based on the number of lighting equipment. For example, when using multiple (above a threshold) lighting equipment, normal photography is selected as a lighting state where specular reflection is suppressed. Furthermore, specular reflection can also be suppressed using a diffuser. Therefore, the optimal photographic method can also be selected based on whether a diffuser is used. Furthermore, specular reflection can also be suppressed by setting the brightness. Therefore, the optimal photographic method can be selected by setting the brightness.

[0193] Furthermore, the structure can be set as follows: Information about the lighting environment, such as whether the lighting can be adjusted (information on whether the lighting can be adjusted), is acquired as information about the photographic environment, and the optimal photographic method (recommended photographic method) is determined based on this information. For example, in environments where the lighting cannot be adjusted (e.g., outdoors), polarized photography is preferred. Therefore, information about whether the lighting can be adjusted is acquired, and the photographic method is determined based on this information. Specifically, in environments where the lighting can be adjusted, normal photography is selected; in environments where the lighting cannot be adjusted, polarized photography is selected.

[0194] Furthermore, it can be structured as follows: acquire information about whether the lighting is sufficient to cause specular reflection on the object as lighting environment information, and determine the optimal photographic method (recommended photographic method) based on the acquired information. For example, in cases where the lighting causes specular reflection on the object, polarized photography is selected; otherwise, normal photography is selected.

[0195] Thus, any information about the lighting conditions or the environment of the subject being photographed is sufficient to determine whether specular reflection has occurred or to what extent.

[0196] [Methods for obtaining lighting status information]

[0197] In the above embodiments, the structure is designed for manual input of lighting status information by the user, but it can also be designed for automatic acquisition. For example, it can be designed as follows: based on an image of the object taken, it determines whether specular reflection occurs and / or its degree, and automatically acquires lighting status information. The image of the object taken can utilize multi-view images (using at least one of the multiple images constituting the multi-view images to determine whether specular reflection occurs).

[0198] Methods for generating 3D models

[0199] In the above embodiments, the example described is the generation of a three-dimensional model of an object by photogrammetry and based on multi-viewpoint images, but the same applies when generating a three-dimensional model using other methods (e.g., the view volume cross method).

[0200] Furthermore, it can also be applied to situations where 3D models are generated using other 3D scanning methods. For example, when generating a 3D model using LiDAR, the structure can be set as follows: acquire information related to the environment of the object being measured, determine the optimal measurement method based on the acquired information, and then prompt the user with this information.

[0201] [Photography Aid Information Tips]

[0202] In the above embodiment, the structure is configured to display photography assistance information (information on the optimal photography method) on the display unit 105 of the 3D model generation device 100 and notify the user, but the method of notifying the user of the photography assistance information is not limited to this. For example, the structure could also be configured to send the photography assistance information to the photography device 10 and display it on the display unit of the photography device 10.

[0203] Furthermore, in the above embodiment, the example described is that the 3D model generation apparatus 100 has a photography assistance function, but the photography assistance function can also be independently integrated into other devices. For example, the photography assistance function can be integrated into the photography apparatus 10. Alternatively, the photography assistance function can be integrated into mobile devices such as smartphones and tablets.

[0204] [Photography Aid]

[0205] In the case of polarization photography, information regarding the setting of the polarization filter can also be provided to the user. For example, when taking a picture while the camera is fixed in place and the object is rotated, the optimal setting angle of the polarization filter can be calculated and provided to the user.

[0206] [Fourth Implementation]

[0207] When generating a 3D model of an object based on images of the object, it is necessary to photograph the object from multiple viewpoints.

[0208] As a photographic method for multi-viewpoint images, there are two methods: shooting from the perspective of a moving object and shooting from the perspective of a moving photographic device.

[0209] In shooting moving objects, a turntable is typically used. In turntable shooting, the object is placed on the turntable and rotated, and then photographed from a fixed position using a camera.

[0210] On the other hand, in shooting with a mobile camera device, handheld shooting is the most common method. In handheld shooting, the photographer (user) holds the camera device in their hand and shoots the subject from various directions.

[0211] Handheld shooting has the advantage of being simple and easy to perform, but it also has the disadvantage of being prone to image shakiness due to hand tremors. Shaky images negatively impact the generation of 3D models.

[0212] On the other hand, the effects of hand shakiness can be suppressed by changing shooting conditions, such as increasing the shutter speed.

[0213] In this embodiment, a system is proposed that enables even users lacking knowledge of photography to capture high-quality images suitable for the generation of 3D models.

[0214] [3D Model Generation System]

[0215] Figure 10 This is a diagram showing the general structure of a 3D model generation system.

[0216] The three-dimensional model generation system 1 of this embodiment is configured as a system for generating a three-dimensional model M of an object O based on multi-viewpoint images.

[0217] like Figure 10 As shown, the 3D model generation system 1 includes a photography device 10 and a 3D model generation device 100.

[0218] [Photographic installation]

[0219] The photographic device 10 is a device for photographing the object O. The photographic device 10 is composed of a conventional digital camera.

[0220] [3D Model Generation Device]

[0221] The hardware structure of the 3D model generation device 100 is the same as that of the 3D model generation device 100 in the first embodiment described above. That is, it is composed of a computer such as a PC, and includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, an operation unit 104, a display unit 105, and an interface unit 106, etc. (see reference). Figure 2 ).

[0222] Figure 11 This is a block diagram of the main functions of a 3D model generation device.

[0223] The 3D model generation device 100 has the function of generating a 3D model of an object based on multi-viewpoint images (3D model generation function) and the function of assisting in the photography of multi-viewpoint images (photography assistance function).

[0224] [3D Model Generation Function]

[0225] The 3D model generation function is the same as that in the 3D model generation apparatus 100 of the third embodiment described above. Therefore, the description is omitted.

[0226] [Photography Assistance Functions]

[0227] In this embodiment, as a photography assistance function, information about the photography method is obtained from the user, the photography conditions are determined based on the obtained information, and then the user is prompted accordingly. For example... Figure 11 As shown, the 3D model generation device 100 has functions such as a photography mode information acquisition unit 100J, a photography condition determination unit 100K, and a notification unit 100E as a photography auxiliary function.

[0228] The photography mode information acquisition unit 100J acquires photography mode information (photography mode information). The photography mode information acquisition unit 100J acquires photography mode information from the user's input of photography mode information. In this embodiment, as photography mode information, it acquires information on whether handheld shooting is used. The photography mode information acquisition unit 100J, for example, displays a predetermined input screen on the display unit 105 to receive input from the user regarding whether handheld shooting is used. For example, it displays a screen on the display unit 105 asking the user to select "yes" or "no" regarding whether handheld shooting is used, and acquires information on whether handheld shooting is used based on the user's selection of "yes" or "no". The selection operation is performed via the operation unit 104. In this embodiment, the photography mode information (information on whether handheld shooting is used) acquired by the photography mode information acquisition unit 100J is an example of first information.

[0229] The photography condition determination unit 100K determines the photography conditions based on information about the photography method obtained from the user. In this embodiment, when shooting handheld, a high shutter speed is determined as the photography condition. "Shooting at a high shutter speed" means setting the shutter speed to a relatively high (fast) setting for shooting. On the other hand, when not shooting handheld, a low shutter speed is determined as the photography condition. "Shooting at a low shutter speed" means setting the shutter speed to a relatively low (slow) setting for shooting.

[0230] "High shutter speed" and "low shutter speed" are determined from the perspective of suppressing the effects of camera shake. That is, in handheld shooting, a shutter speed that can suppress the effects of camera shake is considered a "high shutter speed." Typically, a speed faster than 1 / focal length [second] is set to suppress the effects of camera shake. Therefore, "shooting at a high shutter speed" means shooting at a shutter speed faster than 1 / focal length [second]. On the other hand, "low shutter speed" means shooting at a shutter speed slower than 1 / focal length [second]. Furthermore, a shutter speed faster than 1 / 60 [second] can be set as a "high shutter speed," and a shutter speed slower than 1 / 60 [second] can be set as a "low shutter speed." Additionally, the shutter speed that can suppress the effects of camera shake varies depending on whether camera shake correction is present. Therefore, it is preferable to set a shutter speed based on whether camera shake correction is present as a reference for high or low shutter speed.

[0231] In this embodiment, the information on the photographic conditions determined by the photographic condition determination unit 100K is an example of the second information.

[0232] The notification unit 100E notifies the user of the photography conditions determined by the photography conditions determination unit 100K as photography assistance information. As an example, in this embodiment, the determined photography conditions information (photography assistance information) is displayed on the display unit 105 to notify the user.

[0233] [Generation of 3D Models]

[0234] Next, the method for generating a three-dimensional model M of an object O based on multi-view images using the three-dimensional model generation system 1 of this embodiment will be described.

[0235] (1) Provision of photographic auxiliary information

[0236] First, the 3D model generation device 100 receives photographic aid information.

[0237] Figure 12 This is a flowchart illustrating the sequence of processes that provide photographic aid information in a 3D model generation apparatus.

[0238] First, information about the shooting method is obtained from the user (step S31). In this embodiment, the shooting method information is obtained from the user's input of the shooting method information. The shooting method information is whether it is handheld shooting.

[0239] The 3D model generation device 100 determines the shooting conditions based on the acquired information about the shooting method (whether it is handheld shooting). First, it determines whether it is handheld shooting (step S32). If it is handheld shooting, the 3D model generation device 100 decides to shoot with a high shutter speed as the shooting condition (step S33). On the other hand, if it is not handheld shooting, the 3D model generation device 100 decides to shoot with a low shutter speed as the shooting condition (step S34).

[0240] The 3D model generation apparatus 100 notifies the user of the determined photographic conditions as photographic assistance information (step S35). In this embodiment, the determined photographic conditions (photographic assistance information) are displayed on the display unit 105 to notify the user. The photographic assistance information is provided to the user as setting information for optimal photography based on the photographic method (recommended photography setting information).

[0241] (2) Generation of three-dimensional models

[0242] The user receives prompts from photographic assistance information to set the exposure and captures multi-viewpoint images of the subject. Therefore, for example, in handheld shooting, a high shutter speed is set to capture the subject. This suppresses image blur and allows for the capture of images suitable for generating 3D models. On the other hand, in non-handheld shooting, a low shutter speed is set to capture the subject.

[0243] Multi-view images obtained through photography are input into the 3D model generation apparatus 100. The 3D model generation apparatus 100 processes the input multi-view images to generate a 3D model.

[0244] As explained above, according to this embodiment, even users lacking knowledge of photography can capture high-quality images suitable for generating 3D models. Therefore, high-quality 3D models can be easily generated.

[0245] [Variation Example]

[0246] [Photography conditions]

[0247] In the above embodiments, the structure is set to determine the shutter speed as a photographic condition, but the items determined as photographic conditions are not limited to this. For example, the shutter speed and sensitivity (ISO sensitivity) can also be determined according to the photographic method. In this case, for example, in handheld shooting, high sensitivity and high shutter speed are set, and in non-handheld shooting, low sensitivity and low shutter speed are set. Furthermore, the shutter speed, aperture value (F-stop), and sensitivity can also be determined according to the photographic method. In this case, for example, in handheld shooting, the aperture value is fixed, and high sensitivity and high shutter speed are set. And in non-handheld shooting, the aperture value is fixed, and low sensitivity and low shutter speed are set.

[0248] [Photography Aid Information Tips]

[0249] In the above embodiment, the structure is configured to notify the user by displaying photography assistance information (information on photography conditions) on the display unit 105 of the 3D model generation apparatus 100, but the method of notifying the user of the photography assistance information is not limited to this. For example, the structure could also be configured to send the photography assistance information to the photography apparatus 10 and display it on the display unit of the photography apparatus 10.

[0250] Furthermore, in the above embodiment, the example described is that the 3D model generation apparatus 100 has a photography assistance function, but the photography assistance function can also be independently integrated into other devices. For example, the photography assistance function can be integrated into the photography apparatus 10. Alternatively, the photography assistance function can be integrated into mobile devices such as smartphones and tablets.

[0251] When the photography device 10 is equipped with a photography assistance function, it can be configured to automatically set the exposure. For example, in handheld shooting, the exposure is automatically set to a high shutter speed and high sensitivity. And, in non-handheld shooting, the exposure is automatically set to a low shutter speed and low sensitivity.

[0252] Furthermore, it can also be configured to communicatively connect the photography device 10 to the 3D model generation device 100, and automatically reflect the photography conditions determined by the 3D model generation device 100 onto the photography device 10.

[0253] Methods for generating 3D models

[0254] In the above embodiments, the example described is the generation of a three-dimensional model of an object by photogrammetry and based on multi-viewpoint images, but the same applies when generating a three-dimensional model using other methods (e.g., the view volume cross method).

[0255] [Fifth Implementation]

[0256] As mentioned above, one method of multi-viewpoint photography is using a turntable. When shooting with a turntable, the subject moves, and if an appropriate shutter speed is not set, subject blur (motion blur) will occur. Images with subject blur negatively impact the generation of 3D models.

[0257] In this embodiment, a system is proposed that enables even users lacking knowledge of photography to capture high-quality images suitable for the generation of 3D models.

[0258] [3D Model Generation System]

[0259] Figure 13 This is a diagram showing the general structure of a 3D model generation system.

[0260] The three-dimensional model generation system 1 of this embodiment is configured as a system for generating a three-dimensional model M of an object O based on multi-viewpoint images.

[0261] like Figure 13 As shown, the 3D model generation system 1 includes a photography device 10 and a 3D model generation device 100.

[0262] [Photographic installation]

[0263] The photographic device 10 is a device for photographing the object O. The photographic device 10 is composed of a conventional digital camera.

[0264] The photographic device 10 is fixed by a tripod or the like, and photographs an object O rotating at a fixed position from a fixed position. As an example, in this embodiment, the object O is placed on a turntable 20 and rotated. The turntable 20 rotates at a constant speed N [rpm].

[0265] It can also be configured to use multiple camera devices 10 for shooting. In this case, for example, multiple camera devices 10 can be used to simultaneously shoot the object O from different positions or directions.

[0266] [3D Model Generation Device]

[0267] The hardware structure of the 3D model generation device 100 is the same as that of the 3D model generation device 100 in the first embodiment described above. That is, it is composed of a computer such as a PC, and includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, an operation unit 104, a display unit 105, and an interface unit 106, etc. (see reference). Figure 2 ).

[0268] Figure 14 This is a block diagram of the main functions of a 3D model generation device.

[0269] The 3D model generation device 100 has the function of generating a 3D model of an object based on multi-viewpoint images (3D model generation function) and the function of assisting in the photography of multi-viewpoint images (photography assistance function).

[0270] [3D Model Generation Function]

[0271] The 3D model generation function is the same as that in the 3D model generation apparatus 100 of the third embodiment described above. Therefore, the description is omitted.

[0272] [Photography Assistance Functions]

[0273] In this embodiment, as a photography assistance function, information about the photography method is obtained from the user, the photography conditions are determined based on the obtained information, and then the user is prompted accordingly. For example... Figure 14 As shown, the 3D model generation device 100 has functions such as a scanning speed information acquisition unit 100L, a photography condition determination unit 100K, and a notification unit 100E as a photography assistance function.

[0274] The scanning speed information acquisition unit 100L acquires scanning speed information (scanning speed information). Scanning speed refers to the speed at which the object O is scanned. As described above, in this embodiment, a rotating object O is photographed from a fixed position, and the object O is scanned. Therefore, the rotational speed of the object O is called the scanning speed. The rotational speed of the object O is the rotational speed (rotational speed N) of the turntable 20. Therefore, the scanning speed information acquisition unit 100L acquires the information of the rotational speed N of the turntable 20 as the scanning speed information. The scanning speed information acquisition unit 100L acquires the scanning speed information from the user's input of the scanning speed (rotational speed of the turntable 20). The scanning speed information acquisition unit 100L receives the scanning speed information from the user, for example, by displaying a predetermined input screen on the display unit 105. The input operation is performed via the operation unit 104. In this embodiment, the scanning speed information (information on photographic conditions) acquired by the scanning speed information acquisition unit 100L is an example of first information.

[0275] The photography condition determination unit 100K determines the photography conditions based on the scanning speed (rotation speed of the turntable 20) information obtained from the user. Specifically, it determines the shutter speed. This shutter speed is one that can suppress blurring of the subject.

[0276] Here, a higher shutter speed improves the suppression of subject blur. On the other hand, increasing the shutter speed requires increasing sensitivity (with a fixed aperture value). Increasing sensitivity increases noise (resulting in a lower SN ratio). Therefore, it is preferable to set the shutter speed to the minimum necessary to suppress subject blur.

[0277] In this embodiment, the information on the photographic conditions (shutter speed information) determined by the photographic condition determination unit 100K is an example of the second information.

[0278] The notification unit 100E notifies the user of the information on the shooting conditions (shutter speed information) determined by the shooting condition determination unit 100K as shooting assistance information. As an example, in this embodiment, the determined shooting condition information (shooting assistance information) is displayed on the display unit 105 to notify the user.

[0279] [Generation of 3D Models]

[0280] Next, the method for generating a three-dimensional model M of an object O based on multi-view images using the three-dimensional model generation system 1 of this embodiment will be described.

[0281] (1) Provision of photographic auxiliary information

[0282] First, the 3D model generation device 100 receives photographic aid information.

[0283] Figure 15 This is a flowchart illustrating the sequence of processes that provide photographic aid information in a 3D model generation apparatus.

[0284] First, the scanning speed information is obtained from the user (step S41). In this embodiment, the scanning speed information is obtained from the user's input of the rotational speed N of the turntable 20.

[0285] The 3D model generation apparatus 100 determines the photographic conditions based on the acquired scanning speed information (step S42). In this embodiment, a shutter speed capable of suppressing subject blur is determined.

[0286] The 3D model generation apparatus 100 notifies the user of the determined photographic conditions (shutter speed) as photographic assistance information (step S43). In this embodiment, the determined photographic conditions (shutter speed) information is displayed on the display unit 105 to notify the user. The photographic assistance information is provided to the user as setting information for the optimal shutter speed based on the scan speed (recommended shutter speed setting information).

[0287] (2) Generation of three-dimensional models

[0288] The user accepts prompts from photographic assistance information to set the exposure and photograph the subject. More specifically, the user sets the shutter speed as prompted by the 3D model generation device 100, and sets the aperture and sensitivity in a way that results in appropriate exposure, and then photographs the subject.

[0289] As described above, the turntable 20 is used for filming. That is, the object O is placed on the turntable 20 and rotated at a constant speed for filming. The photographic apparatus 10 films the rotating object O from a fixed position.

[0290] Multi-view images obtained through photography are input into the 3D model generation apparatus 100. The 3D model generation apparatus 100 processes the input multi-view images to generate a 3D model.

[0291] As explained above, according to this embodiment, even users lacking knowledge of photography can capture high-quality images suitable for generating 3D models. Therefore, high-quality 3D models can be easily generated.

[0292] [Variation Example]

[0293] [Photography conditions]

[0294] In the above embodiment, the structure is set to determine the shutter speed as a photographic condition, but the items determined as photographic conditions are not limited to this. For example, the shooting interval can be determined in addition to or in place of the shutter speed. By adjusting the shooting interval, the overlap rate between adjacent images can be adjusted. Furthermore, by appropriately setting the shooting interval, missed shots can be suppressed.

[0295] [Photography Method]

[0296] In the above embodiments, the structure is designed to photograph a rotating object from a fixed position, but it can also be designed to photograph by rotating the photographing device sideways. For example, it can be designed to photograph an object that rotates at a constant speed (revolution) around a stationary object. In this case, the rotational speed (revolutional speed) of the photographing device becomes the scanning speed. Furthermore, it can also be designed to photograph an object that rotates at a constant speed (revolution) around a rotating object. In this case, the difference between the rotational speed (revolutional speed) of the photographing device and the rotational speed (rotational speed) of the object becomes the scanning speed.

[0297] [Photography Aid Information Tips]

[0298] In the above embodiment, the structure is configured to notify the user by displaying photography assistance information (information on photography conditions) on the display unit 105 of the 3D model generation apparatus 100, but the method of notifying the user of the photography assistance information is not limited to this. For example, the structure could also be configured to send the photography assistance information to the photography apparatus 10 and display it on the display unit of the photography apparatus 10.

[0299] Furthermore, in the above embodiment, the example described is that the 3D model generation apparatus 100 has a photography assistance function, but the photography assistance function can also be independently integrated into other devices. For example, the photography assistance function can be integrated into the photography apparatus 10. Alternatively, the photography assistance function can be integrated into mobile devices such as smartphones and tablets.

[0300] When the photographic device 10 is equipped with a photographic assistance function, it can be set to an automatic exposure setting structure. That is, the exposure is automatically set to the determined shutter speed.

[0301] Furthermore, it can also be configured to communicatively connect the photography device 10 to the 3D model generation device 100, and automatically reflect the photography conditions determined by the 3D model generation device 100 onto the photography device 10.

[0302] Methods for generating 3D models

[0303] In the above embodiments, the example described is the generation of a three-dimensional model of an object by photogrammetry and based on multi-viewpoint images, but the same applies when generating a three-dimensional model using other methods (e.g., the view volume cross method).

[0304] Furthermore, it can also be applied to situations where 3D models are generated using other methods. For example, it can be applied to situations where 3D models are generated using LiDAR, ToF, or structured light methods. For instance, when generating 3D models using LiDAR or ToF methods, the optimal measurement conditions are determined based on the scanning speed information, and the user is prompted with these conditions or the conditions are set automatically.

[0305] [Sixth Implementation]

[0306] The quality of the generated 3D model should ideally be set to correspond to its importance. That is, a higher importance 3D model should be generated with a higher quality (reproducibility, image quality). Generally, increasing the quality of the 3D model requires more images or measurement data for processing. Furthermore, increasing the quality also increases processing time.

[0307] Users lacking knowledge about 3D model generation may find it difficult to set appropriate parameters based on the importance of the 3D model.

[0308] In this embodiment, a system is proposed that allows even users lacking knowledge about the generation of 3D models to appropriately set the necessary processing based on the importance of the 3D model to be generated, and to generate a 3D model of quality corresponding to the importance.

[0309] [3D Model Generation System]

[0310] Figure 16 This is a diagram showing the general structure of a 3D model generation system.

[0311] The three-dimensional model generation system 1 of this embodiment is configured as a system for generating a three-dimensional model M of an object O based on multi-viewpoint images.

[0312] like Figure 16 As shown, the 3D model generation system 1 includes a photography device 10 and a 3D model generation device 100.

[0313] [Photographic installation]

[0314] The photographic device 10 is a device for photographing the object O. The photographic device 10 is composed of a conventional digital camera. Alternatively, it can be configured such that photography is performed by a single photographic device 10, or it can be configured such that multiple photographic devices 10 are used.

[0315] [3D Model Generation Device]

[0316] The hardware structure of the 3D model generation device 100 is the same as that of the 3D model generation device 100 in the first embodiment described above. That is, it is composed of a computer such as a PC, and includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, an operation unit 104, a display unit 105, and an interface unit 106, etc. (see reference). Figure 2 ).

[0317] Figure 17 This is a block diagram of the main functions of a 3D model generation device.

[0318] like Figure 17 As shown, the 3D model generation apparatus 100 has the functions of an image acquisition unit 100A, a 3D model generation unit 100B, an importance information acquisition unit 100M, and a processing mode determination unit 100N. The functions of each unit are implemented by the processor 101 executing a predetermined program.

[0319] The image acquisition unit 100A acquires multi-view images of the object. For example, the multi-view images are stored in the auxiliary storage unit 103, and then read from and acquired from the auxiliary storage unit 103. Alternatively, it can be configured to acquire the images directly from the photographic device 10.

[0320] The 3D model generation unit 100B generates a 3D model based on multi-viewpoint images. As an example, in this embodiment, a 3D model of an object is generated by photogrammetry based on multi-viewpoint images. The 3D model generation unit 100B processes the multi-viewpoint images in a processing mode determined by the processing mode determination unit 100N to generate the 3D model of the object.

[0321] Here, "processing mode" refers to the mode in which a 3D model of an object is generated based on multi-view images. The quality of the generated 3D model changes by changing the processing mode. For example, in this embodiment, there are two processing modes: a "high-quality mode" and a "low-quality mode." The "high-quality mode" generates a high-quality 3D model. The "low-quality mode" generates a 3D model of lower quality compared to the high-quality mode. In each mode, the quality of the generated 3D model is adjusted by changing the settings of specific parameters. For example, in this embodiment, the quality of the generated 3D model is adjusted by changing the number of images used in the generation of the 3D model and the number of repetitions of the recursive estimation process for the 3D shape. In high-quality mode, the number of images used is set to be higher, and the number of recursive estimation processes for the 3D shape is set to be higher. In low-quality mode, the number of images used and the number of recursive estimation processes for the 3D shape are set to the necessary minimum. By setting the number of images used to be higher and the number of recursive estimation processes for the 3D shape to be higher, a high-quality 3D model can be generated, but the processing time increases, requiring more time for generation. On the other hand, by setting the number of images used and the number of repetitions of the recursive estimation process of the 3D shape to the necessary minimum, the generation time can be shortened.

[0322] The importance information acquisition unit 100M acquires information about the importance of the 3D model to be generated (importance information). Here, "importance" is a subjective judgment by the user. The higher the required quality, the higher the importance is set. The importance information acquisition unit 100M acquires importance information from the user's input. For example, in this embodiment, importance is divided into two stages: "high" and "low" to acquire importance information. The importance information acquisition unit 100M, for example, displays a predetermined selection screen (a screen that allows the user to select "high" or "low" regarding importance) on the display unit 105 to receive importance information input (selection) from the user. The selection operation is performed via the operation unit 104. In this embodiment, the importance information acquired by the importance information acquisition unit 100M is an example of first information.

[0323] The processing mode determination unit 100N determines the processing mode based on importance information obtained from the user. In this embodiment, either a high-quality mode or a low-quality mode is selected. Specifically, when the importance is "high," the high-quality mode is selected, and when the importance is "low," the low-quality mode is selected. In this embodiment, the information regarding the processing mode of the 3D model determined by the processing mode determination unit 100N is an example of the second information.

[0324] The processing mode information determined by the processing mode determination unit 100N is provided to the 3D model generation unit 100B. The 3D model generation unit 100B processes the multi-view image and generates a 3D model of the object under the processing mode determined by the generation method determination unit 100D.

[0325] [Generation of 3D Models]

[0326] Next, the method for generating a three-dimensional model M of an actual object O using the three-dimensional model generation system 1 of this embodiment will be described.

[0327] First, the object O is photographed from multiple viewpoints using the photographic device 10. The photography can be performed using one photographic device 10 or multiple photographic devices 10.

[0328] A three-dimensional model is generated by inputting multi-viewpoint images obtained through photography into a three-dimensional model generation device 100.

[0329] Figure 18 This is a flowchart illustrating the sequence of processes for generating a 3D model from multi-view images.

[0330] First, acquire multi-view images of the object (step S51).

[0331] Next, information about the importance of the 3D model to be generated is obtained (step S52). In this embodiment, the importance information is "high" or "low". The 3D model generation apparatus 100 obtains information about the importance of the 3D model to be generated from the user's input (selection) of importance information.

[0332] Next, the processing mode (processing mode) for generating the 3D model is determined based on the obtained importance information (step S53). In this embodiment, either a high-quality mode or a low-quality mode is selected to determine the processing mode. Specifically, when the importance is "high", the high-quality mode is selected, and when the importance is "low", the low-quality mode is selected.

[0333] Next, the multi-view images are processed under the determined processing mode to generate a 3D model (step S54). For example, if a high-quality mode is selected, the multi-view images are processed in high-quality mode to generate a 3D model. On the other hand, if a low-quality mode is selected, the multi-view images are processed in low-quality mode to generate a 3D model. In high-quality mode, a high-quality 3D model is generated. On the other hand, in low-quality mode, a 3D model of lower quality than that in high-quality mode is generated.

[0334] As explained above, the 3D model generation system 1 according to this embodiment automatically sets processing conditions (processing mode) based on the importance of the 3D model to be generated to generate the 3D model. Therefore, even users lacking knowledge of 3D scanning can generate 3D models of appropriate quality corresponding to their importance. That is, high-quality 3D models can be generated for models with high importance, while models with low importance can be generated quickly.

[0335] [Variation Example]

[0336] [Importance Information]

[0337] In the above implementation, the structure is set to obtain the information by dividing the importance into two stages, "high" and "low", but it can also be set to obtain the information by dividing it into multiple stages.

[0338] [Processing conditions when generating 3D models]

[0339] In the above embodiments, the structure is set to determine the processing mode (processing mode) as a processing condition, but the content of the determined processing condition is not limited to this. It can also be set to determine specific processing parameters (such as the number of images used in the generation of the 3D model, the setting of the number of repetitions of the recursive estimation process of the 3D shape, etc.) separately based on the importance information.

[0340] Methods for generating 3D models

[0341] In the above embodiments, an example was given of generating a 3D model of an object using photogrammetry and multi-view images; however, the method for generating a 3D model of an object is not limited to this. Other generation methods can also be used. Furthermore, it is also applicable to situations where 3D models are generated using image measurement data, such as LiDAR or ToF methods.

[0342] [Manual Selection]

[0343] In the above embodiment, the example described is the automatic generation of a 3D model under a determined processing mode. However, it is also possible to manually select a processing mode based on a decision and generate the structure of the 3D model under the selected processing mode. In this case, the user is notified of the determined processing mode. The user refers to the notified information to select a processing mode and generates the 3D model.

[0344] [Other information prompts]

[0345] In the above embodiments, the structure is designed to determine the processing conditions for generating the 3D model based on information about the importance of the 3D model to be generated. However, it can also be designed to determine photographic conditions or measurement conditions based on the importance information and then prompt the user. For example, the number of images to be captured or the amount of data to be measured can be determined based on information about the importance of the 3D model to be generated, and this information can be prompted to the user. Furthermore, the overlap rate between adjacent images or the overlap rate of the measurement range can be determined based on information about the importance of the 3D model to be generated, and this information can be prompted to the user.

[0346] [Other Implementation Methods]

[0347] The hardware of the information processing system implementing this invention can be composed of various processors. These processors include general-purpose processors that execute programs and function as various processing units, such as CPUs (Central Processing Units); processors whose circuit structure can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs); and processors with circuit structures specifically designed for performing specific processes, such as ASICs (Application Specific Integrated Circuits). A processing unit constituting a damage map creation auxiliary device can be composed of one of the aforementioned processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Furthermore, a single processor can also constitute multiple processing units. As an example of a single processor constituting multiple processing units, firstly, as exemplified by a computer such as a client or server, a processor can be composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units. Secondly, there are other approaches, such as System-on-Chip (SoC), which uses a single IC (Integrated Circuit) chip to implement the overall functionality of a system comprising multiple processing units. In this way, various processing units are constructed using one or more of these processors as their hardware structure. More specifically, the hardware structure of these processors is a circuit composed of semiconductor elements and other circuitry.

[0348] Symbol Explanation

[0349] 1-3D model generation system, 10-Photographic device, 20-Turntable, 100-3D model generation device, 100A-Image acquisition unit, 100B-3D model generation unit, 100B1-First processing unit, 100B2-Second processing unit, 100B3-Compositing processing unit, 100C-Object information acquisition unit, 100D-Generation method determination unit, 100E-Notification unit, 100F-Transparent area extraction unit, 100G-Lighting information acquisition unit, 100H-Photographic method determination unit, 100J-Photographic mode information acquisition unit, 100K-Photographic condition determination unit, 100L-Scanning speed information acquisition unit, 100M- Importance information acquisition unit, 100N-processing mode determination unit, 101-processor, 102-main storage unit, 103-auxiliary storage unit, 104-operation unit, 105-display unit, 106-interface unit, M-3D model, O-object, S1~S4-processing order for determining the generation method of the 3D model, S11~S14-processing order for generating the 3D model, S21~S25-processing order for providing photographic assistance information, S31~S35-processing order for providing photographic assistance information, S41~S43-processing order for providing photographic assistance information, S51~S54-processing order for generating the 3D model.

Claims

1. An information processing system comprising at least one processor, The processor performs the following processing: Acquire first information containing at least one of the following: information related to the object, information related to the environment in which the object was photographed or measured, information related to the conditions in which the object was photographed or measured, and information on the importance of the photographing or measurement; and Based on the first information, a second information related to the generation of the three-dimensional data of the object is generated.

2. The information processing system according to claim 1, wherein, The processor obtains the first information from the user's input of the first information.

3. The information processing system according to claim 1, wherein, The information related to the object includes information about transparent areas. Information related to the environment in which the object is photographed or measured includes information about the lighting conditions. Information related to the conditions under which the object is photographed or measured includes at least one of whether the photograph or measurement is handheld and the scanning speed.

4. The information processing system according to claim 1, wherein, The second piece of information is information about the method used in generating the three-dimensional data. The processor determines the method used in generating the three-dimensional data based on the first information.

5. The information processing system according to claim 4, wherein, The first information includes at least information about the transparent area of ​​the object. The processor determines the method used to generate the 3D data based on the information from the transparent area.

6. The information processing system according to claim 5, wherein, The first information includes at least one of the following: information on whether the transparent area exists and information on the proportion of the transparent area in the object. The processor determines the method used to generate the three-dimensional data based on at least one of information about the presence or absence of the transparent region and information about the proportion of the transparent region in the object.

7. The information processing system according to claim 6, wherein, The processor determines whether to use photogrammetry or the apparent volume cross method as the method used to generate the 3D data.

8. The information processing system according to claim 5, wherein, The processor determines the method used to generate the 3D data for each region of the object based on the information from the transparent region.

9. The information processing system according to any one of claims 1 to 8, wherein, The second piece of information is information about the photographic method used to generate the three-dimensional data. The first piece of information includes at least information about the lighting status. The processor determines the photographic method of the image used in generating the 3D data based on the information about the lighting conditions.

10. The information processing system according to claim 9, wherein, The processor determines whether to employ polarization photography as the photographic method.

11. The information processing system according to claim 9, wherein, The lighting status information includes at least one of the following: whether specular reflection occurs or not, and the proportion of the area in the object where specular reflection occurs.

12. The information processing system according to claim 9, wherein, The lighting status information includes whether the lighting is for creating a specular reflection on the object.

13. The information processing system according to claim 9, wherein, The lighting status information includes whether the lighting can be adjusted.

14. The information processing system according to any one of claims 1 to 8, wherein, The first piece of information includes at least whether the photo was taken handheld. The processor determines the photographic conditions of the object based on the information of whether it is a handheld shot, and generates the second information.

15. The information processing system according to claim 14, wherein, The processor determines the shutter speed setting when photographing the object as the photographic condition.

16. The information processing system according to any one of claims 1 to 8, wherein, The second piece of information is the photographic conditions of the object. The first piece of information includes at least information about the scanning speed. The processor determines the imaging conditions based on the scanning speed information.

17. The information processing system according to claim 16, wherein, The processor determines the shutter speed setting when photographing the object as the photographic condition.

18. The information processing system according to any one of claims 1 to 8, wherein, The second piece of information is information about the processing conditions used to generate the three-dimensional data. The first piece of information includes at least information about the importance of the image or measurement. The processor determines the processing conditions based on the information regarding the importance of the captured or measured image.

19. The information processing system according to claim 18, wherein, The processor determines the processing mode for generating the three-dimensional data as the processing condition.

20. An information processing method, comprising the following steps: The step of obtaining first information comprising at least one of information related to the object, information related to the environment in which the object was photographed or measured, information related to the conditions in which the object was photographed or measured, and information on the importance of the photographing or measurement; and The step of generating second information related to the generation of three-dimensional data of the object based on the first information.

21. An information processing program that enables a computer to perform the following functions: The function of acquiring first information containing at least one of the following: information related to the object, information related to the environment in which the object was photographed or measured, information related to the conditions in which the object was photographed or measured, and information on the importance of the photographing or measurement; and The function of generating second information related to the generation of three-dimensional data of the object based on the first information.

22. A recording medium that is non-transitory and computer-readable, the recording medium recording the program of claim 21.

Images