Information processing system, three-dimensional model generation device, optical device, information processing method, and program

Abstract

Provided are an information processing system, a three-dimensional model generation device, an optical device, an information processing method, and a program with which it is possible to provide an excellent 3D model. An information processing system comprising one or more processors, wherein the one or more processors acquire two or more images having different in-focus positions for each of a plurality of imaging directions of an object, combine images that are acquired from the same imaging direction and have different in-focus positions to generate a depth composite image, and generate a 3D model of at least a portion of the object on the basis of the depth synthesized image obtained for each imaging direction.

Description

Information processing system, three-dimensional model generating device, optical device, information processing method and program

[0001] The present invention relates to an information processing system, a three-dimensional model generating device, an optical device, an information processing method, and a program for generating a three-dimensional model.

[0002] Patent document 1 discloses a method for generating a 3D view of an object, which includes capturing image data from multiple viewpoints around the object, analyzing the image data for quality, creating an image dataset based on the image data, filtering the image dataset, generating data reference parameters, and uploading the image dataset to a server via a network.

[0003] Special table 2020-527814 publication

[0004] One embodiment of the technique of the present disclosure provides an information processing system, a three-dimensional model generation device, an optical device, an information processing method, and a program that can provide a good 3D model.

[0005] The information processing system of the first aspect is an information processing system including one or more processors, which acquire two or more images with different focus positions for each of multiple shooting directions of an object, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-synthesized image, and generate a 3D model of at least a portion of the object based on the depth-synthesized image acquired for each shooting direction.

[0006] In the information processing system of the second aspect, in the first aspect, the one or more processors determine whether or not the image is appropriate to be applied to the generation of a 3D model.

[0007] In the information processing system of the third aspect, in the first or second aspect, the one or more processors determine whether or not an image is appropriate for use in generating a depth stacking image.

[0008] An information processing system of a fourth aspect is any one of the first to third aspects, wherein the one or more processors determine whether or not to apply the depth stacking image to the generation of a 3D model.

[0009] An information processing system of a fifth aspect is any one of the second to fourth aspects, wherein the one or more processors perform image evaluation to determine suitability.

[0010] In a sixth aspect of the information processing system, in the fifth aspect, the one or more processors instruct recapture of the image based on the result of the image evaluation or the judgment of suitability.

[0011] An information processing system according to a seventh aspect is any one of the first to sixth aspects, wherein the one or more processors determine conditions for acquiring images of the object with different focus positions.

[0012] An information processing system according to an eighth aspect is the information processing system according to the seventh aspect, wherein the one or more processors determine the conditions based on a preliminary image of the object taken before the image is taken.

[0013] In a ninth aspect of the information processing system, in the eighth aspect, the one or more processors determine the width of the object based on the preliminary image, and determine the condition based on the width.

[0014] In a tenth aspect of the information processing system, in the eighth aspect, the one or more processors obtain a pre-3D model of the object based on the pre-image, and determine the conditions based on the pre-3D model.

[0015] An information processing system of an eleventh aspect is the information processing system of the tenth aspect, wherein the one or more processors generate the pre-3D model by a volume intersection method.

[0016] In a twelfth aspect of the information processing system, in the seventh aspect, the one or more processors calculate a coverage rate of a target area of ​​the object for each shooting direction, and determine the conditions based on the calculation results.

[0017] In a thirteenth aspect of the information processing system, in the seventh aspect, the one or more processors determine, for each imaging direction, an area of ​​the object that does not need to be imaged, and determine the conditions.

[0018] A fourteenth aspect of the information processing system is the seventh aspect, wherein the one or more processors determine conditions in a direction different from one of the imaging directions based on conditions determined in the one direction.

[0019] In a fifteenth aspect of the information processing system, in the eighth aspect, the one or more processors extract feature points of the object based on the preliminary image, and determine the conditions based on the feature points.

[0020] In a sixteenth aspect of the information processing system, in the fifteenth aspect, the one or more processors determine an initial focus position of an image for creating a depth stacking image as a condition based on the feature points.

[0021] In a seventeenth aspect of the information processing system, in the seventh aspect, the one or more processors determine the conditions based on a purpose desired for the 3D model.

[0022] An information processing system of an eighteenth aspect is the information processing system of the seventh aspect, wherein the one or more processors determine the condition based on distance information to the object.

[0023] A 19th aspect of the three-dimensional model generation device is a three-dimensional model generation device that includes an optical device that can capture images of an object, and an information processing device, and is equipped with one or more processors included in either the optical device or the information processing device, and the one or more processors acquire two or more images with different focus positions for each of multiple shooting directions of the object, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-combined image, and generate a 3D model of at least a portion of the object based on the depth-combined images obtained for each shooting direction.

[0024] A 3D model generating device according to a twentieth aspect is the 19th aspect, wherein the one or more processors determine whether or not it is appropriate to apply the image to the generation of the 3D model.

[0025] A 3D model generating device according to a 21st aspect is the 19th or 20th aspect, wherein the one or more processors determine whether or not an image is appropriate for use in generating a depth stacking image.

[0026] A 3D model generating device of a 22nd aspect is any one of the 19th to 21st aspects, wherein one or more processors determine whether or not it is appropriate to apply a depth stacking image to the generation of a 3D model.

[0027] A 3D model generating device of a 23rd aspect is any one of the 20th to 22nd aspects, wherein the one or more processors perform image evaluation to determine suitability.

[0028] A 3D model generating device according to a 24th aspect is the 23rd aspect, wherein the one or more processors instruct recapture of the image based on the result of the image evaluation or suitability determination.

[0029] A 3D model generating device according to a 25th aspect is any one of the 19th to 24th aspects, wherein the one or more processors determine conditions for acquiring images of the object with different focus positions.

[0030] An optical device of the 26th aspect is an optical device that has one or more processors and is capable of photographing an object, wherein the one or more processors acquire two or more images with different focus positions for each of multiple shooting directions of the object, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-combined image, and generate a 3D model of at least a portion of the object based on the depth-combined image acquired for each shooting direction.

[0031] The optical device of the 27th aspect is the 26th aspect, wherein the one or more processors determine conditions for acquiring images of the object with different focus positions.

[0032] The information processing method of the 28th aspect is an information processing method executed by an information processing system having one or more processors, in which the one or more processors acquire images of an object with different focus positions from multiple shooting directions, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-synthesized image, and generate a 3D model based on the depth-synthesized image obtained for each shooting direction.

[0033] The program of the 29th aspect is a program that executes an information processing method of an information processing system that has one or more processors, and causes the one or more processors to acquire images of an object with different focus positions from multiple shooting directions, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-combined image, and generate a 3D model based on the depth-combined image obtained for each shooting direction.

[0034] 1 is a diagram illustrating an overview of a system for generating a 3D model. FIG. 2 is a diagram illustrating an overview of a system different from that of FIG. 1 for generating a 3D model. FIG. 3 is a block diagram illustrating a schematic configuration of a camera. FIG. 4 is a block diagram illustrating a schematic configuration of a computer. FIG. 5 is a flowchart illustrating a method for generating a three-dimensional model. FIG. 6 is a diagram illustrating an example of FBK photography. FIG. 7 is a block diagram illustrating functions related to the generation of depth stacked images. FIG. 8 is a block diagram illustrating functions related to 3D model generation. FIG. 9 is a flowchart illustrating a method for identifying FBK images to be excluded. FIG. 10 is a flowchart illustrating a method for identifying depth stacked images to be excluded. FIG. 11 is a flowchart illustrating a method for generating a three-dimensional model including pre-photography. FIG. 12 is a diagram illustrating an example of setting conditions for FBK photography. FIG. 13 is a diagram illustrating an example of thinning photography. FIG. 14 is a diagram illustrating an example of matching between an object and a still image (2D image). FIG. 15 is a diagram illustrating a procedure for determining other preferable conditions for FBK photography. FIG. 16 is a diagram illustrating a procedure for determining other preferable conditions for FBK photography.

[0035] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings.

[0036] [Summary] In 3D modeling using photogrammetry, it is desirable to photograph the object to be 3D modeled at a high resolution and to ensure that the photographed image is free of blur or blur.

[0037] Improving the quality of the 3D model you create requires more detailed images (higher pixel counts), and using a wide-angle or macro lens to capture close-up shots of the subject results in a shallow depth of field. To prevent camera shake, a faster shutter speed is desirable. Because high-sensitivity noise affects the image quality of 3D models, it's best to shoot at a low sensitivity. This ultimately requires using a smaller f-number (wider aperture), resulting in a shallow depth of field. Therefore, to create a better 3D model, the images used for depth stacking must be captured with optimal camera settings (shooting conditions). However, currently, this can be time-consuming, requiring an increased number of shots or time-consuming setup.

[0038] 1 is a diagram showing a system 1 for generating a 3D model according to an embodiment. The system 1 includes a camera 10 and a computer 20. An object 30 to be the subject of a 3D model is placed on a shooting stand 40.

[0039] The camera 10 can move around the object 30. The camera 10 is configured to capture images of the object 30 from multiple directions as it moves around the object 30. When generating a 3D model of the entire object 30, the camera 10 moves around the object 30 (e.g., 360° or more). When generating a 3D model of only a portion of the object 30, the camera 10 moves within a specific range of the object 30 (a range less than 360°). The camera 10 can be moved around the object 30 by a user holding the camera 10 or by a mobile object supporting the camera 10. The mobile object is, for example, an arm attached to the imaging stand 40. This arm is configured to be able to move around the imaging stand 40 by a motor or the like. Furthermore, when the object 30 is large, the mobile object may be a vehicle or a drone.

[0040] The camera 10 is configured to capture images of the object 30 at each position of the camera 10 while moving around the object 30. The camera 10 can capture still images and moving images. The camera 10 is also configured to enable focus bracketing (FBK) photography. By performing FBK photography, the camera 10 can capture two or more images of the object 30 at different focus positions in one photography direction. Images captured by FBK photography are called FBK images. FBK photography is photography in which the focus position is moved in one photography direction to obtain multiple images.

[0041] The camera 10 can capture images of the object 30 from multiple shooting directions. The camera 10 can capture two or more images with different focus positions for each of the multiple shooting directions. The camera 10 is also configured to store images of the object 30. The camera 10 is configured to capture images of the object 30 in response to a user instruction (operation), that is, manually, or automatically. The camera 10 can also be equipped with a distance measuring device that can measure the distance to the object 30.

[0042] The computer 20 has a display and a keyboard. The display is an example of a display device that displays various information. The keyboard is an example of an input device that allows a user to input instructions. The computer 20 is configured to process various data including images, input and output various data, and store various data. The computer 20 is configured to store programs and the like for executing its functions and to be able to execute the programs and the like.

[0043] The computer 20 is configured to execute a program to perform depth stacking on the FBK images acquired from the camera 10. Depth stacking is a technique for synthesizing multiple FBK images with different focus positions to generate a composite image in which the entire object 30 is in focus.

[0044] The computer 20 is configured to execute a program to generate a 3D model from multiple images of the object 30 using photogrammetry. Photogrammetry is a technology that analyzes multiple images of the object 30 taken from different angles and synthesizes the multiple images to generate (restore) a three-dimensional shape or structure (3D model).

[0045] The object 30 is not particularly limited in shape or size as long as it is an object for which a 3D model is to be generated. The object 30 may be any object having a physical shape.

[0046] The imaging table 40 has a plurality of marks 41 on its placement surface (top surface). The marks 41 serve as indicators for the imaging direction. The distance between two marks 41 serves as a reference for the size when creating a 3D model. The imaging table 40 has a cylindrical shape with a flat top surface. The shape of the imaging table 40 is not limited as long as it can accommodate the object 30. The imaging table 40 may be a rectangular parallelepiped. However, the imaging table 40 is not essential when considering conditions such as the size of the object 30 and the location where the object 30 is placed.

[0047] Fig. 2 is a diagram showing an overview of system 2 that generates a 3D model different from system 1. In Fig. 2, parts common to the above-described system 1 are given the same reference numerals, and their description will be omitted. System 2 in Fig. 2 includes a camera 10 and a computer 20. The camera 10 is fixed to a tripod 44 and is in a stationary state. The tripod 44 is an example of a support member that keeps the camera 10 in a stationary state.

[0048] The object 30 is placed on an imaging table 42 having a mark 43. The imaging table 42 is rotatable, unlike the imaging table 40 of the system 1. The imaging table 42 can rotate the object 30 at any angle.

[0049] When generating a 3D model of the entire object 30, the imaging table 42 rotates the object 30 (for example, 360° or more). When generating a 3D model of a portion of the object 30, the imaging table 42 rotates the object 30 within a specific range (a range smaller than 360°). The imaging table 42 includes a motor or the like and can rotate at any speed.

[0050] The camera 10 is configured to capture images at each position of the object 30 while the object 30 is rotated by the imaging platform 42. The camera 10 can manually or automatically capture images of the object 30 from multiple imaging directions.

[0051] The camera 10 and computer 20 of system 2 are essentially the same as the camera 10 and computer 20 of system 1 .

[0052] [Camera] Fig. 3 is a block diagram showing a schematic configuration of the camera 10. As shown in Fig. 3, the camera 10 includes a lens device 100. The lens device 100 includes an imaging optical system 102 including a lens group 104 and an aperture 106, a lens driver 110, an aperture driver 112, and the like.

[0053] The lens device 100 may be detachable from the camera 10 or may be integrated with the camera 10 .

[0054] The lens group 104 includes at least a focus lens that is movable in the optical axis direction. This focus lens is a lens for focus adjustment. The focus of the imaging optical system 102 is adjusted by moving the focus lens back and forth along the optical axis. The focus lens is driven and operated by the lens driver 110. The focus lens can be moved to a focus position where it focuses on the target object 30.

[0055] The diaphragm 106 is configured by, for example, an iris diaphragm. The amount of light passing through the imaging optical system 102 is adjusted by the diaphragm 106. The diaphragm 106 is driven by a diaphragm driver 112 to operate.

[0056] As shown in FIG. 3, the camera 10 includes an image sensor 130, a shutter 132, a shutter driver 134, a memory 136, a digital signal processor 138, an input / output interface 140, a display 142, an operation unit 144, and a system controller 146.

[0057] The image sensor 130 is, for example, a complementary metal-oxide semiconductor (CMOS) image sensor having a predetermined color filter array (e.g., a Bayer array). In the camera 10 of this embodiment, the image sensor 130 includes a drive unit, an analog-to-digital converter (ADC), a signal processor, and the like. In this case, the image sensor 130 is driven by the built-in drive unit. The signals from each pixel are converted into digital signals by the built-in ADC. Furthermore, the signals from each pixel are subjected to processes such as correlated double sampling, gain processing, and correction processing as needed by the built-in signal processor. The signal processing can be performed on the analog signals of each pixel, or on the digital signals of each pixel.

[0058] The imaging element 130 may be configured as an organic thin-film imaging element, an XY address type, a CCD (Charged Coupled Device) type, or other image sensor, in addition to a CMOS type image sensor.

[0059] The shutter 132 is disposed between the aperture 106 and the image sensor 130. The shutter 132 is driven to operate by a shutter driver 134. The shutter driver 134 controls the opening and closing of the shutter 132, and controls the exposure time (shutter speed) at the image sensor 130.

[0060] The memory 136 includes a flash memory, a read-only memory (ROM), a random access memory (RAM), an auxiliary storage device, etc. The flash memory and ROM store a camera control program, a focus bracket shooting program for executing shooting in focus bracket shooting mode, an image processing program, various data necessary for camera control, etc. The RAM temporarily stores shooting data and functions as a work area for processing by the system control unit 146. It also temporarily stores the camera control program, image processing program, etc. stored in the flash memory, etc. Note that part of the memory 136 (RAM) may be built into the system control unit 146.

[0061] The digital signal processing unit 138 performs signal processing such as offset processing, gamma correction processing, demosaic processing, and RGB / YCrCb conversion processing on the image obtained by shooting, thereby generating image data.

[0062] The input / output interface 140 includes a connection unit connectable to an external display device, a connection unit connectable to an external recording device, a card connection unit for inserting and removing a memory card, a communication unit connectable to a network, etc. For example, the input / output interface 140 may be a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), or the like.

[0063] The display unit 142 is used as a playback monitor when playing back captured images, a live view monitor that displays live view images during capture, and a setting monitor when performing various settings. The display unit 142 is configured with a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode), for example.

[0064] The operation unit 144 is configured to include various operation members for operating the camera 10. The operation members include a power button, a shutter button, and various other operation buttons. The various operation buttons include a button for turning the function of the image stabilization mechanism 150 on and off. Furthermore, if the display unit 142 is configured as a display unit with a touch panel, the operation members that make up the operation unit 144 include the touch panel. The operation unit 144 outputs signals corresponding to the operation of each operation member to the system control unit 146. For example, the user can set the conditions for FBK photography from the operation unit 144.

[0065] The system control unit 146 performs overall control of the camera 10. The system control unit 146 also calculates various physical quantities required for control. The system control unit 146 is configured, for example, by a microcomputer equipped with a processor, memory, etc. The processor is configured, for example, by a CPU (Central Processing Unit), etc. The system control unit 146 can execute a program for FBK photography. The system control unit 146 is responsible for controlling the image stabilization mechanism 150, the distance measurement mechanism 160, etc.

[0066] The camera 10 may include a shake correction mechanism 150. The shake correction mechanism 150 is a BIS (Body Image Stabilizer) control type shake correction mechanism that corrects shake by shifting (including rotating) the image sensor 130 in a plane perpendicular to the optical axis in the direction opposite to the shake direction. The shake correction mechanism 150 includes a shake control unit 152, an image sensor drive unit 154, a shake detection unit 156, and a position detection unit 158.

[0067] The image sensor driver 154 may include an actuator that moves the image sensor 130. The shake detector 156 may include an acceleration sensor, a gyro sensor, etc. The position detector 158 may include a Hall element that generates a voltage signal corresponding to the position of the image sensor 130. The shake controller 152 controls the image sensor driver 154 based on the signals from the shake detector 156 and the position detector 158 to shift the image sensor 130 in a plane perpendicular to the optical axis so as to offset shake of the camera 10. The shake correction mechanism may be provided in the lens apparatus 100.

[0068] The camera 10 may include a ranging mechanism 160. The ranging mechanism 160 can measure the distance or direction to an object. The ranging mechanism 160 includes a ranging device 162 and a ranging device control unit 164. The ranging device 162 is driven to operate by the ranging device control unit 164. The ranging mechanism 160 is, for example, a light detection and ranging (LiDAR). The LiDAR includes a laser light source and an optical sensor. The LiDAR irradiates the object with laser light and detects the laser light that hits the object 30 and bounces back using the optical sensor. The LiDAR measures the time it takes for the laser light to hit the object 30 and bounce back. This allows the LiDAR to measure the distance and direction to the object 30. The camera 10 is an example of an optical device of the present invention.

[0069] [Computer] FIG. 4 is a block diagram showing an example of the hardware configuration of the computer 20. As shown in FIG. 4, the computer 20 is configured to include a CPU 200, a RAM 202, a ROM 204, an auxiliary storage device 206, an input / output interface (IF) 207, an input device 208, and a display device 209. The ROM 204 and / or the auxiliary storage device 206 store programs executed by the CPU 200 and various data. The auxiliary storage device 206 is configured, for example, by a hard disk drive (HDD) or a solid state drive (SSD). The input device 208 is configured, for example, by a keyboard, a mouse, a touch panel, etc. The display device 209 is configured, for example, by an LCD or an OLED. Images captured by the camera 10 are imported into the computer 200 via the input / output interface 207 and stored, for example, in the auxiliary storage device 206.

[0070] The computer 20 stores in the auxiliary storage device 206 a program for generating a 3D model from multiple images of the object 30. The CPU 200 executes the program to generate a 3D model by photogrammetry.

[0071] [Method of Generating a Three-Dimensional Model] Next, a method of generating a three-dimensional model using the camera 10, the computer 20, and the photographing stand 40 will be described. Fig. 5 is a flowchart showing the method of generating a three-dimensional model (information processing method).

[0072] 5, a user prepares a camera 10 capable of FBK photography, a computer 20 capable of depth stacking and 3D model creation, and an object 30. The user places the object 30 on a photography stand 40 (step S1).

[0073] The user holds the camera 10 and moves to a shooting position to start shooting (step S2). The user fixes the object 30 to the shooting stand 40, and while holding the camera 10 and moving around the object 30, the user takes FBK images of the object 30 from each shooting direction. The user can arbitrarily determine the shooting position to start shooting. Here, the shooting position is defined as P, and a specific shooting position is defined as Pm (m is an integer). For example, P1 indicates the shooting start position.

[0074] The user takes an FBK image of the object 30 with the camera 10 at the shooting position (step S3). The user sets the camera 10 to a mode for capturing two-dimensional images for generating a 3D model. At the shooting position (P1), the user captures the object 30 placed on the shooting stand 40 in the shooting range with the camera 10 from a shooting direction (θ1). Here, the shooting direction is θ, and the shooting direction at a specific shooting position Pm is θm. The shooting direction (θm) is an angle that represents the relative positional relationship between the camera 10 and the object 30 at the shooting position (Pm).

[0075] When the user half-presses the shutter button of the camera 10, the camera 10 sets the FBK shooting conditions for that shooting direction based on the information about the shooting equipment of the camera 10 and the analysis results of the live view image.

[0076] The steps by which the camera 10 automatically determines the FBK imaging condition settings will be described below. The system control unit 146 of the camera 10 analyzes the live view image and extracts the area of ​​the object 30. A known contour extraction process can be applied to the area extraction. The camera 10 may exclude areas outside the area of ​​the object 30 from the processing target. This can be done by masking or masking.

[0077] The focus lens of camera 10 is moved to change the focal position (focus position), and the distance that is the farthest from camera 10 and the distance that is the nearest from camera 10 are found among the focal positions where the contrast within the outline is greatest. As a result, camera 10 acquires the distance between camera 10 and the focal position for focusing on the farthest point of object 30 (referred to as the farthest focal position distance) and the distance between camera 10 and the focal position for focusing on the nearest point of object 30 (referred to as the nearest focal position distance). In other words, the distance between camera 10 and the focal position at the nearest point where the focal position is focused on some part of object 30, and the distance at the farthest side, are found while changing the focal position. It is also possible to find the nearest focal position where no part of object 30 is focused, and the nearest focal position.

[0078] The system control unit 146 calculates the depth of field from the F-number setting of the lens group 104, the innermost focus position distance, and the innermost focus position distance. Next, the system control unit 146 calculates the necessary focus position and the number of times to capture images, and sets the capture conditions for FBK capture. When the setting of the capture conditions for FBK capture is completed, the camera 10 may notify the user that the setting is complete.

[0079] When the user realizes that photography is now possible, he or she fully presses the shutter button of the camera 10, and the camera 10 performs FBK photography at the set focus position and number of shots. The system control unit 146 controls the lens drive unit to move the focus lens and capture images of the object 30 at two or more different focus positions.

[0080] FIG. 6 is a diagram showing an example of FBK photography. FIG. 6 shows a case where four images are captured by FBK photography from one shooting direction at a shooting position. FP indicates the focus position, and D indicates the depth of field. 6-1 in FIG. 6 shows a case where the focus position of the captured image is the foreground focus position FP determined by the system control unit 146. 6-4 in FIG. 6 shows a case where the focus position of the captured image is the background focus position determined by the system control unit 146.

[0081] As shown in 6-1 in Fig. 6, the focus position is determined to be the foreground focus position, and the camera 10 captures the first image of the object 30. Next, as shown in 6-2 in Fig. 6, the camera 10 captures the second image of the object 30 at the next focus position in accordance with the shooting conditions. The focus positions for the second and subsequent images are calculated by the system control unit 146 as step amounts from the focus position and the number of times shooting has been performed.

[0082] Next, as shown in 6-3 of FIG. 6, the camera 10 captures a third image of the object 30 at the next focus position. Finally, as shown in 6-4 of FIG. 6, the camera 10 captures a fourth image of the object 30 at the farthest focus position, thereby completing FBK photography. The camera 10 may notify the user that FBK photography has ended. The focus position may be moved from the nearest focus position to the farthest focus position, or from the farthest focus position to the nearest focus position, or may be moved randomly.

[0083] The FBK images captured by FBK photography are associated with the photography direction (θ) and the number of the image, and are stored in memory 136. For example, the photography direction (θ) and the FBK images are associated and stored as shown by Img(θ,1), Img(θ,2), Img(θ,3)... Img(θ,n) (n is an integer indicating the number of images captured).

[0084] Furthermore, when storing an image, the camera 10 may also store information (meta information) related to the shooting conditions. The meta information may include shooting direction data of the object 30 acquired from a sensor or the like of the camera 10. When the ranging mechanism 160 measures the distance to the object 30 in the shooting direction, the meta information may include the distance to the object 30.

[0085] Next, after step S3, it is determined whether or not the photographing of the object 30 has been completed (step S4). If the result of this determination is that the photographing has not been completed (step S4: No), the camera 10 is moved to the next photographing position (P) (step S5). Then, returning to step S3, the user photographs the object 30 with the camera 10 in the photographing direction (θ) at the next photographing position (P) using FBK (step S3). The processes from step S3 to step S5 are repeatedly executed until the photographing of the object 30 is completed.

[0086] In step S4, the system control unit 146 may determine whether the image capturing operation in the set image capturing direction has been completed. The system control unit 146 may also display images that have already been captured on the display unit 142 so that the user can easily determine whether the image capturing operation has been completed.

[0087] On the other hand, if the photographing of the object 30 has been completed (step S4: Yes), the camera 10 ends the FBK photographing.

[0088] Next, the computer 20 acquires two or more images (FBK images) with different focus positions for each of the multiple shooting directions of the object 30 stored in the camera 10 (step S6). The computer 20 acquires the FBK images via the input / output interface 207 from the memory 136 of the camera 10 via a wired or wireless connection, or from a recording medium such as a memory card.

[0089] All FBK images are stored in the auxiliary storage device 206 of the computer 20, with the imaging direction (θ) and the image order associated with each other as shown in Table 1 below. In Table 1, the number of images in each imaging direction is n, but the number of images in each imaging direction may be different.

[0090]

[0091] Next, the computer 20 performs depth stacking on the FBK images for each shooting direction to generate a depth stacked image for each shooting direction (step S7). Note that the depth stacking technique itself is a well-known technique. The outline of the process is as follows.

[0092] 7 shows functional blocks related to the generation of depth stacked images by the CPU 200. As shown in Fig. 7, the image acquisition unit 210 acquires all FBK images (Img(θ1,1), Img(θ1,2) ... Img(θ1,n), Img(θ2,1), Img(θ2,2) ... Img(θ2,n), ... Img(θm,1), Img(θm,2) ... Img(θm,n)) for each shooting direction from the auxiliary storage device 206.

[0093] The alignment unit 211 aligns the positions of multiple FBK images used for focus stacking for each shooting direction. For example, if camera shake or the like occurs, multiple FBK images may be aligned in the same coordinate system using a geometric transformation such as an affine transformation. The presence or absence of camera shake may be determined, for example, from information from the shake detection unit 156 of the camera 10.

[0094] The region extraction unit 212 of the CPU 200 extracts in-focus pixels (in-focus regions) from the FBK images for each shooting direction (for example, Img(θ1,1), Img(θ1,2) ... Img(θ1,n)). When extracting in-focus regions, the region extraction unit 212 determines the region with the highest contrast value as the in-focus region among regions corresponding to the same position in multiple FBK images in the shooting direction, and extracts this region for generating a depth stacking image.

[0095] The depth-combined image generation unit 213 combines the extracted in-focus areas of each FBK image to generate a single depth-combined image (Img_syn(θ1), Img_syn(θ2) ... Img_syn(θm)) that is in focus overall. The methods for extracting and combining in-focus areas are not particularly limited, and known techniques can be applied. When depth composition is completed for each shooting direction, the CPU 200 stores the depth-combined image (Img_syn(θ1), Img_syn(θ2) ... Img_syn(θm)) in the auxiliary storage device 206.

[0096] Next, the computer 20 generates a 3D model from the multiple depth stacked images (step S8). Note that the technology for generating a 3D model using photogrammetry is a well-known technology. The outline of this process is as follows.

[0097] 8 shows functional blocks related to 3D model generation by the CPU 200. As shown in Fig. 8, the image acquisition unit 220 acquires depth stacked images (Img_syn(θ1), Img_syn(θ2), ... Img_syn(θm)) for each shooting direction from the auxiliary storage device 206.

[0098] The point cloud data generation unit 221 analyzes multiple depth-stitched images and generates three-dimensional point cloud data of feature points. The point cloud data generation unit 221 extracts feature points from each depth-stitched image. Next, the point cloud data generation unit 221 sets corresponding feature points between different depth-stitched images as corresponding points and matches the corresponding points. The point cloud data generation unit 221 estimates camera parameters (e.g., fundamental matrix, fundamental matrix, internal parameters, etc.) of the camera and estimates the shooting position and orientation based on the estimated camera parameters. Then, the three-dimensional positions of the feature points of the object 30 are determined. Bundle adjustment is performed as necessary. Point cloud data (point cloud) is generated by combining the three-dimensional coordinates of the estimated feature points.

[0099] The 3D patch model generation unit 222 performs processing to generate a 3D patch model of the subject based on the three-dimensional point cloud data of the object 30 generated by the point cloud data generation unit 221. Specifically, patches (meshes) are generated from the generated three-dimensional point cloud to generate a 3D patch model. This makes it possible to represent the undulations of the surface with a small number of points.

[0100] The 3D model generation unit 223 generates a textured 3D model (three-dimensional model) by performing texture mapping on the 3D patch model generated by the 3D patch model generation unit 222. The 3D model generation unit 223 maps the texture onto the mesh, thereby imparting the realistic appearance of the object 30 to the 3D patch model.

[0101] The data of the generated 3D model is stored in the auxiliary storage device 206, etc. Furthermore, the data of the 3D model is displayed on the display device 209 as necessary.

[0102] In the embodiment, the case of the system 1 has been described in which the object 30 is placed on the imaging stand 40 and the camera 10 moves around the object 30. However, the present invention is not limited to this, and a 3D model can also be generated in the system 2.

[0103] In the embodiment, a 3D model is generated by photogrammetry from multiple depth synthesis images that are in focus from the foreground to the background of the object 30, so a highly accurate 3D model can be generated.

[0104] When creating a 3D model, distance information between two marks 41 on the imaging stand 40 can be provided as a reference for the size of the 3D model. The positional relationship between the camera 10 and the object 30 may be determined based on the multiple marks 41 on the imaging stand 40, and a 3D model may be generated. The computer 20 is an example of an information processing system of the present invention.

[0105] [Preferred Embodiment] Next, a preferred embodiment will be described. In a preferred embodiment, the computer 20 identifies FBK images to be excluded when generating a depth stacking image. The computer 20 also identifies depth stacking images to be excluded when generating a 3D model.

[0106] 9 shows a flowchart for identifying FBK images to be excluded when generating a depth stacking image. As shown in FIG. 9, a plurality of FBK images are acquired in each shooting direction (step S11). Specifically, the image acquisition unit 210 acquires a plurality of FBK images.

[0107] Next, the multiple FBK images are aligned in each of the shooting directions (step S12). Specifically, the alignment unit 211 performs the alignment. For example, a reference image is selected from the multiple FBK images, and feature points are extracted from this reference image. The positions to which corresponding points in the reference image have moved in the remaining FBK images are tracked. Based on the results, the remaining FBK images are subjected to translation, rotation, enlargement, and reduction processing using affine transformation, etc. The multiple FBK images are aligned. This makes it possible to align the FBK images even if there is some camera shake, etc.

[0108] Next, the FBK images that could not be aligned are excluded from the targets of focus stacking (step S13). Specifically, the alignment unit 211 excludes the FBK images from the targets of focus stacking. The alignment unit 211 can determine that the FBK images that could not be aligned with the reference image in step S12 are not to be applied to focus stacking. The alignment unit 211 associates information indicating that the FBK images are not to be applied to focus stacking with the FBK images, and stores the information in the auxiliary storage device 206.

[0109] Next, an in-focus area is extracted from the FBK image (step S14). Specifically, the area extraction unit 212 extracts an in-focus area from the FBK image. As described above, among areas of the FBK image corresponding to the same position, the area with the highest contrast value is determined as the in-focus area and extracted for generating a depth stacking image. However, the FBK image does not include FBK images excluded from the focus stacking target. This shortens the time required to extract the in-focus position.

[0110] Next, the FBK image for which the focus position could not be extracted is excluded from the focus stacking targets (step S15). Specifically, the region extraction unit 212 can determine not to apply (exclude) the FBK image for which the focus position could not be extracted in step S14 to focus stacking. The region extraction unit 212 associates information indicating that the FBK image will not be applied to focus stacking with the FBK image and stores the information in the auxiliary storage device 206.

[0111] Finally, the in-focus areas extracted from each FBK image are combined to generate a depth-combined image (step S16). Specifically, the depth-combined image generation unit 213 generates the depth-combined image. However, the FBK image does not include the FBK images excluded from the depth-combining target in step S13 or step S15. The depth-combined image generation unit 213 can also extract the presence or absence of camera shake (shooting conditions) in the FBK image and exclude images determined to have camera shake from the depth-combining target. The presence or absence of camera shake can be determined from the high-frequency components of the FBK image, etc. In the step of excluding an FBK image from the depth-combining target, it can be determined whether to exclude it based on the image evaluation of the FBK image.

[0112] By excluding images that are not suitable for depth stacking, high-quality depth stacking images can be generated, and therefore high-quality 3D models can be generated from high-quality depth stacking images.

[0113] In the flow shown in FIG. 9 , when an FBK image to be excluded from the focus stacking target is determined, the computer 20 can also prompt the camera 10 to re-capture the image. Information on the position and orientation of the camera 10 that captured the FBK image can be obtained from information accompanying the FBK image when it is acquired. The computer 20 can notify the user of the position and orientation of the camera 10. The user can easily know the location of the object 30 for which the FBK image needs to be re-captured.

[0114] 10 shows a flowchart for specifying FBK images to be excluded when generating a 3D model. As shown in FIG. 10, a depth stacking image is acquired (step S21). Specifically, the image acquisition unit 220 acquires a plurality of depth stacking images.

[0115] Next, feature points are extracted from the plurality of depth stacked images (step S22). The feature points are detected as corresponding points from the plurality of depth stacked images (step S23). Specifically, the point cloud data generation unit 221 extracts feature points from the plurality of depth stacked images and compares the feature points of the plurality of depth stacked images to detect matching feature points as corresponding points.

[0116] Next, the depth stacked image is excluded from the target of the 3D model (step S24). Specifically, the point cloud data generation unit 221 can determine not to apply a depth stacked image for which feature points cannot be extracted in step S22 or a depth stacked image for which corresponding points cannot be detected in step S23 to the 3D model. Furthermore, the point cloud data generation unit 221 can determine an image to be excluded from the target of the 3D model based on high-frequency components of the depth stacked image, etc. The point cloud data generation unit 221 associates information indicating that the image will not be applied to the generation of the 3D model with the depth stacked image and stores the information in the auxiliary storage device 206.

[0117] Next, a 3D model is generated from the depth composite image (step S25). Specifically, the point cloud data generation unit 221 generates point cloud data from the depth composite image by combining the camera parameters, the shooting position and orientation, and the three-dimensional coordinates of the estimated feature points, and the 3D patch model generation unit 222 and the 3D model generation unit 223 generate a 3D model. However, the depth composite image does not include depth composite images that have been excluded from the target of the 3D model. Since depth composite images that are not suitable for generating a 3D model are excluded, a high-quality 3D model can be generated.

[0118] 10 , when determining a depth stacking image to be excluded from the target of the 3D model, the computer 20 can also prompt the camera 10 to re-capture the image. Information on the position and orientation of the camera 10 that captured the FBK image applied to the depth stacking image can be acquired from information accompanying the depth stacking image. The computer 20 can notify the user of the position and orientation of the camera 10. The user can easily know the location of the object 30 for which the FBK image required to generate the depth stacking image needs to be re-captured.

[0119] <Setting Conditions for FBK Photography> The determination of the setting conditions for FBK photography will be described.

[0120] Depth of field (DOF) refers to the area within a certain range in front and behind the in-focus position (focus point) in a captured image. Within this range, the object is captured relatively clearly, resulting in an in-focus image. On the other hand, areas outside this range are blurred, resulting in an out-of-focus image. Depth of field is a value that is physically and optically determined by the lens focal length, pixel size (the size of one pixel on the image sensor), camera F-number (aperture value), the distance from the pixel sensor to the object 30 (focus position), and the permissible circle of confusion. Therefore, depth of field changes depending on the F-number and the distance between the object and the camera.

[0121] In FBK photography, the same object is photographed at multiple focus positions, multiple FBK images are obtained, and the multiple FBK images are depth-composited to obtain an image with an expanded depth of field for the object (depth-composited image).

[0122] In FBK photography, when shooting at multiple focus positions (assuming the distance between the subject and the camera is fixed), if the F-number is increased (narrowed), the depth of field becomes deeper. As a result, the number of FBK images taken decreases, reducing the effort required for FBK photography and the amount of movement between FBK images.

[0123] On the other hand, since the amount of light reaching the image sensor is reduced, it becomes necessary to take photographs under one of the following shooting conditions: (1) increase the shutter speed and, instead, do not increase the ISO sensitivity to suppress high-sensitivity noise, or (2) decrease the shutter speed and, instead, increase the ISO sensitivity to suppress camera shake.

[0124] However, there is a trade-off between (1) and (2), so in the case of (1) there is the problem of camera shake, and in the case of (2) there is the problem of high-sensitivity noise.

[0125] Furthermore, the shooting environment also affects the shooting conditions. For example, if the lighting is bright, increasing the F-number or lowering the ISO sensitivity will not affect the image quality.

[0126] In the embodiment, the FBK shooting conditions are set for each shooting direction to reduce the effort required for shooting and maintain the quality of the generated 3D model. For example, the FBK shooting conditions can be determined taking into account the F-number, ISO sensitivity, and shutter speed.

[0127] If the depth of the object relative to the shooting direction is short, even if the depth of field is somewhat shallow (narrow), the number of images taken can be reduced, thereby reducing the amount of work required. Therefore, the computer 20 can set a small F-number, a short shutter speed (exposure time), and a low ISO sensitivity as the conditions for FBK shooting. The conditions for setting the FBK shooting conditions can include a depth of field that results in a small F-number, a short shutter speed, and a low ISO sensitivity within a range that reduces the number of FBK images taken.

[0128] These FBK imaging conditions can be stored in advance as a table in the auxiliary storage device 206 of the computer 20. Furthermore, learning data can be prepared in the CPU of the computer 20, and an AI (Artificial Intelligence) model that has been trained can be installed and trained.

[0129] The above description is based on the assumption that FBK photography is performed with different F-numbers and the like for each shooting direction. However, variations in image quality and the like for each shooting direction may affect the quality of the generated 3D model. Therefore, pre-shooting may be performed before FBK photography to determine common FBK photography settings according to the object 30 (regardless of the shooting direction). Furthermore, settings that optimize the FBK images obtained from each shooting direction may be set for each shooting direction. Alternatively, settings that optimize the images obtained from each shooting direction overall may be set as a common setting. FBK photography settings can be determined appropriately based on the purpose of 3D data generation.

[0130] As another example of setting the conditions for FBK photography, a beginner mode can be set. In beginner mode, to reduce the effort required for photography, the F-number is increased, the depth of field is deepened, and the number of shots taken is reduced. However, the incidence of camera shake is higher for beginners than for advanced users. Therefore, a condition for fast shutter speed can be set as a condition for FBK photography, even if some high-sensitivity noise occurs.

[0131] As another example, when setting the conditions for FBK photography, if the results of pre-photography show that the depth of the object 30 is quite narrow no matter where you photograph it, i.e., the object 30 is small, then even if you make the depth of field shallower, the number of shots will not increase, so you can reduce the F-number. Since the F-number can be reduced, you can shorten the shutter speed to prevent camera shake. Depending on the brightness of the surrounding lighting, you can also lower the ISO sensitivity to improve image quality. The conditions for FBK photography can also be set to achieve the best balance.

[0132] Next, a description will be given of a case where the object 30 is pre-photographed by the camera 10. Fig. 11 is a flowchart showing a method for generating a three-dimensional model, including pre-photographing.

[0133] Prepare a camera 10 capable of FBK photography, a computer 20 capable of depth stacking and 3D model creation, and an object 30. As shown in Fig. 11, the user places the object 30 on a photography stand 40 (step S41).

[0134] The user holds the camera 10 and moves to a shooting position for starting pre-shooting, fixes the object 30 to the shooting stand 40, and moves around the object 30 with the camera 10 while taking pre-shooting images of the object 30 from each shooting direction (step S42). Pre-shooting refers to taking images of the object 30 before FBK shooting (step S45). The pre-shooting may be a still image or a moving image of the object 30. It may also be a live view image. The distance between the object 30 and the camera 10 can also be obtained.

[0135] Next, the computer 20 acquires a pre-image captured by the camera 10 and acquires information for determining the conditions for FBK photography (step S43). Examples of information that can be acquired include the position and orientation of the camera 10 at a certain coordinate position and a pre-3D model of the pre-photographed object 30. The pre-3D model refers to a 3D model generated based on the pre-image before the 3D model is generated. The pre-3D model does not require high accuracy as long as the approximate size of the object 30 can be grasped. When generating the pre-3D model, for example, if rapid photography is desired, it is preferable to use a parallax volume intersection method, which has high processing speed. Note that the parallax volume intersection method itself is a well-known technique that estimates a 3D model based on silhouette information of the object 30. The acquired information is stored in the auxiliary storage device 206.

[0136] The user takes the camera 10 and moves to a shooting position to start shooting (step S44). At the shooting position, the user uses the camera 10 to shoot FBK images of the object 30 (step S45). The pre-3D model acquired in step S43 has a completed, actual three-dimensional shape. Therefore, the actual depth distance when viewing the object 30 from each direction can be obtained from the pre-3D model. When the user shoots FBK images of the object 30 from a certain direction, the FBK shooting conditions (e.g., depth of field, number of FBK images, etc.) can be automatically set based on these distances.

[0137] The distance to the object 30 is calculated by the CPU 200 of the computer 20. The setting for FBK photography is executed by the system control unit 146 (CPU) of the camera 10. However, this is not limited to these, and either the CPU 200 or the system control unit 146 may execute the setting.

[0138] FIG. 12 is a diagram illustrating an example of setting conditions for FBK photography. A pre-3D model is generated before FBK photography. In 12-1 of FIG. 12, the depth distance L1 of the object 30 in this direction is acquired from the pre-3D model. Because the distance L1 is relatively short, the camera 10 or the computer 20 determines, as the conditions for FBK photography, conditions such as the F-number, ISO sensitivity, and shutter speed that result in a shallow depth of field.

[0139] 12-2, the depth distance L2 of the object 30 in this direction is acquired from the pre-3D model. Because the distance L2 is relatively long, the camera 10 or the computer 20 determines the F-number, ISO sensitivity, shutter speed, and other conditions that result in a deep depth of field as the FBK shooting conditions. These FBK shooting conditions can reduce the number of FBK images.

[0140] Although the case where the depth of the object 30 is obtained from a pre-3D model has been described, the depth in each direction of the object 30 can also be obtained from a pre-image (for example, a 2D image: still image) or the like.

[0141] Next, thinning-out photography in FBK photography will be described. Thinning-out photography means that when FBK photography is performed from multiple photography directions, FBK photography is not performed from a certain photography direction. By performing thinning-out photography, the number of FBK photography operations can be reduced, thereby saving time and effort.

[0142] Fig. 13 is a diagram for explaining an example of thinning-out photography. 13-1 in Fig. 13 shows a state in which FBK photography of the object 30 is performed by the camera 10 from a first photography direction. 13-2 in Fig. 13 shows a state in which FBK photography of the object 30 is performed by the camera 10 from a second photography direction different from the first photography direction. 13-3 in Fig. 13 shows a state in which FBK photography of the object 30 is performed by the camera 10 from a third photography direction different from the first photography direction and the second photography direction.

[0143] For example, the camera 10 or the computer 20 can estimate FBK images obtained by FBK photography from the photography directions shown in the first photography direction (13-1) to the third photography direction (13-3) based on the pre-photography or pre-3D model. The FBK image obtained in the second photography direction (13-2) can be determined to be included in the FBK images obtained in the first photography direction (13-1) and the third photography direction (13-3).

[0144] The camera 10 or the computer 20 can determine that FBK images of the object 30 taken from the second imaging direction (13-2) are areas where imaging is not required. The camera 10 or the computer 20 can also display FBK images that are expected to be acquired from the first imaging direction (13-1) to the third imaging direction (13-3) on the display device 209. The user can determine areas where imaging is not required based on the expected FBK images.

[0145] The camera 10 or the computer 20 can calculate the coverage rate of the target area of ​​the object 30 based on the FBK images expected to be acquired from the first imaging direction (13-1) to the third imaging direction (13-3). The conditions for FBK imaging can be determined based on the calculation results. For example, the coverage rate can be used to determine whether the FBK image acquired from the second imaging direction (13-2) is included in the FBK images acquired from the first imaging direction (13-1) and the third imaging direction (13-3). If sufficient FBK images can be acquired from the first imaging direction (13-1) and the third imaging direction (13-3), the second imaging direction (13-2) can be excluded from the FBK imaging. Furthermore, the number of FBK images acquired when capturing FBK images from the second imaging direction (13-2) can be reduced.

[0146] If pre-photography has been performed or a pre-3D model has been generated, it is preferable to photograph the object 30 with the camera 10 from the position and direction of the pre-photography when performing FBK photography (step S45). This makes it easier to use information from the pre-photography or pre-3D model.

[0147] The procedure for thinning out FBK photography is described in 13-1 to 13-3 of FIG. 13 . Next, another procedure for determining FBK photography conditions will be described using the same diagram. FBK photography is performed from the photography directions shown in the first photography direction (13-1) to the third photography direction (13-3). First, the conditions for FBK photography from the first photography direction (13-1) are determined. Next, the conditions for FBK photography from the second photography direction (13-2) are determined. At this time, it can be determined that the second photography direction (13-2) has not changed significantly compared to the first photography direction (13-1). As a result, the conditions for FBK photography from the second photography direction (13-2) can be made the same as the conditions for FBK photography from the first photography direction (13-1). The conditions for FBK photography from the second photography direction (13-2) can be determined without spending a lot of time.

[0148] On the other hand, it can be determined that the third shooting direction (13-3) has changed significantly with respect to the first shooting direction (13-1). The FBK shooting conditions from the third shooting direction (13-3) are determined without being affected by the FBK shooting conditions from the first shooting direction (13-1). In other words, based on the FBK shooting conditions determined in one shooting direction, the FBK shooting conditions in a different direction can be determined. Note that whether or not there has been a significant change can be determined by tracking feature points, etc.

[0149] FIG. 14 is a diagram showing an example of matching between an object 30 and a still image (2D image).

[0150] For example, 14-1 in FIG. 14 illustrates a state in which a user is attempting to perform FBK photography of an object 30 with the camera 10. A live view image 31 of the object 30 is displayed on the display unit 142 of the camera 10. The camera 10 stores in the memory 136 still images (2D images) captured for pre-photography or pre-3D modeling. The camera 10 uses template matching to determine and display a 2D image 32 that most closely resembles the image of the object 30 displayed in the live view image 31. Information on the position and direction of the camera 10 when the 2D image 32 was captured is acquired. This allows the current distance between the object 30 and the camera 10 to be determined, which can be used to determine the conditions for FBK photography. While the camera 10 displays the 2D image 32, the 2D image 32 may not be displayed by notifying the user that template matching is complete.

[0151] 14-2 in FIG. 14 shows a state in which a user is attempting to perform FBK photography of an object 30 with the camera 10. A live view image 31 of the object 30 is displayed on the display unit 142 of the camera 10. As shown in 14-2, the display unit 142 displays the outline of a 2D image 33. The 2D image 33 prompts the user to move relative to the object 30 in a direction (indicated by an arrow) that matches the outline of the 2D image 33. This allows the current distance between the object 30 and the camera 10 to be determined, which can be used when determining the conditions for FBK photography.

[0152] Next, the determination of other preferable conditions for FBK photography will be described. Fig. 15 is a diagram for explaining the procedure for determining other preferable conditions for FBK photography.

[0153] As shown in 15-1 of Fig. 15, unlike the object 30, the object 30A has a characteristic portion 30B. As shown in 15-2 of Fig. 15, in the first shooting direction, the camera 10 extracts the characteristic portion 30B based on the live view image, which is a preliminary image, focuses on the characteristic portion 30B, and uses that position as a reference. The FBK shooting conditions are determined by a method that shifts the focus position forward or backward using the characteristic portion 30B as a reference.

[0154] Similarly, as shown in 15-3 of FIG. 15 , in the second shooting direction, the camera 10 extracts the feature portion 30B based on the live view image, which is a preliminary image, focuses on the feature portion 30B, and uses that position as a reference. The FBK shooting conditions are determined by a method that shifts the focus position forward or backward using the feature portion 30B as a reference. That is, as the FBK shooting conditions, the initial focus position of the FBK image for creating a depth stacked image is determined to be the feature portion 30B. Because the feature portion 30B is an important element when generating a 3D model, by setting the initial focus position of the FBK image to the feature portion 30B, it is possible to obtain a highly accurate depth stacked image that is important for generating a 3D model.

[0155] Next, the determination of other preferable FBK imaging conditions will be described. FIG. 16 is a diagram illustrating the procedure for determining other preferable FBK imaging conditions. In FIG. 16, FBK imaging of the object 30 is performed from a first imaging direction and from a second imaging direction. When FBK imaging is performed from the first imaging direction, a distance L3 of the width of the object 30 can be obtained. Next, when FBK imaging is performed from the second imaging direction, the distance L3 can be used to determine the number of FBK images, the depth of field, the imaging interval, etc.

[0156] 11 , after step S45, it is determined whether or not the photographing of the object 30 has been completed (step S46). If the result of this determination is that the photographing has not been completed (step S46: No), the camera 10 is moved to the next photographing position (P) (step S47). At the next photographing position (P), the user photographs the object 30 using the camera 10 in the photographing direction (θ) using FBK (step S45). The processes from step S45 to step S47 are repeated until the photographing of the object 30 is completed.

[0157] If the photographing of the object 30 has been completed (step S46: Yes), the camera 10 ends the FBK photographing.

[0158] Next, the computer 20 acquires two or more images (FBK images) with different focus positions for each of the multiple shooting directions of the object 30 stored by the camera 10 (step S48).

[0159] Next, the computer 20 performs depth stacking on the FBK images for each shooting direction to generate a depth stacked image for each shooting direction (step S49).

[0160] Next, the computer 20 generates a 3D model from the plurality of depth stacked images (step S50). When the generation of the 3D model is complete, the process ends.

[0161] In the above embodiment, systems 1 and 2 that constitute a 3D model generation device that generates a 3D model have been described, including the camera 10 and the computer 20. The camera 10 includes a system control unit 146, and the computer 20 includes a CPU, and the functions of the system control unit 146 and the CPU 200 are not particularly limited. For example, the determination of FBK shooting conditions may be executed by either the CPU 200 or the system control unit 146. In other words, as long as the camera 10 and the computer 20 can generate a 3D model, it is sufficient for either the CPU 200 or the system control unit 146 to fulfill its functions.

[0162] The present invention also includes an information processing system that generates a 3D model configured by a computer 20. It also includes an optical device that performs FBK shooting, depth stacking of FBK images, and generation of a 3D model using only a camera 10. It also includes a 3D model generation device that is configured by a camera 10 (optical device) and a computer 20 (information processing device). Furthermore, although the example has been given of FBK shooting using one camera 10, FBK shooting can also be performed using multiple cameras 10.

[0163] [Configuration of the system control unit 146 and CPU 200] The functions of the system control unit 146 and CPU 200 are realized by various processors. The various processors include a CPU, which is a general-purpose processor that executes a program and functions as various processing units, a GPU (Graphic Processing Unit), a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically to execute specific processing. The term "program" is synonymous with "software."

[0164] A single processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Furthermore, multiple processing units may be composed of a single processor. Examples of multiple processing units composed of a single processor include: a first configuration, as typified by computers used as clients or servers, in which a single processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple processing units; a second configuration, as typified by system-on-chip (SoC), in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip; and a hardware structure, in which the various processing units are composed of one or more of the above-mentioned various processors.

[0165] Although the present invention has been described above, the present invention is not limited to the above examples, and various improvements and modifications may be made without departing from the spirit of the present invention.

[0166] 1 System 2 System 10 Camera 20 Computer 30 Object 30A Object 30B Feature part 31 Live view image 32 2D image 33 2D image 40 Shooting stand 41 Mark 42 Shooting stand 43 Mark 44 Tripod 100 Lens device 102 Imaging optical system 104 Lens group 106 Aperture 110 Lens driving unit 112 Aperture driving unit 130 Imaging element 132 Shutter 134 Shutter driving unit 136 Memory 138 Digital signal processing unit 140 Input / output interface 142 Display unit 144 Operation unit 146 System control unit 150 Shake correction mechanism 152 Shake control unit 154 Imaging element driving unit 156 Detection unit 158 ​​Position detection unit 160 Distance measurement mechanism 162 Distance measuring device 164 Distance measuring device control unit 200 CPU 202 RAM 204 ROM 206 Auxiliary storage device 207 Input / output interface 208 Input device 209 Display device 210 Image acquisition unit 211 Alignment unit 212 Area extraction unit 213 Depth synthesis image generation unit 220 Image acquisition unit 221 Point cloud data generation unit 222 3D patch model generation unit 223 3D model generation unit D Depth of field FP Focus position L1 Distance L2 Distance L3 Distance

Claims

1. An information processing system having one or more processors, wherein the one or more processors acquire two or more images with different focus positions for each of a plurality of shooting directions of an object, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-combined image, and generate a 3D model of at least a portion of the object based on the depth-combined image acquired for each shooting direction.

2. The information processing system according to claim 1, wherein the one or more processors determine whether or not the image is suitable for use in generating the 3D model.

3. The information processing system according to claim 1, wherein the one or more processors determine whether or not the image is suitable for use in generating the depth stacking image.

4. The information processing system according to claim 1, wherein the one or more processors determine whether or not the depth synthesis image is to be applied to the generation of the 3D model.

5. An information processing system according to any one of claims 2 to 4, wherein the one or more processors perform image evaluation to determine suitability.

6. The information processing system according to claim 5, wherein the one or more processors instruct the image to be re-taken based on the result of the image evaluation or the suitability judgment.

7. The information processing system according to claim 1, wherein the one or more processors determine conditions for acquiring images with different focus positions of the object.

8. The information processing system according to claim 7, wherein the one or more processors determine the condition based on a pre-image taken of the object before the image is taken.

9. The information processing system according to claim 8, wherein the one or more processors determine a width of the object based on the preliminary image, and determine the condition based on the width.

10. The information processing system of claim 8, wherein the one or more processors obtain a pre-3D model of the object based on the pre-image, and determine the condition based on the pre-3D model.

11. The information processing system according to claim 10, wherein the one or more processors generate the pre-3D model using a volume intersection method.

12. The information processing system according to claim 7, wherein the one or more processors calculate a shooting coverage rate of a target area for the object for each shooting direction, and determine the condition based on the calculation result.

13. The information processing system according to claim 7, wherein the one or more processors determine, for each shooting direction, an area of ​​the object that does not need to be photographed, and determine the conditions.

14. The information processing system according to claim 7, wherein the one or more processors determine the conditions in a direction different from the one direction based on the conditions determined in the one shooting direction.

15. The information processing system according to claim 8, wherein the one or more processors extract feature points of the object based on the preliminary image, and determine the condition based on the feature points.

16. The information processing system according to claim 15, wherein the one or more processors determine, as the condition, an initial focus position of the image for creating the depth synthesis image based on the feature points.

17. The information processing system according to claim 7, wherein the one or more processors determine the conditions based on a desired purpose for the 3D model.

18. The information processing system according to claim 7, wherein the one or more processors determine the condition based on distance information to the object.

19. A three-dimensional model generation device including an optical device capable of capturing images of an object and an information processing device, the three-dimensional model generation device comprising one or more processors included in either the optical device or the information processing device, the one or more processors acquiring two or more images having different focus positions for each of a plurality of shooting directions of the object, synthesizing the images having different focus positions acquired from the same shooting direction to generate a depth composite image, and generating a 3D model of at least a portion of the object based on the depth composite image acquired for each shooting direction.

20. The three-dimensional model generating device according to claim 19, wherein the one or more processors determine whether or not the image is suitable for application to the generation of the 3D model.

21. The three-dimensional model generating device according to claim 19, wherein the one or more processors determine whether or not the image is suitable for application to the generation of the depth synthesis image.

22. The three-dimensional model generating device according to claim 19, wherein the one or more processors determine whether or not the depth synthesis image is appropriate for generating the 3D model.

23. A three-dimensional model generating device according to any one of claims 20 to 22, wherein the one or more processors perform image evaluation to determine suitability.

24. The three-dimensional model generating device according to claim 23, wherein the one or more processors instruct the image to be re-taken based on the result of the image evaluation or the suitability determination.

25. The three-dimensional model generating device according to claim 19, wherein the one or more processors determine conditions for acquiring images with different focus positions of the object.

26. An optical device equipped with one or more processors and capable of photographing an object, wherein the one or more processors acquire two or more images with different focus positions for each of a plurality of shooting directions of the object, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth composite image, and generate a 3D model of at least a portion of the object based on the depth composite image acquired for each shooting direction.

27. The optical apparatus of claim 26, wherein the one or more processors determine conditions for acquiring images with different in-focus positions of the object.

28. An information processing method executed by an information processing system having one or more processors, wherein the one or more processors acquire images of an object with different focus positions from a plurality of shooting directions, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-composited image, and generate a 3D model based on the depth-composited image acquired for each shooting direction.

29. A program for executing an information processing method of an information processing system having one or more processors, the program causing the one or more processors to acquire images of an object with different focus positions from a plurality of shooting directions, synthesize the images with different focus positions acquired from the same shooting direction to generate a depth-combined image, and generate a 3D model based on the depth-combined image acquired for each shooting direction.

30. A non-transitory computer-readable recording medium having the program according to claim 29 recorded thereon.

Images