Information processing device, method for operating information processing device, and three-dimensional model generation system

Abstract

One aspect of the present invention provides an information processing device, a method for operating the information processing device, and a three-dimensional model generation system. An information processing device according to one aspect of the present invention comprises a processor. The processor acquires feature information indicating a feature of an object for which a three-dimensional model is to be generated, and outputs background determination information, which is information for determining a first background to be used in first photography for generating a three-dimensional model, on the basis of the feature information.

Description

Information processing device, operating method of information processing device, and three-dimensional model generation system

[0001] The present invention relates to an information processing device, an operating method of an information processing device, and a three-dimensional model generation system, and more particularly to photography for generating a three-dimensional model.

[0002] Regarding the generation of a three-dimensional model, for example, Patent Document 1 describes a technique for generating a three-dimensional model based on a plurality of images of an object taken from different directions.

[0003] International Publication No. 2023 / 223958

[0004] One embodiment of the technique of the present disclosure provides an information processing device, an operating method of the information processing device, and a three-dimensional model generation system.

[0005] An information processing device according to a first aspect of the present invention is an information processing device including a processor, which acquires feature information indicating the features of an object for which a three-dimensional model is to be generated, and outputs background determination information, which is information for determining a first background to be used in a first photographing operation for generating the three-dimensional model, based on the feature information.

[0006] In the first aspect, the "background determination information," i.e., "information for determining the first background," does not have to be the background image itself. Furthermore, in the first aspect and the following aspects, the terms "first," "second," etc. do not specify the chronological order of processing.

[0007] In the information processing device according to a second aspect of the present invention, the processor acquires feature information based on at least one image obtained by second photography of the object prior to the first photography.

[0008] An information processing device according to a third aspect is the second aspect, wherein the processor acquires feature information from a plurality of images obtained by performing second photography of the object from a plurality of directions.

[0009] An information processing device according to a fourth aspect is based on any one of the first to third aspects, and wherein the feature information is information about a surface of the object.

[0010] In the information processing device according to a fifth aspect, in any one of the first to fourth aspects, the processor acquires information indicating at least one of the color and pattern of the object as the feature information.

[0011] In the information processing device according to a sixth aspect, in any one of the first to fifth aspects, the processor outputs information indicating at least one of a color and a pattern of the first background as background determination information.

[0012] An information processing device according to a seventh aspect is the sixth aspect, in which the processor outputs background determination information that satisfies at least one of the following conditions: the color is different from the color of the object, and the pattern is different from the pattern of the object.

[0013] In the information processing device of the eighth aspect, in the seventh aspect, the processor distinguishes between the object and the first background in the first image obtained by the first photographing based on the fact that at least one of the color and pattern in the background determination information is different from at least one of the color and pattern of the object.

[0014] In an information processing device according to a ninth aspect, in any one of the first to eighth aspects, the processor selects a first background from among a plurality of backgrounds having predetermined colors and patterns based on background determination information.

[0015] In the information processing device according to a tenth aspect, in any one of the first to ninth aspects, the processor generates the first background based on background determination information.

[0016] An information processing device according to an eleventh aspect is the information processing device of the tenth aspect, wherein the processor generates two or more first backgrounds that differ in at least one of color and pattern.

[0017] An information processing device according to a twelfth aspect is any one of the first to eleventh aspects, wherein the processor acquires information according to the orientation of the object as the feature information.

[0018] In an information processing device according to a thirteenth aspect, in any one of the first to twelfth aspects, the processor causes the output device to output information prompting the user of the information processing device to take the first photograph in an environment corresponding to the background determination information.

[0019] A three-dimensional model generation system according to a fourteenth aspect includes an information processing device according to any one of the first to thirteenth aspects, a display device that displays a first background, a photographing device that performs a first photograph of an object against the first background to obtain a plurality of images, and a generation unit that generates a three-dimensional model of the object using the plurality of images.

[0020] A three-dimensional model generation system according to a fifteenth aspect is the fourteenth aspect, wherein the photographing device performs first photographing of the object from a plurality of directions to obtain a plurality of images.

[0021] A three-dimensional model generation system according to a sixteenth aspect is the fourteenth or fifteenth aspect, wherein the generation unit generates the three-dimensional model by referring to information indicating the relationship between the background determination information and the plurality of images.

[0022] A three-dimensional model generation system according to a seventeenth aspect is any one of the fourteenth to sixteenth aspects, wherein the generation unit generates the three-dimensional model taking into consideration differences between the background determination information and the characteristics of the object.

[0023] The three-dimensional model generation system according to an eighteenth aspect is any one of the fourteenth to seventeenth aspects, wherein the photographing device performs a first photographing operation and a second photographing operation of the object prior to the first photographing operation.

[0024] An operation method according to a 19th aspect is an operation method of an information processing device having a processor, in which the processor acquires feature information indicating the features of an object for which a three-dimensional model is to be generated, and outputs background determination information, which is information for determining a first background to be used in a first photographing to generate the three-dimensional model, based on the feature information.

[0025] In addition, a program that causes a computer to execute the operating method according to the nineteenth aspect, and a non-transitory tangible recording medium on which computer-readable code of such a program is recorded, can also be cited as aspects of the present invention.

[0026] FIG. 1 is a diagram showing the configuration of a 3D model generation system according to a first embodiment. FIG. 2 is a diagram showing the configuration of a controller. FIG. 3 is a diagram showing the functional configuration of a processor. FIG. 4 is a diagram showing an example of information recorded in a recording device. FIG. 5 is a flowchart showing processing in the 3D model generation system. FIG. 6 is a diagram showing an example of output of information suggesting an appropriate shooting environment. FIG. 7 is a diagram showing the acquisition of feature information and an example (part 1) of a background image based thereon. FIG. 8 is a diagram showing the acquisition of feature information and an example (part 2) of a background image based thereon. FIG. 9 is a diagram showing the acquisition of feature information and an example (part 3) of a background image based thereon. FIG. 10 is a diagram showing an icon display of a sample background image corresponding to background determination information. FIG. 11 is a diagram showing an example of a configuration for performing actual shooting from multiple directions. FIG. 12 is a diagram showing another example of a configuration for performing actual shooting from multiple directions.

[0027] [Generating 3D Models Using Photogrammetry] In recent years, technological advances have led to an increase in businesses and consumer services that utilize 3D (three-dimensional) model data. There are two main methods for generating 3D models: creating 3D CG (Computer Graphics) from scratch, or scanning the actual object. One of the methods for scanning the actual object is photogrammetry, which creates 3D models from photographs. The advantages of photogrammetry include its ability to precisely reproduce complex objects that would be difficult to create in 3D CG, and its ease of generating 3D models by simply taking a photograph. On the other hand, a disadvantage is that accuracy depends on the quality of the photograph.

[0028] When creating a 3D model from a photographed image, it is necessary to photograph the object from all directions. However, in order to improve the accuracy of the 3D model and reduce the time it takes to generate it, it is necessary to accurately grasp the object in the image. Depending on the color scheme and pattern of the object (real object) you want to create a 3D model of, it may be difficult to distinguish the boundary between the object and the background in the resulting image, which will result in a decrease in the quality of the 3D model during subsequent software processing.

[0029] In light of these circumstances, the inventors of the present application have conducted extensive research and have discovered that "a high-quality 3D model can be generated by determining a background that allows for accurate separation of the boundary between the object and the background in terms of color scheme, pattern, and shape, and by using this information (background determination information)." The present invention was created based on this discovery, and specific aspects of an information processing device, an operating method for an information processing device, and a 3D model generation system according to the present invention will be described below.

[0030] [First Embodiment] [Overall Configuration of 3D Model Generation System] Fig. 1 is a diagram showing the configuration of a 3D model generation system 10 (information processing device, 3D model generation system) according to a first embodiment. As shown in Fig. 1, the 3D model generation system 10 includes a controller 100 (processor, information processing device, generation unit), a photographing camera 200 (photographing device), a background display device 300 (display device), and a lighting device 400. The photographing camera 200 may include some or all of the configuration of the controller 100. Note that an object 99A (object) is an example of an object for which a 3D model is generated.

[0031] [Configuration of Controller] FIG. 2 is a diagram showing the configuration of the controller 100 (processor, information processing device, generation unit) in the first embodiment. As shown in FIG. 2, the controller 100 includes a processor 110 (processor, information processing device, generation unit), a ROM 130 (ROM: Read Only Memory, a non-transitory and tangible recording medium), a RAM 140 (RAM: Random Access Memory), an operation unit 150, a speaker 160 (output device), a display 170 (output device), an input / output interface 180, and a recording device 190, and these components are connected by a bus 195. The controller 100 can communicate with a photographing camera 200, a background display device 300, a lighting device 400, and various external devices via the input / output interface 180.

[0032] [Processor Configuration] Fig. 3 is a diagram showing the functional configuration of the processor 110. As shown in Fig. 3, the processor 110 includes a feature information acquisition unit 112, a background information determination unit 114, a background generation unit 116, a background selection unit 118, an imaging control unit 120, a background display control unit 122, an illumination control unit 124, a three-dimensional model generation unit 126, and an input / output control unit 128.

[0033] The processor 110 is configured with various processors and electrical circuits, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), a programmable logic device (PLD), etc. When these processors and electrical circuits execute software (programs), the software to be executed is stored in a non-transitory, tangible recording medium such as a code ROM 130 that can be read by a computer (for example, various processors and electrical circuits that constitute the processor, and / or a combination thereof), and the computer references the software.

[0034] The software stored in the non-transitory, tangible recording medium includes the program of the present invention (a program that causes a computer to execute the method for operating an information processing device and the method for generating a three-dimensional model of the present invention) and data used in executing the program. Instead of the ROM 130, the code may be recorded in a non-transitory, tangible recording medium such as a flash ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory). Note that this "non-transitory, tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves. During processing using the software, the RAM 140 is used as a temporary storage area or working area.

[0035] The functions of the processor 110 described above may be realized by various types of AI (Artificial Intelligence). Such AI may be, for example, AI that performs image recognition, segmentation, feature extraction, etc., or AI that generates an image from given information (for example, AI that generates a background image from background determination information).

[0036] [Configuration of Operation Unit and Display] The operation unit 150 is composed of devices such as a keyboard, mouse, buttons, and switches (not shown). The user can issue instructions to the controller 100 via these devices, and the processor 110 accepts the instructions and performs processing in accordance with the accepted instructions. The display 170 may be composed of a touch panel device so that the user can issue instructions via the touch panel. The display 170 is composed of such a touch panel device or a device such as a liquid crystal display device, and can display feature information of an object, conditions for determining background information, determined background information, generated and selected background images, captured images and videos, generated three-dimensional models, and the like. The display 170 can also display information recorded in the recording device 190.

[0037] [Configuration of Input / Output Interface] The input / output interface 180 is composed of terminals and slots for connecting external devices such as a display, printer, and recording medium, and communication interfaces such as Wi-Fi (registered trademark) and Bluetooth (registered trademark). The controller 100 can acquire image data, video data, background content, and the like from external devices (server devices, recording devices, databases, imaging devices, etc.) via the input / output interface 180. The external devices may be connected to the controller 100 via a wired or wireless connection. Furthermore, the external devices may be connected via a network such as the Internet.

[0038] [Configuration of the recording device] The recording device 190 (recording device) is composed of recording media (non-transitory and tangible recording media) such as semiconductor memories such as hard disks and SSDs (Solid-State Drives), and various types of magneto-optical recording media, and their control units, and records or saves various types of information.

[0039] FIG. 4 is a diagram showing an example of information recorded in the recording device 190. Feature information 190A (feature information) is information indicating the features of the object for which a 3D model is to be generated. Preliminary image 190B is an image obtained by preliminary shooting (second shooting) performed prior to shooting (first shooting) of an image (actually shot image 190C) used to generate a 3D model of the object. The actual shot image 190C is an image used to generate a 3D model of the object. Background determination information 190D is output based on the feature information of the object and is information used in the actual shooting (first shooting) to generate a 3D model. Background image 190E is an image generated or selected based on the background determination information 190D and can be displayed on the background display device 300. 3D model 190F is a 3D model of the object generated using multiple images acquired in the actual shooting.

[0040] [Configuration of Background Display Device] The background display device 300 (display device) is a device that displays a background image (first background) used in the main shooting (first shooting) under the control of the controller 100. The background display device 300 may be provided in the direction of a wall (behind or to the side of the object), in the direction of the ceiling (above the object), or in the direction of the floor (below the object). The background display device 300 may be composed of a monitor such as a liquid crystal display, an LED display, an organic EL display, or a plasma display, or may be composed of a projector and a screen. A single monitor may be used, or multiple monitors may be used in combination. Projection by the projector can be performed from the front, back, side, above, or below.

[0041] [Configuration of the Lighting Device] The configuration of the lighting device 400 (lighting device) is not particularly limited. For example, a lighting device configured with LEDs of multiple colors (e.g., red, blue, and green) can be used. The lighting device 400 is preferably a device that can change the brightness and color (e.g., color temperature) in addition to turning it on and off. It is also preferable that the lighting device 400 be a device that can change the illumination position and illumination direction. In this case, for example, a configuration similar to that of changing the viewpoint position and imaging direction of the imaging camera 200, such as a dolly 210 (described later with respect to part (a) of Figure 11), can be used. Furthermore, the lighting device 400 may be disposed not only above the object 99A as in the example of Figure 1, but also below (e.g., on the floor) or to the side. The number of lighting devices 400 is also not particularly limited. The processor 110 (lighting control unit 124) can control the illumination by the lighting device 400 in accordance with the control of the display of the background image (first background) on the background display device 300.

[0042] [Processing in the Three-Dimensional Model Generation System] FIG. 5 is a flowchart showing processing (acquisition of characteristic information, determination of background information, actual photography, generation of a three-dimensional model, etc.) in the three-dimensional model generation system 10 configured as described above.

[0043] [Acquisition of Feature Information] The processor 110 (feature information acquisition unit 112) acquires feature information indicating the features of the object 99A (the object for which a 3D model is to be generated) (step S100). The processor 110 can acquire information indicating at least one of the color and pattern of the object as the "feature information" of the object.

[0044] [Acquisition of Feature Information Based on Preliminary Captured Images] The processor 110 (feature information acquisition unit 112, capture control unit 120, background display control unit 122, lighting control unit 124, etc.) can acquire feature information based on at least one image obtained by preliminarily capturing an image of the object 99A (second capture; capture performed prior to the main capture for acquiring feature information). Multiple images may be acquired. The processor 110 may capture a moving image of the object 99A and acquire feature information based on the moving image.

[0045] The processor 110 (controller 100) can control the imaging camera 200, background display device 300, lighting device 400, etc. to perform preliminary imaging. The processor 110 may automatically determine the conditions for preliminary imaging (second imaging conditions) or may determine them in response to a user's instruction. This "automatic determination" includes selecting from among predetermined conditions. The conditions for preliminary imaging may also include, for example, the viewpoint position and imaging direction, background, lighting, exposure, background, etc. In preliminary imaging, a background including multiple colors, such as a color chart, may be used.

[0046] It is preferable that the processor 110 (controller 100) takes preliminary images of the object 99A from multiple directions and acquires feature information corresponding to each direction from the multiple images obtained. This is because the features of the object 99A and the appropriate background for them may differ depending on the direction. The processor 110 may take preliminary images of one direction under multiple conditions (e.g., multiple conditions with different background colors and / or patterns). If the object 99A has a complex pattern, using images taken under multiple conditions allows for appropriate feature extraction.

[0047] The preliminary photographing (second photographing) and the main photographing (first photographing) may be performed using the same photographing device (such as the photographing camera 200) or different photographing devices. The processor 110 can acquire information about and results of preliminary photographing performed using a photographing device other than the photographing camera 200 (including the conditions for the preliminary photographing and the photographed images) via a network (not shown) or the input / output interface 180. The processor 110 (such as the input / output control unit 128) can also display the information and results of the preliminary photographing on the display 170 and record them in the recording device 190.

[0048] [Acquisition of Feature Information Based on User Input] The processor 110 may acquire feature information based on a user input via the operation unit 150 .

[0049] [Contents of Feature Information] The processor 110 (such as the feature information acquisition unit 112) can acquire information about the surface of the object 99A as feature information. Specifically, the feature information may be information indicating at least one of the color and pattern of the object 99A. Alternatively, the feature information may be information indicating that the object does not contain a specific color and / or a specific pattern. When determining which colors the object 99A contains in abundance (or does not contain), the processor 110 can evaluate the statistical values ​​(such as variance values) of pixel values ​​in the image obtained by preliminary photography by setting a threshold value. By performing such quantitative color analysis, a background can be set according to the photography environment and for each user.

[0050] Regarding the characteristic information, the "color" may be a single color or multiple colors, or may include color variations (hues). Furthermore, the processor 110 may acquire, as characteristic information, whether the object 99A is transparent or translucent, based on a pre-captured image or a user input.

[0051] It is preferable that the processor 110 acquires information according to the orientation of the object as the feature information, and more specifically, it is preferable that the processor 110 acquires a plurality of pieces of information corresponding to a plurality of orientations of the object as the feature information. This is because, when generating a three-dimensional model, images are taken from a plurality of viewpoints and shooting directions, and the optimal background may differ depending on the viewpoints, shooting directions, and other conditions.

[0052] [Output of Background Determination Information] The processor 110 (background information determination unit 114) outputs background determination information based on the feature information of the object 99A (step S110). The background determination information is information for determining the background (first background) to be used in the main photographing (first photographing) to generate a 3D model. The processor 110 can use information indicating at least one of the color and pattern of the background as background determination information. The "background determination information" may be any information that can determine the background and does not have to be the background image itself. For example, the background may include information such as "green, solid color, no pattern" or "blue polka dots on a white background." Note that, as with the feature information described above, the "color" of the background determination information may be a single color or multiple colors, or may include color variations (hues).

[0053] The processor 110 preferably outputs background determination information such that at least one of the color and pattern of the background is different from at least one of the color and pattern of the object 99A. This means that at least one of the following conditions is met: "The color of the background is different from the color of the object 99A" and "The pattern of the background is different from the pattern of the object 99A." Based on this background determination information, when generating a 3D model using the actual captured image, the object and the background can be easily separated or identified. Furthermore, if information according to the orientation of the object is acquired as feature information as described above, the processor 110 preferably also outputs background determination information according to the orientation of the object, and the output background determination information is preferably recorded in association with the orientation of the object.

[0054] The processor can display the background determination information on the display 170 or record it in the recording device 190 (background determination information 190D in FIG. 4).

[0055] [Message to User] The processor 110 (background information determination unit 114, input / output control unit 128, etc.) can output a message regarding the background determination information (information encouraging the user of the 3D model generation system 10 (information processing device) to perform the actual photographing (first photographing) in an environment corresponding to the background determination information). For example, as shown in parts (a) and (b) of FIG. 6 , the processor 110 can display on the display 170 (output device) a screen 950 containing a message 952 saying, "The object does not appear to have any green areas. Using a green background will generate a better 3D model." Alternatively, the processor 110 can display a screen 960 containing a message 962 saying, "It would be a good idea to move to a place with lots of greenery and perform the actual photographing." In this case, the user can perform the actual photographing on, for example, a lawn. The processor 110 can output these messages as audio from the speaker 160 (output device).

[0056] [Generation or Selection of Background Image] The processor 110 (background information determination unit 114, background generation unit 116, background selection unit 118) determines whether to generate a background image (step S120). The processor 110 can make this determination by considering, for example, whether an image of a background determined by the background determination information or an image of a background having information similar to the background determination information exists in the recording device 190 or an external device. If the required background image or an image similar to it already exists, it is not necessary to generate a new background image. The processor 110 may determine the "closeness" of the background based on distance in a color space or other feature space. Depending on the determination result, the processor 110 generates (step S130) or selects (step S135) a background image.

[0057] The processor 110 may generate two or more background images (backgrounds for actual photography, first backgrounds) that differ in at least one of color and pattern. Furthermore, the two or more background images may be generated corresponding to different orientations (or different feature information) of the object. The processor 110 may generate the background images using AI in an image generation system. The generated background images may be displayed on the display 170 or may be recorded in the recording device 190. During the actual photography described below, the background images can be displayed on a background display device 300 (display device).

[0058] When a background image is displayed on the background display device 300, the background color may change due to ambient light. For this reason, it is preferable to generate or select a background that will have a uniform color when photographed, and it is also preferable to control the color, intensity, and irradiation direction of the illumination light from the lighting device 400 in accordance with the background image so that the background has a uniform color. Furthermore, in order to reduce the influence of ambient light, a light-shielding plate or light-shielding sheet may be provided around the object or the background display device 300.

[0059] [Acquisition of Feature Information and Example of Background Image (Part 1)] Figure 7 is a diagram showing an example (Part 1) of the acquisition of feature information and a background image based thereon. In the preliminary captured image 910 (at least one image obtained by the second capture, the preliminary captured image) shown in part (a) of Figure 7, the background 912 (second background) is "gray (single color) without pattern." In this case, the processor 110 (feature information acquisition unit 112, etc.) extracts a region 914 including the object 99B from the preliminary captured image 910, as shown in parts (a) and (b) of Figure 7. This region extraction can be performed by cutting out a region of a predetermined shape (e.g., rectangular, circular, elliptical, etc.) including the object using normal image processing, or by segmenting the preliminary captured image 910 using AI such as a neural network.

[0060] The processor 110 acquires, as feature information, information indicating at least one of the color and pattern of the object 99B in the extracted region 914 (step S100 described above). Here, it is assumed that the acquired feature information for the surface of the object 99B is "red (single color), no pattern." The type of object 99B (here, an apple) may be acquired as feature information using AI or image processing.

[0061] 7, since the object 99B does not include any white or black parts, the background can be set to white (or black). That is, the processor 110 (background information determination unit 114, etc.) outputs "white (single color), no pattern" as background determination information 916 for the actual photographing (step S110 described above), and can generate or select a background image having this background determination information 916 (steps S130 and S135 described above), as shown in part (c) of FIG.

[0062] In Figure 7 and the following figures, parenthesized words such as "(gray)" are used for convenience in describing the color of the object or background, and do not mean that such words are actually included in the object or background.

[0063] [Acquisition of Feature Information and Example of Background Image (Part 2)] Figure 8 is a diagram showing an example (part 2) of acquiring feature information and a background image based thereon. In a preliminary captured image 920 (at least one image obtained by second capturing, a preliminary captured image) shown in part (a) of Figure 8, the background 922 (second background) is "gray (single color) with no pattern." In this case, the processor 110 (the feature information acquisition unit 112, etc.) extracts an area 924 including the object 99A from the preliminary captured image 920, as shown in parts (a) and (b) of Figure 8.

[0064] The processor 110 acquires, as feature information, information indicating at least one of the color and pattern of the object 99A in the extracted region 924 (step S100 described above). Here, it is assumed that the acquired feature information for the surface of the object 99A is "white and black, striped." The type of the object 99A (here, a zebra model) may be acquired as feature information using AI or image processing.

[0065] In the example of Fig. 8, the object 99A includes white and black areas, so the background is set to a different color (green in this example). That is, the processor 110 outputs "green (single color), no pattern" as background determination information 926 for the actual photographing, as shown in part (c) of Fig. 8 (step S110 described above), and can generate or select a background image having this background determination information 926 (steps S130 and S135 described above).

[0066] [Acquisition of Feature Information and Example of Background Image (Part 3)] Figure 9 is a diagram showing an example (Part 3) of acquisition of feature information and a background image based thereon. In the example shown in part (a) of Figure 9, a background image having "blue, dot pattern" can be generated or selected for a red object 99B as background determination information 930. The "blue, dot pattern" of the background determination information 930 is a different color and pattern from the "red (solid color), no pattern" of the object 99B.

[0067] 9B, for an object 99C containing multiple colors (a cube with each face having a different color), a background image having "blue polka dots on a white background" can be generated or selected as background determination information 940. The "blue polka dots on a white background" of the background determination information 940 has a different color and pattern from the "yellow, purple, and red" of object 99B.

[0068] [Example of Background Image Using Selected Pattern] When generating a background image (first background image for actual shooting) based on background determination information, an image may be generated or selected based on a user's selection. For example, as shown in Figure 10, sample background images 902 to 908 corresponding to the background determination information are displayed as icons on a screen 900 of the display 170, and a background pattern of a sample selected by the user can be generated or selected.

[0069] [Setting Shooting Conditions for Main Shooting] Once the background image is generated or selected (steps S130 and S135), the processor 110 (shooting control unit 120, etc.) sets the shooting conditions for the main shooting (first shooting) (step S140). These shooting conditions are for acquiring multiple images (first images) to be used in generating a 3D model, and may include conditions such as viewpoint position, shooting direction, lighting, exposure, and background. It is preferable that the processor 110 sets the shooting conditions to acquire multiple images by performing main shooting from multiple directions. Such settings can improve the accuracy of the generated 3D model.

[0070] FIG. 11 is a diagram showing an example of a configuration for performing actual photography from multiple directions. In the example shown in part (a) of the figure, the photography camera 200 is mounted on a dolly 210 that can pan, tilt, expand and contract vertically, and move. The dolly 210 can be used to change the viewpoint position and photography direction of the photography camera 200, allowing actual photography to be performed from multiple directions. The photography camera 200 may be mounted on a crane, drone, or the like instead of the dolly 210 to change the viewpoint position and photography direction. The processor 110 (photography control unit 120, etc.) can control changes in the viewpoint position and photography direction caused by the dolly 210, etc. Furthermore, the configuration of the dolly 210, crane, drone, etc. may allow the processor 110 to control the movement of the lighting device 400 and the lighting direction.

[0071] 11(b) is a diagram (viewed from vertically above) showing another example of a configuration for performing actual photography from multiple directions. In the example shown in this part, by placing an object (not shown) on a stage 500 and rotating the stage 500, actual photography can be performed from multiple directions without moving the photography camera 200. The processor 110 can control changes in the viewpoint position and photography direction caused by the stage 500.

[0072] 12 is a diagram showing another example of a configuration for taking actual photographs from multiple directions (as viewed from the side of the background display device 300). The diagram shows how photographs are taken using multiple photographing cameras 200 with different viewpoint positions and photographing directions. Photographs may be taken by combining the configuration of FIG. 11 and the configuration of FIG. 12.

[0073] [Actual Shooting] After the shooting conditions for the actual shooting are set in step S140, the processor 110 (shooting control unit 120, background display control unit 122, lighting control unit 124, etc.) changes the viewpoint position and shooting direction of the shooting camera 200 (this may be a relative change), controls the lighting by the lighting device 400, etc., in accordance with the shooting conditions, and also displays the generated or selected background image on the background display device 300 (step S150). These controls realize the set shooting conditions. It is preferable that the processor 110 controls the color, intensity, irradiation direction, etc. of the illumination light in accordance with the display of the background image.

[0074] The processor 110 performs main photography from multiple directions to acquire multiple images (step S160). The processor 110 may perform main photography from one direction at a time, or may perform main photography from multiple directions at once using multiple photographing cameras 200. The processor 110 repeats the main photography until it has acquired the multiple images necessary to generate a 3D model (while the result of step S170 is NO). During the repetition, the processor 110 can sequentially change the photography conditions (the viewpoint position and photography direction of the photographing camera 200, the lighting by the lighting device 400, the background image displayed on the background display device 300, etc.) using the configuration described above with reference to Figures 11 and 12. If multiple background images corresponding to the orientation of the object have been generated or selected, the processor 110 can switch the background image as the viewpoint position or photography direction changes. This is because the optimal background may differ depending on the viewpoint position and photography direction.

[0075] The processor 110 can also output the multiple images obtained by this actual photography in association with the photography conditions (for example, by displaying them on the display 170 or recording them in the recording device 190 as the actual photography image 190C in FIG. 4). This "association of the photography conditions with the multiple images" may include information indicating the relationship between the background determination information (first background) and the multiple images obtained by the actual photography.

[0076] In the three-dimensional model generation system 10, the above-mentioned preliminary photography and main photography may be performed using the same camera, or different cameras.

[0077] [Generation of 3D Model] After acquiring the plurality of images (step S170), the processor 110 (the 3D model generator 126) generates a 3D model of the object using the plurality of images (step S180). The 3D model may include information on not only the shape of the object but also its color and pattern.

[0078] When generating a 3D model, processor 110 refers to information indicating the relationship between a background image (an image generated or selected based on background determination information) and the multiple images from the multiple images acquired in the actual photographing, and can obtain boundary information between the object and the background by taking into consideration differences between the characteristics of the background image and the object. Furthermore, as described above, if "at least one of the color being different from the color of the object and the pattern being different from the pattern of the object" is established, processor 110 can easily distinguish between the object and the background (first background) in the actual photographing image (first image acquired by the first photographing) based on this establishment.

[0079] [Generation of 3D Models by Photogrammetry] The processor 110 can use photogrammetry to generate 3D models. In photogrammetry, the position and orientation of a camera are estimated from images using SfM (Structure from Motion), and a 3D model is generated using MVS (Multi-View Stereo). Images captured by the capturing camera 200 can be used to generate the 3D model.

[0080] Photogrammetry involves matching feature points between images and generating a 3D model using the results of that processing. For this reason, objects that do not have a sufficient number of feature points available from the images, such as transparent or translucent objects, are considered unsuitable for photogrammetry.

[0081] [Generation of 3D Models Using Visual Volume Intersection Method] The processor 110 may generate a 3D model using visual volume intersection method. Visual volume intersection method, also known as silhouette method, is a technique for generating a 3D model based on the silhouette of an object. In visual volume intersection method, silhouettes of an object photographed from multiple viewpoints are used to create a cone (visual volume) with each viewpoint as a vertex and the silhouette as a cross section, and the silhouettes are back-projected into three-dimensional space to determine their intersections (common portions) to generate a 3D model. As with photogrammetry, images captured by the photographing camera 200 can be used to generate the 3D model.

[0082] [Generating a 3D model by combining two methods] In contrast to the above-mentioned photogrammetry, the volume intersection method is a method for generating a 3D model using a silhouette, so even if the object has transparent or semi-transparent parts, a 3D model can be generated as long as the silhouette can be obtained. However, the accuracy of the reconstructed 3D shape may be inferior to that of photogrammetry.

[0083] Therefore, in the 3D model generation system 10, the processor 110 may combine these techniques to generate a 3D model. Specifically, using boundary information between the object and the background, the 3D shape of a certain portion of the object may be estimated using volume intersection, and the 3D shape of the remaining portion may be estimated using photogrammetry, and a final 3D model may be generated based on the 3D shapes obtained using these two techniques. In this case, for example, volume intersection may be used for transparent or semitransparent portions of the object, and photogrammetry may be used for the remaining portions. The processor 110 may acquire which portions are transparent or semitransparent as the above-mentioned feature information, and determine the 3D model generation method based on this feature information.

[0084] [Output of 3D Model] The processor 110 (3D model generation unit 126, input / output control unit 128, etc.) can output the generated 3D model to an output device (step S190). The processor 110 may display the 3D model on the display 170, or may record it in the recording device 190 as a 3D model 190F (see FIG. 4). The 3D model may also be printed by a 3D printer (not shown) via the input / output interface 180.

[0085] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible.

[0086] 10 3D model generation system 99A Object 99B Object 99C Object 100 Controller 110 Processor 112 Feature information acquisition unit 114 Background information determination unit 116 Background generation unit 118 Background selection unit 120 Shooting control unit 122 Background display control unit 124 Lighting control unit 126 3D model generation unit 128 Input / output control unit 150 Operation unit 160 Speaker 170 Display 180 Input / output interface 190 Recording device 190A Feature information 190B Preliminary captured image 190C Main captured image 190D Background determination information 190E Background image 190F 3D model 195 Bus 200 Shooting camera 210 Dolly 300 Background display device 400 Lighting device 500 Stage 900 Screen 902 Sample 903 Sample 904 Sample 905 Sample 906 Sample 907 Sample 908 Sample 910 Preliminary captured image 912 Background 914 Area 916 Background determination information 920 Preliminary captured image 922 Background 924 Area 926 Background determination information 930 Background determination information 940 Background determination information 950 Screen 952 Message 960 Screen 962 Message S100 to S190 Processes in the 3D model generation system

Claims

1. An information processing device having a processor, wherein the processor acquires feature information indicating the features of an object for which a three-dimensional model is to be generated, and outputs background determination information, which is information for determining a first background to be used in a first photographing operation for generating the three-dimensional model, based on the feature information.

2. The information processing device according to claim 1, wherein the processor acquires the feature information based on at least one image obtained by a second photographing of the object prior to the first photographing.

3. The information processing device according to claim 2, wherein the processor acquires the feature information from a plurality of images obtained by performing the second photographing of the object from a plurality of directions.

4. An information processing device according to any one of claims 1 to 3, wherein the feature information is information about the surface of the object.

5. An information processing device according to any one of claims 1 to 3, wherein the processor acquires information indicating at least one of the color and pattern of the object as the feature information.

6. An information processing device according to any one of claims 1 to 3, wherein the processor outputs information indicating at least one of the color and pattern of the first background as the background determination information.

7. The information processing device according to claim 6, wherein the processor outputs the background determination information such that at least one of the color is different from the color of the object and the pattern is different from the pattern of the object is true.

8. An information processing device as described in claim 7, wherein the processor distinguishes between the object and the first background in the first image obtained by the first photographing based on the fact that at least one of the color and pattern in the background determination information is different from at least one of the color and pattern of the object.

9. An information processing device according to any one of claims 1 to 3, wherein the processor selects the first background from among a plurality of backgrounds having predetermined colors and patterns based on the background determination information.

10. An information processing device according to any one of claims 1 to 3, wherein the processor generates the first background based on the background determination information.

11. The information processing device according to claim 10, wherein the processor generates two or more first backgrounds that differ in at least one of color and pattern.

12. An information processing device according to any one of claims 1 to 3, wherein the processor acquires information according to the orientation of the object as the feature information.

13. An information processing device according to any one of claims 1 to 3, wherein the processor causes an output device to output information that prompts a user of the information processing device to take the first photograph in an environment corresponding to the background determination information.

14. A three-dimensional model generation system comprising: an information processing device according to any one of claims 1 to 3; a display device that displays the first background; a photographing device that performs the first photographing of the object against the first background to obtain a plurality of images; and a generation unit that generates the three-dimensional model of the object using the plurality of images.

15. A three-dimensional model generation system according to claim 14, wherein the photographing device performs the first photographing of the object from a plurality of directions to obtain the plurality of images.

16. A three-dimensional model generation system according to claim 14, wherein the generation unit generates the three-dimensional model by referring to information indicating the relationship between the background determination information and the plurality of images.

17. A three-dimensional model generation system according to claim 14, wherein the generation unit generates the three-dimensional model taking into consideration differences between the background determination information and the characteristics of the object.

18. A three-dimensional model generation system according to claim 14, wherein the photographing device performs the first photographing and a second photographing of the object prior to the first photographing.

19. An operating method of an information processing device having a processor, wherein the processor acquires feature information indicating the features of an object for which a three-dimensional model is to be generated, and outputs background determination information, which is information for determining a first background to be used in a first photographing to generate the three-dimensional model, based on the feature information.

Images