Image capturing system and program

Abstract

To photograph a specular reflection image of the entire surface of an object to be photographed when the object is photographed by a photographing device having lighting. [Solution] When capturing a specular reflection image of a photographing object, a control unit 33 first measures the distance between a mobile terminal device 10 and the photographing object and the shape of the photographing object. Then, when the mobile terminal device 10 moves to a position and orientation where a partial area of ​​the surface of the photographing object, whose distance and shape have been measured, satisfies the specular reflection conditions when the photographing object is captured by the mobile terminal device 10, the control unit 33 performs processing to assist the user in capturing a specular reflection image.

Description

[Technical Field]

[0001] The present disclosure relates to an image capturing system and a program. [Background technology]

[0002] Patent Document 1 discloses a gloss reading method in which an illuminating light source and an imaging means that images the subject in at least one channel are moved relative to each other while the subject is photographed multiple times with the imaging means, the channel value data for each pixel of the multiple images obtained by photographing are compared to determine the maximum value of each channel value, and a maximum value image is created that shows the maximum channel value for each pixel, thereby making it possible to read the specular reflection intensity of the entire surface of a subject even when the subject has a large curved surface.

[0003] Patent Document 2 discloses a display system that can display the texture of a large area with simple calculations using less image data than when a large amount of image data is acquired by taking photographs with a camera from multiple angles and then using this image data to display the texture of the object surface.

[0004] Patent Document 3 discloses an inspection device that acquires image information representing continuously captured images by moving a scanning unit equipped with a camera, a laser sensor, a white light source, and an invisible light laser light source relative to an object, extracts area images corresponding to a predetermined range for the specular reflection angle from the image information, and synthesizes the extracted area images to generate a defect detection image for detecting defects in the object, thereby obtaining an image that is more suitable for inspecting defects in the external surface of an object than when an image of specular reflection is captured. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-050015 [Patent Document 2] Japanese Patent Application Publication No. 2019-082838 [Patent Document 3] Japanese Patent Publication No. 2022-047371 Summary of the Invention [Problem to be solved by the invention]

[0006] Three-dimensional models of photographed objects are generated using images of the object captured from various angles. In order to reproduce the texture of the object in such a 3D model, it is sometimes desirable to include the glossiness of the object in the 3D model. To generate a 3D model that includes the glossiness, it is necessary to capture all specular reflection images of the entire surface of the object. Here, a specular reflection image refers to an image captured when light irradiated from a light source is reflected from a portion of the object, and the specular reflection light of the reflected light components is directly incident on the camera. However, when the surface shape of the object is complex, such as when it includes a curved surface, it is difficult to determine the position at which the object should be photographed under specular reflection conditions.

[0007] An object of the present disclosure is to provide an image capturing system and program that are capable of capturing a specular reflection image of the entire surface of an object when capturing an image of the object using a capturing device with lighting. [Means for solving the problem]

[0008] The image capturing system of the first aspect of the present disclosure includes a portable image capturing device having lighting, a camera, and a measuring unit capable of measuring the distance between the image capturing device and an object to be captured and the shape of the object to be captured; a processor, The processor measures a distance between the image capture device and the object to be captured and a shape of the object to be captured; When the photographing device moves to a position and orientation where a partial area of ​​the surface of the object to be photographed, whose distance and shape have been measured, creates specular reflection conditions when the object to be photographed is photographed by the photographing device, the device assists the user in photographing a specular reflection image.

[0009] In a second aspect of the image capturing system of the present disclosure, in the image capturing system of the first aspect, the processor assists the user in capturing a specular reflection image by notifying the user that the position and orientation of the capturing device are such that a portion of the surface of the object being captured will be in a specular reflection condition when the object is captured.

[0010] In a third aspect of the image capturing system of the present disclosure, in the image capturing system of the first aspect, the processor assists the user in capturing a specular reflection image by capturing an image of the object to be captured using the capturing device without accepting user operation when the position and orientation of the capturing device are such that a portion of the surface of the object to be captured becomes a specular reflection condition when the object is captured.

[0011] In a fourth aspect of the image capturing system of the present disclosure, in the image capturing system of the first aspect, the processor divides the surface of the object to be captured, the shape of which has been identified, into a plurality of regions, and among the plurality of regions, regions where images have been captured at a position and orientation that satisfies specular reflection conditions and uncaptured regions where images have not been captured at a position and orientation that satisfies specular reflection conditions are displayed in different display modes, thereby assisting the user in capturing specular reflection images.

[0012] In a fifth aspect of the image capturing system of the present disclosure, in the image capturing system of the fourth aspect, when the uncaptured area is selected, the processor displays a guidance display to guide the uncaptured area to a position and orientation that allows it to capture the uncaptured area under specular reflection conditions, based on the current position and orientation of the capturing device determined from information on the distance between the capturing device and the object to be captured measured by the measurement unit.

[0013] An image capturing system of a sixth aspect of the present disclosure is the image capturing system of the first aspect, wherein the processor divides the surface of the object to be captured into smaller areas as the surface shape of the object to be captured becomes more complex, as the accuracy required for the captured specular reflection image becomes higher, or as the illumination area of ​​the lighting becomes smaller.

[0014] An image capturing system according to a seventh aspect of the present disclosure is the image capturing system according to the first aspect, wherein the measuring unit is a sensor capable of measuring distances between the capturing device and a plurality of points on the object to be captured.

[0015] An image capturing system according to an eighth aspect of the present disclosure is the image capturing system according to the seventh aspect, wherein the sensor is a LiDAR sensor capable of measuring the distance to the object to be captured for each pixel captured by the camera.

[0016] In the image capturing system of the ninth aspect of the present disclosure, in the image capturing system of the first aspect, the processor calculates the position and orientation of the capturing device that can capture the area under specular reflection conditions using information on the positional and angular relationship between the lighting and camera in the capturing device and the position of the area to be captured under specular reflection conditions on the surface shape of the object to be captured, the shape of which has been identified.

[0017] A program according to a tenth aspect of the present disclosure includes a step of measuring, by a portable photographing device having lighting, a camera, and a measurement unit capable of measuring the distance between the photographing device and the object to be photographed and the shape of the object to be photographed, the distance between the photographing device and the object to be photographed and the shape of the object to be photographed; The computer is caused to execute a step of assisting the user in capturing a specular reflection image when the photographing device moves to a position and orientation where a partial area of ​​the surface of the object to be photographed, the distance and shape of which have been measured, will create specular reflection conditions when the object to be photographed is photographed by the photographing device. [Effects of the Invention]

[0018] According to the image capturing system of the first aspect of the present disclosure, when capturing an image of a target object using a capturing device having lighting, it is possible to capture a specular reflection image of the entire surface of the target object.

[0019] According to the image capturing system of the second aspect of the present disclosure, a user can, simply by moving the capturing device, grasp that the position and orientation of the capturing device have become such that a portion of the surface of the object being captured meets the specular reflection conditions.

[0020] According to the image capturing system of the third aspect of the present disclosure, a user can automatically capture a specular reflection image of a partial area of ​​the surface of an object to be captured simply by moving the capturing device.

[0021] According to the image capturing system of the fourth aspect of the present disclosure, it is possible to allow the user to recognize areas where images have not been acquired at positions and orientations that satisfy specular reflection conditions.

[0022] According to the image capturing system of the fifth aspect of the present disclosure, the user can move the capturing device to a position and orientation for capturing an image of the selected unacquired area under specular reflection conditions simply by following the guidance display.

[0023] According to the image capturing system of the sixth aspect of the present disclosure, the accuracy of the captured specular reflection image can be brought closer to the required accuracy compared to when the size of the areas dividing the surface of the object to be captured is constant.

[0024] According to the image capturing system of the seventh aspect of the present disclosure, when capturing an image of a target object using a capturing device with lighting, it is possible to capture a specular reflection image of the entire surface of the target object without omission.

[0025] According to the image capturing system of the eighth aspect of the present disclosure, when capturing an image of a target object using a capturing device with lighting, it is possible to capture a specular reflection image of the entire surface of the target object without omission.

[0026] According to the image capturing system of the ninth aspect of the present disclosure, it is possible to calculate the position and orientation of the capturing device that can capture an area under specular reflection conditions simply by setting the positional and angular relationship between the lighting and the camera.

[0027] According to the program of the tenth aspect of the present disclosure, when an object is photographed using an imaging device having lighting, it is possible to photograph a specular reflection image of the entire surface of the object. [Brief explanation of the drawings]

[0028] [Figure 1] 1 is a diagram showing a state in which an image of a photographing object 50 is photographed by a mobile terminal device 10 which is an image photographing system according to an embodiment of the present disclosure. [Figure 2] 1 is a block diagram showing a hardware configuration of a mobile terminal device 10 according to an embodiment of the present disclosure. [Figure 3] 1 is a block diagram showing a functional configuration of a mobile terminal device 10 according to an embodiment of the present disclosure. [Figure 4] 10 is a flowchart for explaining an operation when capturing a specular reflection image of the entire surface of a target object 50 by a mobile terminal device 10 according to an embodiment of the present disclosure. [Figure 5] 10 is a diagram for explaining a method of calculating the position and angle of the mobile terminal device 10 that satisfy the specular reflection conditions for a certain block. FIG. [Figure 6] FIG. 10 is a diagram for explaining how the position and angle of the mobile terminal device 10 that constitutes the specular reflection condition differs for each block. [Figure 7] FIG. 10 is a diagram showing an example of a specular reflection image captured when a certain block of a photographing object 50 is in a specular reflection condition. [Figure 8] FIG. 10 is a diagram showing an example in which blocks for which specular reflection images have not yet been acquired and blocks for which specular reflection images have been acquired are displayed in different colors. [Figure 9] FIG. 10 is a diagram showing how a block from among blocks for which a specular reflection image has not yet been acquired is selected on the touch panel. [Figure 10] FIG. 10 is a diagram showing a first example of a guidance display that prompts the user to move the mobile terminal device 10 to a position where the selected block can be photographed under specular reflection conditions. [Figure 11] FIG. 10 is a diagram showing a second example of a guidance display that prompts the user to move the mobile terminal device 10 to a position where the selected block can be photographed under specular reflection conditions. [Figure 12] FIG. 10 is a diagram showing a third example of a guidance display that prompts the user to move the mobile terminal device 10 to a position where the selected block can be photographed under specular reflection conditions. [Figure 13] FIG. 10 is a diagram showing an example in which lighting is provided externally to the mobile terminal device 10. [Figure 14] FIG. 10 is a diagram showing another example in which lighting is provided externally to the mobile terminal device 10. [Figure 15] FIG. 10 is a diagram showing another example in which lighting is provided externally to the mobile terminal device 10. [Figure 16] FIG. 10 is a diagram showing another example in which lighting is provided externally to the mobile terminal device 10. DETAILED DESCRIPTION OF THE INVENTION

[0029] Next, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0030] FIG. 1 shows how an image of a photographing object 50 is captured by a mobile terminal device 10, which is an image capturing system according to an embodiment of the present disclosure.

[0031] Here, a 3D model of a photographed object is generated using images of the photographed object taken from various angles. In order to reproduce the texture of the photographed object in such a 3D model, it may be desirable to include the glossiness of the photographed object in the 3D model. To generate a 3D model that includes the glossiness, it is necessary to capture all specular reflection images of the entire surface of the photographed object. Here, a specular reflection image refers to an image captured when light irradiated from a light source is reflected from a portion of the photographed object, and the specular reflection light among the reflected light components is directly incident on the camera. Note that a specular reflection image refers to an image in which a specific region of the photographed object is subject to specular reflection conditions. A specular reflection image does not refer to an image in which the entire region of the photographed object is subject to specular reflection conditions. In other words, a single specular reflection image includes both regions that are subject to specular reflection conditions and regions that are not subject to specular reflection conditions.

[0032] In this embodiment, the case where the object to be photographed is a three-dimensional object 50 whose surface shape includes a curved surface, as shown in FIG. 1, will be described. However, the object to be photographed includes not only three-dimensional objects but also objects that are similar to a sheet-like plane. Furthermore, the present disclosure is applicable not only to cases where the entire object is the object to be photographed, but also to cases where only a part of the object is the object to be photographed. For example, the present disclosure is applicable to cases where only the tabletop is the object to be photographed, rather than the entire table.

[0033] In the image capturing system of this embodiment, in order to generate a 3D model of the object 50 as shown in Fig. 1, the object 50 is captured from various angles. Then, in order to include the glossiness of the object 50 in the generated 3D model, all specular reflection images of the entire surface of the object 50 are captured. However, when attempting to generate a 3D model of the object 50 as shown in Fig. 1, it is difficult to determine at what position the object 50 should be captured under specular reflection conditions because the surface shape of the object 50 includes curved surfaces.

[0034] Therefore, in the image capturing system of this embodiment, the processing described below is performed so that when capturing an image of the object to be captured using a portable terminal device 10 equipped with lighting, a specular reflection image of the entire surface of the object to be captured 50 can be captured.

[0035] Here, the mobile terminal device 10 is realized by, for example, a mobile phone such as a smartphone (hereinafter abbreviated as "smartphone"), a tablet terminal device, or the like. The mobile terminal device 10 is a portable photographing device having lighting, a camera, and a measurement unit capable of measuring the distance between the device itself and a photographing object and the shape of the photographing object. The measurement unit is a sensor capable of measuring the distance between the mobile terminal device 10 and multiple points on the photographing object. Specifically, this sensor is a LiDAR (Light Detection And Ranging) sensor capable of measuring the distance to the photographing object for each pixel captured by the camera. Although the LiDAR sensor has been described as being capable of measuring the distance between the mobile terminal device 10 and the photographing object, in reality, it measures the distance between the LiDAR sensor and the photographing object.

[0036] Furthermore, although it has been described that the LiDAR sensor can measure the shape of a photographed object, the case where the shape of the photographed object is identified from information about the distance to the photographed object is also included in measuring the distance to the photographed object and the shape of the photographed object. Furthermore, the distance to the photographed object and the shape of the photographed object may be measured separately.

[0037] Next, the hardware configuration of the mobile terminal device 10 that constitutes the image capturing system of this embodiment is shown in FIG.

[0038] As shown in FIG. 2, the mobile terminal device 10 has a CPU 11, a memory 12, a storage device 13 such as a flash memory, a communication interface (abbreviated as IF) 14 that transmits and receives data to and from external devices via a wireless communication line, a user interface (abbreviated as UI) device 15 that includes a touch panel or LCD display, a camera 16, a LiDAR sensor 17, and lighting 18.

[0039] The CPU 11 is a processor that controls the operation of the mobile terminal device 10 by executing predetermined processes based on a control program stored in the memory 12 or the storage device 13. In the present embodiment, the CPU 11 is described as reading and executing the control program stored in the memory 12 or the storage device 13, but this is not limiting. The control program may be provided in a form recorded on a computer-readable recording medium. For example, the program may be provided in a form recorded on an optical disc such as a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM, or in a form recorded on a semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. The control program may also be acquired from an external device via a communication line connected to the communication interface 14. The control program may be provided as standalone application software, or may be incorporated into the software of each device of the mobile terminal device 10 as a function of the device.

[0040] FIG. 3 is a block diagram showing the functional configuration of the mobile terminal device 10 realized by executing the above control program.

[0041] As shown in Fig. 3, the mobile terminal device 10 of this embodiment includes an operation input unit 31, a display unit 32, a control unit 33, a data storage unit 34, a camera 16, a LiDAR sensor 17, and lighting 18. Note that Fig. 3 omits functional components such as communication functions that are not related to the technology of the present disclosure. Furthermore, in this embodiment, the camera 16 and the LiDAR sensor 17 are described as being separate components, but they may be integrated into a camera with LiDAR.

[0042] The display unit 32 displays various information to the user under the control of the control unit 33. The operation input unit 31 inputs various operation information performed by the user. The operation input unit 31 and the display unit 32 form a touch panel.

[0043] The control unit 33 controls the operations of the camera 16, the LiDAR sensor 17, and the lighting 18 to capture an image of the object to be photographed, and the image is stored in the data storage unit 34. The control unit 33 also performs an operation to receive operation information from the operation input unit 31 and an operation to display various information on the display unit 32.

[0044] When capturing a specular reflection image of a photographing object, the control unit 33 first measures the distance between the mobile terminal device 10 and the photographing object and the shape of the photographing object. Then, when the mobile terminal device 10 moves to a position and orientation where a partial area of ​​the surface of the photographing object, the distance and shape of which have been measured, satisfies the specular reflection conditions when the photographing object is captured by the mobile terminal device 10, the control unit 33 performs processing to assist the user in capturing a specular reflection image.

[0045] For example, the control unit 33 assists the user in capturing a specular reflection image by notifying the user that the position and orientation of the mobile terminal device 10 are such that a portion of the surface of the object to be photographed will be in a specular reflection condition when the object is photographed.

[0046] In addition, when the position and orientation of the mobile terminal device 10 are such that a portion of the surface of the object to be photographed will be in a specular reflection condition when the object is photographed, the control unit 33 assists the user in photographing a specular reflection image by photographing the object to be photographed using the camera 16 of the mobile terminal device 10 without accepting any operation from the user.

[0047] Furthermore, the control unit 33 divides the surface of the object to be photographed, the shape of which has been identified, into multiple regions, and among the multiple regions, displays in different display modes those regions where images have been acquired at a position and orientation that satisfies the specular reflection conditions and those regions where images have not been acquired at a position and orientation that satisfies the specular reflection conditions, thereby assisting the user in photographing a specular reflection image.

[0048] At that time, when an uncaptured area is selected by the user, the control unit 33 may display a guidance display to guide the user to a position and orientation that allows the uncaptured area to be photographed under specular reflection conditions, based on the current position and orientation of the mobile terminal device 10 determined from information on the distance between the mobile terminal device 10 and the object being photographed measured by the LiDAR sensor 17.

[0049] Here, the control unit 33 divides the surface of the object to be photographed into smaller areas as the surface shape of the object to be photographed becomes more complex, as the accuracy required for the captured specular reflection image becomes higher, or as the illumination area of ​​the light 18 becomes smaller.

[0050] The control unit 33 uses the positional and angular relationship between the lighting 18 and the camera 16 in the mobile terminal device 10 and information on the position of the area to be photographed under specular reflection conditions on the surface shape of the object to be photographed, the shape of which has been identified, to calculate the position and orientation of the mobile terminal device 10 at which the area can be photographed under specular reflection conditions.

[0051] Next, the operation of the mobile terminal device 10 of this embodiment when capturing a specular reflection image of the entire surface of the object 50 shown in FIG. 1 will be described in detail with reference to FIG.

[0052] First, in step S101, the control unit 33 acquires information on the relative positional relationship and angle between the camera 16 and the lighting 18, and the size of the lighting 18. Specifically, the control unit 33 may acquire this information from the user via the operation input unit 31. Alternatively, the control unit 33 may photograph a mirror using the mobile terminal device 10 to acquire information such as the positional relationship between the camera 16 and the lighting 18 in the device itself. Note that when the camera 16 and lighting 18 provided in the mobile terminal device 10 are used, the control unit 33 may acquire information such as the positional relationship between the camera 16 and the lighting 18 using preset information.

[0053] Next, in step S102, the control unit 33 measures the shape of the entire circumference of the object 50 to be photographed using the LiDAR sensor 17.

[0054] Then, in step S103, the control unit 33 divides the measured surface of the object 50 to be photographed into a plurality of blocks, which are a plurality of regions, and calculates the position and angle of the mobile terminal device 10 that satisfies the specular reflection conditions for each block.

[0055] Here, a method for calculating the position and angle of the mobile terminal device 10 that satisfies the specular reflection conditions for a certain block will be described with reference to FIG.

[0056] 5, a certain block satisfies the specular reflection condition means that the positional relationship is such that light irradiated from the lighting 18 is reflected at a certain block of the object 50 to be photographed, and specularly reflected light among the reflected light components is directly incident on the camera 16. In other words, as shown in FIG. 6, if the surface shape of the object 50 to be photographed is curved, the position and angle of the mobile terminal device 10 that satisfies the specular reflection condition for each block will be different.

[0057] FIG. 7 shows an example of a specular reflection image captured when a certain block of the object 50 is in specular reflection conditions. Referring to FIG. 7, it can be seen that the image shows light specularly reflected from a certain block of the object 50 entering the camera 16. In other words, a specular reflection image is an image in which a specific region of the object is in specular reflection conditions and is shining, rather than an image in which the entire object is shining. In this way, a single specular reflection image includes both regions that are in specular reflection conditions and regions that are not in specular reflection conditions. By capturing specular reflection images of the entire object in which such a certain block is shining and generating a 3D model from such images, it becomes possible to generate a 3D model with a glossy appearance.

[0058] Next, the control unit 33 instructs the user to capture an image of the surroundings of the object 50, thereby capturing an image of the object 50 in step S104.

[0059] Then, in step S105, the control unit 33 determines whether or not specular reflection images of all blocks have been captured.

[0060] If it is determined in step S105 that specular reflection images of all blocks have been captured, the control unit 33 ends the process.

[0061] However, if it is determined in step S105 that specular reflection images have not been captured for all blocks, the control unit 33 executes a process to assist in capturing images of blocks for which specular reflection images have not yet been acquired under specular reflection conditions. For example, the control unit 33 displays blocks for which specular reflection images have not yet been acquired and blocks for which specular reflection images have been acquired in different display modes, for example, by changing the display color.

[0062] An example of this case where blocks for which specular reflection images have not yet been acquired and blocks for which specular reflection images have been acquired are displayed in different colors is shown in Figure 8. Referring to Figure 8, blocks for which specular reflection images have been acquired are displayed with black diagonal lines, and blocks for which specular reflection images have not yet been acquired are displayed in white. By looking at this display, the user can grasp the positions of blocks for which specular reflection images have not yet been acquired.

[0063] Then, in step S106, the control unit 33 displays a guidance message to move the mobile terminal device 10 to a position and angle for capturing an image of a block for which a specular reflection image has not yet been acquired under specular reflection conditions. For example, as shown in Fig. 9, a block from among the blocks for which a specular reflection image has not yet been acquired is selected on the touch panel. The control unit 33 then displays a guidance message to move the mobile terminal device 10 to a position where the block selected by the user can be captured under specular reflection conditions.

[0064] Specific examples of such guidance displays are shown in Fig. 10 to Fig. 12. In Fig. 10, the guidance display shows the direction in which the mobile terminal device 10 is moved and the direction in which it is rotated by using arrows, and the size of the arrows shows the amount of movement and rotation. In Fig. 11, the arrow in the center of the touch panel shows the direction in which the mobile terminal device 10 is moved, and the arrows at the edge of the touch panel show the direction in which the mobile terminal device 10 is rotated. In Fig. 11, the guidance display shows the amount of movement and the amount of rotation as numerical values.

[0065] 12 shows an example of a guide display in which a virtual mobile terminal device 10 is displayed in AR (Augmented Reality) at a position and angle to which the mobile terminal device 10 should be moved on the touch panel. By moving the mobile terminal device 10 so that it overlaps with the position of the AR-displayed mobile terminal device 10, the user can move the mobile terminal device 10 to a position where the selected block can be photographed under specular reflection conditions.

[0066] Alternatively, the control unit 33 may assist the user in capturing a specular reflection image by having the user capture a photograph of the entire circumference of the object 50 to be photographed again, and then notifying the user by, for example, generating a notification sound, that the position and orientation of the mobile terminal device 10 are now such that a block for which a specular reflection image has not yet been acquired can be captured under specular reflection conditions.

[0067] In addition, when the position and orientation of the mobile terminal device 10 are such that a block for which a specular reflection image has not yet been acquired can be photographed under specular reflection conditions, the control unit 33 may assist the user in photographing a specular reflection image by automatically photographing the object 50 to be photographed using the camera 16.

[0068] Then, the image capturing of the object 50 is repeated by the process described above, and if it is determined in step S105 that specular reflection images of all blocks have been captured, the control unit 33 ends the process.

[0069] In this way, the user can photograph the object 50 according to the guidance display, thereby capturing a specular reflection image of the entire surface of the object 50 without omission.

[0070] In the embodiment described above, an image is captured by irradiating the object 50 with light using the lighting 18 built into the mobile terminal device 10. However, an external lighting may be provided outside the mobile terminal device 10 to capture an image of the object 50.

[0071] Specific examples of such external lighting provided for the mobile terminal device 10 are shown in FIGS. 13 to 16. FIG.

[0072] FIG. 13 is a diagram showing a state in which a spherical light 51 is attached externally to the mobile terminal device 10. FIG. 14 is a diagram showing a state in which a wide bar light 52 is attached externally to the mobile terminal device 10. FIG. 15 is a diagram showing a state in which two bar lights 53 are attached externally to the left and right sides of the mobile terminal device 10. Finally, FIG. 16 is a diagram showing a state in which a ring light 54 is attached externally to the mobile terminal device 10. Even when an external light is attached to the mobile terminal device 10 in this way, the above-described processing can be performed by setting in advance information such as the relative positional relationship and angle between the camera 16 and the light, and the size of the light.

[0073] As explained above, by providing an external light to the mobile terminal device 10, the illumination area can be increased, and the size of the blocks used to divide the surface of the object to be photographed can be increased. As a result, the time required to photograph the specular reflection image of the entire surface of the object to be photographed is reduced, and images can be photographed efficiently.

[0074] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0075] Furthermore, the operations of the processor in each of the above embodiments may be performed not only by a single processor but also by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processor is not limited to the order described in each of the above embodiments and may be changed as appropriate.

[0076] In this embodiment, the term "system" includes both a system made up of multiple devices and a system made up of a single device.

[0077] The techniques of this disclosure are also applicable to programs and program products.

[0078] [Note] (((1))) a portable photographing device having lighting, a camera, and a measuring unit capable of measuring the distance between the device and a photographing object and the shape of the photographing object; a processor, The processor measures a distance between the image capture device and the object to be captured and a shape of the object to be captured; assisting the user in capturing a specular reflection image when the photographing device is moved to a position and orientation where a partial area of ​​the surface of the photographing object, the distance and shape of which have been measured, will be in a specular reflection condition when the photographing device captures the photographing object; Image capture system. (((2))) the processor notifies the user that the position and orientation of the photographing device are such that a partial area of ​​the surface of the photographing object becomes a specular reflection condition when the photographing object is photographed, thereby assisting the user in photographing the specular reflection image; The imaging system according to (((1))). (((3))) When the position and orientation of the photographing device are such that a partial area of ​​the surface of the photographing object becomes a specular reflection condition when the photographing object is photographed, the processor assists the user in photographing a specular reflection image by photographing the photographing object with the photographing device without receiving an operation from the user. The imaging system according to (((1))). (((4))) The processor divides the surface of the object to be photographed, the shape of which has been identified, into a plurality of regions, and among the plurality of regions, displays in different display modes a region where an image has been captured at a position and orientation that satisfies specular reflection conditions and a region where an image has not been captured at a position and orientation that satisfies specular reflection conditions, thereby assisting the user in photographing a specular reflection image. The imaging system according to (((1))). (((5))) When the unacquired area is selected, the processor displays a guide to guide the image capturing device to a position and orientation that allows the image capturing device to capture an image of the unacquired area under specular reflection conditions, based on the current position and orientation of the image capturing device determined from information on the distance between the image capturing device and the object to be captured measured by the measurement unit. The imaging system according to (((4))). (((6))) The processor divides the surface of the object into smaller regions as the surface shape of the object becomes more complex, as the accuracy required for the captured specular reflection image becomes higher, or as the illumination area of ​​the lighting becomes smaller. An imaging system according to any one of (((1))) to (((5))). (((7))) the measuring unit is a sensor capable of measuring distances between the photographing device and a plurality of points on the photographing object, An imaging system according to any one of (((1))) to (((6))). (((8))) The sensor is a LiDAR sensor capable of measuring the distance to the object for each pixel captured by the camera, The imaging system according to (((7))). (((9))) the processor calculates a position and orientation of the photographing device that can photograph the area under specular reflection conditions using information on the positional relationship and angular relationship between the lighting and the camera in the photographing device and the position of the area to be photographed under specular reflection conditions on the surface shape of the photographing object, the shape of which has been specified; An imaging system according to any one of (((1))) to (((8))). (((10))) a step of measuring a distance between the photographing device and the object to be photographed and a shape of the object to be photographed by a portable photographing device having lighting, a camera, and a measuring unit capable of measuring the distance between the photographing device and the object to be photographed and the shape of the object to be photographed; a step of assisting a user in capturing a specular reflection image when the photographing device is moved to a position and orientation where a partial area of ​​the surface of the photographing object, the distance and shape of which have been measured, will be in a specular reflection condition when the photographing device captures the photographing object; A program that causes a computer to execute the following.

[0079] According to the image capturing system (((1))), when a subject is captured using a capturing device with lighting, it is possible to capture a specular reflection image of the entire surface of the subject. According to the image capturing system of (((2))), a user can grasp, simply by moving the capturing device, that the position and orientation of the capturing device have become such that a portion of the surface of the object being captured satisfies the specular reflection conditions. According to the image capturing system of (((3))), a user can automatically capture a specular reflection image of a partial area on the surface of an object to be captured simply by moving the capturing device. According to the image capturing system (((4))), it is possible to allow the user to grasp the area where the image has not been captured at the position and orientation that satisfies the specular reflection condition. According to the image capturing system of (((5))), the user can move the capturing device to a position and orientation that will capture an image of the selected uncaptured area under specular reflection conditions simply by following the guidance display. According to the image capturing system (((6))), the accuracy of the captured specular reflection image can be made closer to the required accuracy compared to when the size of the areas dividing the surface of the object to be captured is constant. According to the image capturing system (((7))), when capturing an image of an object using a capturing device with lighting, it is possible to capture a mirror-reflected image of the entire surface of the object without omission. According to the image capturing system (((8))), when capturing an image of an object using a capturing device with lighting, it is possible to capture a mirror-reflected image of the entire surface of the object without omission. According to the image capturing system (((9))), it is possible to calculate the position and orientation of the capturing device that can capture an area under specular reflection conditions simply by setting the positional and angular relationship between the lighting and the camera. According to the program (((10))), when an object is photographed using a photographing device having lighting, it is possible to photograph a specular reflection image of the entire surface of the object. [Explanation of symbols]

[0080] 10. Portable terminal device 11 CPU 12 Memory 13 Storage device 14 Communication Interface 15 User Interface Device 16 Camera 17 LiDAR sensors 18. Lighting 31 Operation input section 32 Display section 33 Control Unit 34 Data storage unit 50 Subject of Photography 51 Lighting 52, 53 Bar lighting 54 Ring Light

Claims

1. a portable photographing device having lighting, a camera, and a measuring unit capable of measuring the distance between the device and a photographing object and the shape of the photographing object; a processor, The processor measures a distance between the image capture device and the object to be captured and a shape of the object to be captured; assisting the user in capturing a specular reflection image when the photographing device is moved to a position and orientation where a partial area of ​​the surface of the photographing object, the distance and shape of which have been measured, will be in a specular reflection condition when the photographing device captures the photographing object; Image capture system.

2. the processor notifies the user that the position and orientation of the photographing device are such that a partial area of ​​the surface of the photographing object becomes a specular reflection condition when the photographing object is photographed, thereby assisting the user in photographing the specular reflection image; The imaging system of claim 1 .

3. When the position and orientation of the photographing device are such that a partial area of ​​the surface of the photographing object becomes a specular reflection condition when the photographing object is photographed, the processor assists the user in photographing a specular reflection image by photographing the photographing object with the photographing device without receiving an operation from the user. The imaging system of claim 1 .

4. The processor divides the surface of the object to be photographed, the shape of which has been identified, into a plurality of regions, and among the plurality of regions, displays in different display modes a region where an image has been captured at a position and orientation that satisfies specular reflection conditions and a region where an image has not been captured at a position and orientation that satisfies specular reflection conditions, thereby assisting the user in photographing a specular reflection image. The imaging system of claim 1 .

5. When the unacquired area is selected, the processor displays a guide to guide the image capturing device to a position and orientation that allows the image capturing device to capture an image of the unacquired area under specular reflection conditions, based on the current position and orientation of the image capturing device determined from information on the distance between the image capturing device and the object to be captured measured by the measurement unit.

5. The imaging system according to claim 4.

6. The processor divides the surface of the object into smaller regions as the surface shape of the object becomes more complex, as the accuracy required for the captured specular reflection image becomes higher, or as the illumination area of ​​the lighting becomes smaller. The imaging system of claim 1 .

7. the measuring unit is a sensor capable of measuring distances between the photographing device and a plurality of points on the photographing object, The imaging system of claim 1 .

8. The sensor is a LiDAR sensor capable of measuring the distance to the object for each pixel captured by the camera, 8. The imaging system according to claim 7.

9. the processor calculates a position and orientation of the photographing device that can photograph the area under specular reflection conditions using information on the positional relationship and angular relationship between the lighting and the camera in the photographing device and the position of the area to be photographed under specular reflection conditions on the surface shape of the photographing object, the shape of which has been specified; The imaging system of claim 1 .

10. a step of measuring a distance between the photographing device and the object to be photographed and a shape of the object to be photographed by a portable photographing device having lighting, a camera, and a measuring unit capable of measuring the distance between the photographing device and the object to be photographed and the shape of the object to be photographed; a step of assisting a user in capturing a specular reflection image when the photographing device is moved to a position and orientation where a partial area of ​​the surface of the photographing object, the distance and shape of which have been measured, will be in a specular reflection condition when the photographing device captures the photographing object; A program that causes a computer to execute the following.

Images