Information processing systems and programs

Abstract

This invention provides an information processing system and program that facilitates the verification of a three-dimensional model compared to individually comparing the subject itself, or an image of the subject, with the three-dimensional model of the subject. [Solution] A three-dimensional model of the subject is generated from an image of the subject, and information comparing the image of the subject and the rendered image of the three-dimensional model of the subject is displayed.

Description

Technical Field

[0001] The present disclosure relates to an information processing system and a program.

Background Art

[0002] Patent Document 1 describes an image processing apparatus that removes noise generated due to the accuracy of three-dimensional shape data in a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an object from a plurality of viewpoints and three-dimensional shape data of the object.

[0003] Patent Document 2 describes a system that generates a virtual viewpoint image using a plurality of viewpoint images and background 3D data representing the three-dimensional shape of the background in a shooting scene. The system generates a simulation image corresponding to the viewpoint from the camera using the background 3D data, detects the difference between the generated simulation image and the image actually captured from the viewpoint of the camera, and updates the background 3D data based on the detection result. <**********>

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] [[ID=*]] A process of generating a three-dimensional model of a subject from an image of the subject is known. In such a process, there is a possibility that a difference different from the subject may occur in the generated three-dimensional model.

[0006] However, it is time-consuming to individually compare images of the subject with a 3D model to check for any differences between the subject and the 3D model.

[0007] The purpose of this disclosure is to provide an information processing system and program that facilitates the verification of a three-dimensional model compared to individually comparing the subject itself or an image of the subject with the three-dimensional model of the subject. [Means for solving the problem]

[0008] The first embodiment of the information processing system includes a processor, which generates a three-dimensional model of a subject from an image of the subject, and displays information comparing the image of the subject and an image of the rendered three-dimensional model of the subject.

[0009] In the second embodiment of the information processing system, the processor displays an image of the subject and an image of a rendered three-dimensional model of the subject on the same screen.

[0010] In the third embodiment of the information processing system, the processor displays on the same screen an image of the subject taken and an image of a three-dimensional model of the subject rendered from the same direction as the direction in which the image was taken.

[0011] In the fourth embodiment of the information processing system, the processor generates an image rendering a three-dimensional model of the subject using at least one of the following pieces of information: information about the photographing device that photographed the subject, and information about the lighting at the time the subject was photographed.

[0012] The information processing system of the fifth embodiment, in the information processing system of the first embodiment, includes a processor that compares an image of the subject with an image of a rendered three-dimensional model of the subject, extracts areas in the three-dimensional model of the subject that differ from the subject, and displays the extracted areas of difference.

[0013] The information processing system of the sixth embodiment, in the information processing system of the fifth embodiment, extracts differences in at least one piece of information of the subject's color, brightness, and pattern between an image of the subject taken and an image rendered from the same direction as the image taken, thereby extracting areas in the three-dimensional model of the subject that differ from the subject.

[0014] The information processing system of the seventh embodiment is an information processing system of the fifth embodiment in which the processor accepts input of the difference extraction level.

[0015] The information processing system of the eighth embodiment is an information processing system of the fifth embodiment in which the processor displays the type of extracted difference.

[0016] The information processing system of the ninth embodiment, in the information processing system of the fifth embodiment, modifies the portion of the three-dimensional model of the subject that differs from the subject.

[0017] In the tenth aspect of the information processing system, the processor in the ninth aspect of the information processing system accepts input of a difference correction level.

[0018] The program of the 11th embodiment causes the computer to perform the steps of generating a three-dimensional model of the subject from an image of the subject, and displaying information comparing the image of the subject and an image rendered from the three-dimensional model of the subject. [Effects of the Invention]

[0019] According to the information processing system of the first aspect, it is possible to facilitate the confirmation of the three-dimensional model as compared with the case of separately comparing and confirming the subject itself or the image of the subject taken and the three-dimensional model of the subject.

[0020] According to the information processing system of the second aspect, it is possible to simultaneously visually recognize the image of the subject taken and the image obtained by rendering the three-dimensional model of the subject.

[0021] According to the information processing system of the third aspect, it is possible to facilitate the confirmation of the three-dimensional model as compared with the case of displaying an image rendered from a direction different from the shooting direction of the original image of the three-dimensional model.

[0022] According to the information processing system of the fourth aspect, it is possible to generate a rendering image similar to the original image of the three-dimensional model.

[0023] According to the information processing system of the fifth aspect, it is possible to make the user visually recognize the difference portion from the subject in the three-dimensional model.

[0024] According to the information processing system of the sixth aspect, it is possible to extract the difference portion from the subject in the three-dimensional model by comparing two-dimensional images.

[0025] According to the information processing system of the seventh aspect, it is possible to arbitrarily let the user select how much difference is considered a difference by comparing the subjects in the image of the subject taken and the image rendered from the same direction as the shooting direction of the image.

[0026] According to the information processing system of the eighth aspect, it is possible to make the user grasp the type of the extracted difference.

[0027] According to the information processing system of the ninth aspect, it is possible to correct the difference of the three-dimensional model without making the user perform a correction operation.

[0028] According to the information processing system of the tenth embodiment, the user can arbitrarily select the degree of correction processing.

[0029] According to the program of the 11th embodiment, it is possible to verify the three-dimensional model more easily than when the subject itself, or an image of the subject, is individually compared with the three-dimensional model of the subject. [Brief explanation of the drawing]

[0030] [Figure 1] This figure shows the system configuration of a three-dimensional model generation system according to one embodiment of the present disclosure. [Figure 2] This diagram illustrates the shooting direction within the vertical plane of the shooting device of the above three-dimensional model generation system. [Figure 3] This diagram illustrates the horizontal imaging direction of the imaging device of the three-dimensional model generation system described above. [Figure 4] This figure illustrates the multi-view images acquired by the imaging device of the three-dimensional model generation system described above. [Figure 5] This block diagram shows the hardware configuration of the image forming apparatus in the three-dimensional model generation system described above. [Figure 6] This figure shows an example of the three-dimensional model display screen in the three-dimensional model generation system described above. [Figure 7] This figure shows an example of the three-dimensional model modification screen in the three-dimensional model generation system described above. [Figure 8] This figure shows an example of the three-dimensional model modification screen in the three-dimensional model generation system described above. [Figure 9] This figure shows an example of the detailed lighting condition setting screen in the three-dimensional model generation system described above. [Figure 10] This figure shows an example of the detailed difference settings screen in the three-dimensional model generation system described above. [Figure 11] This figure shows an example of the detailed difference settings screen in the three-dimensional model generation system described above. [Figure 12]This flowchart illustrates the user procedure for modifying a three-dimensional model in the three-dimensional model generation system described above. [Figure 13] This figure illustrates the effect of minute surface irregularities noise on a three-dimensional model on the rendered image. [Figure 14] This figure shows an example of the three-dimensional model modification screen in the three-dimensional model generation system described above. [Modes for carrying out the invention]

[0031] Hereinafter, examples of embodiments for carrying out the technology of this disclosure will be described in detail with reference to the drawings. Figure 1 is a diagram showing the system configuration of a three-dimensional model generation system of one embodiment.

[0032] As shown in Figure 1, the three-dimensional model generation system of this embodiment consists of an imaging device 10 and an image processing device 20.

[0033] The photographing device 10 comprises a photographing unit 11 for photographing the subject 30, a lighting unit 12 for illuminating the subject 30 with illumination light, and a mounting stand 13 for placing the subject 30.

[0034] The imaging unit 11 is equipped with multiple cameras 11a that photograph the subject 30 placed on the mounting base 13. The multiple cameras 11a are each installed at different height positions. As shown in Figure 2, the imaging unit 11 can acquire images of the subject 30 taken from different height positions in one direction within the horizontal plane by simultaneously photographing the subject 30 with the multiple cameras 11a.

[0035] In the example shown in Figure 2, the imaging unit 11 includes, for example, five cameras 11a. Each camera 11a is positioned such that its imaging direction in the vertical plane is 20°, 10°, 0°, -10°, and -20° relative to the subject 30.

[0036] The lighting unit 12 is equipped with multiple lights 12a that illuminate the subject 30 placed on the mounting base 13. The multiple lights 12a are each installed at different height positions.

[0037] The mounting base 13 is configured to be rotatable in a horizontal plane. As shown in Figure 3, by rotating the mounting base 13 on which the subject 30 is mounted at a set angular pitch, the relative shooting direction of the shooting unit 11 to the subject 30 can be changed.

[0038] In this embodiment, as an example, the angle pitch of the mounting base 13 is set to 10°. In Figure 3, the indication of the shooting direction is partially omitted and shown in 30° increments.

[0039] The imaging device 10 can acquire multi-view images of the subject 30 as shown in Figure 4 by changing the direction of photography from the imaging unit 11 to the subject 30 and taking multiple shots with the imaging unit 11.

[0040] In this embodiment, within a single horizontal plane, images are captured in five vertical planes at angles of 20°, 10°, 0°, -10°, and -20°. Furthermore, images are captured from 36 horizontal planes with a rotation angle pitch of 10°, ranging from 0° to 350° relative to the subject 30. As a result, multi-view images of the subject 30 from 180 different directions can be obtained.

[0041] For each image in the multi-view imagery, the shooting conditions, such as the shooting direction in the vertical plane, the shooting direction in the horizontal plane, and the lighting position, are known. Each image in the multi-view imagery, along with these shooting conditions, is recorded in the image processing device 20.

[0042] The image processing device 20 generates a three-dimensional model of the subject 30 using multi-view images of the subject 30 acquired by the imaging device 10. For example, a personal computer can be used as the image processing device 20. The image processing device 20 is an example of an information processing system in the technology of this disclosure.

[0043] Next, the hardware configuration of the image processing device 20 in this embodiment will be described. Figure 5 is a block diagram showing the hardware configuration of the image processing device 20.

[0044] As shown in Figure 5, the image processing device 20 comprises a control unit 21, a communication interface (abbreviated as communication IF) 22, a user interface device (abbreviated as UI device) 23, a monitor 24, and an input / output interface (abbreviated as input / output IF) 25. These components are connected to each other via a control bus 26.

[0045] The control unit 21 comprises a processor 21a, a memory 21b, and a storage unit 21c. The processor 21a executes predetermined processing based on a program read from the storage unit 21c and expanded into the memory 21b. The storage unit 21c is composed of, for example, a ROM, HDD, or SSD. Various programs and data are stored in the storage unit 21c.

[0046] In this embodiment, the processor 21a is described as reading and executing a program stored in the memory unit 21c, but it is not limited to this. This program may be provided in the form of a computer-readable recording medium as described above. Alternatively, this program may be obtained from an external device via a communication line.

[0047] The communication IF 22 transmits and receives data to and from external devices. The UI device 23 is a device for the user to receive and input information, such as a keyboard and / or mouse. The monitor 24 displays information such as images generated by the control unit 21. The input / output IF 25 is an interface for connecting peripheral devices such as the imaging device 10.

[0048] The control unit 21 of the image processing device 20 generates a three-dimensional model of the subject 30 using multi-view images of the subject 30 acquired by the imaging device 10.

[0049] In the three-dimensional model generation system of this embodiment, the generation of a three-dimensional model of the subject 30 is performed as follows, as an example.

[0050] First, the control unit 21 uses a technique called SFM (Structure From Motion) to estimate the position and shooting direction of the camera that captured each of the multi-view images of the subject 30.

[0051] Next, the control unit 21 performs point cloud measurement using multi-view image measurement based on triangulation, employing a method called MVS (Multi View Stereo), to acquire a three-dimensional point cloud of the subject 30.

[0052] Next, the control unit 21 connects the three-dimensional point cloud with a triangular or quadrilateral mesh to form a surface. Furthermore, the control unit 21 uses multi-view images to generate a texture corresponding to the formed surface.

[0053] Since these processes are all publicly known technologies, detailed explanations will be omitted.

[0054] As described above, the generated three-dimensional model may have differences from the subject 30. However, it is time-consuming to individually compare the image of the subject 30 with the three-dimensional model to confirm the differences between the subject 30 and the three-dimensional model.

[0055] To resolve these issues, the control unit 21 in the image processing device 20 generates a three-dimensional model of the subject 30 from an image of the subject 30, and displays information comparing the image of the subject 30 and the rendered image of the three-dimensional model of the subject 30.

[0056] Here, "displaying information comparing an image of subject 30 and an image of a rendered three-dimensional model of subject 30" can be described in two ways, for example:

[0057] In one embodiment, the control unit 21 may display the image of the subject 30 and the rendered image of the three-dimensional model of the subject 30 on the same screen, as shown in Figure 6.

[0058] In this case, the control unit 21 preferably displays on the same screen, as shown in Figure 6, an image of the subject 30 taken and an image of a three-dimensional model of the subject 30 rendered from the same direction as the image taken.

[0059] In the following, each image of the subject 30, that is, each of the multi-view images of the subject 30 acquired by the imaging device 10, will be referred to as the subject image. In addition, the image obtained by rendering the three-dimensional model of the subject 30 will be referred to as the rendered image.

[0060] As an example, the three-dimensional model display screen 40 shown in Figure 6 includes a display direction input unit 41, a subject image display unit 42, a rendering image display unit 43, and a subject image selection button 44.

[0061] The display direction input unit 41 is for inputting the shooting direction to be displayed for the subject image and the rendered image. The three-dimensional model of the subject 30 is displayed on the display direction input unit 41.

[0062] The three-dimensional model of the subject 30 displayed in the display direction input unit 41 may be the same as the rendered image displayed in the rendering image display unit 43 described later, or it may be an image with lower image quality than the rendered image.

[0063] The subject image display unit 42 is an area that displays one of the subject images among the multiple subject images that make up the multi-view image.

[0064] The rendering image display unit 43 is an area that displays an image rendered from a three-dimensional model.

[0065] The subject image selection button 44 is a button used to select the subject image to be displayed on the subject image display unit 42.

[0066] There are two ways to select the images to be displayed on the subject image display unit 42 and the rendering image display unit 43.

[0067] First, we will explain the case in which the subject image and the rendered image are displayed based on the display direction input in the display direction input unit 41.

[0068] When the user moves the mouse vertically in the display direction input unit 41, the control unit 21 rotates the three-dimensional model of the subject 30 vertically according to the amount of mouse movement. Also, when the user moves the mouse horizontally in the display direction input unit 41, the control unit 21 rotates the three-dimensional model of the subject 30 horizontally according to the amount of mouse movement.

[0069] When the display direction is changed in the display direction input unit 41, the control unit 21 displays the subject image in the shooting direction corresponding to the input display direction on the subject image display unit 42.

[0070] Here, to obtain the shooting direction of each subject image, for example, the shooting direction information estimated by the SFM processing described above for each subject image may be used.

[0071] Furthermore, as described above, each subject image is recorded along with the shooting conditions at the time of shooting, such as the shooting direction in the vertical plane, the shooting direction in the horizontal plane, and the lighting position. Therefore, the shooting direction of each subject image may be obtained from the shooting condition information recorded together with each subject image. Alternatively, the shooting direction of each subject image may be obtained by using both the shooting direction information estimated by SFM processing and the shooting condition information recorded together with each subject image.

[0072] Furthermore, it is preferable that the shooting direction of the subject image displayed on the subject image display unit 42 is exactly the same as the display direction input in the display direction input unit 41, but it may be slightly off to account for errors.

[0073] For example, the horizontal error may include an error of, for example, ± rotational angle pitch / 2 (°), taking into account the rotational angle pitch of the mounting base 13 when the imaging device 10 captures multi-view images of the subject 30.

[0074] Specifically, if the horizontal rotation angle pitch of the mounting base 13 is set to 10° and 36 shots are taken around the entire circumference of the subject 30, there may be an error of ±5° in the horizontal direction.

[0075] Furthermore, the vertical error may include, for example, an error of ±shooting angle pitch / 2(°), taking into account the arrangement pitch of the cameras 11a in the shooting unit 11 of the shooting device 10 and the shooting angle pitch determined from the distance from the shooting unit 11 to the mounting base 13.

[0076] Specifically, if the shooting direction pitch in the vertical plane is 10°, it may include an error of ±5° in the vertical direction.

[0077] Furthermore, it is preferable to display the subject image taken from the closest shooting direction, based on the display direction input in the display direction input unit 41.

[0078] Furthermore, the control unit 21 updates the rendering image displayed on the rendering image display unit 43 when the display direction is changed in the display direction input unit 41.

[0079] The rendering image displayed on the rendering image display unit 43 may be a rendering image taken from the same direction as the subject image displayed on the subject image display unit 42. Alternatively, the rendering image displayed on the rendering image display unit 43 may be a rendering image taken from the same direction as the display direction input in the display direction input unit 41.

[0080] However, in order to compare the subject image and the rendered image, it is preferable that the shooting direction of the rendered image displayed on the rendered image display unit 43 is the same as the shooting direction of the subject image displayed on the subject image display unit 42.

[0081] At this time, the control unit 21 may generate an image of a rendered three-dimensional model of the subject 30 using the shooting condition information recorded together with each subject image. By using the shooting condition information, it becomes possible to generate a rendered image with the same shooting direction and lighting conditions as the subject image.

[0082] Based on the above, in this embodiment, for example, if a display direction of 20° horizontally and 10° vertically is input to the display direction input unit 41, the control unit 21 displays the subject image captured with a shooting direction of 20° horizontally and 10° vertically on the subject image display unit 42. In addition, the control unit 21 displays the rendered image captured with a shooting direction of 20° horizontally and 10° vertically on the rendered image display unit 43.

[0083] Furthermore, if a display direction of 27° horizontally and 12° vertically is input to the display direction input unit 41, the control unit 21 displays the subject image captured in the shooting direction of 30° horizontally and 10° vertically, which is closest to the input display direction, on the subject image display unit 42. In addition, the control unit 21 displays the rendered image captured in the same shooting direction of 30° horizontally and 10° vertically as the subject image on the rendered image display unit 43.

[0084] Next, we will explain how to display the subject image and the rendered image based on the subject image selected from the subject image selection button 44.

[0085] When the subject image selection button 44 is selected, the control unit 21 displays an image file selection screen and accepts input from the user for the subject image to be displayed.

[0086] The control unit 21 displays the subject image input by the user on the subject image display unit 42. The control unit 21 also displays a rendered image, taken from the same direction as the subject image displayed on the subject image display unit 42, on the rendered image display unit 43.

[0087] Next, we will explain the second aspect of "displaying information comparing an image of subject 30 with an image of a rendered three-dimensional model of subject 30."

[0088] In a second embodiment, as shown in Figures 7 and 8, the control unit 21 may compare an image of the subject 30 with an image of a rendered three-dimensional model of the subject 30, extract areas in the three-dimensional model of the subject 30 that differ from the subject 30, and display the extracted areas of difference.

[0089] Here, the control unit 21 may extract the differences between the three-dimensional model of the subject 30 and the image of the subject by extracting the difference between at least one piece of information such as the color, brightness, and pattern of the subject in the image of the subject and the image rendered from the same direction as the image being taken.

[0090] Here, the control unit 21 may accept input for the difference extraction level, or it may display the type of difference that has been extracted.

[0091] Furthermore, the control unit 21 may correct any differences between the three-dimensional model of the subject 30 and the actual subject 30. In this case, the control unit 21 may accept input for the difference correction level.

[0092] As an example, the three-dimensional model modification screen 50 shown in Figure 7 includes a display direction input unit 51, a confirmation button 52, an image display unit 53, a lighting condition setting button 54, a difference detection level setting button 55, a difference extraction button 56, and a cancel button 57.

[0093] The display direction input unit 51 is for inputting the shooting direction to be displayed for the subject image and the rendered image. The display direction input unit 51 displays a three-dimensional model of the subject 30.

[0094] The confirmation button 52 is a button for displaying a group of comparison images consisting of a subject image, a rendered image, and a difference-highlighted rendered image that highlights the differences between the subject image and the rendered image, corresponding to the display direction input in the display direction input unit 51.

[0095] The shooting direction of each image displayed at this time—the subject image, the rendered image, and the difference-enhanced rendering image—is the same as that of the three-dimensional model display screen 40 described above.

[0096] However, before the difference detection button 56 (described later) is pressed and the control unit 21 performs difference detection between the subject image and the rendered image, the difference-highlighted rendered image will not be displayed even if the confirmation button 52 is pressed.

[0097] The image display unit 53 is an area that displays a group of comparison images for each display direction.

[0098] The lighting condition setting button 54 is a button for selecting the subject image to be displayed on the subject image display unit 42.

[0099] When the lighting condition setting button 54 is pressed, the control unit 21 displays the lighting condition detailed setting screen 70 as shown in Figure 9. On the lighting condition detailed setting screen 70, it is possible to set the position, angle, and illuminance of the lighting used when generating the rendering image. In addition, it is possible to set the position and angle of the camera used when generating the rendering image on the lighting condition detailed setting screen.

[0100] The difference detection level setting button 55 is used to set the detection level when detecting differences between the subject image and the rendered image.

[0101] When the difference detection level setting button 55 is pressed, the control unit 21 displays a difference detection setting screen (not shown). On the difference detection setting screen, it is possible to set how much difference is required between the subject image and the rendered image to be recognized as a difference. For inputting the degree of difference, for example, buttons for large, medium, and small may be provided, or a slider bar may be provided.

[0102] The difference detection button 56 is a button used to perform difference detection between the subject image and the rendered image.

[0103] The cancel button 57 is used to exit the 3D model modification screen 50.

[0104] In the 3D model modification screen 50, when the difference extraction button 56 is pressed, the control unit 21 extracts the differences between the subject image and the rendered image for each type of difference. In the 3D model modification screen 50, as an example, three types of differences are detected: holes, chips, and debris.

[0105] Here, a "hole" refers to a missing area in the three-dimensional model's shape that is completely surrounded by a normal area. A "chip" refers to a missing area in the three-dimensional model's shape that is only partially surrounded by a normal area. And "debris" refers to an object that does not exist in the subject in the three-dimensional model's shape.

[0106] These differences are detected by extracting the difference in at least one piece of information regarding the shape, color, brightness, and pattern of the subject 30 between the subject image and the rendered image, both taken from the same shooting direction.

[0107] Furthermore, when detecting differences between the subject image and the rendered image, it is possible to detect not only differences in the shape of the three-dimensional model as described above, but also differences in the textures reflected in the three-dimensional model. Texture differences may be detected by extracting the difference of at least one piece of information such as the color, brightness, and pattern of the subject 30 from the subject image and the rendered image taken from the same shooting direction. In addition, if texture differences are detected, the texture image may be modified to approximate the subject image, similar to the modification of the shape of the three-dimensional model described later.

[0108] Specifically, regarding the shape of the three-dimensional model, for example, this can be done by extracting the contour lines of subject 30 from subject images taken from the same shooting direction and from the rendered image of subject 30, and comparing the two. Regarding the texture reflected in the three-dimensional model, for example, this can be done by comparing the same region of subject 30 in subject images and rendered images taken from the same shooting direction. Furthermore, the detection of differences is not limited to the methods described above; any method may be used. Since these processes are all publicly known techniques, detailed explanations are omitted.

[0109] After extracting the differences, the control unit 21 displays a three-dimensional model of the subject 30 with the differences shown in the display direction input unit 51, as shown in Figure 8.

[0110] When the confirmation button 52 is pressed, the control unit 21 displays a group of comparison images 65 on the image display unit 53, consisting of a subject image 65a, a rendering image 65b, and a difference-highlighted rendering image 65c that highlights the differences between the subject image and the rendering image, corresponding to the display direction input in the display direction input unit 51.

[0111] When the difference-highlighted rendering image 65c in the comparison image group 65 is pressed, the control unit 21 displays the difference detail settings screen 80 as shown in Figure 10.

[0112] As shown in Figure 10, the difference details setting screen 80 includes a difference highlighting rendering image display unit 81 and a group of operation buttons 82.

[0113] The difference-highlighted rendering image display unit 81 displays an enlarged image of the difference-highlighted rendering image. As an example, Figure 10 shows the state in which the differences in holes 86a, 86b, 86c and the differences in chips 87 have been extracted from the three-dimensional model of the coffee can 85, which is the subject 30.

[0114] In Figure 10, the differences in holes 86a, 86b, and 86c, and the difference in chips 87 are shown in the same color, but different colors may be used for each type of difference. Also, as described later, after correcting the differences, the corrected areas may be shown in a different color from the areas with differences.

[0115] The operation button group 82 includes buttons for selection, addition, deletion, scaling, and confirmation, enabling the user to perform the following operations.

[0116] On the detailed difference settings screen, users can manually add and delete differences. Furthermore, on the detailed difference settings screen, users can change the scope of differences by clicking on the image of the difference.

[0117] Furthermore, if the information regarding the differences is changed on the detailed difference settings screen, the control unit 21 will also reflect the correction results in the images of other shooting directions displayed on the image display unit 53.

[0118] Furthermore, as shown in Figure 8, after extracting the differences, the control unit 21 displays the correction strength setting unit (hole) 58, the correction strength setting unit (chip) 59, and the correction strength setting unit (debris) 60 on the three-dimensional model correction screen 50.

[0119] The correction strength setting section (hole) 58 is a group of buttons for setting the correction strength for the difference in holes, and includes buttons for large, medium, small, and detailed.

[0120] The large, medium, and small buttons are used to uniformly set the correction strength for differences in all holes to either large, medium, or small.

[0121] The Details button is used to individually set the correction strength for each difference in the holes. When the Details button is pressed, the control unit 21 displays the Difference Details setting screen 90 as shown in Figure 11. On the Difference Details setting screen 90, each difference in the holes is displayed individually, and the correction strength can be set for each difference using a slider bar.

[0122] The correction strength setting section (for chips) 59 is a group of buttons for setting the correction strength of the chip difference, and includes buttons for large, medium, small, and detailed. The function of each button is the same as that of the correction strength setting section (for holes) 58.

[0123] The correction strength setting section (dust) 60 is a group of buttons for setting the correction strength of the dust difference, and includes buttons for large, medium, small, and detailed. The function of each button is the same as that of the correction strength setting section (hole) 58.

[0124] Furthermore, the display of the correction strength setting section is not limited to displaying the correction strength setting section (hole) 58, the correction strength setting section (chip) 59, and the correction strength setting section (debris) 60, regardless of the type of difference extracted as described above.

[0125] For example, only the correction strength setting section corresponding to the type of difference extracted may be displayed. Specifically, if the extracted differences are hole differences 86a, 86b, 86c and chip differences 87, only the correction strength setting section (hole) 58 and the correction strength setting section (chip) 59 may be displayed.

[0126] Furthermore, regarding the display of the correction intensity setting section, it is possible to display the correction intensity setting section for each type of difference in an inactive state before the difference extraction is performed, and then, after the difference extraction is performed, to display only the correction intensity setting section for the extracted difference type in an active state.

[0127] Furthermore, as shown in Figure 8, after extracting the differences, the control unit 21 displays a simple modification execution button 61, a modification execution button 62, and a model save button 63 on the three-dimensional model modification screen 50.

[0128] The quick correction execution button 61 is a button for performing quick corrections on the three-dimensional model of the subject 30. Quick correction performs a uniform strength correction for all differences, regardless of the settings of the correction strength setting unit (holes) 58, correction strength setting unit (chips) 59, and correction strength setting unit (debris) 60. When the quick correction execution button 61 is pressed, the control unit 21 performs quick corrections on the three-dimensional model.

[0129] The correction execution button 62 is a button for executing corrections to the three-dimensional model of the subject 30 according to the settings of the correction intensity setting unit (hole) 58, the correction intensity setting unit (chip) 59, and the correction intensity setting unit (debris) 60. As described above, after extracting the differences and setting the correction intensity of the extracted differences, when the correction execution button 62 is pressed, the control unit 21 executes the correction of the three-dimensional model according to the set content.

[0130] The model save button 63 is used to save a three-dimensional model of the subject 30. When the model save button 63 is pressed, the control unit 21 performs the saving of the current three-dimensional model.

[0131] Next, an example of the user's procedure when modifying a three-dimensional model on the three-dimensional model modification screen 50 will be explained with reference to the flowchart in Figure 12.

[0132] First, in step S01, the user presses the lighting condition setting button 54 to set the lighting conditions for generating the rendered image.

[0133] Next, in step S02, the user presses the difference detection level setting button 55 to set the detection level for detecting differences between the subject image and the rendered image.

[0134] Next, in step S03, the user presses the difference extraction button 56 to perform the extraction of differences between the subject image and the rendered image.

[0135] When the difference detection button 56 is pressed, the control unit 21, in step S04, highlights the differences between the subject image and the rendered image, and displays the correction intensity setting section.

[0136] Next, in step S05, the user presses the confirmation button 52, and in step S06, a group of comparison images consisting of a subject image, a rendering image, and a difference-enhanced rendering image corresponding to the display direction input in the display direction input unit 51 is displayed.

[0137] Here, in step S07, the user determines whether or not the extraction of differences in the three-dimensional model of the subject 30 is appropriate.

[0138] If, in step S07, it is determined that the extraction of differences in the three-dimensional model of subject 30 is inappropriate, the user proceeds to step S02 and restarts the detection level setting process.

[0139] If, in step S07, the user determines that the extraction of differences in the three-dimensional model of the subject 30 is appropriate, the user modifies the three-dimensional model in step S08.

[0140] Here, in step S09, the user determines whether the modification result to the three-dimensional model of the subject 30 is appropriate.

[0141] If, in step S09, the user determines that the modification result for the three-dimensional model of subject 30 is inappropriate, the user proceeds to step S08 and restarts the modification of the three-dimensional model.

[0142] If, in step S09, the user determines that the modification result for the three-dimensional model of the subject 30 is appropriate, the user saves the three-dimensional model in step S10 and terminates the three-dimensional model modification.

[0143] Furthermore, the differences between the subject and the 3D model are not limited to holes, chips, and dust as described above. The microscopic surface irregularities of the 3D model may change due to microscopic noise.

[0144] Figure 13 illustrates the effect of minute surface irregularities noise on a three-dimensional model on the rendered image.

[0145] In Figure 13, Image G1 is a subject image of Subject 30. Image G2 is a rendering image without texture showing the presence of minute surface irregularities noise on the three-dimensional model of Subject 30. Image G3 is a rendering image with texture showing the presence of minute surface irregularities noise on the three-dimensional model of Subject 30. Image G4 is a rendering image without texture showing the absence of minute surface irregularities noise on the three-dimensional model of Subject 30. Image G5 is a rendering image with texture showing the absence of minute surface irregularities noise on the three-dimensional model of Subject 30.

[0146] As shown in Image G1, the subject 30 is a coffee can with a smooth surface. As shown in Image G2, if minute surface irregularities occur on the surface of the three-dimensional model of the coffee can, as shown in Image G3, the surface of the coffee can will appear rough, and the texture will be displayed differently.

[0147] For a three-dimensional model with minute surface irregularities and noise, as shown in image G4, applying a smoothing process results in a final rendered image that closely resembles the surface of the original coffee can, as shown in image G5.

[0148] Conversely, although not illustrated, if a smoothing process is applied to a three-dimensional model of an object with minute irregularities on its surface, such as an unglazed pot, the surface of the pot will become smooth in the final rendered image, resulting in a different appearance of texture.

[0149] The control unit 21 may also modify the minute surface irregularities of such a three-dimensional model.

[0150] As an example, the three-dimensional model modification screen 100 shown in Figure 14 includes a load button 101, a subject image display unit 102, and a group of operation buttons 103. The load button 101 is a button for loading a subject image. The subject image display unit 102 is an area for displaying the loaded subject image. The group of operation buttons 103 is a group of buttons for performing various operations on the loaded subject image.

[0151] The 3D model modification screen 100 also includes a load button 104, a rendering image display unit 105, and a group of operation buttons 106. The load button 104 is a button for loading a rendering image. The rendering image display unit 105 is an area for displaying the loaded rendering image. The group of operation buttons 106 is a group of buttons for performing various operations on the loaded rendering image.

[0152] The 3D model modification screen 100 also includes a setting button 107, an execution button 108, and a rendering image display unit 109. The setting button 107 is for setting the intensity of the smoothing process applied to the 3D model. The execution button 108 is for executing the smoothing process on the 3D model. The rendering image display unit 109 is an area for displaying a rendering image of the 3D model after the smoothing process.

[0153] On the 3D model modification screen 100, the user can set different areas of the subject for both the subject image and the rendered image. For example, the top and side surfaces of the coffee can, which is the subject, can be set to different areas. The area settings can be automatically recognized and set by the control unit 21 when the recognition button of the operation button group 106 is pressed, or they can be set by the user using drag operations.

[0154] Furthermore, in the 3D model modification screen 100, the same parts of the subject can be displayed in correspondence with each other for each region of the subject set in the subject image and the rendering image. The correspondence between regions can be automatically recognized and matched by the control unit 21, or it can be matched by the user's drag operation. In addition, for example, the same parts of the subject can be enclosed by lines of the same color and / or the same line type.

[0155] In this way, by displaying the same areas of the subject 30 in both the subject image and the rendered image in correspondence, the user can determine how much the minute surface irregularities of the three-dimensional model need to be corrected.

[0156] The user presses the settings button 107 to set the intensity of the smoothing process for the 3D model. Next, the user presses the execute button 108.

[0157] When the execute button 108 is pressed, the control unit 21 performs a smoothing process on the three-dimensional model according to the set settings.

[0158] [Differentiation] Although a three-dimensional model generation system according to one embodiment of the present disclosure has been described above, the technology of the present disclosure is not limited to the above embodiment and can be modified as appropriate.

[0159] For example, the method for generating the three-dimensional model of the subject 30 is not limited to the method described in the above embodiment, but can be any method.

[0160] Furthermore, the method of generating a three-dimensional model is not limited to multi-view images that include multiple images of the subject 30 taken from multiple directions, but may also be used, for example, to generate a three-dimensional model from only one image using AI (Artificial Intelligence).

[0161] Furthermore, the highlighting of differences may be done not only on the rendered image, but also on the subject image, or on both the subject image and the rendered image.

[0162] In each of the embodiments described above, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0163] Furthermore, the operation of the processor in the above embodiments may not be performed by a single processor, but may be performed by multiple processors located in physically separate locations working together. Also, the order of the processor operations is not limited to the order described in each of the above embodiments, but may be changed as appropriate.

[0164] Furthermore, in the technology of this disclosure, a system includes both systems composed of multiple devices and systems composed of a single device.

[0165] Furthermore, the technology disclosed herein can also be applied to programs and program products.

[0166] [Note] With regard to the embodiments described above, the following additional information is disclosed.

[0167] (((1))) Equipped with a processor, The aforementioned processor, A three-dimensional model of the subject is generated from an image of the subject, This displays information comparing an image of the subject and an image of a rendered 3D model of the subject. Information processing system.

[0168] (((2))) The aforementioned processor,

[0169] The image of the subject and the rendered 3D model of the subject are displayed on the same screen. The information processing system described in (((1))).

[0170] (((3))) The aforementioned processor, The image of the subject and the image of a three-dimensional model of the subject, rendered from the same direction as the image was taken, are displayed on the same screen. The information processing system described in (((2))).

[0171] (((4))) The aforementioned processor, Using information regarding the camera used to photograph the subject, and information regarding the lighting conditions when the subject was photographed, an image is generated that renders a three-dimensional model of the subject. The information processing system described in (((3))).

[0172] (((5))) The aforementioned processor, The image of the subject is compared with the image of the rendered three-dimensional model of the subject. In the three-dimensional model of the subject, we extract the parts that differ from the subject. Display the extracted differences. An information processing system as described in any one of items (((1))) through (((4))).

[0173] (((6))) The aforementioned processor, By extracting the difference between at least one piece of information regarding the subject's color, brightness, and pattern in an image of the subject and an image rendered from the same direction as the image was taken, the areas in the three-dimensional model of the subject that differ from the subject are extracted. The information processing system described in (((5))).

[0174] (((7))) The aforementioned processor, Accepts input for the difference detection level. The information processing system described in (((5))) or (((6))).

[0175] (((8))) The aforementioned processor, Display the types of differences extracted. An information processing system as described in any one of items (((5))) through (((7))).

[0176] (((9))) The aforementioned processor, In the three-dimensional model of the subject, any differences between the model and the actual subject are corrected. An information processing system as described in any one of items (((5))) through (((8))).

[0177] (((10))) The aforementioned processor, Accepts input for the difference correction level. The information processing system described in (((9))).

[0178] (((11))) A step of generating a three-dimensional model of the subject from an image of the subject, The steps include displaying information comparing an image of the subject with an image of a rendered three-dimensional model of the subject, A program that causes a computer to execute something.

[0179] The effects of the configuration described below are explained below.

[0180] According to the information processing system (((1))), it is possible to verify the three-dimensional model more easily than when comparing the subject itself, or an image of the subject taken, with the three-dimensional model of the subject individually.

[0181] According to the information processing system (((2))), it is possible to simultaneously view an image of the subject taken with an image of a rendered three-dimensional model of the subject.

[0182] According to the information processing system (((3))), it is possible to verify the three-dimensional model compared to displaying an image rendered from a different direction than the original image of the three-dimensional model was captured.

[0183] According to the information processing system (((4))), it is possible to generate a rendered image similar to the original image of the three-dimensional model.

[0184] According to the information processing system (((5))), the user can visually identify the differences between the three-dimensional model and the actual subject.

[0185] According to the information processing system (((6))), by comparing two-dimensional images, it is possible to extract the differences between the two-dimensional images and the subject in a three-dimensional model.

[0186] According to the information processing system (((7))), the user can arbitrarily select how much difference to consider when comparing the subject in an image of the subject taken with an image rendered from the same direction as the image taken.

[0187] According to the information processing system (((8))), the user can be made aware of the types of differences that have been extracted.

[0188] According to the information processing system (((9))), differences in a three-dimensional model can be corrected without requiring the user to perform any correction work.

[0189] According to the information processing system (((10))), the user can arbitrarily select the degree of correction processing.

[0190] According to the program (((11))), it is possible to verify the three-dimensional model more easily than by individually comparing the subject itself or an image of the subject with the three-dimensional model of the subject. [Explanation of Symbols]

[0191] 10. Imaging device 11. Photography Department 11a camera 12 Lighting Section 12a light 13 Installation stand 20 Image Processing Devices 21 Control Unit 21a Processor 21b Memory 21c storage section 22 Communication Interfaces 23 User Interface Device 24 monitors 25 Input / Output Interfaces 26 Control bus

Claims

1. Equipped with a processor, The aforementioned processor, A three-dimensional model of the subject is generated from an image of the subject, This displays information comparing an image of the subject and an image of a rendered 3D model of the subject. Information processing system.

2. The aforementioned processor, The image of the subject and the rendered 3D model of the subject are displayed on the same screen. The information processing system according to claim 1.

3. The aforementioned processor, The image of the subject and the image of a three-dimensional model of the subject, rendered from the same direction as the image was taken, are displayed on the same screen. The information processing system according to claim 2.

4. The aforementioned processor, Using information regarding the camera used to photograph the subject, and information regarding the lighting conditions when the subject was photographed, an image is generated that renders a three-dimensional model of the subject. The information processing system according to claim 3.

5. The aforementioned processor, The image of the subject is compared with the image of the rendered three-dimensional model of the subject. In the three-dimensional model of the subject, we extract the parts that differ from the subject. Display the extracted differences. The information processing system according to claim 1.

6. The aforementioned processor, By extracting the difference between at least one piece of information regarding the subject's color, brightness, and pattern in an image of the subject and an image rendered from the same direction as the image was taken, the areas in the three-dimensional model of the subject that differ from the subject are extracted. The information processing system according to claim 5.

7. The aforementioned processor, Accepts input for the difference detection level. The information processing system according to claim 5.

8. The aforementioned processor, Display the types of differences extracted. The information processing system according to claim 5.

9. The aforementioned processor, In the three-dimensional model of the subject, any differences between the model and the actual subject are corrected. The information processing system according to claim 5.

10. The aforementioned processor, Accepts input for the difference correction level. The information processing system according to claim 9.

11. A step of generating a three-dimensional model of the subject from an image of the subject, The steps include displaying information comparing an image of the subject with an image of a rendered three-dimensional model of the subject, A program that causes a computer to execute something.

Images