Image processing device and image processing method

Abstract

The present technology pertains to an image processing device and an image processing method that enable a user to more suitably perform correction work on shape data. An image processing device according to the present technology comprises: an input control unit that receives, from a user, a designation of a correction range to be corrected by the user, out of a predetermined range including scan data, which is shape data indicating a 3D shape of a subject, acquired using a depth sensor that measures the 3D shape of the subject; and a display control unit that causes a model preview, which is an image obtained by rendering the shape data included in the correction range, to be displayed in a superimposed manner on a reference preview, which is an image that is based on a captured image acquired by an image sensor capturing an image of the subject and that serves as a guide when the user performs correction work. The present technology can be applied to, for example, a 3D modeling system.

Description

Image processing device and image processing method

[0001] The present technology relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method that enable a user to more suitably perform correction work on shape data.

[0002] 3D scanners are known that measure the 3D shape of an object by irradiating it with light from various viewpoints, and acquire shape data that indicates the object's 3D shape. However, due to the principle of measuring the 3D shape of an object by irradiating it with light, 3D scanners may not be able to measure the 3D shape correctly depending on the object.

[0003] If the 3D shape of the subject cannot be measured correctly, for example, the user manually corrects the shape data. Patent Literature 1 describes a technology in which a user uses a pen-type input device to annotate, extend, refine, or modify a 3D model generated based on data obtained by scanning the surface of a physical object with a 3D scanner.

[0004] Special Publication No. 2023-532090

[0005] It is difficult for users to determine the correct 3D shape of a subject just by looking at the shape data, and even after making corrections, it is difficult to obtain shape data that accurately represents the subject's actual 3D shape.

[0006] The present technology has been made in view of such circumstances, and enables a user to more suitably perform correction work on shape data.

[0007] An image processing device according to one aspect of the present technology includes an input control unit that receives from a user a specification of a correction range that is to be modified by the user within a predetermined range that includes scan data, which is shape data indicating the 3D shape of a subject, acquired using a depth sensor that measures the 3D shape of the subject, and a display control unit that displays a model preview, which is an image obtained by rendering the shape data included in the correction range, superimposed on a reference preview, which is an image based on a captured image acquired by an image sensor by photographing the subject, and which serves as a guide for the user when performing the modification work.

[0008] An image processing method according to one aspect of the present technology includes receiving from a user a specification of a correction range that is to be modified by the user within a predetermined range that includes scan data, which is shape data indicating the 3D shape of a subject, acquired using a depth sensor that measures the 3D shape of the subject; and displaying a model preview, which is an image obtained by rendering the shape data included in the correction range, superimposed on a reference preview, which is an image based on a captured image acquired by an image sensor by photographing the subject, and which serves as a guide for the user when performing the modification work.

[0009] In one aspect of the present technology, a user specifies a correction range that is to be modified by the user within a predetermined range that includes scan data, which is shape data indicating the 3D shape of the subject, obtained using a depth sensor that measures the 3D shape of the subject, and the user specifies a model preview, which is an image that renders the shape data included in the correction range, and the model preview is displayed superimposed on a reference preview, which is an image based on a captured image obtained by photographing the subject with an image sensor and serves as a guide for the user when performing the modification work.

[0010] 1 is a block diagram showing an example configuration of a 3D modeling system according to an embodiment of the present technology; FIG. 2 is a diagram showing an example of a viewpoint of an RGB camera; FIG. 3 is a diagram showing an example of shape data acquired using a 3D scanner; FIG. 4 is a diagram showing another example of shape data acquired using a 3D scanner; FIG. 5 is a diagram explaining a correction operation on scan data; FIG. 6 is a block diagram showing an example configuration of an image processing device; FIG. 7 is a block diagram showing a first modified example of the configuration of the image processing device; FIG. 8 is a first diagram showing an example display of a reference preview and a model preview; FIG. 9 is a block diagram showing a second modified example of the configuration of the image processing device; FIG. 10 is a diagram showing an example of shape data having a simple shape; FIG. 11 is a second diagram showing an example display of a reference preview and a model preview; FIG. 11 is a block diagram showing a detailed example configuration of a refinement unit; FIG. 12 is a flowchart explaining processing performed by the 3D modeling system; FIG. 13 is a block diagram showing an example configuration of computer hardware.

[0011] Hereinafter, an embodiment of the present technology will be described. The description will be made in the following order: 1. Overview of 3D modeling system 2. Configuration of image processing device 3. Operation of 3D modeling system

[0012] 1. Overview of 3D Modeling System FIG. 1 is a block diagram showing an example of the configuration of a 3D modeling system according to an embodiment of the present technology.

[0013] 1, the 3D modeling system of the present technology is composed of a 3D scanner 1, an RGB camera 2, and an image processing device 3. The 3D scanner 1 and the RGB camera 2 are examples of devices including a depth sensor and an image sensor, respectively, in the present disclosure.

[0014] The 3D scanner 1 is a device including a depth sensor that uses, for example, a stereo camera system, a structured light system, or a time-of-flight (ToF) system. The 3D scanner 1 performs a 3D scan by illuminating the subject with light, thereby measuring the 3D shape of the subject as the distance to a point (a ranging point) on the surface of the subject. The 3D scanner 1 performs 3D scans from multiple viewpoints and generates shape data indicating the 3D shape of the subject based on the results of the 3D scans from each viewpoint. The 3D scanner 1 measures the 3D shape of the same subject as the one photographed by the RGB camera 2.

[0015] The 3D scanner 1 supplies the generated shape data as scan data to the image processing device 3. Note that the shape data may be generated by the image processing device 3. In this case, the 3D scanner 1 supplies the results of the 3D scan to the image processing device 3.

[0016] The RGB camera 2 photographs an object from a plurality of viewpoints and supplies the photographed images obtained to the image processing device 3. Note that instead of the RGB camera 2, for example, a monochrome camera or an infrared camera may be used.

[0017] FIG. 2 is a diagram showing an example of the viewpoint of the RGB camera 2.

[0018] 2, the RGB camera 2 captures images of a plastic model as the subject Obj1 from multiple viewpoints 2-1 to 2-n that are set to surround the subject Obj1. The 3D scanner 1 measures the 3D shape of the subject Obj1 from multiple viewpoints that are set to surround the subject Obj1, just like the RGB camera 2. Note that the viewpoints of the 3D scanner 1 and the RGB camera 2 do not necessarily have to correspond to each other.

[0019] A user of the 3D modeling system takes photographs and performs 3D scanning while moving, for example, one 3D scanner 1 and one RGB camera 2. If the 3D scanner 1 and the RGB camera 2 are fixed on a turntable, the user can take photographs and perform 3D scanning while moving the 3D scanner 1 and the RGB camera so that they rotate around the subject Obj1. Multiple 3D scanners 1 and multiple RGB cameras 2 may be installed to surround the subject Obj1, and photographs and 3D scanning may be performed from each viewpoint.

[0020] Returning to FIG. 1 , the image processing device 3 is configured with a PC, a cloud server, etc. The image processing device 3 executes a digital content creation tool (DCC tool) and displays a model preview, which is an image rendered from the scan data acquired by the 3D scanner 1, on a display unit (not shown). The user can perform corrections to the scan data by operating the DCC tool. The display unit of the image processing device 3 displays a reference preview, which is an image that serves as a guide for the user when performing corrections, along with the model preview. The reference preview is an image based on the captured image acquired by the RGB camera 2.

[0021] The image processing device 3 generates a 3D model that represents the 3D shape, color, pattern, texture, etc. of the subject by pasting (texture mapping) multiple captured images onto the surface of the 3D shape represented by the corrected shape data (scan data). The 3D model is composed of shape data that represents the 3D shape (geometry information) of the subject using, for example, a polygon mesh or a 3D point cloud, and a UV map that maps the texture of the subject in, for example, two dimensions.

[0022] Due to the principle of measuring the 3D shape of a subject by irradiating it with light, the 3D scanner 1 may not be able to correctly measure the 3D shape of some subjects. Below, we will explain a case where the 3D scanner 1 performs 3D scanning using the Structured Light method, in which the 3D scanner 1 irradiates a pattern of light toward the subject and calculates the distance to the object based on the distortion of the pattern on the surface of the subject.

[0023] For example, if the surface of the subject is dark, the pattern light is absorbed, and the distortion of the pattern on the surface of the subject may not be detected by the 3D scanner 1. For example, if the surface of the subject is made of a light-reflective material such as metal, the pattern light may be reflected in unpredictable directions by the surface of the subject, and the 3D scanner 1 may not be able to correctly measure the 3D shape of the surface of the subject.

[0024] For example, if the surface of the subject is transparent or translucent, the pattern light passes through the surface of the subject or is diffused by the surface of the subject, which may prevent the 3D scanner 1 from correctly detecting distortion of the pattern on the surface of the subject. Due to limitations of the device (e.g., projector) that irradiates the pattern light, an appropriate distance from the 3D scanner 1 to the subject may be determined, and if the distance from the 3D scanner 1 to the subject is not appropriate, the 3D shape of the subject may not be correctly measured.

[0025] FIG. 3 is a diagram showing an example of shape data acquired using the 3D scanner 1.

[0026] For example, as shown on the left side of Figure 3, if subject Obj11 is wearing a black hat, the shape data D1 obtained by 3D scanning of subject Obj11 may be missing 3D points or polygon meshes that indicate the 3D shape of part of the hat, as shown by the dashed ellipse on the right side of Figure 3.

[0027] FIG. 4 is a diagram showing another example of shape data acquired using the 3D scanner 1.

[0028] For example, as shown by the dashed ellipse in Figure 4, 3D points or polygon meshes representing the 3D shape of a part of the subject Obj11 (part of a golf club) outside a range A1 of a predetermined distance from the center of the subject Obj11 may be missing from the shape data D1.

[0029] Generally, when the 3D scanner 1 cannot accurately measure the 3D shape of the subject, the user will imagine the actual 3D shape of the subject and perform corrections such as filling in missing parts, removing unnecessary parts, and smoothing the surface of the 3D shape shown in the shape data acquired using the 3D scanner 1. When correcting shape data, the following issues arise.

[0030] First, the correction work takes time. Second, it is difficult for a user to determine the correct shape of a subject just by looking at the shape data. For example, as shown in FIG. 5, when performing correction work to fill in a missing portion of a hat in shape data D1, it is difficult for a user to determine whether the missing portion should be corrected with a shape Sh1 formed by combining multiple flat surfaces, a shape Sh2 formed by a convex curved surface, or a shape Sh3 formed by combining multiple concave curved surfaces. Third, the correction work requires a high level of skill.

[0031] As described above, correcting shape data is time-consuming and labor-intensive, and even after correction, it is difficult to obtain shape data that accurately represents the actual 3D shape of the subject.

[0032] This technology was conceived with the above points in mind, and by displaying a reference preview when a user makes corrections to shape data acquired using a 3D scanner 1, it allows the user to more efficiently make corrections to the shape data.

[0033] 2. Configuration of Image Processing Device FIG. 6 is a block diagram showing an example of the configuration of the image processing device 3. As shown in FIG.

[0034] As shown in FIG. 6, the image processing device 3 includes a camera parameter estimation unit 21, a shape generation unit 22, a learning unit 23, rendering units 24 and 25, an input / output control unit 26, a correction unit 27, a refinement unit 28, and a synthesis unit 29.

[0035] The camera parameter estimation unit 21 estimates camera parameters indicating the viewpoint of the RGB camera 2 when each captured image was acquired, based on the scan data acquired using the scanner 1 and the multiple captured images acquired by the RGB camera 2. For example, the camera parameter estimation unit 21 estimates the camera parameters of each captured image by performing image processing called SfM (Structure from Motion) using the scan data and the multiple captured images. SfM is a process that calculates correspondences between feature points between captured images captured from multiple viewpoints and estimates camera parameters based on the correspondences of the feature points.

[0036] Here, the viewpoint of the RGB camera 2 includes the position and orientation of the RGB camera 2, and the viewpoint of the RGB camera 2 when capturing each captured image is also referred to as the capturing viewpoint of each captured image. The camera parameters are, for example, three-dimensional coordinates (X camera ,Y camera ,Z camera ) and the three-dimensional coordinates in the world coordinate system (X world ,Y world ,Z world ) is expressed as a matrix showing the relationship between

[0037]

[0038] In formula (1), r 11 From 33 indicates the rotational component (attitude) of the camera parameters, and t ｘ From t ｚ indicates the translation component (position) of the camera parameter.

[0039] The camera parameter estimation unit 21 supplies the camera parameters of each captured image to the shape generation unit 22, the learning unit 23, and the refinement unit 28.

[0040] The shape generation unit 22 performs shape reconstruction (photogrammetry) to reconstruct the 3D shape of the subject based on the multiple captured images acquired by the RGB camera 2 and the camera parameters of each captured image, and generates shape data indicating the 3D shape of the subject. A portion of the shape data generated by the shape generation unit 22 is cut out and used by the correction unit 27 as shape data that will be the basis for complementary part data. The complementary part data is shape data that complements missing parts of the scan data that is the target of the correction work.

[0041] The learning unit 23 generates an image generation model through machine learning using multiple captured images acquired by the RGB camera 2 and camera parameters of each captured image. The image generation model generated by the learning unit 23 is a machine learning model that uses technologies such as NeRF (Representing Scenes as Neural Radiance Fields for View Synthesis), 3DGS (3D Gaussian Splatting), and NeX (Real-time View Synthesis with Neural Basis Expansion), and receives as input information (camera parameters) indicating a desired new viewpoint and outputs an image of the subject seen from the new viewpoint.

[0042] The learning unit 23 supplies the generated image generation model to the rendering unit 24 .

[0043] The rendering unit 24 inputs the camera parameters supplied from the input / output control unit 26 into the image generation model generated by the learning unit 23, thereby generating (rendering) an image of the subject viewed from a viewpoint indicated by the camera parameters. The rendering unit 24 supplies the generated image to the input / output control unit 26 as a reference preview. Here, for example, if the viewpoint indicated by the camera parameters supplied from the input / output control unit 26 does not match any of the shooting viewpoints of the captured image, the rendering unit 24 generates a reference preview using the image generation model. On the other hand, if the viewpoint indicated by the camera parameters supplied from the input / output control unit 26 matches any of the shooting viewpoints of the captured image, the rendering unit 24 supplies the captured image to the input / output control unit 26 as a reference preview.

[0044] The rendering unit 25 generates a rendering image by rendering the scan data and the complementary part data from a viewpoint indicated by the camera parameters supplied from the input / output control unit 26. The rendering unit 25 supplies the generated rendering image as a model preview to the input / output control unit 26. Hereinafter, the viewpoint when rendering the scan data and the complementary part data is also referred to as a preview viewpoint.

[0045] The input / output control unit 26 controls the input and output of information to and from a user I / F (not shown). The user I / F is composed of devices for exchanging information with the user, such as a touch panel, a display, a speaker, a keyboard, and a pointing device.

[0046] For example, the input / output control unit 26 functions as a display control unit that displays a GUI (Graphical User Interface) of a DCC tool, a reference preview, a model preview, etc. on a display unit such as a touch panel or a display that constitutes the user I / F. The input / output control unit 26 also functions as an input control unit that receives input of user operations on a touch panel, keyboard, pointing device, operable GUI, etc. that constitute the user I / F.

[0047] For example, by operating the user I / F, the user can specify a preview viewpoint or perform corrections on the shape data that is the source of the complementary part data. The input / output control unit 26 supplies camera parameters indicating the preview viewpoint specified by the user to the rendering units 24 and 25, and notifies the correction unit 27 of correction instructions corresponding to the corrections performed by the user. Note that the preview viewpoints that the user can specify may be limited to only the shooting viewpoints of the images captured by the RGB camera 2. In this case, the rendering unit 24 does not need to use an image generation model to generate a reference preview, thereby shortening the processing time required for machine learning to generate an image generation model.

[0048] The correction unit 27 acquires the shape data that is the source of the complementary part data from the shape generation unit 22, and generates the complementary part data by correcting the shape data in accordance with the correction instructions notified from the input / output control unit 26. The correction unit 27 can also correct the scan data in accordance with the correction instructions, such as by deleting unnecessary parts of the scan data.

[0049] The correction unit 27 supplies the corrected shape data (shape data for which correction work has not yet been completed or completed complementary part data) to the rendering unit 25. The correction unit 27 also supplies the completed complementary part data to the refinement unit 28. Note that if no correction work has been performed by the user, the correction unit 27 supplies the original shape data of the complementary part to the rendering unit 25 as is.

[0050] The refinement unit 28 refines the complementary part data so as to reduce discontinuity between the scan data and the complementary part data supplied from the correction unit 27. For example, the refinement unit 28 refines the complementary part data based on the captured image acquired by the RGB camera 2 using a differentiable rendering technique such as PyTorch3D. Details of the processing performed by the refinement unit 28 will be described later. The refinement unit 28 supplies the refined complementary part data to the synthesis unit 29.

[0051] The synthesis unit 29 synthesizes the scan data with the complementary part data supplied from the refinement unit 28 to generate final shape data.

[0052] Fig. 7 is a block diagram showing a first modified example of the configuration of the image processing device 3. In Fig. 7, the same components as those in Fig. 6 are denoted by the same reference numerals. Duplicate explanations will be omitted where appropriate.

[0053] The image processing device 3 in FIG. 7 differs from the image processing device 3 in FIG. 6 in that a shape generation unit 51 having a trained shape generation model is provided instead of the shape generation unit 22 .

[0054] The shape generation model possessed by the shape generation unit 51 is a machine learning model that receives a single captured image as input and outputs shape data indicating the 3D shape of the subject appearing in the captured image. The shape generation unit 51 generates shape data by inputting a single captured image acquired by the RGB camera 2 into the shape generation model. A portion of the shape data generated by the shape generation unit 51 is cut out and used in the correction unit 27 as shape data that will be the basis for complementary part data.

[0055] FIG. 8 is a first diagram showing an example of displaying a reference preview and a model preview.

[0056] As shown in the first image from the left in Fig. 8 , the display unit of the image processing device 3 first displays a model preview rendered from the scan data superimposed on a reference preview. As indicated by the dotted ovals in Fig. 8 , at least portions of the middle finger, ring finger, and little finger of the plastic model serving as the subject are missing in the scan data. By comparing this model preview with the reference preview, the user can understand that complementary parts data indicating the 3D shapes of the middle finger, ring finger, and little finger of the plastic model must be created.

[0057] After identifying the parts for which complementary parts data must be created, the user specifies on the screen the correction range to be targeted for correction work within a predetermined range that includes the entire scan data. In the example shown on the left in Figure 8, the user specifies the range that includes, for example, the missing parts of the middle finger, ring finger, and little finger of the plastic model (the range surrounded by the dotted oval) as the correction range.

[0058] After the correction range is specified, a model preview obtained by rendering shape data based on the captured image is displayed on the display unit of the image processing device 3, as shown in the second from the left in Fig. 8. The user specifies on the screen an extraction range A11 that includes a portion of the entire shape data displayed in the model preview that the user wishes to use as the shape data that will be the basis for the complementary part.

[0059] When the cut-out range A11 is specified, the correction unit 27 cuts out the shape data included in the cut-out range A11 from the shape data based on the captured image as shape data that will be the basis for the complementary part. On the display unit of the image processing device 3, a model preview rendered from the cut-out shape data is displayed superimposed on the reference preview, as shown in the third from the left in Fig. 8 .

[0060] While checking the 3D shape of the actual subject by referring to the reference preview, the user can make corrections to the surface of the 3D shape indicated by the shape data displayed in the model preview, such as filling in missing parts, removing unnecessary parts, smoothing, etc. Here, the user can make corrections to the entire shape data that is the basis for the complementary parts by changing the preview viewpoint while making corrections.

[0061] When the user performs the modification work, the display unit of the image processing device 3 displays a model preview rendered from the complementary part data (modified shape data) superimposed on the reference preview, as shown in the fourth figure from the left in Figure 8.

[0062] After the correction work is completed, a refinement unit 28 refines the complementary part data, and a synthesis unit 29 synthesizes the scan data and the refined complementary part data to generate final shape data.

[0063] The fifth image from the left in Fig. 8 shows a model preview obtained by rendering the final shape data and a reference preview superimposed on each other. As shown in the fifth image from the left in Fig. 8, the shape data is corrected and refined so that, when viewed from an arbitrary preview viewpoint, the 3D shape of the actual subject and the 3D shape represented by the shape data substantially match. An image in which the model preview obtained by rendering the final shape data and the reference preview are superimposed on each other may be displayed on the display unit of the image processing device 3.

[0064] Note that a model preview rendered from scan data and a model preview rendered from complementary part data may be simultaneously displayed superimposed on a reference preview. If the display unit of the image processing device 3 is capable of displaying 3D images consisting of a left-eye image and a right-eye image having parallax therebetween, the reference preview and model preview may be displayed as 3D images that can be viewed stereoscopically.

[0065] Fig. 9 is a block diagram showing a second modified example of the configuration of the image processing device 3. In Fig. 9, the same components as those in Fig. 6 are denoted by the same reference numerals. Duplicate explanations will be omitted where appropriate.

[0066] The image processing device 3 in FIG. 9 differs from the image processing device 3 in FIG. 6 in that the shape generating unit 22 is not provided.

[0067] The correction unit 27 generates complementary part data based on shape data having a simple shape, rather than based on shape data based on a captured image. Various shape data having simple shapes are prepared in advance in the image processing device 3. For example, shape data having a rectangular parallelepiped shape shown in A of Fig. 10, shape data having a cylindrical shape shown in B of Fig. 10, shape data having a spherical shape shown in C of Fig. 10, and shape data having a conical shape shown in D of Fig. 10 are prepared in advance.

[0068] FIG. 11 is a second diagram showing an example of displaying a reference preview and a model preview.

[0069] Even when complementary parts data is generated based on shape data having a simple shape, a model preview rendered from the scan data is first displayed superimposed on a reference preview on the display unit of the image processing device 3, as shown in the first image from the left in Fig. 11. By comparing this model preview with the reference preview, the user can understand that complementary parts representing the 3D shapes of the middle finger, ring finger, and little finger of the plastic model need to be created.

[0070] After identifying the parts for which complementary parts data must be created, the user specifies on the screen the correction range to be targeted for correction work within a predetermined range that includes the entire scan data. In the example shown on the left in Figure 11, the user specifies the range that includes, for example, the missing parts of the middle finger, ring finger, and little finger of the plastic model (the range surrounded by the dotted oval) as the correction range.

[0071] After specifying the correction range, the user selects, from various types of shape data prepared in advance, shape data that will serve as the basis for complementary parts corresponding to the middle finger, ring finger, and little finger of the plastic model. Once the shape data that will serve as the basis for the complementary parts is selected, a model preview rendered from the selected shape data is displayed on the display unit of the image processing device 3, superimposed on the reference preview, as shown in the second from the left in Figure 11. Here, the user can change the preview viewpoint and specify the superimposition position of the model preview relative to the reference preview, in other words, the placement position of the complementary part data relative to the scan data.

[0072] After specifying the placement position of the complementary part data relative to the scan data, the user can check the 3D shape of the actual subject by referring to the reference preview, and then perform corrections on the surface of the 3D shape indicated by the shape data displayed in the model preview.

[0073] When the user performs the correction work, the display unit of the image processing device 3 displays a model preview in which the complementary part data is rendered, superimposed on the reference preview, as shown in the third from the left in FIG.

[0074] After the correction work is completed, a refinement unit 28 refines the complementary part data, and a synthesis unit 29 synthesizes the scan data and the refined complementary part data to generate final shape data.

[0075] The fourth image from the left in Fig. 11 shows a model preview obtained by rendering the final shape data and a reference preview superimposed on each other. As shown in the fourth image from the left in Fig. 11, the shape data is corrected and refined so that the 3D shape of the actual subject and the 3D shape represented by the shape data approximately match when viewed from an arbitrary preview viewpoint.

[0076] FIG. 12 is a block diagram showing a detailed configuration example of the refinement unit 28.

[0077] As shown in FIG. 12, the refinement unit 28 includes a synthesis unit 71 , a silhouette rendering unit 72 , a difference acquisition unit 73 , and an update unit 74 .

[0078] The synthesis unit 71 synthesizes the complementary part data and the scan data input to the refinement unit 28 , and supplies the synthesized shape data to the silhouette rendering unit 72 .

[0079] The silhouette rendering unit 72 generates a silhouette image by projecting the shape data supplied from the synthesis unit 71 onto a virtual image plane corresponding to a certain shooting viewpoint. The silhouette rendering unit 72 supplies the generated silhouette image to the difference acquisition unit 73.

[0080] The difference acquisition unit 73 acquires a captured image that was actually captured from the shooting viewpoint used by the silhouette rendering unit 72 to generate the silhouette image, and generates a silhouette image that shows the area of ​​the subject in the captured image. The difference acquisition unit 73 acquires the difference between the silhouette image based on the shape data and the silhouette image based on the captured image, and notifies the acquired difference to the update unit 74.

[0081] The update unit 74 updates (modifies) the complementary part data input to the refinement unit 28 so as to reduce the difference between the silhouette image based on the shape data and the silhouette image based on the photographed image. The update unit inputs the updated complementary part data again to the refinement unit 28 (the synthesis unit 71 and the update unit 74).

[0082] The refinement unit 28 refines the complementary part data by recursively repeating the process of combining the complementary part data with the scan data, obtaining the difference between the silhouette images, and updating the complementary part data while changing the captured image to be used. The refinement unit 28 (update unit 74) outputs the refined complementary part data to a subsequent stage.

[0083] 3. Operation of the 3D modeling system Next, the processing performed by the 3D modeling system having the above configuration will be described with reference to the flowchart in Fig. 13. In particular, the processing performed when the image processing device 3 has the configuration described with reference to Fig. 6 will be described below.

[0084] In step S1, the 3D scanner 1 performs 3D scanning to measure the 3D shape of the subject from multiple viewpoints to obtain scan data.

[0085] In step S2, the RGB camera 2 captures images of the subject from multiple viewpoints to obtain multiple captured images.

[0086] In step S3, the camera parameter estimation unit 21 of the image processing device 3 estimates the camera parameters of each captured image.

[0087] In step S4, the shape generation unit 22 of the image processing device 3 generates shape data indicating the 3D shape of the subject by performing shape restoration based on the multiple captured images and the camera parameters of each captured image.

[0088] In step S5, the learning unit 23 of the image processing device 3 generates an image generation model by machine learning using a plurality of captured images and the camera parameters of each captured image.

[0089] In step S6, the input / output control unit 26 of the image processing device 3 accepts the preview viewpoint designated by the user.

[0090] In step S7, the rendering unit 24 of the image processing device 3 generates a reference preview by inputting camera parameters indicating the preview viewpoint into the image generation model.

[0091] In step S8, the rendering unit 25 of the image processing device 3 generates a model preview by rendering the shape data, complementary part data, scan data, etc. that are the basis of the complementary parts from a preview viewpoint.

[0092] In step S9, the input / output control unit 26 displays the reference preview and the model preview in an overlapping manner.

[0093] In step S10, the input / output control unit 26 accepts correction work performed by the user. Here, the accepted correction work includes specification of the correction range to be subjected to the correction work, specification of the cut-out range from which the shape data that will be the source of the complementary part will be cut out, etc. The correction unit 27 corrects the shape data and scan data that will be the source of the complementary part, etc., in accordance with the correction instructions corresponding to the correction work performed by the user.

[0094] In step S11, the input / output control unit 26 determines whether or not the user has finished the correction work.

[0095] If it is determined in step S11 that the user has not finished the correction work, the process returns to step S6, and the subsequent processes are carried out.

[0096] On the other hand, if it is determined in step S11 that the user has completed the correction work, the refinement unit 28 of the image processing device 3 refines the complementary part data in step S12.

[0097] In step S13, the synthesis unit 29 of the image processing device 3 synthesizes the scan data and the refined complementary part data to generate final shape data.

[0098] As described above, in the image processing device 3 of the present technology, the user specifies a correction range to be modified within a predetermined range including the scan data acquired using the 3D scanner 1, and a model preview, which is an image obtained by rendering shape data included in the correction range, is displayed superimposed on a reference preview, which is an image that serves as a guide for the user when performing the modification work. Since the user can modify the shape data while checking the 3D shape of the actual subject by referring to the reference preview, the user can obtain shape data that precisely represents the actual shape of the subject through the modification work.

[0099] Furthermore, in the image processing device 3 of the present technology, the complementary part data is generated by correcting the shape data that is the source of the complementary part data that complements the scan data included in the correction range in accordance with the correction work accepted by the input / output control unit 26. Compared to creating complementary part data without any shape data such as polygon meshes or 3D points, the user can easily create complementary part data without spending time and effort.

[0100] Furthermore, in the image processing device 3 of the present technology, the complementary part data is refined so as to reduce discontinuity between the complementary part data and the scan data, thereby enabling the user to obtain shape data that precisely represents the actual shape of the subject, without having to perform detailed correction work that requires advanced skills.

[0101] <Regarding the Computer> The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0102] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0103] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0104] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0105] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0106] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0107] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0108] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0109] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0110] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0111] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0112] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0113] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0114] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0115] (1) An image processing device comprising: an input control unit that receives from a user a specification of a correction range to be corrected by the user within a predetermined range including scan data, the scan data being shape data indicating the 3D shape of the subject acquired using a depth sensor that measures the 3D shape of the subject; and a display control unit that displays a model preview, which is an image obtained by rendering the shape data included in the correction range, superimposed on a reference preview, which is an image based on an image acquired by capturing an image of the subject with an image sensor and serves as a guide for the user when performing the correction work. (2) The image processing device according to (1), wherein the input control unit receives the correction work to be performed by the user; and further comprises a correction unit that corrects the shape data included in the correction range in accordance with the correction work accepted by the input control unit. (3) The image processing device according to (2), wherein the correction unit generates the complementary part data by correcting shape data that is a source of complementary part data that complements the scan data included in the correction range in accordance with the correction work accepted by the input control unit; and further comprises a combination unit that combines the complementary part data with the scan data. (4) The image processing device according to (3), further comprising a shape generation unit that generates shape data that is the basis of the complementary parts data based on the captured image. (5) The image processing device according to (4), wherein the shape generation unit generates shape data that is the basis of the complementary parts data based on a plurality of captured images acquired by the image sensor capturing images of the subject from a plurality of viewpoints. (6) The image processing device according to (4), wherein the shape generation unit receives one of the captured images as input and generates shape data that is the basis of the complementary parts data using a machine learning model that acquires shape data that indicates the 3D shape of the subject appearing in the one captured image. (7) The image processing device according to (3), wherein the correction unit corrects shape data selected by the user from pre-prepared shape data as shape data that is the basis of the complementary parts data, in accordance with the correction operation accepted by the input control unit.(8) The image processing device according to any one of (3) to (7), further comprising a refinement unit that refines the complementary part data based on the captured image. (9) The image processing device according to (8), wherein the refinement unit refines the complementary part data based on a difference between a silhouette image based on shape data obtained by combining the complementary part data and the scan data, and the silhouette image based on the captured image. (10) The image processing device according to (8) or (9), wherein the refinement unit refines the complementary part data so as to reduce discontinuity between the complementary part data and the scan data. (11) The image processing device according to (2), wherein the correction unit corrects the scan data included in the correction range in accordance with the correction work accepted by the input control unit. (12) The image processing device according to any one of (1) to (11), wherein the display control unit displays an image of the subject seen from a preview viewpoint specified by the user as the reference preview, and displays an image obtained by rendering shape data included in the correction range from the preview viewpoint as the model preview. (13) The image processing device according to (12), wherein, when the preview viewpoint does not match the shooting viewpoint when the image sensor captured the captured image, the display control unit displays the model preview generated using a machine learning model that takes the preview viewpoint as an input and outputs an image of the subject seen from the preview viewpoint. (14) The image processing device according to (13), wherein the machine learning model that takes the preview viewpoint as an input and outputs an image of the subject seen from the preview viewpoint is generated by machine learning using a plurality of captured images acquired by the image sensor capturing the subject from a plurality of the shooting viewpoints and information indicating the shooting viewpoints for each of the plurality of captured images. (15) The image processing device according to any of (12) to (14), wherein, when the preview viewpoint matches the shooting viewpoint when the image sensor captured the captured image, the display control unit displays the captured image as the reference preview.(16) The image processing device according to any one of (1) to (15), wherein the display control unit displays the reference preview as a 3D image consisting of a left-eye image and a right-eye image having parallax between them. (17) The image processing device according to any one of (1) to (16), wherein the display control unit displays the model preview as a 3D image consisting of a left-eye image and a right-eye image having parallax between them. (18) An image processing method including: receiving from the user designation of a correction range to be corrected by the user within a predetermined range including scan data that is shape data indicating the 3D shape of the subject acquired using a depth sensor that measures the 3D shape of the subject; and superimposing and displaying a model preview that is an image obtained by rendering the shape data included in the correction range on a reference preview that is an image based on a captured image acquired by capturing an image of the subject with an image sensor and that serves as a guide for the user when performing the correction work.

[0116] REFERENCE SIGNS LIST 1 3D scanner, 2 RGB camera, 3 Image processing device, 21 Camera parameter estimation unit, 22 Shape generation unit, 23 Learning unit, 24, 25 Rendering unit, 26 Input / output control unit, 27 Refinement unit, 29 Synthesis unit, 51 Shape generation unit, 71 Synthesis unit, 72 Silhouette rendering unit, 73 Difference acquisition unit, 74 Update unit

Claims

1. An image processing device comprising: an input control unit that receives from a user a specification of a correction range that is to be corrected by the user, within a predetermined range including scan data that is shape data showing the 3D shape of the subject obtained using a depth sensor that measures the 3D shape of the subject; and a display control unit that displays a model preview, which is an image obtained by rendering the shape data included in the correction range, superimposed on a reference preview, which is an image based on a captured image obtained by photographing the subject using an image sensor and serves as a guide for the user when performing the correction work.

2. The image processing device according to claim 1, further comprising a correction unit that receives the correction work performed by the user and corrects shape data included in the correction range in accordance with the correction work received by the input control unit.

3. The image processing device according to claim 2, wherein the correction unit generates the complementary part data by correcting shape data that is the source of complementary part data that complements the scan data included in the correction range in accordance with the correction work accepted by the input control unit, and further comprises a synthesis unit that synthesizes the complementary part data and the scan data.

4. The image processing device according to claim 3, further comprising a shape generating unit that generates shape data that is the basis of the complementary part data based on the captured image.

5. An image processing device as described in claim 4, wherein the shape generation unit generates shape data that is the basis of the complementary part data based on a plurality of captured images obtained by the image sensor photographing the subject from a plurality of viewpoints.

6. The image processing device described in claim 4, wherein the shape generation unit takes one of the captured images as input and generates shape data that serves as the basis for the complementary part data using a machine learning model that acquires shape data indicating the 3D shape of the subject appearing in the one of the captured images.

7. An image processing device as described in claim 3, wherein the correction unit corrects shape data selected by the user from pre-prepared shape data as the shape data that will serve as the basis for the complementary part data, in accordance with the correction work accepted by the input control unit.

8. The image processing device according to claim 3, further comprising a refinement section that refines the complementary part data based on the photographed image.

9. An image processing device according to claim 8, wherein the refinement unit refines the complementary part data based on the difference between a silhouette image based on shape data obtained by combining the complementary part data and the scan data, and a silhouette image based on the captured image.

10. An image processing apparatus according to claim 8, wherein said refinement section refines said complementary part data so as to reduce discontinuity between said complementary part data and said scan data.

11. The image processing device according to claim 2, wherein the correction unit corrects the scan data included in the correction range in accordance with the correction work accepted by the input control unit.

12. An image processing device as described in claim 1, wherein the display control unit displays an image of the subject viewed from a preview viewpoint specified by the user as the reference preview, and displays an image of shape data included in the correction range rendered from the preview viewpoint as the model preview.

13. The image processing device described in claim 12, wherein, when the preview viewpoint does not match the shooting viewpoint when the image sensor acquired the captured image, the display control unit displays the model preview generated using a machine learning model that uses the preview viewpoint as input and outputs an image of the subject as seen from the preview viewpoint.

14. The image processing device described in claim 13, wherein a machine learning model that inputs the preview viewpoint and outputs an image of the subject as seen from the preview viewpoint is generated by machine learning using a plurality of captured images obtained by the image sensor photographing the subject from a plurality of the shooting viewpoints and information indicating the shooting viewpoints for each of the plurality of captured images.

15. An image processing device according to claim 12, wherein the display control unit displays the captured image as the reference preview when the preview viewpoint coincides with the shooting viewpoint when the image sensor captured the captured image.

16. The image processing device according to claim 1, wherein the display control unit displays the reference preview as a 3D image made up of a left-eye image and a right-eye image having a parallax therebetween.

17. The image processing device according to claim 1, wherein the display control unit displays the model preview as a 3D image consisting of a left-eye image and a right-eye image having a mutual parallax.

18. An image processing method including: receiving from a user a specification of a correction range to be modified by the user within a predetermined range including scan data, which is shape data indicating the 3D shape of a subject, acquired using a depth sensor that measures the 3D shape of the subject; and displaying a model preview, which is an image obtained by rendering the shape data included in the correction range, superimposed on a reference preview, which is an image based on a captured image acquired by an image sensor by photographing the subject, and which serves as a guide for the user when performing the modification work.

Images