Image processing apparatus, image processing method, and program

Abstract

To provide an image processing apparatus, a method and a program for appropriately correcting three-dimensional data of an object.SOLUTION: In an image processing system, an image processing apparatus is configured to: acquire first three-dimensional shape data representing a three-dimensional shape of an object, the three-dimensional shape data being generated on the basis of a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by a plurality of imaging processing apparatus imaging the object; acquire second three-dimensional shape data generated by deforming second three-dimensional shape data representing a basic shape of the object stored in advance on the basis of information on an orientation of the object in the captured images; and correct a loss in the first three-dimensional shape data on the basis of the second three-dimensional shape data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a process for generating three-dimensional shape data of an object. [Background technology]

[0002] One method involves using multiple imaging devices installed in different locations to capture images of an object while time-synchronizing the images, and then using a volume intersection method to generate data that represents the three-dimensional shape of the object in voxels (called the volume shape) from the multiple captured images obtained by the capture.

[0003] Patent Document 1 describes a method of estimating the orientation of an object and generating data (called the orientation shape) representing the three-dimensional shape of the object based on the estimated orientation. It also describes a method of using the intersection between the visual volume shape and the orientation shape as the final three-dimensional shape data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2021 / 0065430 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method of Patent Document 1, if a voxel that should be present in the visual volume shape is missing, three-dimensional shape data is generated in which the missing voxel in the visual volume shape is not corrected, even if the voxel that should be present in the posture shape is present. [Means for solving the problem]

[0006] The image processing device of the present disclosure is characterized by having a first acquisition means for acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing an object with a plurality of imaging devices; a second acquisition means for acquiring second three-dimensional shape data being generated by deforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured images; and a correction means for correcting defects in the first three-dimensional shape data based on the second three-dimensional shape data. Effect of the Invention

[0007] According to the present disclosure, three-dimensional shape data can be appropriately corrected. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram showing an example of the functional configuration of an imaging processing device and an image processing device. [Diagram 2] FIG. 2 is a diagram for explaining the arrangement of an imaging processing device. [Diagram 3] A diagram showing the basic principle of volume intersection. [Figure 4] 11A and 11B are diagrams for explaining the influence of a foreground silhouette image on a visual volume shape. [Diagram 5] FIG. 4 is a diagram for explaining posture information. [Figure 6] FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus. [Figure 7] 11 is a flowchart for explaining the flow of a process for generating a view volume shape. [Figure 8] FIG. 4 is a diagram for explaining the process of a comparison unit. [Figure 9] FIG. 2 is a diagram showing an example of the functional configuration of an imaging processing device and an image processing device. [Figure 10] 11 is a flowchart for explaining the flow of a process for generating a view volume shape. [Figure 11] 6A and 6B are diagrams for explaining a process of generating a visual volume shape. [Figure 12] FIG. 2 is a diagram showing an example of the functional configuration of an imaging processing device and an image processing device. [Figure 13] 3A to 3C are diagrams showing a distance image, a foreground silhouette image, and a foreground texture image. [Figure 14] 10 is a flowchart for explaining a flow of processing up to generating a virtual viewpoint image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the technology of the present disclosure is not limited to the configurations shown in the drawings.

[0010] <Embodiment 1> In this embodiment, a method for correcting a three-dimensional model of an object generated by the volume intersection method based on the orientation information of the object will be described.

[0011] [System Configuration] 1 is a diagram showing an example of the functional configuration of each device included in the image processing system of this embodiment. The image processing system includes an image capture processing device 110 and an image processing device 100 which is an information processing device.

[0012] [Functional configuration of imaging processing device] The functional configuration of the imaging processing device 110 will be described. The imaging processing device 110 includes an imaging unit 111 and a foreground image generating unit 112. The imaging unit 111 is an imaging device such as a digital video camera, and has an interface for outputting an image signal such as a serial digital interface (SDI). The imaging unit 111 captures an image of an imaging target to obtain a captured image. The captured image obtained by imaging the imaging unit 111 is output to the foreground image generating unit 112 via the image signal interface.

[0013] FIG. 2 is a diagram for explaining the arrangement of the imaging processing device 110. As shown in FIG. 2, the imaging processing device 110 is composed of a plurality of imaging processing devices 110a to 110e. The imaging processing devices 110a to 110e are arranged so that the imaging units 111 capture images of the imaging area 200 from different positions. The imaging processing devices 110a to 110e are synchronized with one piece of time information. Therefore, each of the imaging units 111 can capture images of an object 201 such as a person present in the imaging area 200 from various angles while being time-synchronized. The moving images captured by the imaging processing devices 110a to 110e are provided with imaging time information for each frame.

[0014] 2 shows an example in which five imaging processing devices 110 are included, but the number of imaging processing devices 110 is not limited to five. For example, when the imaging area 200 is a rugby or soccer stadium, several tens to several hundreds of imaging processing devices 110 may be installed around the stadium. The imaging processing devices 110 do not have to be installed all around the imaging area 200, and may be installed only in a partial direction of the imaging area 200. Unless otherwise specified, when referring to any of the imaging processing devices 110a to 110e, the alphabet of the reference symbol is omitted and the imaging processing device 110 is written.

[0015] Returning to FIG. 1, the description of the functions of the image capture processing device 110 will be continued. The foreground image generating unit 112 is an image processing unit corresponding to the image capture unit 111, and is composed of, for example, a calculation device represented by a CPU capable of calculation to generate an image. A captured image obtained as a result of image capture by the corresponding image capture unit 111 is input to the foreground image generating unit 112. The foreground image generating unit 112 performs image processing on the input captured image. The image processing performed by the foreground image generating unit 112 includes processing to extract a foreground region, which is a region of an object included in the input captured image. Then, the foreground image generating unit 112 generates foreground silhouette images 202a to 202e, which indicate the foreground region and non-foreground region included in the captured image in binary. Furthermore, the captured image is cut out to match the circumscribing rectangle of the foreground region included in the foreground silhouette image, to generate a foreground texture image.

[0016] A method of extracting a foreground region where an object exists from a captured image includes a method of using background difference information. In this method, for example, a state of the captured image region 200 where no object exists is captured in advance and stored as a background image. Then, a region where the difference value of pixel values ​​between the captured image and the background image is greater than a threshold value is determined to be a foreground region. Note that the method of extracting a foreground region in a captured image is not limited to the method of using background difference information. Other methods of extracting a foreground region may include a method of using parallax, a method of using a feature amount, or a method of using machine learning. The generated foreground silhouette image and foreground texture image are output to the image processing device 100.

[0017] [Functional configuration of image processing device] The functions of the image processing device 100 will be described with reference to Fig. 1. The image processing device 100 includes a data acquisition unit 101, a visual volume shape generation unit 102, a posture estimation unit 103, a posture shape generation unit 104, a comparison unit 105, and a correction unit 106.

[0018] The data acquisition unit 101 acquires imaging data such as a captured image, a foreground silhouette image, a foreground texture image, and camera parameters captured or generated by each of the imaging processing devices 110. The camera parameters refer to internal parameters, external parameters, and distortion parameters of the imaging unit 111 included in the imaging processing device 110.

[0019] In the above description, the foreground image generating unit 112 included in the imaging processing device 110 performs image processing on the captured image, but the function of the foreground image generating unit 112 may be included in the image processing device 100. For example, the data acquiring unit 101 may perform image processing on the captured image to generate a foreground silhouette image and a foreground texture image. In this case, the data acquiring unit 101 receives the captured image from each of the imaging processing devices 110a to 110e and generates a foreground silhouette image and a foreground texture image corresponding to each of the imaging processing devices 110a to 110e.

[0020] The data acquisition unit 101 may also calculate camera parameters of the imaging units 111 of each of the imaging processing devices 110a to 110e. For example, the data acquisition unit 101 extracts feature points from a marker image (e.g., a checkerboard) for camera calibration captured in advance by each of the imaging units 111 of the imaging processing devices 110a to 110e, and performs correspondence. The data acquisition unit 101 then calculates the camera parameters by calibrating each of the imaging units 111 while optimizing the corresponding points so that an error when projecting them onto each of the imaging units 111 is minimized. The calibration method may be any existing method. The camera parameters may be acquired in synchronization with the captured images, or may be acquired in a preparation stage, or may be acquired asynchronously with the captured images as necessary.

[0021] The visual volume shape generation unit 102 estimates the three-dimensional shape of the object 201 present in the imaging area 200 by a visual volume intersection method using the foreground silhouette images generated by the foreground image generation units 112 of the respective imaging processing devices 110a to 110e. The three-dimensional shape data (three-dimensional model) representing the three-dimensional shape of the object obtained by the visual volume intersection method is referred to as a visual volume shape.

[0022] Fig. 3 is a diagram for explaining the basic principle of the visual volume intersection method. Fig. 3(a) is a diagram when an imaging unit (camera) captures an image of a target object C, which is an object present in an imaging area. A foreground silhouette image including a two-dimensional silhouette (foreground area) of the target object C is obtained by binarizing the captured image obtained by capturing an image of the target object C based on the difference in color or brightness between the captured image and a background image.

[0023] FIG. 3(b) is a diagram showing a cone extending in three-dimensional space from the projection center (Pa) of the imaging unit through each point on the contour of the foreground region Da. This cone is called the visual volume Va of the imaging unit. FIG. 3(c) is a diagram showing how the visual volume shape of an object is obtained from multiple visual volumes. As shown in FIG. 3(c), multiple visual volumes are obtained for each imaging unit from the foreground region Da based on images captured synchronously by multiple different imaging units at different positions. In generating the visual volume shape using the visual volume intersection method, the visual volume shape of an object is generated by determining the intersection (common area) of multiple visual volumes corresponding to multiple imaging units.

[0024] The visual volume shape is expressed by a set of elements called voxels, which are tiny rectangular parallelepipeds. Specifically, a three-dimensional space is tiled with voxels. Then, when each voxel in the three-dimensional space is back-projected onto the plane of each imaging unit, voxels that are back-projected into the foreground region of the foreground silhouette image of a predetermined number or more of imaging units are left as the foreground, and other voxels are deleted. The predetermined number of imaging units is, for example, all imaging units that capture an object. In this way, the visual volume shape of the object is generated by the voxels by removing voxels that do not fit into the foreground region of the foreground silhouette image. Each voxel has coordinates (x, y, z) in the three-dimensional space. The size of the voxel may be set in advance by the user using a GUI (Graphical User Interface) or may be set using a text file or the like.

[0025] FIG. 4 is a diagram for explaining the influence of a foreground silhouette image on a visual volume shape when the visual volume shape is generated by the visual volume intersection method using a foreground silhouette image with insufficient accuracy. FIG. 4(a)-(c) are diagrams showing foreground silhouette images 401-403 generated from captured images obtained by three imaging units 111 capturing an image of object 201. In foreground silhouette images 401-403, a white area represents a foreground area that is an area of ​​object 201, and a black area represents an area other than the foreground area. Foreground silhouette images 402 and 403 are images in which the foreground area appropriately represents the area of ​​object 201. Meanwhile, the foreground area of ​​foreground silhouette image 401 includes a missing area 404. Missing area 404 occurs because foreground image generating unit 112 erroneously extracts the foreground area, and refers to an area where an area that should be extracted as the foreground area is an area other than the foreground area. Missing area 404 may occur regardless of the method of generating the foreground silhouette image.

[0026] FIG. 4(d) shows a volume shape 405 of the object 201 generated by the volume intersection method using at least the foreground silhouette images 401 to 403. When the volume shape is generated by the volume intersection method using the foreground silhouette image including the missing region 404, the voxels that would be part of the object 201 are removed to generate the volume shape. For this reason, as shown in FIG. 4(d), the volume shape 405 includes an area 406 in which voxels corresponding to the missing region 404 are missing. In this way, when the foreground silhouette image includes a missing region, the voxels of the part that should be the object are missing in the volume shape of the object due to the missing region. In addition to missing regions, the volume intersection method may also cause voxels that are left uncut (hereinafter referred to as phantom volume) due to the influence of concave shapes and occlusion.

[0027] The posture estimation unit 103 estimates the posture of the object 201 in the captured image by estimating the positions of the parts of the object 201 using the captured image and camera parameters. Then, posture information representing the estimated posture is generated. As a posture estimation method, for example, a posture estimation method using machine learning can be mentioned, but is not limited to the method using machine learning. In addition, for example, the posture may be estimated by identifying each part of a person based on a general motion capture technique, distance measurement using a laser, or the like.

[0028] Fig. 5 is a diagram for explaining posture information. The posture indicated by the posture information means a bone model indicating the positions of a plurality of parts such as the head, chest, arms, and legs, as shown in Fig. 5. The parts included in the posture information are 13 parts: head (B1), chest (B2), right shoulder (B3), left shoulder (B4), right elbow (B5), left elbow (B6), right hand (B7), left hand (B8), abdomen (B9), right thigh (B10), left thigh (B11), right foot (B12), and left foot (B13). Any part among the 13 parts is represented as part Bi.

[0029] The posture information includes position information for each of the 13 parts. The position information is, for example, the three-dimensional coordinates of the center position of the part Bi. The posture information may include information on the rotation of the part Bi, the connection with other parts, etc., instead of the position of the part Bi.

[0030] The posture estimation unit 103 may have a tracking function. The method of tracking an object based on posture information may involve, for example, finding the difference between the position of each part of a bone model in a target object and the position of each part of a bone model in each of all objects in the previous frame. Then, the object with the smallest position difference is searched for. In addition, object recognition processing using bounding box information of a bone model and a foreground texture image may be used for the tracking processing.

[0031] Based on the posture information generated by posture estimation unit 103, posture shape generation unit 104 generates three-dimensional shape data representing the three-dimensional shape of object 201. The three-dimensional shape data (three-dimensional model) representing the three-dimensional shape of the object generated based on the posture information is also referred to as posture shape.

[0032] As a method of generating a posture shape, three-dimensional shape data representing a basic shape of the object 201 is stored in advance. For example, a three-dimensional model (three-dimensional shape data) such as a human body model of a standard template of a human body or a human body model obtained by 3D scanning a human body such as the object 201 is prepared in advance. Then, a posture shape is generated by fitting posture information to the three-dimensional model so that the basic shape represented by the three-dimensional model is transformed into a posture state indicated by the posture information. Alternatively, a view volume shape of the same object 201 generated in one or more previous frames may be used as a three-dimensional model representing the basic shape. In that case, for example, a view volume shape of an object to be generated for a posture shape is selected based on a result of tracking by the posture estimation unit 103, and the selected view volume shape is used as a three-dimensional model for generating a posture shape.

[0033] The comparison unit 105 compares the view volume shape generated by the view volume shape generation unit 102 with the orientation shape generated by the orientation shape generation unit 104. For each part Bi, the comparison unit 105 compares the voxels belonging to the part Bi of the view volume shape and the orientation shape. Then, the comparison unit 105 identifies a part of the part Bi of the view volume shape where a defect such as a defect or a phantom volume occurs. Details of the processing by the comparison unit 105 will be described later.

[0034] The correction unit 106 corrects the portion of the view volume shape where a defect occurs, based on the result of the comparison by the comparison unit 105. The correction method will be described later. The correction unit 106 also functions as an output unit that outputs the corrected view volume shape.

[0035] [Hardware configuration] 6 is a block diagram showing a hardware configuration of the image processing device 100. The image processing device 100 includes a CPU 601, a ROM 602, a RAM 603, an auxiliary storage device 604, a display unit 605, an operation unit 606, a communication I / F 607, and a bus 608.

[0036] The CPU 601 realizes each function of the image processing device 100 by controlling the entire image processing device 100 using computer programs and data stored in the ROM 602 and the RAM 603. Note that one or more dedicated hardware components different from the CPU 601 may be provided, and at least a part of the processing by the CPU 601 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0037] The ROM 602 stores programs that do not require modification. The RAM 603 temporarily stores programs and data supplied from an auxiliary storage device 604, and data supplied from the outside via a communication I / F 607. The auxiliary storage device 604 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

[0038] The display unit 605 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) etc. for the user to operate the image processing device 100. The operation unit 606 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user and inputs various instructions to the CPU 601. The CPU 601 operates as a display control unit that controls the display unit 605, and as an operation control unit that controls the operation unit 606.

[0039] The communication I / F 607 is used for communication with an external device of the image processing device 100. For example, when the image processing device 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 607. When the image processing device 100 has a function of wireless communication with an external device, the communication I / F 607 is provided with an antenna. The bus 608 connects each unit in the device to transmit information. In the present embodiment, the display unit 605 and the operation unit 606 are described as being present inside the device, but at least one of the display unit 605 and the operation unit 606 may be present as a separate device outside the image processing device 100. The display unit 605 and the operation unit 606 are not essential components of the image processing device 100, and the image processing device 100 may have a system configuration that allows remote operation from, for example, an external controller (not shown).

[0040] Each functional unit in the image processing device 100 in Fig. 1 is realized by the CPU 601 executing a predetermined program, but is not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) for accelerating calculations may also be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0041] [flowchart] Fig. 7 is a flowchart for explaining the flow of the visual volume shape correction process executed in the image processing device 100. The series of processes shown in the flowchart in Fig. 7 are performed by the CPU 601 of the image processing device 100 expanding the program code stored in the ROM 602 into the RAM 603 and executing it. Also, some or all of the functions of the steps in Fig. 7 may be realized by hardware such as an ASIC or electronic circuit. Note that the symbol "S" in the explanation of each process indicates a step in the flowchart, and the same applies to the subsequent flowcharts.

[0042] In S701, the data acquisition unit 101 acquires camera parameters of the imaging units 111 of the imaging processing devices 110a to 110e from each of the imaging processing devices 110a to 110e. As described above, the data acquisition unit 101 may calculate the camera parameters from the captured images obtained from each of the imaging processing devices 110a to 110e.

[0043] In S702, the data acquisition unit 101 acquires a foreground silhouette image and a foreground texture image from each of the imaging processing devices 110a to 110e. As described above, the data acquisition unit 101 may generate a foreground silhouette image and a foreground texture image from the captured images obtained from each of the imaging processing devices 110a to 110e.

[0044] In S703, the view volume shape generation unit 102 generates a view volume shape of the object based on a view volume intersection method using the foreground silhouette image obtained in S702 and the camera parameters obtained in S701. As a result of this processing, voxel information for expressing the three-dimensional shape of the object 201 in the view volume shape is obtained. The view volume shape generation unit 102 may obtain the view volume shape of the object 201 generated by an external device.

[0045] In S704, the orientation estimation unit 103 estimates the orientation of the object 201 based on the foreground texture image and foreground silhouette image obtained in S702 and the camera parameters obtained in S701, and generates orientation information.

[0046] In S705, the posture shape generation unit 104 generates a posture shape based on the posture information generated in S704. As a result of this process, voxel information for expressing the three-dimensional shape of the object 201 in the posture shape is obtained. The posture shape generation unit 104 may obtain a posture shape of the object 201 generated by an external device.

[0047] In S706, the comparison unit 105 compares the visual volume shape with the posture shape for each of the 13 parts Bi included in the posture information.

[0048] The comparison unit 105 applies the orientation information obtained in S704 to the view volume shape obtained in S703, and groups (divides) the voxels that represent the view volume shape so that they belong to one of the 13 parts. For example, each voxel is grouped as belonging to the part Bi with the closest center position among the centers of the parts Bi. A voxel in the view volume shape (ShapeVolume; SV) that belongs to the i-th part Bi is grouped as SV Bi This is expressed as:

[0049] The voxels representing the posture shape are grouped so that they belong to one of the 13 parts. The voxels in the posture shape (PoseVolume; PV) that belong to the i-th part Bi are called PV Bi This is expressed as:

[0050] The comparison unit 105 then calculates true positives, false positives, false negatives, and true negatives, which are also used as evaluation indices for machine learning, for each of the 13 sites Bi. Bi can be calculated from equation (1). False positive FP Bi is calculated from equation (2). False negative FN Bi is calculated from equation (3). True negative TN Bi can be calculated from equation (4).

[0051]

number

[0052] false positive FP Bi SV Bi It exists in PV Bi represents the sum of voxels that do not exist in FP Bi can be used to determine the shape of a phantom volume, etc.

[0053] False negative FN Bi SV Bi It does not exist in PV Bi represents the total number of voxels present. Bi can be used to determine defects in the visual volume shape.

[0054] true negative TN Bi is the SV in three-dimensional coordinates (x, y, z). Bi But P.V. Bi In this embodiment, the true negative TN Bi does not have to be calculated.

[0055] Fig. 8 is a diagram for explaining the processing of comparison unit 105. Fig. 8(a) is a diagram showing in a simplified manner the posture shape generated in S705. Fig. 8(b) is a diagram showing in a simplified manner the view volume shape generated in S703. For the sake of explanation, Figs. 8(a) and 8(b) show the three-dimensional shape of the object with a reduced number of voxels. The voxels constituting the posture shape and the view volume shape are grouped into 13 parts, and voxels within thick frames in Fig. 8 indicate voxels belonging to the same part.

[0056] In FIG. 8, voxel PV belonging to part B1 (head) in the posture shape B1 The number of voxels SV belonging to part B1 in the visual volume shape is 16. B1 PV Bi 16 voxels as above, plus PV Bi In this case, there is one voxel 805 that does not exist in the region B1. B1 , false positive FP B1 , false negative FN B1 is calculated as follows:

[0057] TP B1 =16 FP B1 =1 FN B1 =0 In addition, in FIG. 8, the voxel PV B9 The number of voxels is 20. B9 As a PV B9 There are voxels corresponding to the 16 voxels other than the four voxels 801 to 804. However, SV B9 PV B9 In this case, there are no voxels corresponding to the four voxels 801 to 804 of the region B9. B9 , false positive FP B9 , false negative FN B9 is calculated as follows:

[0058] TP B9 =16 FP B9 =0 FN B9 =4 In addition, if the voxel size is small, there is a possibility that an error will occur in the voxels of the contour when comparing the view volume shape and the posture shape. For example, the view volume shape is generated slightly smaller than the posture shape. In this case, the posture shape has SV Bi It does not exist in PV Bi In this state, there is a thin layer of voxels around the contour. Bi When is calculated, the influence of the error in the contour part is included, so the false negative FN Bi is calculated as a value larger than it should be. This false negative FN Bi If this is used to determine whether a defect has occurred, there is a risk that it will not be possible to appropriately determine whether a defect has occurred.

[0059] Therefore, the comparison unit 105 compares PV BiAfter the contraction and expansion process, the voxels that make up the shape are contracted and then expanded. When the contraction and expansion process is performed, the voxels in the contour area can be removed. Therefore, if there are voxels in the contour area of ​​the posture shape that have been generated due to errors, the voxels that have been generated due to errors are removed. As a result, a false negative FN with the effect of errors in the contour area suppressed is obtained. Bi In this way, the comparison unit 105 calculates the true positive TP Bi , false positive FP Bi , false negative FN Bi、 true negative TN Bi Before calculating, a contraction / expansion process may be performed on the voxels that make up each element.

[0060] In S707, the comparison unit 105 specifies, for each part, whether or not a defect such as a missing part or a phantom volume has occurred in the voxels constituting the view volume shape, based on the comparison result in S706.

[0061] Regarding whether a defect occurs in a voxel belonging to a region Bi in the visual volume shape, the comparison unit 105 performs a false negative (FN Bi If the threshold is greater than or equal to the threshold, it is determined that the data is missing. Bi As a result of this determination, the site Bi where the defect has occurred can be identified.

[0062] Regarding whether the voxels constituting the part Bi include the phantom volume, the comparison unit 105 detects a false positive FP Bi is the threshold value Threshold_FP Bi If so, it can be determined that a phantom volume is included.

[0063] The threshold is set based on, for example, the voxel size and the number of voxels. For example, a threshold Threshold_FN for determining whether a defect occurs in the region Bi is Biis set as a predetermined percentage (e.g., 10 to 30%) of the total number of voxels belonging to the part Bi in the posture shape. If the part B9 in the posture shape is composed of 1000 voxels and the threshold is set to 10% of the total number of voxels, the threshold Threshold_FN for determining a defect in the part B9 is B9 is set to 100. And the false negative FN B9 If the calculated number of defects is 100 or more, it is determined that there is a defect in the region B9 of the visual volume shape.

[0064] In order to determine whether or not a defect or a phantom volume occurs in each part of the visual volume shape, the recall rate and the accuracy rate may be used, or information obtained by the comparison unit 105 may be used. Bi can be calculated from equation (5). Bi , can be obtained from equation (6).

[0065]

number

[0066] When the comparison unit 105 determines that there is a region where a defect or phantom volume occurs (YES in S707), the process proceeds to S708. In S708, the correction unit 106 corrects only the region identified as having a defect or phantom volume among the voxels constituting the view volume shape.

[0067] The correction unit 106 performs a process of correcting the loss of a voxel possibly in a part Bi identified as having a loss in the view volume shape so as to restore the voxel possibly in the part Bi. For example, the correction unit 106 corrects the loss of the view volume shape by replacing a voxel belonging to the part Bi identified as having a loss in the view volume shape with a voxel belonging to the same part Bi in the posture shape. Alternatively, the correction unit 106 corrects the loss of the view volume shape by replacing a false negative FN in the view volume shape so as to restore the voxel possibly in the view volume shape. Bi For example, in the case of Fig. 8, processing may be performed such that voxels belonging to part B9 of the visual volume shape are corrected so that voxels 801 to 804 of the posture shape are restored in the visual volume shape.

[0068] The correction unit 106 corrects the phantom volume of the view volume shape by replacing voxels belonging to a part Bi that is identified as having a phantom volume in the view volume shape with voxels belonging to the same part Bi in the posture shape. Alternatively, the correction unit 106 corrects the phantom volume of the view volume shape by replacing false positive FP Bi For example, in the case of Fig. 8, a process may be performed in which the voxels belonging to the part B1 of the view volume shape are corrected so that the voxel 805 of the view volume shape is deleted.

[0069] On the other hand, if it is determined that neither a defect nor a phantom volume has occurred (NO in S707), the process of this flowchart ends.

[0070] The posture shape may not be able to express the shape of a part smaller than a preset part exactly as the actual object. On the other hand, since the view volume shape is generated based on a captured image of the actual object, it often expresses a shape closer to the three-dimensional shape of the actual object than the posture shape. For this reason, for example, the view volume shape is used rather than the posture shape to generate a virtual viewpoint image to be described later. According to this embodiment, the view volume shape and posture shape of the object are compared for each part, and the view volume shape can be corrected for each part based on the result. Therefore, it is possible to suppress the part of the view volume shape that does not have a defect such as a defect or a phantom volume being replaced by the posture shape. That is, according to this embodiment, it is possible to generate a view volume shape in which only the defective part is corrected.

[0071] <Embodiment 2> In the first embodiment, a method was described in which a view volume shape of an object is generated, and then a defective portion of the generated view volume shape is corrected based on the posture shape. In this embodiment, a method is described in which a defective foreground silhouette image is corrected based on the posture shape, and the corrected foreground silhouette image is used to generate a view volume shape of the object, thereby generating a view volume shape with a corrected defect. This embodiment will be described mainly with respect to differences from the first embodiment. Portions not specifically stated have the same configuration and processing as the first embodiment.

[0072] Fig. 9 is a functional configuration diagram of an image processing device 900 in this embodiment. The same components as those in Fig. 1 will be described with the same reference numerals. The difference from the first embodiment is that a reprojection unit 901, a silhouette image comparison unit 902, and a silhouette image correction unit 903 are provided.

[0073] The reprojection unit 901 performs processing based on the camera parameters acquired by the data acquisition unit 101 so that the posture shape is reprojected onto the imaging unit 111 of the image capture processing device 110, and generates a posture shape silhouette image that represents the silhouette of the posture shape. The reprojection unit 901 arranges the posture shape in a virtual space, and arranges a virtual camera based on the camera parameters so that the position and orientation of the virtual camera with respect to the posture shape are the same as the position and orientation of the imaging processing devices 110a to 110e with respect to the object 201. The reprojection unit 901 performs image processing so that the posture shape is captured from the virtual camera, thereby generating posture silhouette images corresponding to the actual imaging processing devices 110a to 110e. The posture shape silhouette image can be generated by performing a flow reverse to the flow for generating a visual volume shape from a silhouette image by a visual volume intersection method.

[0074] The silhouette image comparison unit 902 compares the posture shape silhouette image generated by the reprojection unit 901 with the foreground silhouette image generated by the imaging processing device 110 and acquired by the data acquisition unit 101. The comparison is made between the posture shape silhouette image and the foreground silhouette image corresponding to the imaging processing device 110 to be processed among the imaging processing devices 110a to 110e. Details will be described later.

[0075] A silhouette image correcting unit 903 corrects the foreground silhouette image using the posture shape silhouette image based on the result of comparison by the silhouette image comparing unit 902. The correction method will be described later.

[0076] 10 is a flowchart for explaining the flow of processing up to generating a view volume shape in this embodiment. S1001 to S1002 are the same as S701 to S702, so their explanations will be omitted. Also, S1003 to S1004 are the same as S704 to S705, so their explanations will be omitted.

[0077] In S1005, the reprojection unit 901 generates a posture shape silhouette image corresponding to each of the image capturing processing devices 110a to 110e by performing processing such that the posture shapes are reprojected onto each of the image capturing devices 111 based on the camera parameters. The posture shape silhouette image includes information indicating from which part Bi of the posture shape a pixel indicating a foreground region has been reprojected.

[0078] A pose silhouette image is, for example, an image in which the pixel values ​​of each pixel are made up of two channels. The value of one of the two channels indicates whether each pixel is in the foreground or not. The value of the other channel indicates which part Bi each pixel belongs to. In other words, the foreground region in a pose silhouette image is grouped so that it belongs to one of 13 parts. A pixel that belongs to the ith part Bi in a pose silhouette image (PoseSilhouette; PS) is called a PS Bi This is expressed as:

[0079] In S1006, the silhouette image comparison unit 902 compares the foreground silhouette image of the processing target imaging processing device 110 among the imaging processing devices 110a to 110e with the posture shape silhouette image for each of the 13 parts Bi included in the posture information.

[0080] In this step, the silhouette image comparison unit 902 applies the posture information to the foreground silhouette image of the image capturing device 110 to be processed, and groups the pixels in the foreground region of the foreground silhouette image so that they belong to one of the parts. The pixels in the foreground silhouette image (FS) that belong to the i-th part Bi are classified into groups according to the FS Bi This is expressed as:

[0081] For example, each pixel is grouped as belonging to the nearest part Bi. Alternatively, each pixel in the foreground silhouette image is projected onto a posture shape or a visual volume shape in three-dimensional space, and the part to which the voxel onto which the pixel is projected belongs is taken as the part to which the pixel belongs, thereby grouping the foreground region.

[0082] The silhouette image comparison unit 902 calculates true positives, false positives, false negatives, and true negatives for the pixels of each part Bi of the foreground silhouette image to be processed and the posture silhouette image corresponding to the foreground silhouette image to be processed. True positives, false positives, false negatives, and true negatives are elements that make up a confusion matrix that is also used as an evaluation index for machine learning. In this embodiment, the true positive TP for the i-th part Bi is Bi is calculated from equation (7). False positive FP Bi is calculated from equation (8). False negative FN Bi is calculated from equation (9). True negative TN Bi can be calculated from equation (10).

[0083]

number

[0084] false positive FP Bi FS Bi However, PS Bi represents the sum of pixels that do not exist in the image. Bi can be used to determine whether there are any regions in the foreground silhouette image that have been erroneously extracted as foreground regions, such as the shadow of object 201.

[0085] False negative FN Bi FS Bi does not exist in PS Bi represents the total number of pixels present. Bi can be used to determine defects in the foreground silhouette image.

[0086] true negative TN Bi FS Bi But P.S. Bi In this embodiment, the true negative TN Bidoes not have to be calculated.

[0087] The silhouette image comparison unit 902 compares true positive TPs Bi , false positive FP Bi , false negative FN Bi、 true negative TN Bi Before calculating, a contraction / expansion process may be applied to the group of pixels that make up the foreground region. As a result, the influence of errors in the contour portion can be suppressed, as in the first embodiment.

[0088] In S1007, the silhouette image comparison unit 902 determines whether or not a defect such as a loss occurs in the foreground region of the foreground silhouette image to be processed for each part based on the comparison result.

[0089] FIG. 11 is a diagram for explaining the processing of this flowchart. FIG. 11(a) shows a foreground silhouette image 1100 to be processed, which is generated from an image obtained by actually capturing an image of an object 201. The foreground silhouette image 1100 includes a foreground region 1102 showing the region of the object 201, and a region 1103 of pixels belonging to a part B9 of the abdomen includes a region 1104 where a defect occurs. FIG. 11(b) shows a posture silhouette image 1120 corresponding to the foreground silhouette image of FIG. 11(a) among posture silhouette images generated from the posture of the object 201. In the posture silhouette image 1120, a white region shows a foreground region 1126 showing the posture. There is no defect in a region 1127 of the abdomen B9 in the posture silhouette image 1120.

[0090] Whether there is a defect in the pixels belonging to the part Bi in the foreground silhouette image to be processed is determined by a false negative (FN) Bi is the threshold Threshold_FN Bi If the threshold is greater than or equal to the threshold value, it can be determined that the data is missing. Bi may be different from that in the first embodiment. In the cases of FIG. 11(a) and (b), the false negative FN B9 is the threshold Threshold_FN B9Since it is determined that the above is the case, it is determined that there is a defect in the abdominal region B9.

[0091] The false positive FP is used to check whether the pixels belonging to the part Bi in the foreground silhouette image to be processed include pixels that are mistakenly extracted as the foreground region due to shadows, etc. Bi is the threshold value Threshold_FP Bi It can be specified by determining whether the threshold value Threshold_FP is equal to or greater than the threshold value Threshold_FP. Bi may be different from the value in the first embodiment.

[0092] In the determination in S1007, the recall calculated by the formula (5) or the precision calculated by the formula (6) may be used.

[0093] If the silhouette image comparison unit 902 determines that the foreground region of the foreground silhouette image to be processed has a portion where a defect or the like has occurred (YES in S1007), the process proceeds to S1008. In S1008, the silhouette image correction unit 903 corrects pixels in the foreground silhouette image to be processed that belong to the portion identified as having a defect such as a loss.

[0094] When it is determined that a pixel belonging to abdominal part B9 has a defect such as a loss, a correction is performed, for example, to replace the pixel belonging to part B9 in the foreground silhouette image with a pixel belonging to part B9 in the corresponding posture silhouette image. In the case of Figs. 11(a) and (b), a correction is performed to replace abdomen B9 region 1103 in foreground region 1102 in foreground silhouette image 1100 with abdomen B9 region 1127 in posture silhouette image 1120. As a result, as shown in Fig. 11(c), a foreground silhouette image 1130 is obtained that includes foreground region 1139 in which the loss in the abdomen B9 region has been corrected. Another method of correcting the loss is to use the FN in the posture silhouette image, which indicates a group of missing pixels. BiThe correction may be performed by adding a pixel group corresponding to the missing region 1104 of the foreground silhouette image. The processes of S1006 to S1008 are executed until the foreground silhouette images of all of the imaging processing devices 110a to 110e are selected as the processing target.

[0095] In S1009, the view volume generator 102 generates a view volume of the object by a view volume intersection method using the foreground silhouette image obtained in S1002 and the camera parameters obtained in S1001. If the foreground silhouette image is corrected in S1008, the corrected foreground silhouette image is used to generate the view volume. That is, instead of foreground silhouette image 1100 in which a defect occurs, corrected foreground silhouette image 1130 is used to generate the view volume. As a result of this processing, a view volume in which the defect or phantom volume has been corrected is generated.

[0096] As described above, according to this embodiment, a foreground silhouette image and a posture shape silhouette image are compared for each part, and the foreground silhouette image can be corrected for each part based on the results of the comparison. This makes it possible to locally improve the accuracy of the visual volume shape.

[0097] <Embodiment 3> In this embodiment, an example of generating a virtual viewpoint image using the corrected view volume shape will be described. The present embodiment will be described focusing on the differences from the first embodiment. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

[0098] The virtual viewpoint image is an image that represents a view from a virtual viewpoint, which is the viewpoint of a non-existent imaging unit different from the viewpoint of the actually existing imaging unit 111. The virtual viewpoint image in this embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. In addition, the virtual viewpoint may be specified by a user operation, or may be automatically specified based on the result of image analysis or the like. The virtual viewpoint image may be a video or a still image. The virtual viewpoint image is generated using the visual volume shape of the object, the foreground texture, and information on the virtual viewpoint.

[0099] Fig. 12 is a functional configuration diagram of an image processing device 1200 in this embodiment. The same components as those in Fig. 1 are denoted by the same reference numerals and will be described. The difference from Fig. 1 is that it includes a visibility determination unit 1201 and a virtual viewpoint image generation unit 1202.

[0100] The visibility determination unit 1201 performs a visibility determination process to determine the visible and invisible areas of the imaging units 111 of each of the imaging processing devices 110a to 110e and generate visibility information. When the correction unit 106 performs a correction process to restore a missing area of ​​the visual volume shape, the visibility determination unit 1201 performs a process to determine an area containing a missing area as an invisible area. A specific method will be described later.

[0101] The virtual viewpoint image generation unit 1202 generates a virtual viewpoint image based on the foreground texture image and camera parameters acquired by the data acquisition unit 101, the visual volume shape corrected by the correction unit 106, and the visibility information generated by the visibility determination unit 1201.

[0102] For example, the virtual viewpoint image generating unit 1202 colors voxels corresponding to pixels determined to be visible in the visibility information using the color of the pixel of the foreground texture image corresponding to the pixel determined to be visible in the visibility information. The virtual viewpoint image generating unit 1202 generates a virtual viewpoint image by rendering the colored view volume shape and background so as to form an image of an object viewed from a virtual viewpoint. The generated virtual viewpoint image is transmitted to the display device 1210. The display device 1210 displays the received virtual viewpoint image.

[0103] [About visibility information] The visibility information is information indicating the visible and invisible areas in each imaging unit. A distance image in which the pixel value of each pixel is the distance value from the target imaging unit to the object is used as the visibility information. Since the visibility information in the virtual viewpoints can be shared, multiple virtual viewpoint images can be easily generated.

[0104] The visibility determination unit 1201 generates distance images corresponding to each of the imaging processing devices 110a to 110e as shown in Fig. 2. For example, the virtual camera and the corrected view volume shape are arranged in the virtual space based on the camera parameters so that the position and orientation of the virtual camera with respect to the corrected view volume shape are the same as the position and orientation of the actual imaging processing device 110 with respect to the object 201. Then, a distance image indicating the position from each virtual camera to the corrected view volume shape is generated. The generated distance images become distance images corresponding to the imaging units 111 of each of the imaging processing devices 110a to 110e. The generated distance images become the distance images before correction.

[0105] The distance image of this embodiment is an unsigned 16-bit 1-channel image, and will be described as an image having pixel values ​​ranging from 0 to 65535. An area with a pixel value of 65535 represents an area where no object exists. An area with pixel values ​​of 1 to 65534 represents a visible area of ​​the target imaging unit within an area where an object exists. A pixel with a pixel value of 0 represents an invisible area of ​​the target imaging unit within the area of ​​the object.

[0106] FIG. 13(a) is a diagram showing an example of a distance image before correction generated by the visibility determination unit 1201. The distance image 1301 is a distance image generated based on a virtual camera corresponding to an imaging unit (target imaging unit) of one of the imaging processing devices 110a to 110e. A white area in the distance image 1301 represents an area where no object exists, with a pixel value of 65535. A gray area represents a visible area with pixel values ​​of 1 to 65534. A black area represents an invisible area with a pixel value of 0, but is not included in the distance image 1301 of FIG. 13(a). The visibility determination unit 1201 generates a distance image by assigning a pixel value of 1 to 65534, which represents distance, to pixels in an area where a voxel exists in the view volume shape. Therefore, the distance image 1301 before correction does not include an invisible area.

[0107] Fig. 13(b) is a diagram showing a foreground silhouette image of the target imaging unit. A foreground silhouette image 1303 in Fig. 13(b) includes a defective region 1305 that should have been extracted as a foreground region but was not. On the other hand, an uncorrected distance image 1301 shown in Fig. 13(a) is generated based on the corrected view volume shape, and is therefore generated so that the defective region 1305 is a visible region.

[0108] 13(c) shows foreground texture image 1306 corresponding to foreground silhouette image 1303. In other words, foreground texture image 1306 is a foreground texture image output from imaging processing device 110 which has output foreground silhouette image 1303. From the viewpoint of data compression, the foreground texture image is an image in which only foreground region 1304, which is a white region in foreground silhouette image 1303, has been cut out from the captured image. For this reason, if there is a defect in foreground region 1304, there will also be a defect in foreground texture 1307.

[0109] When a defective region is included as in the foreground texture image 1306 in FIG. 13(c), it is required that the pixels of the defective region in the foreground texture image 1306 are not used for coloring the visual volume shape. However, in the distance image 1301 in FIG. 13(a), the region corresponding to the defective region 1305 is also a visible region. Therefore, when the distance image 1301 is used as visibility information for generating a virtual viewpoint image, there is a risk that the pixels of the defective region without texture in the foreground texture image will be used for coloring the virtual viewpoint image. If the pixels of the defective region are used for coloring, the quality of the virtual viewpoint image will deteriorate. Therefore, in this embodiment, the visibility determination unit 1201 performs a process of correcting the generated distance image 1301.

[0110] [flowchart] FIG. 14 is a flowchart for explaining the flow of processing executed by the image processing device 100 up to generation of a virtual viewpoint image.

[0111] S1401 is the same step as S701, and therefore the explanation will be omitted.

[0112] S1402 is a step similar to S702, in which a foreground silhouette image and a foreground texture image are obtained. When a foreground silhouette image 1303 having a missing region 1305 as shown in Fig. 13(b) is obtained, a foreground texture image 1306 corresponding to the foreground silhouette image 1303 as shown in Fig. 13(c) is obtained.

[0113] S1403 is a step similar to S703, and a volumetric shape is generated using a volumetric intersection method based on the foreground silhouette image and camera parameters output from each imaging processing device 110. When a volumetric shape is generated using a foreground silhouette image 1303 having a missing region 1305 in the abdomen of a foreground region 1304 as shown in Fig. 13(b), a missing voxel occurs in the abdomen in the generated volumetric shape.

[0114] S1404 to S1407 are similar to S704 to S707, and therefore the description will be omitted.

[0115] S1408 is a step similar to S708, and a correction process for the view volume is performed based on the comparison result of S1406. For example, if a defect occurs in the abdominal part of the view volume, the defect is corrected.

[0116] In S1409, the visibility determination unit 1201 generates distance images for each of the imaging processing devices 110a to 110e based on the foreground silhouette image and camera parameters acquired by the data acquisition unit 101 and the visual volume shape corrected by the correction unit 106. As a result, an uncorrected distance image 1301 as shown in FIG. 13(a) is generated.

[0117] Then, visibility determination unit 1201 performs a process on uncorrected distance image 1301, by referring to corresponding foreground silhouette image 1303, to set missing region 1305 in distance image 1301 to an invisible region. Specifically, visibility determination unit 1201 performs a correction to set pixel values ​​of regions in distance image 1301 that correspond to regions other than the foreground region in corresponding foreground silhouette image 1303 to 0. However, visibility determination unit 1201 does not correct the pixel value of a region in distance image 1301 where the pixel value is 65535, because this is a region where no object exists. As a result, the pixel value of uncorrected distance image 1301 remains 65535.

[0118] 13(d) shows a distance image 1309 after the uncorrected distance image 1301 has been corrected to make the corresponding missing area 1305 in the foreground silhouette image 1203 invisible. Black areas represent invisible areas whose pixel value is 0. In this embodiment, the corrected distance image 1309 is used as visibility information.

[0119] In S1410, the virtual viewpoint image generating unit 1202 generates a virtual viewpoint image. The virtual viewpoint image is generated based on the foreground texture image and camera parameters acquired by the data acquiring unit 101, the visual volume shape acquired from the correcting unit 106, and the visibility information generated by the visibility determining unit 1201. The position of the virtual viewpoint and the line of sight direction from the virtual viewpoint for generating the virtual viewpoint image may be set in advance, or may be set by the user via a UI (not shown).

[0120] The virtual viewpoint image generating unit 1202 can generate a virtual viewpoint image by using the distance image 1309, in which the missing area of ​​the foreground texture image is set as an invisible area, as visibility information. Therefore, when generating the virtual viewpoint image, the missing area in the foreground texture image is prevented from being used for coloring the visual volume shape. Therefore, deterioration of the image quality of the virtual viewpoint image is suppressed.

[0121] As described above, in this embodiment, visibility information is generated taking into consideration the area where the visual volume shape has been corrected. Therefore, according to this embodiment, a virtual viewpoint image can be generated using appropriate visibility information, and image quality degradation of the virtual viewpoint image can be suppressed.

[0122] In the above description of this embodiment, the visibility determination unit 1201 and the virtual viewpoint image generation unit 1202 are added to the image processing device of embodiment 1. Alternatively, the visibility determination unit 1201 and the virtual viewpoint image generation unit 1202 may be added to the configuration of the image processing device of embodiment 2. In that case, when a missing area in a foreground silhouette image is corrected as in embodiment 2, a process of making the missing area in the distance image an invisible area may be performed by referring to the foreground silhouette image before the missing area is corrected.

[0123] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0124] The disclosure of the above-described embodiment includes the following configurations.

[0125] (Configuration 1) a first acquisition means for acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition means for acquiring second three-dimensional shape data generated by transforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured image; a correction means for correcting a defect in the first three-dimensional shape data based on the second three-dimensional shape data; 13. An image processing device comprising:

[0126] (Configuration 2) a first acquisition means for acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition means for acquiring second three-dimensional shape data generated by transforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured image; a correcting means for correcting a defective silhouette image among the plurality of silhouette images based on the second three-dimensional shape data, The first three-dimensional shape data acquired by the first acquisition means is three-dimensional shape data generated by using a corrected silhouette image instead of the defective silhouette image. 13. An image processing device comprising:

[0127] (Configuration 3) The method further includes a visibility determination unit that determines a visible area and an invisible area of ​​each of the plurality of imaging devices based on the corrected first three-dimensional shape data and the silhouette image. 3. The image processing device according to configuration 1 or 2.

[0128] (Configuration 4) The visibility determination means includes: A region of the three-dimensional shape represented by the corrected first three-dimensional shape data is determined as a visible region, and a region of the determined visible region other than the region of the object in the silhouette image is determined as the invisible region. 4. The image processing device according to configuration 3.

[0129] (Configuration 5) The present invention further includes a generating means for generating a virtual viewpoint image based on the corrected first three-dimensional shape data, the visible area and the invisible area determined by the visibility determining means, and a texture image of the object generated from the captured image. 5. The image processing device according to configuration 3 or 4.

[0130] (Configuration 6) comparing the three-dimensional shape represented by the first three-dimensional shape data and the three-dimensional shape represented by the second three-dimensional shape data for each part representing the posture; The method further includes a comparison means for identifying a portion where the defect occurs in the three-dimensional shape represented by the first three-dimensional shape data, The correction means corrects a portion where the defect occurs in the three-dimensional shape represented by the first three-dimensional shape data. 2. The image processing device according to claim 1,

[0131] (Configuration 7) When the number of voxels in the region to be compared that are not included in the first three-dimensional shape data but are included in the second three-dimensional shape data is equal to or greater than a first threshold value, Identifying that the defect occurs in the comparison target site in the three-dimensional shape represented by the first three-dimensional shape data. 7. The image processing device according to configuration 6,

[0132] (Configuration 8) When the number of voxels in the region to be compared that are in the first three-dimensional shape data but not in the second three-dimensional shape data is equal to or greater than a second threshold value, Identifying that a phantom volume occurs in the comparison target region in the three-dimensional shape represented by the first three-dimensional shape data; The correction means corrects a portion identified as having a phantom volume in the three-dimensional shape represented by the first three-dimensional shape data based on the second three-dimensional shape data. 8. The image processing device according to configuration 7.

[0133] (Configuration 9) The correction means is The first three-dimensional shape data is corrected by replacing a portion in the three-dimensional shape represented by the first three-dimensional shape data where the defect occurs with a corresponding portion in the three-dimensional shape represented by the second three-dimensional shape data. 9. The image processing device according to any one of configurations 1 or 6 to 8.

[0134] (Configuration 10) The defect occurs due to the silhouette image. 10. The image processing device according to any one of configurations 1 or 6 to 9.

[0135] (Configuration 11) The present invention further includes a posture estimation unit that estimates a position of a part of the object based on the captured image and generates posture information including information on the position of the part. 11. The image processing device according to any one of configurations 1 to 10,

[0136] (Configuration 12) The data representing the basic shape is three-dimensional shape data generated in advance, or the first three-dimensional shape data of the object corresponding to one or more previous frames. 12. The image processing device according to any one of configurations 1 to 11,

[0137] (Configuration 13) The object is a person. 13. The image processing device according to any one of configurations 1 to 12.

[0138] (Configuration 14) The correction means is A defective area of ​​the object area in the silhouette image is corrected based on a corresponding area in an image representing a silhouette of the three-dimensional shape represented by the second three-dimensional shape data. 3. The image processing device according to configuration 2.

[0139] (Configuration 15) a first acquisition step of acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition step of acquiring second three-dimensional shape data generated by transforming second three-dimensional shape data representing a basic shape of the object stored in advance based on information on the orientation of the object in the captured image; a correction step of correcting a defect in the first three-dimensional shape data based on the second three-dimensional shape data; 13. An image processing method comprising:

[0140] (Configuration 16) a first acquisition step of acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition step of acquiring second three-dimensional shape data generated by transforming second three-dimensional shape data representing a basic shape of the object stored in advance based on information on the orientation of the object in the captured image; a correcting step of correcting a defective silhouette image among the plurality of silhouette images based on the second three-dimensional shape data, The first three-dimensional shape data acquired in the first acquisition step is three-dimensional shape data generated by using a corrected silhouette image instead of the defective silhouette image. 13. An image processing method comprising:

[0141] (Configuration 17) A program for causing a computer to execute each of the means of the image processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0142] 100 Image processing device 102 Volumetric Shape Generation Unit 104 Posture shape generator 106 Correction section

Claims

1. a first acquisition means for acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition means for acquiring second three-dimensional shape data generated by transforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured image; a correction means for correcting a defect in the first three-dimensional shape data based on the second three-dimensional shape data; 13. An image processing device comprising:

2. a first acquisition means for acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition means for acquiring second three-dimensional shape data generated by transforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured image; a correcting means for correcting a defective silhouette image among the plurality of silhouette images based on the second three-dimensional shape data, The first three-dimensional shape data acquired by the first acquisition means is three-dimensional shape data generated by using a corrected silhouette image instead of the defective silhouette image.

13. An image processing device comprising:

3. The method further includes a visibility determination unit that determines a visible area and an invisible area of ​​each of the plurality of imaging devices based on the corrected first three-dimensional shape data and the silhouette image.

2. The image processing device according to claim 1,

4. The visibility determination means includes: A region of the three-dimensional shape represented by the corrected first three-dimensional shape data is determined as a visible region, and a region of the determined visible region other than the region of the object in the silhouette image is determined as the invisible region.

4. The image processing device according to claim 3.

5. The present invention further includes a generating means for generating a virtual viewpoint image based on the corrected first three-dimensional shape data, the visible area and the invisible area determined by the visibility determining means, and a texture image of the object generated from the captured image.

4. The image processing device according to claim 3.

6. a comparison means for comparing the three-dimensional shape represented by the first three-dimensional shape data with the three-dimensional shape represented by the second three-dimensional shape data for each part representing the posture, and identifying a part in the three-dimensional shape represented by the first three-dimensional shape data where the defect occurs; The correction means corrects a portion where the defect occurs in the three-dimensional shape represented by the first three-dimensional shape data.

2. The image processing device according to claim 1,

7. The comparison means determines that the defect has occurred in the comparison target region in the three-dimensional shape represented by the first three-dimensional shape data when the number of voxels in the comparison target region that are not in the first three-dimensional shape data but are in the second three-dimensional shape data is equal to or greater than a first threshold value.

7. The image processing device according to claim 6,

8. the comparing means determines that a phantom volume is generated in the comparison target region in the three-dimensional shape represented by the first three-dimensional shape data when the number of voxels in the comparison target region that are present in the first three-dimensional shape data but not in the second three-dimensional shape data is equal to or greater than a second threshold value; The correction means corrects a portion identified as having a phantom volume in the three-dimensional shape represented by the first three-dimensional shape data based on the second three-dimensional shape data.

8. The image processing device according to claim 7,

9. The correction means is The first three-dimensional shape data is corrected by replacing a portion in the three-dimensional shape represented by the first three-dimensional shape data where the defect occurs with a corresponding portion in the three-dimensional shape represented by the second three-dimensional shape data.

2. The image processing device according to claim 1,

10. The defect occurs due to the silhouette image.

2. The image processing device according to claim 1,

11. The present invention further includes a posture estimation unit that estimates a position of a part of the object based on the captured image and generates posture information including information on the position of the part.

2. The image processing device according to claim 1,

12. The data representing the basic shape is three-dimensional shape data generated in advance, or the first three-dimensional shape data of the object corresponding to one or more previous frames.

2. The image processing device according to claim 1,

13. The object is a person.

2. The image processing device according to claim 1,

14. The correction means is A defective area of ​​the object area in the silhouette image is corrected based on a corresponding area in an image representing a silhouette of the three-dimensional shape represented by the second three-dimensional shape data.

3. The image processing device according to claim 2.

15. a first acquisition step of acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition step of acquiring second three-dimensional shape data generated by transforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured image; a correction step of correcting a defect in the first three-dimensional shape data based on the second three-dimensional shape data; 13. An image processing method comprising:

16. a first acquisition step of acquiring first three-dimensional shape data representing a three-dimensional shape of the object, the first three-dimensional shape data being generated based on a plurality of silhouette images representing a region of the object generated from a plurality of captured images obtained by capturing images of the object by a plurality of imaging devices; a second acquisition step of acquiring second three-dimensional shape data generated by transforming pre-stored second three-dimensional shape data representing a basic shape of the object based on information on the orientation of the object in the captured image; a correcting step of correcting a defective silhouette image among the plurality of silhouette images based on the second three-dimensional shape data, The first three-dimensional shape data acquired in the first acquisition step is three-dimensional shape data generated by using a corrected silhouette image instead of the defective silhouette image.

13. An image processing method comprising:

17. A program for causing a computer to execute each of the means of the image processing apparatus according to any one of claims 1 to 14.

Images