Parameter estimation device, parameter estimation method, and parameter estimation program

Abstract

To improve estimation accuracy further, in a technique for estimating a parameter group on the basis of, two-dimensional coordinates of feature points detected in each of images captured by a plurality of cameras.SOLUTION: A parameter estimation apparatus includes: an acquisition unit which acquires subject attitude group data indicating data obtained by associating two-dimensional coordinates of a same feature point of a same subject in images captured by a plurality of cameras from different positions; a weight coefficient calculation unit which calculates, on the basis of, the subject attitude group data acquired by the acquisition unit, a screen occupancy indicating a ratio that a region of the feature point occupies with respect to an entire region of the image, and calculates a weight coefficient on the basis of, the calculated screen occupancy; and a parameter estimation unit which estimates, on the basis of calculation using the weight coefficient calculated by the weight coefficient calculation unit, a value of a parameter including coordinate values of three-dimensional coordinates of the feature point.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a parameter estimation device, a parameter estimation method, and a parameter estimation program. [Background technology]

[0002] Bundle Adjustment is a technology that detects feature points from images of the same subject captured by multiple cameras, associates the detected feature points between the cameras (between images), and then estimates the camera parameters and the 3D coordinates of each feature point. Bundle Adjustment estimates the optimal camera parameters and the 3D coordinates of each feature point by performing iterative calculations using a cost function that minimizes the reprojection error of each feature point associated between the cameras.

[0003] FIG. 14 is a schematic diagram for explaining the outline of Bundle Adjustment. FIG. 14 shows, as an example, a case where the same subject is captured by two cameras (camera j and camera j+1) from different positions. As shown in FIG. 14, first, in Bundle Adjustment, for example, feature points are detected from an image captured by camera j and an image captured by camera j+1. The feature points detected from both images are associated with each other for each feature point of the same location of the subject. Here, the two-dimensional coordinates of the feature points detected from the image captured by camera j are denoted by u. i,j Let u be the two-dimensional coordinates of the feature points detected from the image captured by camera j+1. i,j+1 Let the three-dimensional coordinates of a feature point be p i Let us assume that.

[0004] In the bundle adjustment, the 3D coordinates p of the feature points reprojected onto the screen of the camera j are calculated by the cost function expressed by the following equation (1). i Let u be the 2D coordinates of the feature points reprojected onto the screen of camera j. i,jand the 3D coordinates p of the feature point reprojected onto the screen of camera j+1. i+1 Let u be the 2D coordinates of the feature points reprojected onto the screen of camera j+1. i,j+1 Iterative calculations are performed to minimize the reprojection error between the 3D coordinates of the feature points p i is estimated. For the iterative calculation, the Gauss-Newton method, for example, is used.

[0005]

number

[0006] In the above formula (1), the function F(p i |f j ,r j ,d j ,R j ,t j ) is the 3D coordinate of the feature point p i is a function that projects the image onto the screen of camera j. Here, f j ,r j ,d j ,R j ,t j are the camera parameters of camera j. Specifically, f j is the camera focal length, r j is the camera resolution, d j is the camera distortion, R j is the camera rotation angle, and t j is the camera position. Similarly, for the camera parameters of camera j+1, j+1 is the camera focal length, r j+1 is the camera resolution, d j+1 is the camera distortion, R j+1 is the camera rotation angle, and t j+1 is the camera position.

[0007] In general, when an object for which the 3D coordinates of feature points are to be acquired is a geometric structure such as a building, a method of detecting feature points and associating between cameras using SIFT (Scale-Invariant Feature Transform) features is widely used. On the other hand, when the object for which the 3D coordinates of feature points are to be acquired is a person, a 2D pose estimation technique such as OpenPose (see Non-Patent Document 1) and PersonLab (see Non-Patent Document 2) is used. These techniques detect parts of a person, such as joints, eyes, nose, and ears. Then, for example, under the assumption that the length of a person's skeleton is invariant, the detected parts of the person are regarded as feature points and estimation is performed by Bundle Adjustment (see Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Z. Cao, G. Hidalgo, T. Simon, SE Wei, and Y. Sheikh, “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, arXiv: 1812.08008v2 [cs.CV], pp.1-14, May 2019. [Non-Patent Document 2] G. Papandreou, T. Zhu, LC Chen, S. Gidaris, J. Tompson, and K. Murphy, “PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model”, ECCV, 2018. [Non-Patent Document 3] K. Takahashi, D. Mikami, M Isogawa, and H. Kimata, “Human Pose as Calibration Pattern; 3D Human Pose Estimation with Multiple Unsynchronized and Uncalibrated Cameras”, CVPR, pp.1888-1895, 2018. [Non-Patent Document 4] T. Ohashi1, Y. Ikegami, and Y. Nakamura, “Synergetic Reconstruction from 2D Pose and 3D Motion for Wide-Space Multi-Person Video Motion Capture in the Wild”, arXiv: 2001.05613v2 [cs.CV], pp.1-12, Oct 2020. Summary of the Invention [Problem to be solved by the invention]

[0009] In the Bundle Adjustment, a method is widely used in which parameters are updated to minimize the mean square error (MSE) of the reprojection error in the process of iterative calculation to optimize each parameter. However, even in the process of detecting feature points by the above-mentioned OpenPose and PersonLab, estimation errors are superimposed on the two-dimensional coordinates of the feature points, so that each feature point of the subject in an image (e.g., each frame of a video) has only a maximum of the same number of sampling points as the number of cameras. Therefore, if estimation errors of the two-dimensional coordinates of the feature points are superimposed due to large noise, the estimation accuracy of the camera parameters and the three-dimensional coordinates of each feature point may be significantly impaired.

[0010] In particular, as shown in FIG. 15, even if the estimation errors of both feature points are about the same (for example, an estimation error of 2 pixels for both) when the subject appears small on the screen and when it appears large on the screen, the estimation errors are different in three-dimensional space. In this case, the smaller the subject appears on the screen, the larger the estimation error will be, and the larger the subject appears on the screen, the smaller the estimation error will be. Therefore, in the conventional method in which an object that appears small on the screen is treated as an object that appears large on the screen and estimated, the estimation accuracy of the camera parameters and the three-dimensional coordinates of each feature point may decrease.

[0011] In particular, as shown in Fig. 16, there may be cases where an occluded image area exists because the image area of ​​the feature point to be detected overlaps, for example, an image area of ​​another part of the human subject, or an image area of ​​another person captured in the same image, etc. In this case, the estimation accuracy of the camera parameters and the three-dimensional coordinates of each feature point may decrease in the conventional method that performs estimation by treating a subject with an occluded image area in the same way as a subject with no occluded image area.

[0012] In this way, conventional bundle adjustment detects feature points of a subject from images captured by multiple cameras, and estimates the three-dimensional coordinates and camera parameters of each feature point (hereinafter, these are collectively referred to as a "parameter group") by matching the two-dimensional coordinates of each feature point between images. However, there is a problem in that if the detection of feature points contains estimation errors, the estimation accuracy of the parameter group can be significantly impaired.

[0013] In view of the above circumstances, an object of the present invention is to provide a parameter estimation device, a parameter estimation method, and a parameter estimation program that can improve the estimation accuracy in a technique for estimating a group of parameters based on two-dimensional coordinates of feature points detected from images captured by multiple cameras. [Means for solving the problem]

[0014] One aspect of the present invention is a parameter estimation device that includes an acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the same subject are associated with each other in each image of the same subject captured by a plurality of cameras from different positions; a weighting coefficient calculation unit that calculates a screen occupancy indicating the proportion of the area of ​​the feature point to the entire area of ​​the image based on the subject posture group data acquired by the acquisition unit and calculates a weighting coefficient based on the calculated screen occupancy; and a parameter estimation unit that estimates a value of a parameter including three-dimensional coordinate values ​​of the feature point based on a calculation using the weighting coefficient calculated by the weighting coefficient calculation unit.

[0015] Moreover, one aspect of the present invention is a parameter estimation method having: an acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the same subject are associated with each other in each image captured by a plurality of cameras of the same subject from different positions; a weighting coefficient calculation step of calculating an in-screen occupancy indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired in the acquisition step, and calculating a weighting coefficient based on the calculated in-screen occupancy; and a parameter estimation step of estimating a value of a parameter including a coordinate value of the three-dimensional coordinate of the feature point based on a calculation using the weighting coefficient calculated in the weighting coefficient calculation step.

[0016] Another aspect of the present invention is a parameter estimation program for causing a computer to function as the parameter estimation device described above. Effect of the Invention

[0017] According to the present invention, it is possible to further improve the estimation accuracy in a technique for estimating a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing the overall configuration of an object 3D reconstruction device 100 according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart showing the operation of the object 3D reconstruction device 100 according to the first embodiment of the present invention. [Diagram 3] 1 is a block diagram showing a functional configuration of an optimization unit 102 of an object 3D reconstruction device 100 according to a first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an example of a connection relationship between feature points in a human skeleton or the like. [Diagram 5] 11 is a diagram for explaining a method of deriving a weighting coefficient based on an occupancy rate within a screen. FIG. [Figure 6] 5 is a flowchart showing the operation of an optimization unit 102 of the object 3D reconstruction device 100 according to the first embodiment of the present invention. [Figure 7] FIG. 11 is a block diagram showing the overall configuration of an object 3D reconstruction device 100a according to a second embodiment of the present invention. [Figure 8] 10 is a flowchart showing the operation of the object 3D reconstruction device 100a according to the second embodiment of the present invention. [Figure 9] FIG. 11 is a block diagram showing the functional configuration of an optimization unit 102a of an object 3D reconstruction device 100a according to a second embodiment of the present invention. [Figure 10] 13 is a diagram for explaining a method of deriving a weighting factor based on a visibility rate for self-occlusion. FIG. [Figure 11] 11 is a diagram for explaining a method of deriving a weighting factor based on a visibility rate with respect to occlusion by another person; FIG. [Figure 12] 10 is a flowchart showing the operation of an optimization unit 102a of the object 3D reconstruction device 100 according to the second embodiment of the present invention. [Figure 13] 1 is a block diagram showing an example of a hardware configuration of an object 3D reconstruction device according to the first to third embodiments of the present invention. [Figure 14]FIG. 1 is a schematic diagram for explaining an overview of Bundle Adjustment. [Figure 15] 1A and 1B are diagrams for explaining differences in the size of a subject within a screen and two-dimensional estimation errors and three-dimensional estimation errors. [Figure 16] 1A and 1B are diagrams for explaining a decrease in the estimation accuracy of two-dimensional coordinates due to obstruction by another person. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Hereinafter, embodiments of a parameter estimation device, a parameter estimation method, and a parameter estimation program according to the present invention will be described with reference to the drawings.

[0020] <First embodiment> The object 3D reconstruction device 100 according to the first embodiment of the present invention will be described below. Note that the object 3D reconstruction device 100 described below is one example of an embodiment of a parameter estimation device of the present invention, and the present invention is not necessarily limited to the configuration of the object 3D reconstruction device 100 described below.

[0021] [Configuration of 3D object reconstruction device] The overall configuration of the object 3D reconstruction device 100 will be described below. FIG. 1 is a block diagram showing the overall configuration of the object 3D reconstruction device 100 in the first embodiment of the present invention. As shown in FIG. 1, the object 3D reconstruction device 100 includes an inter-camera correspondence unit 101 and an optimization unit 102. Note that in the object 3D reconstruction device 100 shown in FIG. 1, the optimization unit 102 has a different configuration from that of a conventional general object 3D reconstruction device, and the other configurations are the same as those of the conventional object 3D reconstruction device.

[0022] The camera correspondence unit 101 accepts input of subject posture group data output from an external device (not shown). The subject posture group data here refers to data generated from images of a person as a subject captured simultaneously from different positions by multiple cameras. The subject posture group data includes data indicating the positions of the joints, eyes, nose, ears, etc. of the person as a subject in each image, detected by a two-dimensional posture estimation technique represented by OpenPose, PersonLab, etc. The camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the subject in each image as feature points based on the input subject posture group data. The camera correspondence unit 101 associates feature points of the same part of the same person in images captured simultaneously by different cameras (for example, feature points indicating the positions of right elbows, or feature points indicating the positions of left knees, etc.). Hereinafter, data in which these feature points are associated is referred to as "camera-correlated subject posture group data." The camera correspondence unit 101 generates camera-correlated subject posture group data and outputs it to the optimization unit 102.

[0023] The optimization unit 102 receives input of the camera-correlated subject posture group data output from the camera-correlated unit 101. The optimization unit 102 executes a bundle adjustment using feature points based on the camera-correlated subject posture group data to optimize a parameter group. The parameter group here refers to the camera parameter values ​​of each camera that captured the original image and the three-dimensional coordinates of each feature point. The camera parameters include, for example, the camera position, imaging direction, rotation angle, viewing angle, focal length, resolution, and distortion.

[0024] The optimization unit 102 outputs the optimized parameter group to an external device (not shown). The optimized parameter group here refers to a parameter group optimized for the purpose of minimizing the mean square error of the difference (reprojection error) between the actual 3D coordinates of the feature points and the estimated 3D coordinates.

[0025] Note that the inter-camera correspondence unit 101 may be provided in an external device. That is, the inter-camera corresponded subject posture group data may be generated in an external device (not shown), and the subject 3D reconstruction device 100 may acquire the inter-camera corresponded subject posture group data from the external device.

[0026] [Operation of the 3D object reconstruction device] Hereinafter, a description will be given of an example of the operation of object 3D reconstruction device 100. Fig. 2 is a flowchart showing the operation of object 3D reconstruction device 100 in the first embodiment of the present invention. This flowchart starts when object posture group data output from an external device is input to object 3D reconstruction device 100.

[0027] The camera correspondence unit 101 of the object 3D reconstruction device 100 accepts input of object posture group data output from an external device. The camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the object in each image based on the input object posture group data as feature points. The camera correspondence unit 101 associates feature points of the same part of the same person in images captured simultaneously by different cameras (step S101). The camera correspondence unit 101 generates camera-correlated object posture group data and outputs it to the optimization unit 102.

[0028] The optimization unit 102 receives input of the inter-camera associated subject posture group data output from the inter-camera association unit 101. The optimization unit 102 executes a bundle adjustment using feature points based on the inter-camera associated subject posture group data, and optimizes the parameter group (step S102). The optimization unit 102 outputs the optimized parameter group to an external device. This completes the operation of the object 3D reconstruction device 100 shown in the flowchart of FIG. 2.

[0029] [Configuration of the optimization section] The configuration of the optimization unit 102 will be described in more detail below. Fig. 3 is a block diagram showing the functional configuration of the optimization unit 102 of the object 3D reconstruction device 100 in the first embodiment of the present invention. As shown in Fig. 3, the optimization unit 102 includes a parameter initialization unit 1021, a weighting factor initialization unit 1022, a parameter optimization unit 1023, an object orientation reprojection unit 1024, and a weighting factor calculation unit 1025.

[0030] In the optimization unit 102 shown in FIG. 3, the configuration differs from the conventional technology in that the weight coefficient calculation unit 1025 calculates weight coefficients based on the screen occupancy rate, as described below, of feature points derived for each camera and each feature point, and the parameter optimization unit 1023 applies weighting calculations to the cost function of the Bundle Adjustment using the calculated weight coefficients. The other configurations are the same as those of the optimization unit of the conventional object 3D reconstruction device.

[0031] The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correspondence unit 101. The parameter initialization unit 1021 initializes the camera parameters for each camera and the three-dimensional coordinates for each feature point based on the acquired camera-correlated subject posture group data. That is, the parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates". The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023.

[0032] The weighting coefficient initialization unit 1022 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The weighting coefficient initialization unit 1022 initializes weighting coefficients for each camera and each feature point based on the acquired camera-correlated subject posture group data. That is, the weighting coefficient initialization unit 1022 initializes a weighting coefficient of "number of cameras x number of feature points". The weighting coefficient initialization unit 1022 outputs the initialized weighting coefficient group to the parameter optimization unit 1023.

[0033] The parameter optimization unit 1023 acquires the camera-correlated object posture group data output from the camera-corresponding unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025. The parameter optimization unit 1023 updates the parameter group based on the acquired camera-correlated object posture group data and weighting coefficient group. The parameter optimization unit 1023 outputs the updated parameter group to the object posture reprojection unit 1024 or an external device (not shown).

[0034] The subject pose reprojection unit 1024 acquires the updated parameter group output from the parameter optimization unit 1023. The subject pose reprojection unit 1024 projects the three-dimensional coordinates of each feature point on the screen of each camera. The subject pose reprojection unit 1024 connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera and each subject. The subject pose reprojection unit 1024 outputs the generated camera and subject silhouette group to the weighting coefficient calculation unit 1025.

[0035] The weighting coefficient calculation unit 1025 acquires the group of camera-specific and subject-specific silhouettes output from the subject orientation reprojection unit 1024. The weighting coefficient calculation unit 1025 calculates the screen occupancy rate of each feature point from a silhouette based on the acquired group of camera-specific and subject-specific silhouettes, and calculates a weighting coefficient. The weighting coefficient calculation unit 1025 outputs the group of calculated weighting coefficients for each camera and feature point (i.e., consisting of weighting coefficients of the number of cameras x the number of feature points) to the parameter optimization unit 1023.

[0036] When the parameter optimization unit 1023 acquires the weighting coefficient group output from the weighting coefficient calculation unit 1025, the parameter optimization unit 1023 updates the parameter group based on the weighting coefficient group and the inter-camera associated subject posture group data output from the inter-camera association unit 101. The optimization unit 102 performs an iterative calculation to update the above-mentioned parameter group a predetermined number of times. Alternatively, the optimization unit 102 performs an iterative calculation to update the above-mentioned parameter group until the values ​​of the parameter group converge. When the iterative calculation is completed, the parameter optimization unit 1023 sets the updated parameter group as an optimized parameter group, and outputs the parameter group to an external device (not shown).

[0037] More specifically, for example, during iterative calculation in the bundle adjustment, the object posture reprojection unit 1024 reprojects onto the screen of each camera using each camera parameter in the optimization process in each step of the iterative calculation and the 3D coordinates of each feature point. The weighting factor calculation unit 1025 calculates the screen occupancy rate of the feature point relative to the camera resolution (for example, the ratio of the number of pixels of the silhouette of the feature point to the number of pixels of the camera). The weighting factor calculation unit 1025 derives a weighting factor based on the screen occupancy rate. The parameter optimization unit 1023 incorporates the weighting factor into a cost function to perform iterative calculation. As a result, the object 3D reconstruction device 100 in the first embodiment can suppress random noise that is likely to be superimposed on an object that appears small on the screen.

[0038] When reprojecting onto the screen of each camera, subject orientation reprojection unit 1024 not only projects feature points as points, but also draws line segments between connected feature points as line segments with width, circles, etc., taking into consideration the connection relationships between feature points (such as the skeleton of a person) as shown in Fig. 4. Subject orientation reprojection unit 1024 draws a silhouette that fits the shape of the periphery of the subject's feature points using line segments with width, circles, etc.

[0039] The weighting coefficient calculation unit 1025 generates, as a partial silhouette, a figure in which a line segment connecting a target feature point and another feature point connected to the target feature point is truncated at a certain internal division ratio, as shown in Fig. 5, for example. For example, Fig. 5 shows a partial silhouette in the case where a line segment connecting a target feature point and another feature point connected to the target feature point is truncated at the center position of the line segment. The weighting coefficient calculation unit 1025 derives the screen occupancy rate of the feature point (silhouette) by calculating the number of pixels of the partial silhouette relative to the number of pixels of the camera (or the area ratio of the partial silhouette to the entire screen).

[0040] The smaller the feature point of the object is displayed on the screen, the smaller the screen occupancy value becomes, and the larger the feature point of the object is displayed on the screen, the larger the screen occupancy value becomes. The weighting coefficient calculation unit 1025 uses, for example, the value of the screen occupancy rate as the weighting coefficient. For example, when the screen occupancy rate v i,j can be expressed as the following equation (2).

[0041]

number

[0042] Here, s i,j represents the number of pixels (or area) of the partial silhouette, and S represents the number of pixels of the camera (or the area of ​​the screen). The weighting coefficient w i,j can be expressed as a function of the following equation (3).

[0043]

number

[0044] The function w(v) can be expressed as the following equation (4).

[0045]

number

[0046] If the screen occupancy rate is greater than a predetermined value, the weighting coefficient calculation unit 1025 may limit the weighting coefficient. In this case, the function w(v) can be expressed, for example, as in the following equation (5).

[0047]

number

[0048] The cost function of the bundle adjustment in the first embodiment incorporating the above weighting coefficients is expressed as the following equation (6).

[0049]

number

[0050] Here, W i,j is a weighting coefficient. As in equation (1), in equation (2), the function F(p i |f j ,r j ,d j ,R j ,t j ) is the 3D coordinate of the feature point p i is a function that projects the image onto the screen of camera j. Here, f j ,r j ,d j ,R j ,t j are the camera parameters of camera j. Specifically, f j is the camera focal length, r j is the camera resolution, d j is the camera distortion, R j is the camera rotation angle, t j is the camera position.

[0051] [Optimization section operation] 6 is a flowchart showing the operation of the optimization unit 102 of the object 3D reconstruction device 100 in the first embodiment of the present invention. The operation shown in this flowchart is started when the inter-camera associated object posture group data output from the inter-camera association unit 101 is input to the optimization unit 102.

[0052] The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates" based on the acquired camera-correlated subject posture group data (step S201). The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023.

[0053] The weighting coefficient initialization unit 1022 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The weighting coefficient initialization unit 1022 initializes the "number of cameras x number of feature points weighting coefficient" by setting it to a value of, for example, 1.0 based on the acquired camera-correlated subject posture group data (step S202). Note that this initialization is intended to normalize the weighting coefficient group, and the initialized value is not limited to 1.0. The weighting coefficient initialization unit 1022 outputs the initialized weighting coefficient group to the parameter optimization unit 1023.

[0054] The parameter optimization unit 1023 acquires the inter-camera associated subject posture group data output from the inter-camera association unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025. The parameter optimization unit 1023 updates the parameter group based on the acquired inter-camera associated subject posture group data and weighting coefficient group.

[0055] Here, the parameter optimization unit 1023 calculates the cost using a predetermined cost function (for example, the cost function expressed by the above-mentioned formula (6)) and updates the parameter group so as to minimize the cost. At this time, for example, the parameter optimization unit 1023 counts the number of times the parameter group is updated. Alternatively, for example, the parameter optimization unit 1023 calculates the maximum value of the absolute error before and after updating the parameter group (step S203).

[0056] If the number of times the parameter group is updated is less than a predetermined number, or if the maximum value of the absolute error before and after the parameter group is updated is greater than a predetermined value (step S204: NO), the parameter optimization unit 1023 outputs the updated parameter group to the subject pose reprojection unit 1024. The subject pose reprojection unit 1024 acquires the optimized parameter group output from the parameter optimization unit 1023. The subject pose reprojection unit 1024 projects the three-dimensional coordinates of each feature point on the (virtual) screen of each camera. The subject pose reprojection unit 1024 connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates a silhouette for each camera and each subject (step S205). The subject pose reprojection unit 1024 outputs the generated camera and subject silhouette group to the weighting coefficient calculation unit 1025.

[0057] The weighting coefficient calculation unit 1025 acquires the group of camera-specific and subject-specific silhouettes output from the subject orientation reprojection unit 1024. The weighting coefficient calculation unit 1025 calculates the screen occupancy rate of each feature point from a silhouette based on the acquired group of camera-specific and subject-specific silhouettes, and calculates a weighting coefficient for each camera and feature point based on the screen occupancy rate (step S206). The weighting coefficient calculation unit 1025 outputs the calculated weighting coefficient group consisting of the number of cameras x the number of feature points to the parameter optimization unit 1023. Then, the process returns to step S203, and the update of the parameter group is repeated.

[0058] On the other hand, if the number of updates of the parameter group reaches a predetermined number, or if the maximum value of the absolute error before and after the update of the parameter group is equal to or less than a predetermined value (step S204: YES), the parameter optimization unit 1023 outputs the optimized parameter group to an external device (not shown). This ends the operation of the optimization unit 102 shown in the flowchart of FIG.

[0059] When calculating the weighting coefficient group in the process of step S206, the weighting coefficient calculation unit 1025 may refer to the absolute error before and after the update of the parameter group described above and adjust the weighting coefficient according to the value of the absolute error. For example, if the absolute error before and after the update of the parameter group is relatively large, it is considered that the optimization of each parameter is not sufficient. If the weighting coefficient is considered at such a stage where the uncertainty factor is high, there is a possibility that the coordinates will converge to an unintended coordinate. Therefore, it is desirable for the weighting coefficient calculation unit 1025 to calculate the weighting coefficient group after the absolute error before and after the update of the parameter group becomes small to a certain extent (for example, after it becomes equal to or larger than a predetermined value).

[0060] Although the bundle adjustment can simultaneously optimize both the camera parameters and the three-dimensional coordinates of the feature points, it may be configured to optimize only the three-dimensional coordinates of the feature points. That is, the values ​​of the camera parameters of each camera may be known, and only the three-dimensional coordinates of the feature points may be optimized. In this case, the object 3D reconstruction device 100 acquires a parameter group for each camera from an external device (not shown). Then, the parameter initialization unit 1021 and the parameter optimization unit 1023 perform processing using the acquired known camera parameters.

[0061] Non-Patent Document 4 proposes a method of determining the maximum value of Part Confidence MAP (PCM) superimposed from a plurality of viewpoint directions as the coordinates of a person's joints. This method can also be configured to perform optimization taking into account the screen occupancy rate by changing the following formulas (7) and (8) described in Non-Patent Document 4 into the following formulas (9) and (10), respectively. Unlike the above-mentioned formulas, the following formula (9) uses the letter x as a variable representing a weighting coefficient.

[0062]

number

[0063]

number

[0064]

number

[0065]

number

[0066] As described above, the object 3D reconstruction device 100 in the first embodiment of the present invention performs reprojection onto the screen of each camera using each camera parameter and the 3D coordinates of each feature point in each step of the iterative calculation during the iterative calculation of the optimization process in the bundle adjustment. When performing reprojection onto the screen of each camera, the object 3D reconstruction device 100 not only projects the feature points as points for each object, but also draws a silhouette that fits the shape of the object using a line segment or a circle with width, taking into account the connection relationship of the feature points, as shown in FIG. 4, for example. The object 3D reconstruction device 100 generates a partial silhouette that targets the center of a line segment from a target feature point to another feature point connected to the target feature point, as shown in FIG. 5, for example. The object 3D reconstruction device 100 calculates the screen occupancy rate by the ratio of the number of pixels of the partial silhouette to the number of pixels of the camera (or the area ratio of the partial silhouette to the entire screen). The object 3D reconstruction device 100 derives a weighting coefficient based on the calculated screen occupancy rate, incorporates it into a predetermined cost function, and performs iterative calculation.

[0067] With such a configuration, object 3D reconstruction device 100 in the first embodiment of the present invention can suppress random noise that tends to overlap with an object that appears relatively small on the screen. Therefore, object 3D reconstruction device 100 can further improve estimation accuracy in a technique for estimating a group of parameters based on two-dimensional coordinates of feature points detected from images captured by multiple cameras.

[0068] <Second embodiment> The object 3D reconstruction device 100a according to the second embodiment of the present invention will be described below. Note that the object 3D reconstruction device 100a described below is one example of an embodiment of a parameter estimation device of the present invention, and the present invention is not necessarily limited to the configuration of the object 3D reconstruction device 100a described below.

[0069] [Configuration of 3D object reconstruction device] The overall configuration of the object 3D reconstruction device 100a will be described below. FIG. 7 is a block diagram showing the overall configuration of the object 3D reconstruction device 100a in the second embodiment of the present invention. As shown in FIG. 7, the object 3D reconstruction device 100a includes an inter-camera correspondence unit 101 and an optimization unit 102a. In the object 3D reconstruction device 100a shown in FIG. 7, the optimization unit 102a has a different configuration from that of a conventional general object 3D reconstruction device, and the other configurations are the same as those of the conventional object 3D reconstruction device.

[0070] The camera correspondence unit 101 accepts input of subject posture group data output from an external device (not shown). The subject posture group data here refers to data generated from images of a person as a subject captured simultaneously from different positions by multiple cameras. The subject posture group data includes data indicating the positions of the joints, eyes, nose, ears, etc. of the person as a subject in each image, detected by a two-dimensional posture estimation technique represented by OpenPose, PersonLab, etc.

[0071] The camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the subject in each image based on the input subject posture group data as feature points. The camera correspondence unit 101 associates feature points of the same part of the same person in images captured simultaneously by different cameras (for example, feature points indicating the positions of right elbows or feature points indicating the positions of left knees, etc.). The camera correspondence unit 101 generates camera-correlated subject posture group data in which these feature points are associated, and outputs the data to the optimization unit 102a.

[0072] The optimization unit 102a receives input of the camera-correlated subject posture group data output from the camera-correlated unit 101. The optimization unit 102a executes a bundle adjustment using feature points based on the camera-correlated subject posture group data to optimize a parameter group. The parameter group here refers to the camera parameter values ​​of each camera that captured the original image and the three-dimensional coordinates of each feature point. The camera parameters are, for example, the camera position, image capture direction, rotation angle, viewing angle, focal length, resolution, and distortion. The optimization unit 102a outputs the optimized parameter group to an external device (not shown). The optimized parameter group here refers to a parameter group optimized for the purpose of minimizing the mean square error of the difference (reprojection error) between the actual three-dimensional coordinates of the feature point and the estimated three-dimensional coordinates of the feature point.

[0073] Note that the inter-camera correspondence unit 101 may be provided in an external device. That is, the inter-camera corresponded subject posture group data may be generated in an external device (not shown), and the subject 3D reconstruction device 100a may acquire the inter-camera corresponded subject posture group data from the external device.

[0074] [Operation of the 3D object reconstruction device] An example of the operation of object 3D reconstruction device 100a will be described below. Fig. 8 is a flowchart showing the operation of object 3D reconstruction device 100a in the second embodiment of the present invention. This flowchart starts when object posture group data output from an external device is input to object 3D reconstruction device 100a.

[0075] The camera correspondence unit 101 of the object 3D reconstruction device 100a receives input of object posture group data output from an external device. The camera correspondence unit 101 recognizes the positions of the joints, eyes, nose, ears, etc. of the person who is the object in each image based on the input object posture group data as feature points. The camera correspondence unit 101 associates feature points of the same part of the same person in images captured simultaneously by different cameras (step S301). The camera correspondence unit 101 generates camera-correlated object posture group data and outputs it to the optimization unit 102a.

[0076] The optimization unit 102a receives input of the inter-camera associated subject posture group data output from the inter-camera association unit 101. The optimization unit 102a executes a bundle adjustment using feature points based on the inter-camera associated subject posture group data, and optimizes the parameter group (step S302). The optimization unit 102a outputs the optimized parameter group to an external device. This completes the operation of the object 3D reconstruction device 100a shown in the flowchart of FIG. 8.

[0077] [Configuration of the optimization section] The configuration of the optimization unit 102a will be described in more detail below. Fig. 9 is a block diagram showing the functional configuration of the optimization unit 102a of the object 3D reconstruction device 100a in the second embodiment of the present invention. As shown in Fig. 9, the optimization unit 102a includes a parameter initialization unit 1021, a weighting coefficient initialization unit 1022, a parameter optimization unit 1023a, a weighting coefficient calculation unit 1025a, an invisible area including partial silhouette derivation unit 1026, and a visible area partial silhouette derivation unit 1027.

[0078] In the optimization unit 102a shown in FIG. 9, the configuration differs from the conventional technology in that the weight coefficient calculation unit 1025a calculates weight coefficients based on the visibility rates of feature points derived for each camera and each feature point, as described below, and the parameter optimization unit 1023a applies weighting calculations to the cost function of the Bundle Adjustment using the calculated weight coefficients. The other configurations are the same as those of the optimization unit of the conventional object 3D reconstruction device.

[0079] The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes the camera parameters for each number of cameras and the three-dimensional coordinates for each feature point based on the acquired camera-correlated subject posture group data. That is, the parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates". The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023a.

[0080] The weighting coefficient initialization unit 1022 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The weighting coefficient initialization unit 1022 initializes weighting coefficients for each camera and each feature point based on the acquired camera-correlated subject posture group data. That is, the weighting coefficient initialization unit 1022 initializes a weighting coefficient of "number of cameras x number of feature points". The weighting coefficient initialization unit 1022 outputs the initialized weighting coefficient group to the parameter optimization unit 1023.

[0081] The parameter optimization unit 1023a acquires the camera-correlated subject posture group data output from the camera-corresponding unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient calculation unit 1025a. The parameter optimization unit 1023a updates the parameter group based on the acquired camera-correlated subject posture group data and weighting coefficient group. The parameter optimization unit 1023a outputs the updated parameter group to the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027, or to an external device (not shown).

[0082] The invisible area including part silhouette derivation unit 1026 acquires the updated parameter group output from the parameter optimization unit 1023a. The invisible area including part silhouette derivation unit 1026 projects the three-dimensional coordinates of each feature point on the screen of each camera. The invisible area including part silhouette derivation unit 1026 connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera, each subject, and each feature point.

[0083] Then, the invisible-area-including partial silhouette derivation unit 1026 generates a partial silhouette, which is a figure obtained by cutting a line segment connecting a target feature point with another feature point connected to the target feature point at a certain internal division ratio (for example, at the center of the line segment). The partial silhouette generated by the invisible-area-including partial silhouette derivation unit 1026 also includes a part of the subject that cannot be seen due to, for example, obstruction by a person appearing in the foreground. The invisible-area-including partial silhouette derivation unit 1026 outputs the generated partial silhouette to the weighting coefficient calculation unit 1025a as a "group of invisible-area-including partial silhouettes by camera, by subject, and by feature point."

[0084] The visible area partial silhouette derivation unit 1027 acquires the optimized parameter group output from the parameter optimization unit 1023a. The visible area partial silhouette derivation unit 1027 reprojects the three-dimensional coordinates of each feature point onto the screen of each camera. At this time, the visible area partial silhouette derivation unit 1027 reprojects the feature points onto the screen of each camera in order from the most distant feature point, taking into account the depth. The visible area partial silhouette derivation unit 1027 then connects the feature points that are connected to each other with, for example, a line having a predetermined thickness, and generates silhouette images for each camera, each subject, and each feature point.

[0085] Then, the visible area partial silhouette derivation unit 1027 generates a partial silhouette, which is a figure obtained by truncating a line segment connecting a target feature point with another feature point connected to the target feature point at a certain internal division ratio (for example, at the center of the line segment). The partial silhouette generated by the visible area partial silhouette derivation unit 1027 does not include a part of the subject that cannot be seen due to obstruction by, for example, a person appearing in the foreground. The visible area partial silhouette derivation unit 1027 outputs the partial silhouette of only the visible area to the weighting coefficient calculation unit 1025a as a "group of visible area partial silhouettes by camera, subject, and feature point."

[0086] The weighting coefficient calculation unit 1025a acquires the group of invisible area including partial silhouettes by camera, by subject, and by feature point output from the invisible area including partial silhouette derivation unit 1026. The weighting coefficient calculation unit 1025a also acquires the group of visible area partial silhouettes by camera, by subject, and by feature point output from the visible area partial silhouette derivation unit 1027. The weighting coefficient calculation unit 1025a calculates the visibility of each feature point from the partial silhouette based on the acquired group of invisible area including partial silhouettes by camera, by subject, and by feature point, and the partial silhouette based on the group of visible area partial silhouettes by camera, by subject, and by feature point, and calculates a weighting coefficient. The weighting coefficient calculation unit 1025a outputs the calculated weighting coefficient group for each camera and each feature point (i.e., consisting of a weighting coefficient of the number of cameras x the number of feature points) to the parameter optimization unit 1023a.

[0087] When the parameter optimization unit 1023a acquires the weighting coefficient group output from the weighting coefficient calculation unit 1025a, the parameter optimization unit 1023a updates the parameter group based on the weighting coefficient group and the inter-camera associated subject posture group data output from the inter-camera association unit 101. The optimization unit 102a performs an iterative calculation to update the above-mentioned parameter group a predetermined number of times. Alternatively, the optimization unit 102a performs an iterative calculation to update the above-mentioned parameter group until the value of the parameter group converges. When the iterative calculation is completed, the parameter optimization unit 1023a sets the updated parameter group as an optimized parameter group and outputs the parameter group to an external device (not shown).

[0088] More specifically, for example, during iterative calculation in the bundle adjustment, the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027 reproject onto the screen of each camera using each camera parameter in the optimization process in each step of the iterative calculation and the three-dimensional coordinates of each feature point. The weighting coefficient calculation unit 1025a calculates the visibility rate of the feature point (i.e., the ratio of the visible area partial silhouette to the invisible area including partial silhouette). The weighting coefficient calculation unit 1025a derives a weighting coefficient based on the visibility rate. The parameter optimization unit 1023a incorporates the weighting coefficient into a cost function to perform iterative calculation. As a result, the object 3D reconstruction device 100a in the second embodiment can suppress noise due to self-occlusion and occlusion by others.

[0089] When reprojecting onto the screen of each camera, the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027 not only project feature points as points, but also take into consideration the connection relationship between feature points (such as a person's skeleton) as shown in Fig. 4, and draw line segments between connected feature points as line segments with width, circles, etc. The invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027 draw a silhouette that fits the shape of the subject using line segments with width, circles, etc.

[0090] The weighting coefficient calculation unit 1025a generates, as a partial silhouette, a figure in which a line segment connecting a target feature point and another feature point connected to the target feature point is truncated at a certain internal division ratio, as shown in Figures 10 and 11. For example, Figures 10 and 11 show a partial silhouette in which a line segment connecting a target feature point and another feature point connected to the target feature point is truncated at the center of the line segment.

[0091] Fig. 10 is a diagram showing a case where self-occlusion occurs. Fig. 10 shows an example of an image showing two people. In Fig. 10, a part of the partial silhouette of feature point j, which corresponds to the position of the right hip joint of the person appearing on the left side of the image, is occluded by the partial silhouette of feature point j, which corresponds to the position of the left thigh of the person.

[0092] Fig. 11 is a diagram showing a case where occlusion by another person occurs. Fig. 10 shows an example of an image showing two people. The person on the left side of the image is located closer to the viewer (as viewed from the position of the camera that captured the image) than the person on the right side of the image. Therefore, as shown in Fig. 10, part of the partial silhouette of feature point j, which corresponds to the position of the right elbow of the person on the right, is occluded by the partial silhouette of feature point j, which corresponds to the position of the fist of the person on the left.

[0093] The weighting coefficient calculation unit 1025a derives the visibility of each feature point by calculating the ratio of the number of pixels in the partial silhouette of the visible area to the number of pixels in the partial silhouette including the invisible area (or the area ratio of the partial silhouette of the visible area to the partial silhouette including the invisible area).

[0094] The visibility rate becomes smaller as the ratio of the self-occluded area or the area occluded by another person becomes larger, and becomes larger as the ratio of the self-occluded area or the area occluded by another person becomes smaller. The weighting coefficient calculation unit 1025a uses, for example, the value of the visibility rate as the weighting coefficient. For example, when the visibility rate v i,j can be expressed as the following equation (11).

[0095]

number

[0096] Here, s i,j represents the number of pixels (or area) of the visible partial silhouette, and S represents the number of pixels (or area) of the entire partial silhouette including the invisible parts. i,j can be expressed as a function of the following equation (12).

[0097]

number

[0098] The function w(v) can be expressed as the following equation (13).

[0099]

number

[0100] If the visibility rate is greater than a predetermined value, the weighting coefficient calculation unit 1025 may limit the weighting coefficient. In this case, the function w(v) can be expressed, for example, as in the following equation (14).

[0101]

number

[0102] The cost function of the bundle adjustment in the second embodiment incorporating the above weighting coefficients is expressed, for example, as in the following equation (15).

[0103]

number

[0104] Here, W i,jis a weighting coefficient. As in (1), in (15), the function F(p i |f j ,r j ,d j ,R j ,t j ) is the 3D coordinate of the feature point p i is a function that projects the image onto the screen of camera j. Here, f j ,r j ,d j ,R j ,t j are the camera parameters of camera j. Specifically, f j is the camera focal length, r j is the camera resolution, d j is the camera distortion, R j is the camera rotation angle, t j is the camera position.

[0105] [Optimization section operation] An example of the operation of the optimization unit 102 will be described in more detail below. Fig. 12 is a flowchart showing the operation of the optimization unit 102a of the object 3D reconstruction device 100a in the second embodiment of the present invention. The operation shown in this flowchart is started when the inter-camera associated object posture group data output from the inter-camera association unit 101 is input to the optimization unit 102a.

[0106] The parameter initialization unit 1021 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The parameter initialization unit 1021 initializes the parameters of "number of cameras x number of camera parameters + number of feature points x three-dimensional coordinates" based on the acquired camera-correlated subject posture group data (step S401). The parameter initialization unit 1021 outputs the initialized parameter group to the parameter optimization unit 1023a.

[0107] The weighting coefficient initialization unit 1022 acquires the camera-correlated subject posture group data output from the camera-correlated unit 101. The weighting coefficient initialization unit 1022 initializes the "number of cameras x number of feature points weighting coefficient" by setting it to a value of, for example, 1.0 based on the acquired camera-correlated subject posture group data (step S402). Note that this initialization is intended to normalize the weighting coefficient group, and the initialized value is not limited to 1.0. The weighting coefficient initialization unit 1022 outputs the initialized weighting coefficient group to the parameter optimization unit 1023a.

[0108] The parameter optimization unit 1023a acquires the inter-camera associated subject posture group data output from the inter-camera association unit 101, the parameter group output from the parameter initialization unit 1021, and the weighting coefficient group output from the weighting coefficient initialization unit 1022 or the weighting coefficient calculation unit 1025a. The parameter optimization unit 1023a updates the parameter group based on the acquired inter-camera associated subject posture group data and weighting coefficient group.

[0109] Here, the parameter optimization unit 1023a calculates the cost using a predetermined cost function (for example, the cost function expressed by the above-mentioned formula (15)) and updates the parameter group so as to minimize the cost. At this time, for example, the parameter optimization unit 1023a counts the number of times the parameter group is updated. Alternatively, for example, the parameter optimization unit 1023a calculates the maximum value of the absolute error before and after the parameter group is updated (step S403).

[0110] If the number of times the parameter group has been updated is less than a predetermined number, or if the maximum value of the absolute error before and after the parameter group update is greater than a predetermined value (step S404: NO), the parameter optimization unit 1023a outputs the updated parameter group to the invisible area including partial silhouette derivation unit 1026 and the visible area partial silhouette derivation unit 1027.

[0111] The invisible area including part silhouette derivation unit 1026 acquires the updated parameter group output from the parameter optimization unit 1023a. The invisible area including part silhouette derivation unit 1026 reprojects each feature point for each camera. The invisible area including part silhouette derivation unit 1026 projects the three-dimensional coordinates of each feature point on the screen of each camera based on the updated parameter group. The invisible area including part silhouette derivation unit 1026 connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera, each subject, and each feature point. Then, the invisible area including part silhouette derivation unit 1026 generates a figure in which a line segment connecting a target feature point and another feature point connected to the target feature point is cut at a certain internal division ratio (for example, at the center position of the line segment) as a partial silhouette.

[0112] The visible area partial silhouette derivation unit 1027 acquires the updated parameter set output from the parameter optimization unit 1023a. The visible area partial silhouette derivation unit 1027 reprojects each feature point for each camera in order from the farthest feature point, taking into account the depth. The visible area partial silhouette derivation unit 1027 projects the three-dimensional coordinates of each feature point on the screen of each camera based on the updated parameter set. The visible area partial silhouette derivation unit 1027 then connects the mutually connected feature points with, for example, a line having a predetermined thickness, and generates an image of a silhouette for each camera, each subject, and each feature point. The visible area partial silhouette derivation unit 1027 then generates a figure as a partial silhouette by cutting off a line segment connecting a target feature point and another feature point connected to the target feature point at a certain internal division ratio (for example, at the center position of the line segment) (step S405).

[0113] The invisible-area-including partial silhouette derivation unit 1026 outputs the generated partial silhouette including the invisible area to the weighting coefficient calculation unit 1025a as a "group of invisible-area-including partial silhouettes by camera, by subject, and by feature point". Also, the visible-area partial silhouette derivation unit 1027 outputs the generated partial silhouette including only the visible area to the weighting coefficient calculation unit 1025a as a "group of visible-area partial silhouettes by camera, by subject, and by feature point".

[0114] The weighting coefficient calculation unit 1025a acquires the group of invisible area inclusive partial silhouettes by camera, by subject, and by feature point output from the invisible area inclusive partial silhouette derivation unit 1026. The weighting coefficient calculation unit 1025a also acquires the group of visible area partial silhouettes by camera, by subject, and by feature point output from the visible area partial silhouette derivation unit 1027. The weighting coefficient calculation unit 1025a calculates the visibility of each feature point from the partial silhouette based on the group of invisible area inclusive partial silhouettes by camera, by subject, and by feature point acquired and the partial silhouette based on the group of visible area partial silhouettes by camera, by subject, and by feature point, and calculates the weighting coefficient by camera and by feature point based on the visibility (step S406). The weighting coefficient calculation unit 1025a outputs the calculated weighting coefficient group consisting of the number of cameras x the number of feature points to the parameter optimization unit 1023a. Then, the process returns to step S403, and the update of the parameter group is repeated.

[0115] On the other hand, if the number of updates of the parameter group reaches a predetermined number, or if the maximum value of the absolute error before and after the update of the parameter group is equal to or less than a predetermined value (step S404: YES), the parameter optimization unit 1023a outputs the optimized parameter group to an external device (not shown). This ends the operation of the optimization unit 102a shown in the flowchart of FIG.

[0116] When calculating the weighting coefficient group in the process of step S406, the weighting coefficient calculation unit 1025a may refer to the absolute error before and after the update of the parameter group described above and adjust the weighting coefficient according to the value of the absolute error. For example, if the absolute error before and after the update of the parameter group is relatively large, it is considered that the optimization of each parameter is not sufficient. If the weighting coefficient is considered at such a stage where the uncertainty factor is high, there is a possibility that the coordinates will converge to an unintended coordinate. Therefore, it is desirable for the weighting coefficient calculation unit 1025a to calculate the weighting coefficient group after the absolute error before and after the update of the parameter group becomes small to a certain extent (for example, after it becomes equal to or larger than a predetermined value).

[0117] Although the bundle adjustment can simultaneously optimize both the camera parameters and the three-dimensional coordinates of the feature points, it may be configured to optimize only the three-dimensional coordinates of the feature points. That is, the values ​​of the camera parameters of each camera may be known, and only the three-dimensional coordinates of the feature points may be optimized. In this case, the object 3D reconstruction device 100a acquires a parameter group for each camera from an external device (not shown). Then, the parameter initialization unit 1021 and the parameter optimization unit 1023a perform processing using the acquired known camera parameters.

[0118] As described above, the object 3D reconstruction device 100a in the second embodiment of the present invention performs reprojection on the screen of each camera, sorted in consideration of depth, using each camera parameter and the 3D coordinates of each feature point in each step of the iterative calculation during the iterative calculation of the optimization process in the bundle adjustment. When performing reprojection on the screen of each camera, the object 3D reconstruction device 100a not only projects the feature points as points for each object, but also draws a silhouette that fits the shape of the object using line segments or circles with width, taking into consideration the connection relationship of the feature points, as shown in FIG. 4, for example. The object 3D reconstruction device 100a generates a partial silhouette that targets the center of a line segment from a target feature point to another feature point connected to the target feature point, as shown in FIG. 10 and FIG. 11, for example. The object 3D reconstruction device 100a calculates the visibility rate by the ratio of the number of pixels of the partial silhouette in the visible area to the number of pixels of the entire partial silhouette including the invisible area (or the area ratio of the partial silhouette in the visible area to the entire partial silhouette including the invisible area). The object 3D reconstruction device 100a derives weighting coefficients based on the calculated visibility rates, and incorporates the weighting coefficients into a predetermined cost function to perform iterative calculations.

[0119] With such a configuration, object 3D reconstruction device 100a in the second embodiment of the present invention can suppress noise due to self-occlusion and occlusion by others. Therefore, object 3D reconstruction device 100a can further improve estimation accuracy in a technique for estimating a group of parameters based on two-dimensional coordinates of feature points detected from images captured by multiple cameras.

[0120] <Third embodiment> Hereinafter, an object 3D reconstruction device according to the third embodiment of the present invention will be described.

[0121] The object 3D reconstruction device 100 in the first embodiment described above is configured to apply weighting calculation to the cost function of the bundle adjustment using a weighting factor calculated based on the in-screen occupancy rate of the feature points derived for each camera and each feature point. Also, the object 3D reconstruction device 100a in the second embodiment described above is configured to apply weighting calculation to the cost function of the bundle adjustment using a weighting factor calculated based on the visibility rate of the feature points derived for each camera and each feature point. The object 3D reconstruction device may be configured to perform both the weighting calculation based on the in-screen occupancy rate and the weighting calculation based on the visibility rate.

[0122] For example, the object 3D reconstruction device may be configured to perform a weighting calculation based on the screen occupancy rate and then further perform a weighting calculation based on the visibility rate, or conversely, the object 3D reconstruction device may be configured to perform a weighting calculation based on the screen occupancy rate and then further perform a weighting calculation based on the visibility rate.

[0123] Alternatively, for example, the object 3D reconstruction device may be configured to perform a weighting calculation based on the screen occupancy rate and a weighting calculation based on the visibility rate in parallel, and perform an iterative calculation based on the results of both weighting calculations. For example, the object 3D reconstruction device may be configured to use an average value of the parameter values ​​obtained by both weighting calculations in the next iterative calculation.

[0124] As described above, the object 3D reconstruction device in the third embodiment of the present invention performs reprojection on the screen of each camera, sorted in consideration of depth, using each camera parameter and the 3D coordinates of each feature point in each step of the iterative calculation during the iterative calculation of the optimization process in the bundle adjustment. When performing reprojection on the screen of each camera, the object 3D reconstruction device not only projects the feature points as points for each object, but also draws a silhouette that fits the shape of the object using a line segment or a circle with width, taking into consideration the connection relationship of the feature points, as shown in FIG. 4, for example. The object 3D reconstruction device generates a partial silhouette that targets the center of a line segment from a target feature point to another feature point connected to the target feature point, as shown in FIG. 5, FIG. 10, and FIG. 11, for example. The object 3D reconstruction device calculates the screen occupancy rate by the ratio of the number of pixels of the partial silhouette to the number of pixels of the camera (or the area ratio of the partial silhouette to the entire screen). The object 3D reconstruction device also calculates the visibility rate based on the ratio of the number of pixels in the partial silhouette of the visible region to the number of pixels in the entire partial silhouette including the invisible region (or the area ratio of the partial silhouette of the visible region to the entire partial silhouette including the invisible region). The object 3D reconstruction device derives a weighting factor based on the calculated screen occupancy rate and visibility rate, and incorporates it into a predetermined cost function to perform an iterative calculation.

[0125] With such a configuration, the object 3D reconstruction device in the third embodiment of the present invention can suppress random noise that is likely to be superimposed on an object that appears relatively small on the screen. Also, with such a configuration, the object 3D reconstruction device in the third embodiment of the present invention can suppress noise due to self-occlusion and occlusion by others. Therefore, the object 3D reconstruction device can further improve the estimation accuracy in the technology for estimating a group of parameters based on the two-dimensional coordinates of feature points detected from images captured by multiple cameras.

[0126] An example of a hardware configuration of the object 3D reconstruction device in the first to third embodiments configured by a computer and a software program is shown below. Fig. 13 is a block diagram showing an example of a hardware configuration of the object 3D reconstruction device in the first to third embodiments of the present invention. As shown in Fig. 13, the object 3D reconstruction device is configured by connecting a CPU (Central Processing Unit) 10, a memory 11, a data storage unit 12, a program storage device 13, and an optimal parameter group output unit 15 via a bus.

[0127] The CPU 10 executes a program. The memory 11 is a storage medium such as a RAM (Random Access Memory) in which programs and data accessed by the CPU 10 are stored. The data storage unit 12 stores various input data. The data storage unit 12 may be a storage unit such as a disk device. The program storage unit 13 stores a three-dimensional reconstruction program 14 of an object, which is a software program that causes the CPU 10 to execute the three-dimensional reconstruction process of the object in each of the above-mentioned embodiments. The optimal parameter group output unit 15 outputs an optimized parameter group generated by the CPU 10 executing the three-dimensional reconstruction program 14 of the object loaded in the memory 11, for example, via a network. The optimal parameter group output unit 15 may be a storage unit that stores an optimized parameter group, such as a disk device.

[0128] According to the above-described embodiment, the parameter estimation device includes an acquisition unit, a weighting factor calculation unit, and a parameter estimation unit. For example, the parameter estimation device is object 3D reconstruction device 100 in the embodiment, the acquisition unit is parameter initialization unit 1021, weighting factor initialization unit 1022, and parameter optimization unit 1023 in the embodiment, the weighting factor calculation unit is weighting factor calculation unit 1025 in the embodiment, and the parameter estimation unit is parameter optimization unit 1023 in the embodiment.

[0129] The acquisition unit acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature points of the subject are associated with each other in each image captured by the multiple cameras from different positions of the same subject. For example, the multiple cameras are camera j and camera j+1 in the embodiment, and the subject posture group data is subject posture group data with inter-camera association in the embodiment.

[0130] The weighting coefficient calculation unit calculates an in-screen occupancy rate indicating the ratio of the area of ​​the feature point to the entire area of ​​the image based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated in-screen occupancy rate. For example, the ratio of the area of ​​the feature point to the entire area of ​​the image is the number of pixels of the partial silhouette relative to the number of pixels of the camera in the embodiment (or the area ratio of the partial silhouette to the entire screen).

[0131] The parameter estimation unit estimates the values ​​of parameters including the coordinate values ​​of the three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit. For example, the calculation using the weighting coefficients is an iterative calculation using a cost function expressed by equation (6) in the embodiment, and the values ​​of the parameters including the coordinate values ​​of the three-dimensional coordinates of the feature points are a parameter group consisting of the camera parameter values ​​of each camera and the three-dimensional coordinates of each feature point in the embodiment.

[0132] In the parameter estimation device, the screen occupancy rate may be calculated from the ratio of the number of pixels of a silhouette figure drawn based on the positional relationship between a first feature point of a subject and a second feature point of the subject to the number of pixels of the image. For example, the first feature point is a target feature point in the embodiment, and the second feature point is another feature point connected to the target feature point in the embodiment.

[0133] In the above parameter estimation device, the silhouette figure may be a figure generated by adding width to a line segment in a range from the position of the first feature point to a position at a predetermined internal division ratio of the line segment connecting the first feature point and the second feature point.

[0134] In the parameter estimation device, the weighting coefficient calculation unit may calculate the weighting coefficient by further using a visibility rate indicating the proportion of the area of ​​the feature point that is not occluded by areas other than the feature point. For example, the feature point area is the area of ​​the entire partial silhouette in the embodiment, and the area that is not occluded by areas other than the feature point is the area of ​​the partial silhouette in the embodiment excluding the invisible area (i.e., the visible area).

[0135] In the parameter estimation device, the parameters may further include at least one of an external parameter of the camera and an internal parameter of the camera.

[0136] According to the above-described embodiment, the parameter estimation device includes an acquisition unit, a weighting factor calculation unit, and a parameter estimation unit. For example, the parameter estimation device is the object 3D reconstruction device 100a in the embodiment, the acquisition unit is the parameter initialization unit 1021, the weighting factor initialization unit 1022, and the parameter optimization unit 1023a in the embodiment, the weighting factor calculation unit is the weighting factor calculation unit 1025a in the embodiment, and the parameter estimation unit is the parameter optimization unit 1023a in the embodiment.

[0137] The acquisition unit acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature points of the subject are associated with each other in each image captured by the multiple cameras from different positions of the same subject. For example, the multiple cameras are camera j and camera j+1 in the embodiment, and the subject posture group data is subject posture group data with inter-camera association in the embodiment.

[0138] The weighting coefficient calculation unit calculates a visibility ratio indicating the proportion of the area of ​​the feature point that is not occluded by areas other than the feature point based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated visibility ratio. For example, the area of ​​the feature point is the area of ​​the entire partial silhouette in the embodiment, and the area that is not occluded by areas other than the feature point is the area of ​​the partial silhouette in the embodiment excluding the invisible area (i.e., the visible area).

[0139] The parameter estimation unit estimates the values ​​of parameters including the coordinate values ​​of the three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit. For example, the calculation using the weighting coefficients is an iterative calculation using a cost function expressed by equation (15) in the embodiment, and the values ​​of the parameters including the coordinate values ​​of the three-dimensional coordinates of the feature points are a parameter group consisting of the camera parameter values ​​of each camera and the three-dimensional coordinates of each feature point in the embodiment.

[0140] In the parameter estimation device, the visibility may be calculated from the ratio of the number of pixels of a region in the silhouette figure that is not occluded by a region other than the feature points to the number of pixels of the silhouette figure that is drawn based on the positional relationship between a first feature point of the subject and a second feature point of the subject. For example, the first feature point is a target feature point in the embodiment, and the second feature point is another feature point connected to the target feature point in the embodiment.

[0141] In the above parameter estimation device, the silhouette figure may be a figure generated by adding width to a line segment in a range from the position of the first feature point to a position at a predetermined internal division ratio of the line segment connecting the first feature point and the second feature point.

[0142] In addition, in the above parameter estimation device, the unoccluded areas may be identified by detecting overlapping areas by reprojecting feature points on a camera screen in order from the most distant feature point, taking into account depth.

[0143] In the parameter estimation device, the weighting coefficient calculation unit may calculate the weighting coefficient by further using a screen occupancy rate indicating the ratio of the area of ​​the feature point to the entire area of ​​the image. For example, the ratio of the area of ​​the feature point to the entire area of ​​the image is the number of pixels of the partial silhouette relative to the number of pixels of the camera in the embodiment (or the area ratio of the partial silhouette to the entire screen).

[0144] In the parameter estimation device, the parameters may further include at least one of an external parameter of the camera and an internal parameter of the camera.

[0145] The object 3D reconstruction device in each of the above-mentioned embodiments may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM (Read Only Memory), a CD-ROM, and a storage device such as a hard disk built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0146] Although the embodiments of the present invention have been described above with reference to the drawings, it is clear that the above-mentioned embodiments are merely examples of the present invention, and the specific configuration of the present invention is not limited to the above-mentioned embodiments. Therefore, addition, omission, substitution, and other design changes of components may be made within the technical scope that does not deviate from the spirit and gist of the present invention. [Explanation of symbols]

[0147] 10...CPU, 11...memory, 12...data storage unit, 13...program storage device, 14...3D reconstruction program, 15...optimum parameter group output unit, 100 (100a)...object 3D reconstruction device, 101...camera correspondence unit, 102 (102a)...optimization unit, 1021...parameter initialization unit, 1022...weighting coefficient initialization unit, 1023 (1023a)...parameter optimization unit, 1024...object posture reprojection unit, 1025 (1025a)...weighting coefficient calculation unit, 1026...invisible area including partial silhouette derivation unit, 1027...visible area partial silhouette derivation unit

Claims

1. an acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of the same feature point of the same subject are associated with each other in each of images captured by a plurality of cameras from different positions of the same subject; a weighting coefficient calculation unit that calculates an in-screen occupancy rate indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated in-screen occupancy rate; a parameter estimation unit that estimates values ​​of parameters including coordinate values ​​of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit; Equipped with The screen occupancy rate is calculated from a ratio of the number of pixels of a silhouette figure drawn based on a positional relationship between a first feature point of the subject and a second feature point of the subject to the number of pixels of the image. Parameter estimator.

2. The silhouette figure is a figure generated by adding width to a line segment in a range from the position of the first feature point to a position that is a predetermined internal division ratio of the line segment connecting the first feature point and the second feature point. The parameter estimation device according to claim 1 .

3. An acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of identical feature points of an identical subject in each image captured by a plurality of cameras from different positions of the same subject correspond to each other; a weighting coefficient calculation unit that calculates an in-screen occupancy rate indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated in-screen occupancy rate; a parameter estimation unit that estimates values ​​of parameters including coordinate values ​​of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit; Equipped with The weighting coefficient calculation unit calculates the weighting coefficient by further using a visibility rate indicating a ratio of an area of ​​the feature point that is not occluded by an area other than the feature point. Parameter estimator.

4. An acquisition unit that acquires subject posture group data indicating data in which two-dimensional coordinates of identical feature points of an identical subject in each image captured by a plurality of cameras from different positions of the same subject correspond to each other; a weighting coefficient calculation unit that calculates an in-screen occupancy rate indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired by the acquisition unit, and calculates a weighting coefficient based on the calculated in-screen occupancy rate; a parameter estimation unit that estimates values ​​of parameters including coordinate values ​​of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated by the weighting coefficient calculation unit; Equipped with The parameters further include at least one of an external parameter of the camera and an internal parameter of the camera. Parameter estimator.

5. an acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of the same feature points of the same subject are associated with each other in respective images captured by a plurality of cameras from different positions of the same subject; a weighting coefficient calculation step of calculating an in-screen occupancy rate indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired in the acquisition step, and calculating a weighting coefficient based on the calculated in-screen occupancy rate; a parameter estimation step of estimating values ​​of parameters including coordinate values ​​of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated in the weighting coefficient calculation step; having The screen occupancy rate is calculated from a ratio of the number of pixels of a silhouette figure drawn based on a positional relationship between a first feature point of the subject and a second feature point of the subject to the number of pixels of the image. Parameter estimation methods.

6. An acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of identical feature points of an identical subject in each image captured by a plurality of cameras from different positions of the same subject correspond to each other; a weighting coefficient calculation step of calculating an in-screen occupancy rate indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired in the acquisition step, and calculating a weighting coefficient based on the calculated in-screen occupancy rate; a parameter estimation step of estimating values ​​of parameters including coordinate values ​​of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated in the weighting coefficient calculation step; having The weighting coefficient calculation unit calculates the weighting coefficient by further using a visibility rate indicating a ratio of an area of ​​the feature point that is not occluded by an area other than the feature point. Parameter estimation methods.

7. An acquisition step of acquiring subject posture group data indicating data in which two-dimensional coordinates of identical feature points of an identical subject in each image captured by a plurality of cameras from different positions of the same subject correspond to each other; a weighting coefficient calculation step of calculating an in-screen occupancy rate indicating a ratio of an area of ​​the feature point to an entire area of ​​the image based on the subject posture group data acquired in the acquisition step, and calculating a weighting coefficient based on the calculated in-screen occupancy rate; a parameter estimation step of estimating values ​​of parameters including coordinate values ​​of three-dimensional coordinates of the feature points based on a calculation using the weighting coefficients calculated in the weighting coefficient calculation step; having The parameters further include at least one of an external parameter of the camera and an internal parameter of the camera. Parameter estimation methods.

8. A parameter estimation program for causing a computer to function as the parameter estimation device according to any one of claims 1 to 4.

Images