Three-dimensional position calculation method, device, and program

Abstract

This three-dimensional position calculation device: interpolates a two-dimensional position p2d, pred t, cn in an image 40n (t) (n=0, 1) which is included in a multi-viewpoint image and was captured at a time t, such interpolation being on the basis of two-dimensional positions p2d, obs t-1, cn and p2d, obs t-1, cn of a target detected in an image 40n (t-1) captured one unit time earlier and in an image 40n (t+1) captured one unit time later; uses a camera parameter to calculate a three-dimensional position candidate p3d, cand t for each combination of a two-dimensional position p2d, obs t, cn detected from the image 40n(t) and an interpolated two-dimensional position p2d, pred t, cn; and calculates a three-dimensional position p^3d t of the target at the time t, on the basis of the calculated candidate p3d, cand t.

Description

Three-dimensional position calculation method, device, and program

[0001] The disclosed technology relates to a three-dimensional position calculation method, a three-dimensional position calculation device, and a three-dimensional position calculation program.

[0002] Conventionally, triangulation has been used to calculate the three-dimensional position of an object in a world coordinate system from the two-dimensional position of the object in multi-view images of the object captured from multiple different viewpoints. For example, a system has been proposed that uses two or more cameras to track the path and orientation of a portion of portable sports equipment swung by an athlete. In this system, at least two sets of video images of the portable sports equipment being swung are acquired using at least two different cameras with different positions. Then, motion regions are identified within the video images, and candidate positions in two-dimensional space of an identifiable portion of the portable sports equipment (e.g., the head) are identified within the motion regions. Based on this, possible positions in three-dimensional space of the identifiable portion are identified for each of multiple moments when the portable sports equipment is swung.

[0003] U.S. Patent Application Publication No. 2017 / 0270354

[0004] However, if the multi-view image contains an image in which the object is not detected or an image in which the object is falsely detected, there is a problem in that it may not be possible to accurately calculate the three-dimensional position of the object.

[0005] In one aspect, the disclosed technology aims to improve the accuracy of calculating the three-dimensional position of an object using multi-viewpoint images.

[0006] In one aspect, the disclosed technology acquires a first image captured at one or more times before a target time and a second image captured at the target time using each of multiple cameras that capture images of an object from multiple different viewpoints. The disclosed technology also acquires a third image captured at one or more times after the target time. The disclosed technology then interpolates the two-dimensional position of the object in the second image based on the two-dimensional positions of the object detected from each of the first image and the third image. The disclosed technology also calculates the three-dimensional position of the object at the target time based on the two-dimensional position of the object detected from the second image, the interpolated two-dimensional position of the object in the second image, and camera parameters of each of the multiple cameras.

[0007] As one aspect, it has an effect of improving the accuracy of calculating the three-dimensional position of an object using multi-viewpoint images.

[0008] FIG. 1 is a schematic diagram showing a connection between a three-dimensional position calculation device and a camera according to the present embodiment. FIG. 2 is a diagram for explaining a general method for calculating a three-dimensional position from multi-viewpoint images. FIG. 3 is a diagram for explaining a general method for calculating a three-dimensional position from multi-viewpoint images. FIG. 4 is a diagram for explaining removal of outliers. FIG. 5 is a diagram for explaining problems with a general method for calculating a three-dimensional position from multi-viewpoint images. FIG. 6 is a functional block diagram of a three-dimensional position calculation device according to the present embodiment. FIG. 7 is a diagram for explaining an example of a two-dimensional position of an object. FIG. 8 is a diagram for explaining an example of a two-dimensional position of an object. FIG. 9 is a diagram for explaining interpolation of a two-dimensional position using time-space information. FIG. 10 is a diagram for explaining calculation of a three-dimensional position based on clustering of three-dimensional position candidates. FIG. 11 is a block diagram showing a schematic configuration of a computer functioning as a three-dimensional position calculation device. FIG. 12 is a flowchart showing an example of three-dimensional position calculation processing according to the present embodiment. FIG. 13 is an image diagram showing an example of a calculation result of a three-dimensional position according to the present embodiment. FIG. 14 is an image diagram showing an example of a calculation result of a three-dimensional position according to the present embodiment. FIG. 15 is a diagram for explaining application of a three-dimensional position calculation device according to the present embodiment to a scoring system for gymnastics.

[0009] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0010] As shown in Fig. 1, a three-dimensional position calculation device 10 according to this embodiment is connected to a plurality of cameras 30n that capture images of an object 90 (a person in the example of Fig. 1) at viewpoints n from different directions. In the example of Fig. 1, n = 0, 1, 2, and a camera 300 capturing images from viewpoint 0, a camera 301 capturing images from viewpoint 1, and a camera 302 capturing images from viewpoint 2 are connected to the three-dimensional position calculation device 10. Note that the number of cameras 30n connected to the three-dimensional position calculation device 10 is not limited to the example of Fig. 1, and may be two, four, or more.

[0011] The camera 30n is installed at an angle and position such that the target object 90 falls within the shooting range. The images captured by the camera 30n are sequentially input to the three-dimensional position calculation device 10. A synchronization signal is sent to each camera 30n, and the images captured by each camera 30n are synchronized.

[0012] Here, we will explain a general method and problems involved in using triangulation to calculate the three-dimensional position of an object 90 from multiple images taken from multiple different viewpoints (hereinafter referred to as "multi-viewpoint images").

[0013] As shown in FIG. 2, the two-dimensional position of an object detected in an image 40n captured from a viewpoint n is expressed as p ２ｄ，ｏｂｓ ｃｎ (white circle in Figure 2), the true two-dimensional position is p ２ｄ，ｇｔ ｃｎ In the example of FIG. 2, n=0, 1, 2. The calculated three-dimensional position is expressed as p^ ３ｄ (black circle in Figure 2), the true three-dimensional position is p ３ｄ，ｇｔ (Black star in Figure 2). In Figure 2, "p^" is written as "^ (hat)" above "p". This is the same in the following figures. ２ｄ = [x, y]∈ real number R ２ , p ３ｄ = [X, Y, Z]∈ real number R ３ The three-dimensional position p^ of the object ３ｄis the camera parameters (internal parameters and external parameters) of the camera 30n that captured each image 40n, and the two-dimensional position p ２ｄ，ｏｂｓ ｃｎ It is calculated by triangulation using

[0014] Here, for example, the two-dimensional position p of the object 90 detected from the image 400 ２ｄ，ｏｂｓ ｃ０ If the detection error of p is large, that is, ２ｄ，ｏｂｓ ｃ０ and p ２ｄ，ｇｔ ｃ０ If the difference is large, the calculated three-dimensional position p^ ３ｄ The error of p^ also becomes large. ３ｄ ｎｏｉｓｅ However, the detection error is large. ２ｄ，ｏｂｓ ｃ０ In this case, the two-dimensional position p ２ｄ，ｏｂｓ ｃ０ is excluded as an outlier, and p ２ｄ，ｏｂｓ ｃ１ and p ２ｄ，ｏｂｓ ｃ２ Using the three-dimensional position p ３ｄ ｒｅｆｉｎｅ By calculating the true three-dimensional position p ３ｄ，ｇｔ It is desirable to calculate the three-dimensional position so as to approximate

[0015] A more specific explanation will be given using another example shown in Fig. 3. In the example of Fig. 3, n = 0, 1, 2, 3. The superscript t of each symbol indicates the time when the image 40n was captured, that is, the time information associated with the image (frame) 40n. In the example of Fig. 3, the two-dimensional position p of the object can be determined from the image 402. ２ｄ，ｏｂｓ ｔ，ｃ２ has gone undetected.

[0016] The three-dimensional position of the object p^ ３ｄ ｔ is calculated as shown below using the function cv::sfm::triangulatePoints implemented in OpenCV (reference: https: / / docs.opencv.org / 3.4 / d0 / dbd / group__triangulation.html). ｃｎis a perspective projection matrix representing the camera parameters of the camera 30n.

[0017]

[0018] The two-dimensional position p of the detected object ２ｄ，ｏｂｓ ｔ，ｃｎ If there is an outlier in ３ｄ ｔ and the true three-dimensional position p ３ｄ，ｇｔ ｔ The error with p becomes large. ２ｄ，ｏｂｓ ｔ，ｃｎ In order to remove outliers from the 3D position p̂, for example, RANSAC (Random Sample Consensus) is applied. First, as shown in FIG. 4, the 3D position p̂ ３ｄ ｔ Then, the projected two-dimensional position and p ２ｄ，ｏｂｓ ｔ，ｃｎ The error (indicated by the double arrow in FIG. 4) between the ２ｄ，ｏｂｓ ｔ，ｃｎ In the example of FIG. ２ｄ，ｏｂｓ ｔ，ｃ０ is an outlier. This outlier p ２ｄ，ｏｂｓ ｔ，ｃ０ is excluded and the three-dimensional position is calculated again.

[0019] The problem here is that many p ２ｄ，ｏｂｓ ｔ，ｃｎ If p is excluded as an outlier, as in image 402, ２ｄ，ｏｂｓ ｔ，ｃｎ In such a case, as shown in Fig. 5, there may be insufficient information on the two-dimensional position for calculating the three-dimensional position, and the three-dimensional position may not be calculated accurately.

[0020] Therefore, the three-dimensional position calculation device 10 according to this embodiment calculates the three-dimensional position of the object by using spatiotemporal information, specifically, information on images taken at times before and after the image taken at the target time. The three-dimensional position calculation device 10 according to this embodiment will be described in detail below.

[0021] 6, the three-dimensional position calculation device 10 functionally includes an acquisition unit 12, an interpolation unit 14, and a calculation unit 16. A camera parameter DB (Database) 20 is stored in a predetermined storage area of ​​the three-dimensional position calculation device 10. The camera parameter DB 20 stores internal parameters and external parameters of each camera 30n.

[0022] The acquisition unit 12 acquires time-series multi-viewpoint images captured by a plurality of cameras 30n. Here, in the time-series multi-viewpoint images, an image captured at a time t (t = 0, 1, ..., T, where T is time information of the final frame) that is the target of processing for calculating the three-dimensional position of the object 90 is defined as image 40n(t). Also, an image captured at time t-1, one time before time t, is defined as image 40n(t-1), and an image captured at time t+1, one time after time t, is defined as image 40n(t+1). Image 40n(t-1) is an example of a "first image" of the disclosed technology, image 40n(t) is an example of a "second image" of the disclosed technology, and image 40n(t+1) is an example of a "third image" of the disclosed technology.

[0023] Furthermore, information on the two-dimensional position of the object 90 is assigned to each image 40n included in the multi-view image. The information on the two-dimensional position of the object 90 may be the coordinate values ​​of a predetermined point within a region surrounding the object 90 detected from each image 40n included in the multi-view image using a detection model previously generated by machine learning to detect the region of the object 90 from the image 40n. For example, as shown in FIG. 7, if the region of the object 90 is detected by a two-dimensional bounding box (hereinafter referred to as a "2D-BBOX") 42n, the coordinates of a predetermined position in the 2D-BBOX 42n may be used as information on the two-dimensional position of the object 90. The predetermined position may be, for example, the center, the midpoint of the base, or one of the corners (e.g., the upper left corner) of the 2D-BBOX 42n. The example of FIG. 7 illustrates an example in which the midpoint of the base of the 2D-BBOX 42n (black circle in FIG. 7) is used as the two-dimensional position of the object 90. This is treated as information indicating the position of the feet of the person who is the object 90 .

[0024] Furthermore, the information on the two-dimensional position of the object 90 may be the coordinate values ​​of each part of the object 90 recognized from each image 40n included in the multi-view image using a recognition model that has been generated in advance by machine learning to recognize one or more parts of the person who is the object 90 from the images 40n. For example, as shown in Fig. 8 , when the positions of each joint, etc. of the person who is the object 90 (black circles in Fig. 8 ) are recognized by the recognition model, the coordinate values ​​of the positions of each joint, etc. may be used as information on the two-dimensional position of the object 90.

[0025] In addition, when acquiring a multi-view image that does not have information on the two-dimensional position of the object 90, the acquisition unit 12 may use the above-mentioned detection model or recognition model to acquire information on the two-dimensional position of the object 90.

[0026] The interpolation unit 14 interpolates the two-dimensional position of the object 90 in the image 40n(t) based on the two-dimensional positions of the object 90 detected from each of the images 40n(t-1) and 40n(t+1). Specifically, the interpolation unit 14 predicts and interpolates the two-dimensional position of the object 90 in the image 40n(t) by linearly interpolating the two-dimensional positions of the object 90 detected from each of the images 40n(t-1) and 40n(t+1).

[0027] A specific description will be given using the example of Fig. 9. In the example of Fig. 9, n = 0, 1, and the two-dimensional position of the object 90 is detected from each of the images 40n(t-1), 40n(t), and 40n(t+1) as shown below.

[0028]

[0029] In this case, the interpolation unit 14 calculates the interpolated two-dimensional position p of the object 90 in each image 40n(t) as follows: ２ｄ，ｐｒｅｄ ｔ，ｃｎ (the halftone dotted circle in FIG. 9) is calculated.

[0030]

[0031] The time points of the images used for interpolation are not limited to t-1 and t+1. For example, the interpolation unit 14 may interpolate the two-dimensional position of image 40n(t) using images 40n at times t-5, t-4, t-3, t-2, t-1, t+1, t+2, t+3, t+4, and t+5. In this case, image 40n(t-5), image 40n(t-4), image 40n(t-3), image 40n(t-2), and image 40n(t-1) are examples of the "first image" of the disclosed technology. Furthermore, image 40n(t+1), image 40n(t+2), image 40n(t+3), image 40n(t+4), and image 40n(t+5) are examples of the "third image" of the disclosed technology.

[0032] The calculation unit 16 calculates the three-dimensional position of the object 90 at time t based on the two-dimensional position of the object 90 detected from the image 40n(t), the two-dimensional position interpolated by the interpolation unit 14, and the camera parameters of each camera 30n. Specifically, the calculation unit 16 calculates candidates for the three-dimensional position of the object 90 for each combination of the detected and interpolated two-dimensional positions of the object 90 between the images 40n(t).

[0033] More specifically, the calculation unit 16 selects one of the two-dimensional positions detected and interpolated from the image 40i captured by the camera 30i and calculates p ２ｄ ｉ Then, one of the two-dimensional positions detected and interpolated from the image 40j taken by the camera 30j is selected and p ２ｄ ｊ The calculation unit 16 calculates p ２ｄ ｉ = (x ｉ , y ｉ , 1) and p ２ｄ ｊ = (x ｊ , y ｊ , 1), a perspective projection matrix P ｃｉ and P ｃｊ By solving the following equation using ３ｄ，ｃａｎｄ ｔ Calculate P ｎ is the nth row of P.

[0034]

[0035] In the example of FIG. 9, the calculation unit 16 calculates p ２ｄ，ｏｂｓ ｔ，ｃ０ and p ２ｄ，ｐｒｅｄ ｔ，ｃ０ and p of image 401 ２ｄ，ｏｂｓ ｔ，ｃ１ and p ２ｄ，ｐｒｅｄ ｔ，ｃ１ From the combination of these, four 3D position candidates p ３ｄ，ｃａｎｄ ｔ (the shaded circle in FIG. 9) is calculated.

[0036] The calculation unit 16 calculates the plurality of three-dimensional position candidates p ３ｄ，ｃａｎｄ ｔ Based on this, the three-dimensional position p^ of the object 90 ３ｄ ｔ For example, the calculation unit 16 calculates a plurality of candidates p ３ｄ，ｃａｎｄ ｔ The position of the center of gravity is the three-dimensional position p^ ３ｄ ｔ In this case, the two-dimensional position p detected from the image 40n(t) is calculated as follows: ２ｄ，ｏｂｓ ｔ，ｃｎ , and the interpolated two-dimensional position p ２ｄ，ｐｒｅｄ ｔ，ｃｎ Therefore, the calculation unit 16 calculates the candidate p ３ｄ，ｃａｎｄ ｔ Based on the distance between the candidates p ３ｄ，ｃａｎｄ ｔ Clustering is performed on the candidates p ３ｄ，ｃａｎｄ ｔ The center of gravity of the cluster with the largest number of ３ｄ ｔ This is based on the assumption that the position of the center of gravity of the largest cluster has the highest probability of approximating the true three-dimensional position. Note that a hierarchical clustering method such as the complete linkage method may be applied as the clustering method.

[0037] A specific explanation will be given using the example of Fig. 10. In the example of Fig. 10, n = 0, 1, 2, 3, and the two-dimensional position of the object 90 is detected from each of the images 40n(t-1), 40n(t), and 40n(t+1) as shown below.

[0038]

[0039] In the example of FIG. 10, the interpolation unit 14 calculates the interpolated two-dimensional position p of the object 90 in each image 40n(t) as follows: ２ｄ，ｐｒｅｄ ｔ，ｃｎ (dotted circle in FIG. 10) is calculated.

[0040]

[0041] The calculation unit 16 calculates two candidates p ３ｄ，ｃａｎｄ ｔ If the distance between the two candidates p is smaller than a threshold K, ３ｄ，ｃａｎｄ ｔ In addition, the calculation unit 16 calculates the center of gravity of the cluster and the other candidates p ３ｄ，ｃａｎｄ ｔ If the distance between p and p is less than the threshold K, ３ｄ，ｃａｎｄ ｔ The calculation unit 16 performs this process by assigning candidates p whose distance is smaller than the threshold K to the corresponding cluster. ３ｄ，ｃａｎｄ Between clusters or candidates p ３ｄ，ｃａｎｄ This is repeated until there is no gap between the clusters. In the example of Fig. 10, six clusters (solid ellipses and dashed ellipses in Fig. 10) are generated by clustering.

[0042] The calculation unit 16 calculates the candidate p included in the generated cluster. ３ｄ，ｃａｎｄ ｔ The cluster with the largest number of ３ｄ ｔ In the example of FIG. 10, the cluster indicated by the solid ellipse is selected. As a result, the candidates p belonging to other clusters are calculated as follows. ３ｄ，ｃａｎｄ ｔ is excluded as an outlier, and the true 3D position p ３ｄ，ｇｔｔ The three-dimensional position p^ close to ３ｄ ｔ is calculated.

[0043] The calculation unit 16 calculates the three-dimensional position p^ of the object 90 at each time t. ３ｄ ｔ , and calculate the three-dimensional position p^ at t = 0, 1, ..., T. ３ｄ ｔ (time series of three-dimensional positions p^ ３ｄ ) is output.

[0044] The three-dimensional position calculation device 10 may be realized by, for example, a computer 50 shown in FIG. 11 . The computer 50 includes a CPU (Central Processing Unit) 51, a memory 52 as a temporary storage area, and a non-volatile storage device 53. The computer 50 also includes an input / output device 54 such as an input device and a display device, and an R / W (Read / Write) device 55 that controls reading and writing of data from and to a storage medium 59. The computer 50 also includes a communication I / F (Interface) 56 that is connected to a network such as the Internet. The CPU 51, memory 52, storage device 53, input / output device 54, R / W device 55, and communication I / F 56 are connected to one another via a bus 57.

[0045] The storage device 53 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage device 53 serving as a storage medium stores a three-dimensional position calculation program 60 for causing the computer 50 to function as the three-dimensional position calculation device 10. The three-dimensional position calculation program 60 includes an acquisition process control command 62, an interpolation process control command 64, and a calculation process control command 66. The storage device 53 also includes an information storage area 70 in which information constituting the camera parameter DB 20 is stored.

[0046] The CPU 51 reads the three-dimensional position calculation program 60 from the storage device 53, loads it into the memory 52, and sequentially executes the control instructions contained in the three-dimensional position calculation program 60. The CPU 51 operates as the acquisition unit 12 shown in FIG. 6 by executing the acquisition process control instruction 62. The CPU 51 also operates as the interpolation unit 14 shown in FIG. 6 by executing the interpolation process control instruction 64. The CPU 51 also operates as the calculation unit 16 shown in FIG. 6 by executing the calculation process control instruction 66. The CPU 51 also reads information from the information storage area 70 and loads the camera parameter DB 20 into the memory 52. ​​As a result, the computer 50 that has executed the three-dimensional position calculation program 60 functions as the three-dimensional position calculation device 10. The CPU 51 that executes the program is hardware.

[0047] The functions realized by the three-dimensional position calculation program 60 may be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0048] Next, the operation of the three-dimensional position calculation device 10 according to this embodiment will be described. When time-series multi-viewpoint images are input to the three-dimensional position calculation device 10 and an instruction to calculate the three-dimensional position of the object 90 is given, the three-dimensional position calculation device 10 executes the three-dimensional position calculation process shown in Fig. 12. Note that the three-dimensional position calculation process is an example of a three-dimensional position calculation method of the disclosed technology.

[0049] In step S10, the acquisition unit 12 acquires time-series multi-view images to which information on the two-dimensional position of the object 90 has been assigned. Next, in step S12, the acquisition unit 12 sets a variable t, which represents the time of the processing target, to 1. Next, in step S14, the interpolation unit 14 interpolates the two-dimensional position of the object 90 in image 40n(t) based on the two-dimensional position of the object 90 detected from each of images 40n(t−1) and 40n(t+1) included in the multi-view images.

[0050] Next, in step S16, the calculation unit 16 calculates a candidate three-dimensional position of the object 90 for each combination between the images 40n(t) of the detected and interpolated two-dimensional positions of the object 90, using the camera parameters of the camera 30n that captured the corresponding image 40n(t). Next, in step S18, the calculation unit 16 clusters the candidates based on the distances between the calculated three-dimensional position candidates, and calculates the center of gravity of the cluster that includes the largest number of candidates as the three-dimensional position of the object 90 at time t.

[0051] Next, the acquisition unit 12 determines whether t is smaller than the time information T of the final frame of the multi-view image. If t<T, the process proceeds to step S22, where the acquisition unit 12 increments t by 1, and returns to step S14. On the other hand, if t≧T, the process proceeds to step S24, where the calculation unit 16 outputs the calculated three-dimensional position, and the three-dimensional position calculation process ends.

[0052] As described above, the three-dimensional position calculation device according to this embodiment acquires a first image captured at a time before a target time, a second image captured at a target time, and a third image captured at a time after the target time using each of a plurality of cameras capturing images of an object from multiple viewpoints. The three-dimensional position calculation device also interpolates the two-dimensional position of the object in the second image based on the two-dimensional positions of the object detected in each of the first and third images. The three-dimensional position calculation device then calculates the three-dimensional position of the object at the target time based on the two-dimensional position of the object detected in the second image, the interpolated two-dimensional position, and the camera parameters of each of the plurality of cameras. This supplements the two-dimensional position information used to calculate the three-dimensional position, even when there are many outliers or undetected objects in the two-dimensional positions on the images, thereby improving the accuracy of calculating the three-dimensional position of the object using multi-view images.

[0053] Furthermore, the 3D position calculation device according to this embodiment calculates candidates for the 3D position of the object for each combination of detected and interpolated 2D positions between images. The 3D position calculation device then clusters the candidates based on the distance between the candidates and calculates the center of gravity of the cluster containing the largest number of candidates as the 3D position of the object at a given time. This allows outliers to be appropriately excluded from the candidates, improving the accuracy of calculating the 3D position of the object using multi-view images.

[0054] In the above embodiment, the outliers are removed by clustering the three-dimensional position candidates and selecting the largest cluster. However, this is not limiting. For example, a method such as the above-mentioned RANSAC may be applied. Specifically, the three-dimensional position calculation device calculates the positions of the centers of gravity of multiple candidates as three-dimensional positions and projects the calculated three-dimensional positions onto each of the second images based on the camera parameters of each of the multiple cameras. The three-dimensional position calculation device may then recalculate the three-dimensional position of the object using two-dimensional positions detected and interpolated in the second image whose distance from the projected position is within a predetermined threshold. However, while the method using clustering, as in the above embodiment, functions properly even when the proportion of candidates other than outliers is below 50%, other outlier removal methods, such as RANSAC, may have difficulty removing outliers appropriately.

[0055] Here, an example of the results of calculation of a three-dimensional position by the three-dimensional position calculation device according to the embodiment will be described. Fig. 13 is an image diagram showing an example in which the position of a person in a world coordinate system is calculated based on skeletal information of the person recognized from an image. In both the upper (Shelf) and lower (Campus) examples, it can be seen that the three-dimensional position of each person can be calculated with high accuracy, despite the fact that people are crowded together and overlap on the image.

[0056] 14 is an illustration showing an example in which the three-dimensional position of the midpoint of the base of a 2D-BBOX detected from an image, i.e., the position of a person's feet, is calculated, and the three-dimensional position of the person's feet is mapped onto a map representing the area to be photographed. Even in the example of FIG. 14, the three-dimensional position of each person's feet is calculated with high accuracy, despite the fact that people overlap in the image and are obscured by obstacles such as shelves. The three-dimensional position calculation device according to this embodiment can be applied to a system for acquiring the movement trajectories of customers within a store, as shown in the example of FIG. 14.

[0057] The three-dimensional position calculation device according to the above embodiment can be applied to, for example, a scoring system for gymnastics. Here, an outline of the processing of the scoring system for gymnastics will be described with reference to FIG.

[0058] When multi-view images are input, the scoring system detects a person's area from each image included in the multi-view images. Next, the scoring system determines whether the person indicated by the detected area is a player or a non-player, based on whether the position where the person is located is within the competition area, etc., and identifies the area representing the player. The scoring system tracks the player by matching areas representing the same player in the time-series multi-view images. The scoring system recognizes the player's two-dimensional skeletal information from each of the tracked series of images using a recognition model, etc. The scoring system estimates three-dimensional skeletal information from the two-dimensional skeletal information using camera parameters. The scoring system then performs post-processing, such as smoothing, on the time-series three-dimensional skeletal information to estimate the phases (breaks) of the performance and then recognizes the technique.

[0059] In the processing of the scoring system, the three-dimensional position calculation device according to the above embodiment can be applied to the processing of estimating three-dimensional skeletal information from two-dimensional skeletal information.

[0060] The disclosed technology is not limited to gymnasts as the target object, but can be applied to various people as the target object, such as athletes of other sports, ordinary pedestrians, etc. Furthermore, it can also be applied to animals, vehicles, etc. as the target object other than people.

[0061] In the above embodiment, the three-dimensional position calculation program is stored (installed) in advance in the storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0062] REFERENCE SIGNS LIST 10 Three-dimensional position calculation device 12 Acquisition unit 14 Interpolation unit 16 Calculation unit 20 Camera parameter DB 30n Camera 40n Image 50 Computer 51 CPU 52 Memory 53 Storage device 54 Input / output device 55 R / W device 56 Communication I / F 57 Bus 59 Storage medium 60 Three-dimensional position calculation program 62 Acquisition process control command 64 Interpolation process control command 66 Calculation process control command 70 Information storage area 90 Object

Claims

1. Acquiring a first image taken at one or more times before a target time, a second image taken at the target time, and a third image taken at one or more times after the target time, using each of a plurality of cameras that photograph an object from a plurality of different viewpoints; Interpolating a two-dimensional position of the object in the second image based on the two-dimensional positions of the object detected from each of the first image and the third image; Calculating a three-dimensional position of the object at the target time based on the two-dimensional position of the object detected from the second image, the two-dimensional position of the object in the interpolated second image, and camera parameters of each of the plurality of cameras. A three-dimensional position calculation method, comprising the steps of:

2. 2. The three-dimensional position calculation method according to claim 1, wherein the process of calculating the three-dimensional position of the object includes calculating the three-dimensional position of the object at the target time based on candidates for the three-dimensional position of the object calculated for each combination of the two-dimensional position of the object detected and interpolated from the second image taken by a first camera included in the plurality of cameras and the two-dimensional position of the object detected and interpolated from the second image taken by a second camera included in the plurality of cameras.

3. 3. The three-dimensional position calculation method according to claim 2, wherein the process of calculating the three-dimensional position of the object includes clustering the candidates based on the distance between the candidates, and calculating the center of gravity of the cluster containing the largest number of candidates as the three-dimensional position of the object at the target time.

4. A three-dimensional position calculation method according to any one of claims 1 to 3, in which the three-dimensional position of the object at the target time is recalculated using a two-dimensional position of the object detected from the second image, the distance of which from the calculated three-dimensional position of the object projected onto each of the second images based on the camera parameters of each of the multiple cameras is within a predetermined threshold, and an interpolated two-dimensional position of the object in the second image.

5. The three-dimensional position calculation method according to any one of claims 1 to 3, wherein the process of interpolating the two-dimensional position of the object in the second image includes predicting the two-dimensional position of the object in the second image by linearly interpolating the two-dimensional positions of the object detected from each of the first image and the third image.

6. The three-dimensional position calculation method according to any one of claims 1 to 3, wherein the two-dimensional position of the object is the coordinate value of a predetermined point within an area surrounding the object detected from each of the first image, the second image, and the third image using a detection model generated in advance by machine learning to detect the area of ​​the object from the image, or the coordinate value of a part of the object recognized from each of the first image, the second image, and the third image using a recognition model generated in advance by machine learning to recognize one or more parts of the object from the image.

7. an acquisition unit that acquires, by each of a plurality of cameras that capture an image of an object from a plurality of different viewpoints, a first image captured at one or more times before a target time, a second image captured at the target time, and a third image captured at one or more times after the target time; an interpolation unit that interpolates a two-dimensional position of the object in the second image based on two-dimensional positions of the object detected from each of the first image and the third image; a calculation unit that calculates a three-dimensional position of the object at the target time based on a two-dimensional position of the object detected from the second image, an interpolated two-dimensional position of the object in the second image, and camera parameters of each of the plurality of cameras; A three-dimensional position calculation device comprising:

8. Acquiring a first image taken at one or more times before a target time, a second image taken at the target time, and a third image taken at one or more times after the target time, using each of a plurality of cameras that photograph an object from a plurality of different viewpoints; Interpolating a two-dimensional position of the object in the second image based on the two-dimensional positions of the object detected from each of the first image and the third image; Calculating a three-dimensional position of the object at the target time based on the two-dimensional position of the object detected from the second image, the two-dimensional position of the object in the interpolated second image, and camera parameters of each of the plurality of cameras. A three-dimensional position calculation program for causing a computer to execute a process including the above.