Position estimation method, position estimation device, and program

Abstract

This position estimation method is for estimating the position and orientation of a first camera. The first camera can change position and orientation by rotating on a first rotational axis provided at a position spaced apart from a first image sensor with which the first camera is equipped. In the position estimation method, a first image of a subject captured by the first camera in a first position and orientation and a second image of the subject captured by the first camera in a second position and orientation in which the angle on the first rotational axis is different from that of the first position and orientation are acquired (S11), a first positional relationship between the first position and orientation and the second position and orientation is estimated (S14) by performing feature point matching between the first image and the second image (S13), and the position of the first camera is estimated (S15) on the basis of the first positional relationship.

Description

Position estimation method, position estimation device, and program

[0001] The present disclosure relates to a position estimation method, a position estimation device, and a program for calculating the position of a camera whose shooting direction can be changed by rotation.

[0002] Patent Document 1 discloses a technique for estimating the positions and orientations of a plurality of image capturing devices (i.e., cameras) by performing feature point matching on a plurality of images obtained by the plurality of image capturing devices.

[0003] International Publication No. 2020 / 153264

[0004] The present disclosure provides a position estimation method and the like that can easily estimate the position of a camera whose shooting direction can be changed by rotation.

[0005] A position estimation method according to one aspect of the present disclosure is a position estimation method for estimating the position and orientation of a first camera, wherein the first camera is capable of changing its position and orientation by rotating about a first rotation axis that is positioned away from a first image sensor provided in the first camera, and the position estimation method acquires a first image of a subject captured by the first camera in the first position and orientation, and a second image of the subject captured by the first camera in a second position and orientation that has a different angle on the first rotation axis from the first position and orientation, performs feature point matching on the first image and the second image, thereby estimating a first positional relationship between the first position and orientation and the second position and orientation, and estimates the position of the first camera based on the first positional relationship.

[0006] A position estimation device according to one aspect of the present disclosure is a position estimation device that estimates the position and orientation of a first camera whose position and orientation can be changed by rotating about a first rotation axis located at a position away from a first image sensor, and includes: an acquisition unit that acquires a first image of a subject captured by the first camera in the first position and orientation, and a second image of the subject captured by the first camera in a second position and orientation that has a different angle about the first rotation axis from the first position and orientation; and an estimation unit that estimates a first positional relationship between the first position and orientation and the second position and orientation by performing feature point matching on the first image and the second image, and estimates the position of the first camera based on the first positional relationship.

[0007] These general or specific aspects may be realized as a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a method, a system, an apparatus, an integrated circuit, a computer program, and a non-transitory recording medium.

[0008] The position estimation method and the like disclosed herein can easily estimate the position of a camera whose shooting direction can be changed by rotation.

[0009] Fig. 1 is a diagram for explaining an example of the configuration of an imaging system according to an embodiment. Fig. 2 is a diagram showing an example of the configuration of a camera according to an embodiment. Fig. 3 is a block diagram showing an example of the configuration of a control device according to an embodiment. Fig. 4 is a flowchart showing an example of a process for estimating the position of one camera according to an embodiment. Fig. 5 is a flowchart showing an example of a process for estimating the positions of multiple cameras according to an embodiment.

[0010] (Findings that Form the Basis of the Present Disclosure) The present inventors have found that the following problems occur with the conventional systems described in the "Background Art" section.

[0011] In conventional techniques such as that described in Patent Document 1, the positions of multiple cameras are estimated using SfM (Structure from Motion). SfM is a technique in which any two cameras (hereinafter referred to as a camera pair) are selected from multiple cameras, and the relative positions (x, y, z) and rotation angles (ψ, θ, φ) of the camera pair are found by matching feature points contained in two images (image pair) using the principle of triangulation, thereby estimating alignment that satisfies the relative positions of all camera pairs simultaneously.

[0012] Incidentally, a camera that can rotate at a fixed position may be used as a camera for which position estimation is to be performed. A pan-tilt-zoom camera is known as such a camera. A pan-tilt-zoom camera can rotate around two rotation axes that are oriented in different directions. Generally, even if multiple images obtained by changing the shooting direction at a fixed position are used, the fixed position cannot be estimated because these multiple images are taken from the same viewpoint. Therefore, in conventional technology, the fixed position of a camera cannot be estimated using multiple images obtained by changing the shooting direction at a fixed position.

[0013] Therefore, the present inventors have come up with a position estimation method that can easily estimate the position of a camera whose shooting direction can be changed by rotation.

[0014] A position estimation method according to a first aspect is a position estimation method for estimating the position and orientation of a first camera, wherein the first camera is capable of changing its position and orientation by rotating about a first rotation axis that is positioned away from a first image sensor provided in the first camera, and the position estimation method acquires a first image of a subject captured by the first camera in a first position and orientation, and a second image of the subject captured by the first camera in a second position and orientation that has a different angle on the first rotation axis from the first position and orientation, performs feature point matching on the first image and the second image, thereby estimating a first positional relationship between the first position and orientation and the second position and orientation, and estimates the position of the first camera based on the first positional relationship.

[0015] According to this, the first camera can change its position and orientation by rotating about a first rotation axis provided at a position distant from a first image sensor included in the first camera, and therefore the first image and the second image obtained for the first position and orientation, which have different angles about the first rotation axis, are images captured from different viewpoints. Therefore, by performing feature point matching on the first image and the second image, it is possible to estimate a first positional relationship between the first position and orientation and the second position and orientation, and it is possible to easily estimate the position of the first camera based on the first positional relationship.

[0016] A position estimation method according to a second aspect is the position estimation method according to the first aspect, further comprising: acquiring a first angular difference between the first position and attitude on the first rotation axis and the second position and attitude; and applying a scale obtained from the first angular difference to the first positional relationship, thereby estimating the position of the first camera.

[0017] According to this, since the distance from the first rotation axis to the first image sensor of the first camera is fixed, the distance between the first position and orientation and the second position and orientation can be calculated based on the first angular difference. Therefore, the scale between the 3D model and the real 3D space can be obtained from the first angular difference, and by applying the scale to the first positional relationship, it is possible to easily estimate the position of the first camera in the real 3D space.

[0018] A position estimation method according to a third aspect is the position estimation method according to the second aspect, and in estimating the position of the first camera, an origin and three axes in a Cartesian coordinate system are further set in the first positional relationship, thereby estimating the position of the first camera as an absolute position in the Cartesian coordinate system.

[0019] Therefore, the absolute position of each viewpoint in the Cartesian coordinate system can be easily estimated.

[0020] A position estimation method according to a fourth aspect is the position estimation method according to any one of the first to third aspects, further comprising the steps of: acquiring a third image of the subject captured by the second camera in a third position and orientation; and acquiring a fourth image of the subject captured by the second camera in a fourth position and orientation different from the third position and orientation; performing feature point matching on the third image and the fourth image; estimating a second positional relationship between the third position and orientation and the fourth position and orientation; aligning a first three-dimensional model of the subject generated by performing feature point matching on the first image and the second image with a second three-dimensional model of the subject generated by performing feature point matching on the third image and the fourth image; and estimating the position of the second camera based on the alignment result, the first positional relationship, and the second positional relationship.

[0021] This allows the positional relationship between the first camera and the second camera to be estimated based on the result of aligning the first three-dimensional model of the subject generated based on the multiple images captured by the first camera with the second three-dimensional model of the subject generated based on the multiple images captured by the second camera, the first positional relationship, and the second positional relationship, thereby allowing the position of the second camera to be estimated.

[0022] A position estimation method according to a fifth aspect is the position estimation method according to the fourth aspect, further comprising: acquiring a first angular difference between the first position and attitude about the first rotation axis and the second position and attitude; acquiring a second angular difference between the third position and attitude about the second rotation axis and applying a scale obtained from a most probable angular difference between the first angular difference and the second angular difference to the first positional relationship and the second positional relationship, thereby estimating the position of the first camera and the position of the second camera.

[0023] Therefore, the positions of the first camera and the second camera are estimated by applying the scale obtained from the most probable angle difference among the scales obtained from multiple cameras, so that the positions of the first camera and the second camera can be estimated with high accuracy.

[0024] A position estimation method according to a sixth aspect is the position estimation method according to the fourth aspect, further comprising the steps of: acquiring a first angular difference between the first position and orientation about the first rotation axis and the second position and orientation; acquiring a second angular difference between the third position and orientation about the second rotation axis and statistically processing a first scale obtained from the first angular difference and a second scale obtained from the second angular difference, and applying the scales obtained to the first positional relationship and the second positional relationship, thereby estimating the position of the first camera and the position of the second camera.

[0025] Therefore, the positions of the first camera and the second camera are estimated by applying a scale obtained by statistically processing the scales obtained from multiple cameras, so that the positions of the first camera and the second camera can be estimated with high accuracy.

[0026] A position estimation method according to a seventh aspect is a position estimation method according to any one of the first to third aspects, further comprising acquiring a third image of the subject taken by a second camera, and estimating the position of the second camera by performing feature point matching between the third image and at least one of the first image and the second image.

[0027] This allows the positions of multiple cameras to be estimated.

[0028] A position estimation method according to an eighth aspect is a position estimation method according to any one of the first to seventh aspects, in which the first image and the second image are images captured by the first camera with the same parameters except for position and orientation.

[0029] Therefore, a plurality of images having similar features are captured, and feature point matching can be performed with high accuracy.

[0030] A position estimation method according to a ninth aspect is a position estimation method according to any one of the first to eighth aspects, wherein the first camera is a zoom camera with an adjustable focal length of the lens, and the first image and the second image are images taken by the first camera with the focal length of the lens adjusted to a specified focal length.

[0031] Therefore, the scale can be calculated with high accuracy.

[0032] A position estimation method according to a tenth aspect is a position estimation method according to any one of the first to ninth aspects, wherein the first image and the second image are images captured by the first camera in a state adjusted so that the angular difference on the first rotation axis is a specified rotation angle difference.

[0033] Therefore, the scale can be calculated with high accuracy.

[0034] A position estimation device according to an eleventh aspect is a position estimation device that estimates the position and orientation of a first camera whose position and orientation can be changed by rotating about a first rotation axis that is located away from a first image sensor, and includes an acquisition unit that acquires a first image of a subject captured by the first camera in a first position and orientation, and a second image of the subject captured by the first camera in a second position and orientation that has a different angle about the first rotation axis from the first position and orientation, and an estimation unit that estimates a first positional relationship between the first position and orientation and the second position and orientation by performing feature point matching on the first image and the second image, and estimates the position of the first camera based on the first positional relationship.

[0035] According to this, the first camera can change its position and orientation by rotating about a first rotation axis provided at a position distant from a first image sensor included in the first camera, and therefore the first image and the second image obtained for the first position and orientation, which have different angles about the first rotation axis, are images captured from different viewpoints. Therefore, by performing feature point matching on the first image and the second image, it is possible to estimate a first positional relationship between the first position and orientation and the second position and orientation, and it is possible to easily estimate the position of the first camera based on the first positional relationship.

[0036] A program according to an eleventh aspect is a program for causing a computer to execute the position estimation method according to any one of the first to ninth aspects.

[0037] These general or specific aspects may be realized as a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or may be realized as any combination of a method, a system, an apparatus, an integrated circuit, a computer program, and a non-transitory recording medium.

[0038] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0039] The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0040] (Embodiment) Hereinafter, an embodiment will be described with reference to FIGS.

[0041] [Configuration] FIG. 1 is a diagram illustrating an example of the configuration of an imaging system according to an embodiment.

[0042] In the photography system 1, a specific subject 20 located within a specific photography range 31 within the space 30 is photographed by multiple cameras 11-13 arranged at different positions within the space 30. Of the multiple cameras 11-13, a first camera 11 is arranged at a first position P11, a second camera 12 is arranged at a second position P12, and a third camera 13 is arranged at a third position P13. Because the multiple cameras 11-13 photograph the same subject 20 from different positions, images of the specific subject 20 photographed from multiple different photography directions D1-D3 can be obtained. That is, the first camera 11 photographs the subject 20 in photography direction D1, the second camera 12 photographs the subject 20 in photography direction D2, and the third camera 13 photographs the subject 20 in photography direction D3. Note that the images obtained by the multiple cameras 11-13 may be still images or videos.

[0043] Each of the first camera 11, the second camera 12, and the third camera 13 is a PTZ camera that can pan, tilt, and zoom freely. That is, the first camera 11 can change the shooting direction D1 by rotating in two different rotational directions on different axes at the first position P11. Similarly, the second camera 12 can change the shooting direction D2 by rotating in two different rotational directions on different axes at the second position P12. Similarly, the third camera 13 can change the shooting direction D3 by rotating in two different rotational directions on different axes at the third position. In this way, each of the first camera 11, the second camera 12, and the third camera 13 has a rotation mechanism that changes the shooting direction by rotating in two different rotational directions on different axes.

[0044] FIG. 2 is a diagram illustrating an example of the configuration of a camera according to the embodiment.

[0045] 2, the first camera 11 will be described as an example. The second camera 12 and the third camera 13 have the same configuration, and therefore their description will be omitted.

[0046] As shown in FIG. 2 , the first camera 11 includes an image sensor 11a that is positioned at a different position from the rotation axis AX1. The rotation axis AX1 is, for example, a rotation axis for rotating (i.e., panning) the first camera 11 in the horizontal direction. The rotation axis AX1 is, for example, a rotation axis that extends in the vertical direction and is an example of a first rotation axis. The first camera 11 also rotates on a rotation axis AX2 that intersects (e.g., is perpendicular to) the rotation axis AX1. The rotation axis AX2 is, for example, a rotation axis for rotating (i.e., tilting) the first camera 11 in the vertical direction. The rotation axis AX2 is, for example, a rotation axis that extends in the horizontal direction.

[0047] Because the rotation axis AX1 is disposed at a position different from the position of the image sensor 11a, when the first camera 11 rotates around the rotation axis AX1, the position of the image sensor 11a moves before and after the rotation. For example, (a) of Fig. 2 shows the first camera 11 facing the imaging direction D11, (b) of Fig. 2 shows the first camera 11 facing the imaging direction D12 different from the imaging direction D11, and (c) of Fig. 2 shows a diagram in which the first camera 11 facing the imaging direction D11 and the first camera 11 facing the imaging direction D12 are superimposed.

[0048] 2C, the distance R between the position of the image sensor 11a and the rotation axis AX1 is a fixed distance. Therefore, when the first camera 11 rotates about the rotation axis AX1, the image sensor 11a moves a distance D before and after the rotation that corresponds to the rotation angle θ. In other words, the distance D between the position of the image sensor 11a before the rotation and the position of the image sensor 11a after the rotation corresponds to the rotation angle θ. The rotation angle θ is the angular difference between the angle of the position and orientation of the imaging direction D11 and the angle of the position and orientation of the imaging direction D12, and is an example of a first angular difference.

[0049] The control device 100 controls the operations of the multiple cameras 11 to 13. The control device 100 acquires multiple images captured by the multiple cameras 11 to 13 from the multiple cameras 11 to 13, and estimates the relative positions and attitudes of the multiple cameras 11 to 13 using the acquired multiple images. The control device 100 is an example of a position estimation device. The detailed configuration of the control device 100 will be described using FIG. 3.

[0050] FIG. 3 is a block diagram illustrating an example of the configuration of the control device according to the embodiment.

[0051] The control device 100 includes an acquisition unit 101 and an estimation unit 102. The control device 100 may further include a control unit 103 and an input reception unit 104.

[0052] The acquisition unit 101 acquires a plurality of images from the first camera 11. The plurality of images are images captured at different positions and orientations by rotating the first camera 11 only around AX1. A first image of the plurality of images is an image of the subject 20 captured by the first camera 11 in a first position and orientation. A second image of the plurality of images is an image of the subject 20 captured by the first camera 11 in a second position and orientation. The second position and orientation differ from the first position and orientation in the angle of the first camera 11 about the rotation axis AX1. The second position and orientation have the same parameters (extrinsic parameters) as the first position and orientation, except for the angle of the first camera 11 about the rotation axis AX1.

[0053] The acquisition unit 101 acquires a plurality of images from the second camera 12. The plurality of images are images captured at different positions and orientations by rotating the second camera 12 only around AX1. A third image among the plurality of images is an image of the subject 20 captured by the second camera 12 in the third position and orientation. A fourth image among the plurality of images is an image of the subject 20 captured by the second camera 12 in the fourth position and orientation. The fourth position and orientation differ from the third position and orientation in the angle of the second camera 12 about the rotation axis AX1. The fourth position and orientation have the same parameters (extrinsic parameters) as the third position and orientation, except for the angle of the second camera 12 about the rotation axis AX1.

[0054] The acquisition unit 101 acquires a plurality of images taken from the third camera 13 at different positions and orientations, similar to the first camera 11 and the second camera 12 .

[0055] Furthermore, the acquisition unit 101 acquires a first angular difference between the first position and orientation about the rotation axis AX1 from the first camera 11. Note that the acquisition unit 101 may acquire the first angular difference based on a history of control instructions from the control unit 103. The control unit 103 issues an image capture instruction to the first camera 11 by specifying an image capture direction, and therefore the acquisition unit 101 can also acquire the first angular difference by acquiring the history of image capture instructions. Note that, similar to the first camera 11, the acquisition unit 101 acquires the image capture directions corresponding to multiple images captured by the second camera 12 and the third camera 13, thereby acquiring the angular difference at different position and orientations.

[0056] The estimation unit 102 estimates a first positional relationship between a plurality of positions and orientations corresponding to the plurality of images obtained from the first camera 11 by performing feature point matching on the plurality of images (i.e., the first image and the second image). For example, the estimation unit 102 detects a plurality of feature points in each of the first image and the second image, and matches feature points having corresponding feature amounts (i.e., similar feature points) among the detected plurality of feature points. In this manner, the estimation unit 102 identifies a plurality of pairs of feature points in the first image and the second image, and estimates a first positional relationship between a first position and orientation of the first camera 11 when the first image was captured and a second position and orientation of the first camera 11 when the second image was captured based on the plurality of feature point pairs. Then, the estimation unit 102 estimates the position of the first camera 11 based on the first positional relationship.

[0057] The position of the first camera 11 is indicated by a specific position within the first camera 11. In other words, once the first positional relationship can be determined, the position of the image sensor 11a in the first position and orientation, or the position of the image sensor 11a in the second position and orientation, can be determined. Since the position of the image sensor 11a and the specific position within the first camera 11 have a fixed positional relationship, the position of the first camera 11 can be determined by determining the position of the image sensor 11a in the first position and orientation, or the position of the image sensor 11a in the second position and orientation. Note that the specific position within the first camera 11 may be, for example, a position within the first camera 11 on the rotation axis AX1.

[0058] In addition, the estimation unit 102 may estimate the positions of the second camera 12 and the third camera 13 by performing the same processing as that performed on the first camera 11 on the second camera 12 and the third camera 13, respectively.

[0059] Furthermore, the estimation unit 102 may estimate the position of the first camera 11 by applying a scale obtained from the first angular difference to the first positional relationship. Specifically, the estimation unit 102 applies the distance between the three-dimensional position indicated by the real first position and orientation obtained from the first angular difference and the three-dimensional position indicated by the real second position and orientation as the distance between the three-dimensional position indicated by the estimated first position and orientation and the three-dimensional position indicated by the real second position and orientation, thereby causing the size of the first positional relationship in the virtual space to correspond to the size in the real three-dimensional space. This allows the estimation unit 102 to estimate the position of the first camera 11 in the real three-dimensional space.

[0060] For example, the first positional relationship indicates the ratio of the distance between the first position P11 of the first camera 11 and the positions of multiple points including multiple 3D points representing a 3D model of the subject 20 in virtual space. That is, the estimation unit 102 determines the distance between the 3D position indicated by the first position and orientation and the 3D position indicated by the second position and orientation in the first positional relationship, thereby specifying the distance between the 3D position indicated by the first position and orientation, the 3D position indicated by the second position and orientation, and multiple 3D positions including multiple 3D points on the surface of the 3D model of the subject 20. In this way, the estimation unit 102 generates a first positional relationship in which the distances between the multiple 3D positions are specified. The multiple 3D points on the surface of the 3D model correspond to multiple pairs identified as a result of feature point matching. The 3D point CP1 on the surface of the subject 20 in FIG. 1 is calculated by triangulation based on the pair of the feature point corresponding to the 3D point CP1 on the first image and the feature point corresponding to the 3D point CP1 on the second image.

[0061] Furthermore, the estimation unit 102 may estimate the three-dimensional positions indicated by the first position and attitude and the three-dimensional positions indicated by the second position and attitude as absolute positions in the Cartesian coordinate system by further setting an origin and three axes in the Cartesian coordinate system in the positional relationship in which the distances between the multiple three-dimensional positions are specified. The three axes are, for example, the X-axis, Y-axis, and Z-axis in the Cartesian coordinate system, which are mutually orthogonal. As a result, the three-dimensional positions indicated by the first position and attitude and the three-dimensional positions indicated by the second position and attitude are each expressed by three-dimensional coordinates including X-coordinates, Y-coordinates, and Z-coordinates relative to the set origin. The origin may be set to a position indicated by any one of the multiple camera positions and multiple three-dimensional points, or may be set to an arbitrary point in the virtual space.

[0062] Furthermore, the estimation unit 102 aligns a first three-dimensional model of the subject 20 generated by performing feature point matching on a plurality of images obtained from the first camera 11 with a second three-dimensional model of the subject 20 generated by performing feature point matching on a plurality of images obtained from the second camera. The first three-dimensional model includes a first point cloud including a plurality of three-dimensional points each indicating a three-dimensional position on the surface of the subject 20. The second three-dimensional model includes a second point cloud including a plurality of three-dimensional points each indicating a three-dimensional position on the surface of the subject 20. Each of the plurality of three-dimensional points may include information indicating the color of the subject 20 at the position of the three-dimensional point.

[0063] The estimation unit 102 may then estimate the position of the second camera 12 based on the result of the alignment, a first positional relationship obtained based on the multiple images of the first camera 11, and a second positional relationship obtained based on the multiple images of the second camera 12. If the position of the second camera 12 obtained at this time is estimated using the first positional relationship to which a scale has been applied, it will be a position to which the size of the real three-dimensional space has been applied, and if the position is estimated using the first positional relationship to which a scale has not been applied, it will be a position to which the size of the real three-dimensional space has not been applied. Note that even when the position of the second camera 12 is estimated using the first positional relationship to which a scale has not been applied, it is possible to estimate a position to which the size of the real three-dimensional space has been applied by applying a scale to the estimated positional relationship.

[0064] The control unit 103 instructs the multiple cameras 11 to 13 to capture images. Specifically, the control unit 103 may instruct the first camera 11 to capture multiple images at different positions and orientations by rotating only about the rotation axis AX1. The control unit 103 may instruct the first camera 11 to capture images using the same parameters excluding the position and orientation. In other words, the multiple images captured by the first camera 11 are images captured using the same parameters (prescribed parameters) (e.g., internal parameters) excluding the position and orientation. The control unit 103 also issues the same capture instruction to the second camera 12 and the third camera 13 as the first camera 11. This allows each camera to obtain multiple images captured using the same parameters excluding the position and orientation.

[0065] For example, the control unit 103 may instruct the first camera 11 to take a photograph with the focal length of the lens adjusted to a specified focal length. In other words, the multiple images obtained from the first camera 11 are images taken with the focal length of the lens adjusted to a specified focal length. The control unit 103 may also issue the same instruction to the second camera 12 and the third camera 13 to take a photograph as the instruction to the first camera 11. This allows each camera to obtain multiple images taken with the focal length of the lens adjusted to a specified focal length.

[0066] When the focal length is changed, the calculated distance (converted to a physical distance) between the rotation axis AX1 and the image sensor 11a changes. Specifically, the greater the zoom magnification, the greater the calculated distance between the rotation axis AX1 and the image sensor 11a. For this reason, adjusting the focal length so as to increase the zoom magnification allows for more accurate scale calculation. On the other hand, the greater the zoom magnification, the greater the distance the angle of view moves in response to rotation of the rotation axis AX1, making it less likely that overlapping subjects will be included in the two images captured before and after the rotation. Therefore, the greater the zoom magnification, the smaller the rotation angle must be. In other words, in calculating the scale, a larger zoom magnification is advantageous in terms of the calculated distance between the rotation axis AX1 and the image sensor 11a, but is disadvantageous in terms of the rotation angle, resulting in a trade-off.

[0067] Therefore, in this trade-off relationship, the specified focal length may be set to a focal length determined to provide the highest accuracy in calculating the distance between three-dimensional positions indicated by the position and orientation. Furthermore, the rotation angle for capturing images at different position and orientations may also be set to a rotation angle determined to provide the highest accuracy. For example, the control unit 103 may set the rotation angle about the rotation axis AX1 from the first position and orientation to a specified rotation angle when the first camera 11 captures multiple images at different position and orientations. In other words, the multiple images obtained from the first camera 11 may be images captured by the first camera 11 in a state where the angle difference about the rotation axis AX1 is adjusted to a specified rotation angle difference. Furthermore, the control unit 103 may issue instructions to the second camera 12 and the third camera 13 similar to the capture instructions issued to the first camera 11.

[0068] The control unit 103 may instruct the cameras 11 to 13 to capture images in accordance with input received by the input receiving unit 104, which will be described later. For example, the control unit 103 may instruct a camera specified by a user to capture an image in a direction specified by the user.

[0069] The input receiving unit 104 receives input from the user. The input from the user may be an input for instructing the multiple cameras 11 to 13 to take a picture, or an input for instructing a specific camera to take a picture. The input for instructing a picture to be taken may specify the direction of the picture to be taken, the timing of the picture to be taken, the image quality of the image to be taken, etc. The image quality includes, for example, the resolution and, in the case of a video, the frame rate.

[0070] [Operation] The operation of estimating the positions of the plurality of cameras 11 to 13 by the control device 100 of the photographing system 1 configured as above will be described.

[0071] FIG. 4 is a flowchart illustrating an example of a process for estimating the position of one camera according to the embodiment.

[0072] The control device 100 acquires a first image of the subject 20 taken by the first camera 11 in a first position and orientation, and a second image of the subject 20 taken by the first camera 11 in a second position and orientation that has a different angle on the rotation axis AX1 from the first position and orientation (S11).

[0073] The control device 100 acquires the angular difference between the first position and posture about the rotation axis AX1 and the second position and posture (S12). Note that step S12 may be performed before step S11 or may be performed in parallel with step S11.

[0074] The control device 100 performs feature point matching on the first image and the second image (S13).

[0075] The control device 100 estimates a first positional relationship between the first position and posture and the second position and posture based on the result of the feature point matching (S14).

[0076] The control device 100 estimates the position of the first camera 11 by applying the scale obtained from the angle difference to the first positional relationship (S15).

[0077] After step S15, the control device 100 may further set an origin and three axes in a Cartesian coordinate system in the first positional relationship, thereby estimating the position of the first camera 11 as an absolute position in the Cartesian coordinate system.

[0078] Figure 4 illustrates an example of a position estimation method for estimating the position of the first camera 11, but by performing the same method for each of the second camera 12 and the third camera 13, the positions of each of the second camera 12 and the third camera 13 can be estimated.

[0079] FIG. 5 is a flowchart illustrating an example of a process for estimating the positions of a plurality of cameras according to an embodiment.

[0080] The control device 100 executes a loop for each camera, in which steps S21 to S23 are executed.

[0081] The control device 100 acquires two or more (for example, two) images taken at positions and orientations with different angles about the rotation axis AX1 from the camera to be processed (S21).

[0082] The control device 100 performs feature point matching on the two images (S22).

[0083] Based on the result of feature point matching, the control device 100 estimates the positional relationship between the position and orientation corresponding to one of the two images and the position and orientation corresponding to the other image (S23). At this time, the control device 100 generates a three-dimensional model of the subject 20 as a result of the estimation of the positional relationship.

[0084] The control device 100 acquires the angular difference about the rotation axis AX1 between the two positions and orientations when the two images are captured by the first camera 11, the second camera 12, and the third camera 13 (S24).

[0085] The control device 100 aligns the three-dimensional model generated for the first camera 11, the three-dimensional model generated for the second camera 12, and the three-dimensional model generated for the third camera 13 (S25).

[0086] The control device 100 estimates the positions of the first camera 11, the second camera 12, and the third camera 13 based on the alignment result, the first positional relationship obtained based on the two images of the first camera 11, the second positional relationship obtained based on the two images of the second camera 12, and the third positional relationship obtained based on the two images of the third camera 13 (S26). At this time, the control device 100 may estimate the positions of the first camera 11, the second camera 12, and the third camera 13 by applying a scale obtained using the angular difference obtained from each camera to the positional relationship corresponding to the scale. The scale at this time may be obtained using the angular difference obtained from one of the multiple cameras 11 to 13. In other words, it is not necessary to obtain multiple scales corresponding to each of the multiple cameras 11 to 13. In addition, the control device 100 may integrate the first positional relationship, the second positional relationship, and the third positional relationship using the alignment results, and apply one of the obtained scales to the integrated positional relationship, thereby estimating the position of the first camera 11, the position of the second camera 12, and the position of the third camera 13.

[0087] After step S26, the control device 100 may further set the origin and three axes in a Cartesian coordinate system to the positions of the first camera 11, the second camera 12, and the third camera 13, thereby estimating the positions of the first camera 11, the second camera 12, and the third camera 13 as absolute positions in the Cartesian coordinate system.

[0088] Furthermore, in step S26, the control device 100 may further set an origin and three axes in a Cartesian coordinate system in the first positional relationship, and estimate the positions of the first camera 11, the second camera 12, and the third camera 13 based on the first positional relationship in which the origin and the three axes in the Cartesian coordinate system are set, the second positional relationship obtained based on the two images of the second camera 12, and the third positional relationship obtained based on the two images of the third camera 13. This makes it possible to obtain the absolute position of the first camera 11 in the Cartesian coordinate system, the absolute position of the second camera 12 in the Cartesian coordinate system, and the absolute position of the third camera 13 in the Cartesian coordinate system.

[0089] The scale to be applied when multiple scales are available may be the scale obtained from the most probable angle difference among the angle differences used as the basis for calculating the scale. The most probable angle difference may be the camera angle difference that has the largest evaluation value based on the similarity between the feature point pairs identified in the feature point matching. The evaluation value may be, for example, the sum of the similarities of the extracted feature point pairs, or the average of the similarities obtained between the multiple feature point pairs, extracted by extracting a predetermined number of the similarities obtained between the multiple feature point pairs in descending order.

[0090] Furthermore, when multiple scales are available, the scale to be applied may be a scale obtained by statistically processing the multiple scales, such as the average, weighted average, maximum value, minimum value, or median value of the multiple scales.

[0091] Although two images are processed in FIGS. 4 and 5, three or more images may be processed.

[0092] [Effect] The position estimation method according to this embodiment is a position estimation method for estimating the position and orientation of first camera 11. First camera 11 can change its position and orientation by rotating about a rotation axis AX1 that is located away from image sensor 11a included in first camera 11. In the position estimation method, control device 100 acquires a first image in which subject 20 is captured by first camera 11 in a first position and orientation, and a second image in which subject 20 is captured by first camera 11 in a second position and orientation that has a different angle about rotation axis AX1 from the first position and orientation. Next, control device 100 performs feature point matching on the first image and the second image to estimate a first positional relationship between the first position and orientation and the second position and orientation, and estimates the position of first camera 11 based on the first positional relationship.

[0093] According to this, the position and orientation of the first camera 11 can be changed by rotating about a rotation axis AX1 that is provided at a position away from the image sensor 11a included in the first camera 11, and therefore the first image and the second image obtained at the first position and orientation, which have different angles about the rotation axis AX1, are images captured from different viewpoints. Therefore, by performing feature point matching on the first image and the second image, it is possible to estimate a first positional relationship between the first position and orientation and the second position and orientation, and it is possible to easily estimate the position of the first camera 11 based on the first positional relationship.

[0094] In the position estimation method according to the present embodiment, the control device 100 further acquires a first angular difference between the first position and orientation about the rotation axis AX1 and the second position and orientation. The control device 100 estimates the position of the first camera 11 by applying a scale obtained from the first angular difference to the first positional relationship.

[0095] According to this, since the distance from the rotation axis AX1 to the image sensor 11a of the first camera 11 is fixed, the distance between the first position and orientation and the second position and orientation can be calculated based on the first angular difference. Therefore, the scale between the three-dimensional model and the real three-dimensional space can be obtained from the first angular difference, and by applying the scale to the first positional relationship, it is possible to easily estimate the position of the first camera 11 in the real three-dimensional space.

[0096] Furthermore, in the position estimation method according to this embodiment, when estimating the position of the first camera 11, an origin and three axes in a Cartesian coordinate system are further set in the first positional relationship, and the position of the first camera 11 is estimated as an absolute position in the Cartesian coordinate system.

[0097] Therefore, the absolute position of each viewpoint in the Cartesian coordinate system can be easily estimated.

[0098] In the position estimation method according to this embodiment, the control device 100 further acquires a third image of the subject 20 captured by the second camera 12 in a third position and orientation, and a fourth image of the subject 20 captured by the second camera 12 in a fourth position and orientation different from the third position and orientation, using the second camera 12, which is capable of changing its position and orientation at a position different from that of the first camera 11 by rotating about a rotation axis AX1 located away from the image sensor 11a. The control device 100 then performs feature point matching on the third and fourth images to estimate a second positional relationship between the third and fourth positions and orientations. The control device 100 aligns a first three-dimensional model of the subject 20 generated by performing feature point matching on the first and second images with a second three-dimensional model of the subject 20 generated by performing feature point matching on the third and fourth images. The control device 100 estimates the position of the second camera 12 based on the alignment result, the first positional relationship, and the second positional relationship.

[0099] This makes it possible to estimate the positional relationship between first camera 11 and second camera 12 based on the first positional relationship and the second positional relationship, as well as the result of aligning the first three-dimensional model of subject 20 generated based on the multiple images captured by first camera 11 with the second three-dimensional model of subject 20 generated based on the multiple images captured by second camera 12. Therefore, the position of second camera 12 can be estimated.

[0100] Furthermore, in the position estimation method according to this embodiment, control device 100 further acquires a first angular difference between the first position and orientation about rotation axis AX1 of first camera 11. Control device 100 acquires a second angular difference between the third position and orientation about rotation axis AX1 of second camera 12. Control device 100 estimates the positions of first camera 11 and second camera 12 by applying a scale obtained from the most probable angular difference between the first angular difference and the second angular difference to the first positional relationship and the second positional relationship.

[0101] Therefore, the positions of the first camera 11 and the second camera 12 are estimated by applying the scale obtained from the most probable angle difference among the scales obtained from multiple cameras, so that the positions of the first camera 11 and the second camera 12 can be estimated with high accuracy.

[0102] Furthermore, in the position estimation method according to this embodiment, control device 100 further acquires a first angular difference between the first position and orientation about rotation axis AX1 of first camera 11. Control device 100 acquires a second angular difference between the third position and orientation about rotation axis AX1 of second camera 12. Control device 100 applies scales obtained by statistically processing a first scale obtained from the first angular difference and a second scale obtained from the second angular difference to the first positional relationship and the second positional relationship, thereby estimating the position of first camera 11 and the position of the second camera.

[0103] Therefore, the positions of the first camera 11 and the second camera 12 are estimated by applying a scale obtained by statistically processing the scales obtained from multiple cameras, so that the positions of the first camera 11 and the second camera 12 can be estimated with high accuracy.

[0104] In the position estimation method according to this embodiment, the first image and the second image are images captured by the first camera with the same parameters except for the position and orientation.

[0105] Therefore, a plurality of images having similar features are captured, and feature point matching can be performed with high accuracy.

[0106] In the position estimation method according to the present embodiment, the first camera 11 is a zoom camera with an adjustable focal length of the lens, and the first and second images obtained by the first camera 11 are images captured by the first camera 11 with the focal length of the lens adjusted to a specified focal length.

[0107] Therefore, the scale can be calculated with high accuracy.

[0108] Furthermore, in the position estimation method according to this embodiment, the first image and the second image are images taken by the first camera 11 in a state where the angular difference on the rotation axis AX1 of the first camera 11 is adjusted to be a specified rotation angle difference.

[0109] Therefore, the scale can be calculated with high accuracy.

[0110] [Modifications] While the position estimation method and the position estimation device according to the embodiment of the present disclosure have been described above, the present disclosure is not limited to this embodiment.

[0111] In the above embodiment, the positions of the multiple cameras are estimated by aligning the 3D models obtained for each camera, but this is not limiting. The positions of the multiple cameras may be estimated by performing feature point matching on multiple images obtained from each of the multiple cameras.

[0112] In the above embodiment and modifications, the control device 100 has been described as a device separate from the plurality of cameras 11 to 13, but it may be provided in at least one of the plurality of cameras 11 to 13.

[0113] In the above embodiment and variant examples, all of the multiple cameras 11 to 13 are cameras whose shooting direction can be changed, but it is sufficient that at least one of them is a camera whose shooting direction can be changed, and the other cameras may be cameras whose shooting direction cannot be changed.

[0114] The multiple cameras 11 to 13 in the above embodiment and modified examples do not need to be located at fixed positions, and may be movable. Even in this case, the position of the camera at the destination can be estimated using multiple images obtained by capturing images from different shooting directions without moving at the destination position.

[0115] Furthermore, each processing unit included in the control device according to the above-described embodiments is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or some or all of them may be integrated into a single chip.

[0116] Furthermore, the integrated circuit is not limited to an LSI, but may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI may also be used.

[0117] The present disclosure may also be realized as a position estimation method or the like executed by a control device or the like.

[0118] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in time-sharing by a single piece of hardware or software.

[0119] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

[0120] While the imaging system, control device, and the like according to one or more aspects have been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0121] The present disclosure is useful as a position estimation method that can easily estimate the position of a camera whose shooting direction can be changed by rotation.

[0122] REFERENCE SIGNS LIST 1 Photography system 11 First camera 11a Image sensor 12 Second camera 13 Third camera 20 Subject 30 Space 31 Specific photography range 100 Control device 101 Acquisition unit 102 Estimation unit 103 Control unit 104 Input reception unit AX1, AX2 Rotation axis D, R Distance D1, D2, D3, D11, D12 Photography direction P11 First position P12 Second position P13 Third position

Claims

1. A position estimation method for estimating the position and orientation of a first camera, wherein the first camera is capable of changing its position and orientation by rotating about a first rotation axis provided at a position away from a first image sensor provided in the first camera, the position estimation method comprising: acquiring a first image of a subject captured by the first camera in the first position and orientation; and acquiring a second image of the subject captured by the first camera in a second position and orientation that has a different angle about the first rotation axis from the first position and orientation; estimating a first positional relationship between the first position and orientation and the second position and orientation by performing feature point matching on the first image and the second image; and estimating the position of the first camera based on the first positional relationship.

2. The position estimation method according to claim 1, further comprising: acquiring a first angular difference between the first position and orientation on the first rotation axis and the second position and orientation; and estimating the position of the first camera by applying a scale obtained from the first angular difference to the first positional relationship.

3. The position estimation method according to claim 2, wherein the position of the first camera is estimated as an absolute position in a Cartesian coordinate system by further setting an origin and three axes in the Cartesian coordinate system in the first positional relationship.

4. A position estimation method according to any one of claims 1 to 3, further comprising the steps of: acquiring a third image of the subject taken by the second camera in a third position and orientation, and a fourth image of the subject taken by the second camera in a fourth position and orientation different from the third position and orientation, by rotating about a second rotation axis located away from a second image sensor; estimating a second positional relationship between the third position and orientation and the fourth position and orientation by performing feature point matching on the third image and the fourth image; aligning a first three-dimensional model of the subject generated by performing feature point matching on the first image and the second image with a second three-dimensional model of the subject generated by performing feature point matching on the third image and the fourth image; and estimating the position of the second camera based on the alignment result, the first positional relationship, and the second positional relationship.

5. The position estimation method according to claim 4, further comprising: acquiring a first angular difference between the first position and attitude on the first rotation axis and the second position and attitude; acquiring a second angular difference between the third position and attitude and the fourth position and attitude on the second rotation axis; and applying a scale obtained from the most probable angular difference between the first angular difference and the second angular difference to the first positional relationship and the second positional relationship, thereby estimating the position of the first camera and the position of the second camera.

6. The position estimation method according to claim 4, further comprising: acquiring a first angular difference between the first position and orientation on the first rotation axis and the second position and orientation; acquiring a second angular difference between the third position and orientation on the second rotation axis and the fourth position and orientation; and applying a scale obtained by statistically processing a first scale obtained from the first angular difference and a second scale obtained from the second angular difference to the first positional relationship and the second positional relationship, thereby estimating the position of the first camera and the position of the second camera.

7. A position estimation method according to any one of claims 1 to 3, further comprising: acquiring a third image of the subject taken by a second camera; and estimating the position of the second camera by performing feature point matching between the third image and at least one of the first image and the second image.

8. A position estimation method according to any one of claims 1 to 3, wherein the first image and the second image are images taken by the first camera with the same parameters except for position and orientation.

9. A position estimation method according to any one of claims 1 to 3, wherein the first camera is a zoom camera with an adjustable focal length of the lens, and the first image and the second image are images taken by the first camera with the focal length of the lens adjusted to a specified focal length.

10. A position estimation method described in any one of claims 1 to 3, wherein the first image and the second image are images taken by the first camera in a state where the angular difference on the first rotation axis is adjusted to a specified rotation angle difference.

11. A position estimation device that estimates the position and orientation of a first camera whose position and orientation can be changed by rotating about a first rotation axis provided at a position away from a first image sensor, comprising: an acquisition unit that acquires a first image of a subject captured by the first camera in a first position and orientation, and a second image of the subject captured by the first camera in a second position and orientation that has a different angle about the first rotation axis from the first position and orientation; and an estimation unit that estimates a first positional relationship between the first position and orientation and the second position and orientation by performing feature point matching on the first image and the second image, and estimates the position of the first camera based on the first positional relationship.

12. A program for causing a computer to execute the position estimation method according to any one of claims 1 to 3.