Film camera tracking using IR tracking camera and calibration for virtual production

Abstract

An information processing apparatus includes circuitry configured to: obtain at least one first image of a display screen, the at least one first image being acquired by a first camera; obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being obtained by a second camera; estimating first position information of the first camera relative to the display screen based on the at least one first image; obtaining offset information between the first camera and the second camera; and estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image and the offset information, where a positional relationship between the first camera and the second camera is fixed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Japanese Priority Patent Application JP 2023-040879 filed on Mar. 15, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an information processing apparatus, an information processing method, and a non-transitory computer-readable medium. Background Art

[0004] There are known techniques for estimating the three-dimensional structure of a predetermined object from a plurality of two-dimensional images including the object. Such techniques include a technique called structure from motion (SfM) described in NPL 1, for example.

[0005] Reference List

[0006] Non-patent literature

[0007] NPL 1: Roger Mohr et al., "Relative 3D Reconstruction Using Multiple Uncalibrated Images," The International Journal of Robotics Research, SAGE Publications, 1995, 14(6), pp. 619-632, December 1, 1995. Summary of the Invention

[0008] Technical issues

[0009] Furthermore, there are cases where the three-dimensional position of a marker used to estimate the camera's position and orientation is estimated using SfM. In this case, calibration is required each time the marker's position is moved, which raises concerns about increased processing time.

[0010] Solution to the problem

[0011] According to the present disclosure, an information processing device is provided, which includes a circuit, and the circuit is configured to: obtain at least one first image of a display screen, the at least one first image being obtained by a first camera; obtain at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being obtained by a second camera; estimate first position information of the first camera relative to the display screen based on the at least one first image; obtain offset information between the first camera and the second camera; and estimate second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image and the offset information, wherein the positional relationship between the first camera and the second camera is fixed.

[0012] In addition, according to the present disclosure, an information processing method is provided, including: obtaining at least one first image of a display screen, the at least one first image being acquired by a first camera; obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera; estimating first position information of the first camera relative to the display screen based on the at least one first image; obtaining offset information between the first camera and the second camera; and estimating second position information of the second camera relative to the display screen and third position information of at least one marker relative to the display screen based on the first position information, the at least one second image and the offset information, wherein the positional relationship between the first camera and the second camera is fixed.

[0013] In addition, according to the present disclosure, a non-transitory computer-readable medium is provided, which contains a program, which, when executed by a computer, causes the computer to perform an information processing method, the method including: obtaining at least one first image of a display screen, the at least one first image being acquired by a first camera; obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera; estimating first position information of the first camera relative to the display screen and third position information of at least one marker relative to the display screen based on the at least one first image; obtaining offset information between the first camera and the second camera; and estimating second position information of the second camera relative to the display screen and third position information of at least one marker relative to the display screen based on the first position information, the at least one second image and the offset information, wherein the positional relationship between the first camera and the second camera is fixed. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram for describing an overview of an information processing system according to the present disclosure.

[0015] Figure 22 is an explanatory diagram for describing an embodiment of the functional configuration of the information processing apparatus 20 according to the present disclosure.

[0016] Figure 3 This is an explanatory diagram for describing an outline of a calibration process of an information processing system.

[0017] Figure 4A It is an explanatory diagram for describing details related to the correction of the local coordinate system.

[0018] Figure 4B It is an explanatory diagram for describing details related to the correction of the local coordinate system.

[0019] Figure 4C It is an explanatory diagram for describing details related to the correction of the local coordinate system.

[0020] Figure 5 This is an explanatory diagram for explaining the calibration process related to the installation offset estimation.

[0021] Figure 6 It is an explanatory diagram for describing an embodiment of a process of generating a graph with a pose prior.

[0022] Figure 7 is an explanatory diagram for describing an embodiment of overall processing performed by the information processing apparatus 20 according to the present disclosure.

[0023] Figure 8 2 is an explanatory diagram for describing an embodiment of a mounting offset estimation process performed by the information processing apparatus 20 according to the present disclosure.

[0024] Figure 9 It is an explanatory diagram for describing a modification example of the information processing system according to the present disclosure.

[0025] Figure 10 : is a block diagram showing a hardware configuration example of the information processing apparatus 90 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Preferred embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0027] Note that the description will be given in the following order.

[0028] 1. Implementation Method

[0029] 1.1. Summary

[0030] 1.2. Configuration Example

[0031] 1.3 Details

[0032] 2. Example of Operation Process

[0033] 3. Modifications

[0034] 4. Hardware Configuration Example

[0035] 5. Conclusion

[0036] <<1. Implementation Method>>

[0037] <1.1. Summary>

[0038] As described above, there are cases where the processing results of SfM are used to estimate the position and direction of a camera.

[0039] As an example, on a video production set, it is often necessary to track the position and orientation of a camera in real time for visual effects (VFX) and the like.

[0040] The technology for tracking the position and direction of a camera in real time as described above includes a technology for tracking multiple markers (hereinafter referred to as IR markers) using an infrared (IR) camera to obtain the position and direction of the IR camera, each of the multiple markers including a retroreflective material, etc. arranged in an imaging space.

[0041] While the above technique allows estimation in a low-cost and robust manner, it has the problem that calibration of the markers takes a lot of time and effort.

[0042] Furthermore, there are now studios where display devices such as light-emitting diode (LED) panels are placed on walls or the like in imaging spaces used for virtual production (VP). When VP is performed in such studios, an IR camera that tracks the position and orientation of a cinema camera in real time is sometimes attached to the cinema camera used to capture images of objects adjacent to the LED panels. This system, in which the IR camera tracks the position and orientation of the cinema camera in real time, is called a tracking system.

[0043] Furthermore, in VP, the display device displays a computer graphics (CG) image that matches the position and orientation of the cinema camera. To reflect the position and orientation of the cinema camera in the CG image, the coordinate system of the CG space and the coordinate system of the tracking system must be made common. As a possible embodiment, each coordinate system of the tracking system is unified with the coordinate system of the CG space.

[0044] In order to make the coordinate system of the CG space and the coordinate system of the tracking system common to each other, three independent calibration processes may be required as the initial configuration. According to the comparative example, as the initial configuration, calibration related to map generation is performed first (the first process), then calibration related to installation offset estimation is performed (the second process), and finally calibration related to LED volume alignment is performed (the third process).

[0045] Map generation is a calibration process that restores the three-dimensional position of the IR marker in the imaging space and estimates the physical scale. The technology for map generation according to the comparative example may require the user to perform an initialization operation in order to start map generation in a stable manner. For example, the user holds a tracking camera and moves the tracking camera horizontally by a distance of about 10% of the installation height of the IR marker (approximately) to obtain the physical scale, while allowing the viewpoint to change enough to start map generation. In addition, in order to achieve more precise horizontal movement, the user may also need to pre-install a guide rail with the help of a marker or the like.

[0046] Mounting offset estimation is a calibration process that estimates the coordinate system offset between the cine camera and the tracking camera. In some comparative examples of mounting offset estimation techniques, values ​​calculated based on actual measurements or CAD design values ​​are manually set. Specifically, typical cine cameras have sensor positions printed on their bodies, allowing the user to calculate the offset based on the position of the tracking camera attached to the cine camera and manually set the calculated value. However, this manual method of setting the mounting offset is cumbersome for the user and is prone to setting errors due to human error and other factors.

[0047] LED volume alignment is a calibration process that makes the coordinate system of the CG space (hereinafter also referred to as the LED coordinate system) and the coordinate system of the tracking system share each other. In the technology for LED volume alignment according to the comparative example, the user manually defines the LED coordinate system in the physical space using a marking tool or the like, and sets three reference points in the defined LED coordinate system. Then, when the tracking camera is arranged at each reference point, the user performs alignment using the coordinate value of the LED coordinate system corresponding to the output posture. It takes a lot of time to physically set such an LED coordinate system. In addition, since the position alignment is not based on the information actually displayed on the LED panel, errors are prone to occur. In addition, finally, the user is required to manually fine-tune the translation position and rotation position to balance the error, and this process also requires a certain amount of time and effort. In some cases, the user may need to perform the calibration process again from the graph generation process (first process).

[0048] The three calibration processes described above have the problem of taking a lot of time and effort. For example, in the case of a studio of average size, it is not uncommon for the three calibration processes to take about 2 to 3 hours.

[0049] Furthermore, the position of the marker may need to be moved depending on the imaging scene, but this also requires a time-consuming calibration process.

[0050] From a cost perspective, it can be difficult to keep performers or imaging personnel on call during a lengthy calibration process, and therefore, it is not uncommon for the range of imaging methods to be limited.

[0051] The technical concept according to the present disclosure is conceived by focusing on the above points, and allows reducing the time required for the above calibration process. Figure 1 An overview of an information processing system according to an embodiment of the present disclosure is described.

[0052] Figure 1 : is a diagram for describing the outline of the information processing system according to the present disclosure. Figure 1 As shown, the information processing system according to the present disclosure includes an LED panel 10 , a movie camera K1 , a tracking camera K2 , and an information processing device 20 .

[0053] (LED panel 10)

[0054] The LED panel 10 according to the present disclosure is an embodiment of a display device, and is Figure 1 During the calibration shown, an augmented reality (AR) marker M1 is displayed. In addition, the LED panel 10 displays a CG image. For example, the LED panel 10 displays a CG image based on the position and orientation of the movie camera K1 in the AR marker coordinate system.

[0055] Notice, Figure 1 An embodiment in which the LED panel 10 displays one AR marker M1 is shown, but in reality, the LED panel 10 according to the present disclosure displays a plurality of AR markers M1. In addition, the coordinate values ​​of the coordinate system in the LED panel 10 (hereinafter referred to as the AR marker coordinate system) are defined for each of the plurality of AR markers M1. Note that the AR marker M1 does not have to be displayed on the LED panel 10 and can be set to, for example, printed materials. In this case, some other method is required to achieve conversion between the coordinate system of the CG image and the coordinate system of the AR marker M1.

[0056] (Cinema Camera K1)

[0057] The cine camera K1 according to the present disclosure is an embodiment of a first camera, and is a camera that captures images of objects adjacent to the LED panel 10. For example, the cine camera K1 captures an image of the AR marker M1 displayed on the LED panel 10 during calibration.

[0058] Furthermore, the cinema camera K1 captures an image of an object including a CG image displayed on the LED panel 10. Note that the information processing system according to the present disclosure may include another camera capable of capturing an image of the AR marker M1, instead of the cinema camera K1.

[0059] (Tracking Camera K2)

[0060] The tracking camera K2 according to the present disclosure is an embodiment of a second camera and is attached to the movie camera K1. For example, the tracking camera K2 includes an element capable of capturing infrared light images and captures images of an IR marker M2 placed on the ceiling of the imaging space. Note that the ceiling in the imaging space is an embodiment of a wall in that space.

[0061] The IR markers M2 are irregularly arranged on the ceiling in the imaging space. In addition, each IR marker M2 may be a retroreflective material, or may be a light that can emit infrared light itself.

[0062] Note that in the following description, the embodiment in which the IR markers M2 are arranged on the ceiling of the imaging space will be mainly described. However, the position of each IR marker M2 is not particularly limited as long as the IR markers M2 are arranged to surround the tracking camera K2 in the imaging space. For example, the IR markers M2 may be arranged on the floor or side walls of the imaging space.

[0063] (Information Processing Device 20)

[0064] The information processing device 20 according to the present disclosure is a device that performs various types of calibration processes. For example, the information processing device 20 may be a device such as Figure 1 A personal computer (PC) is shown, or may be another information terminal such as a tablet terminal or a smartphone.

[0065] For example, the information processing device 20 acquires relative position information about the relative positions of the movie camera K1 and the tracking camera K2 .

[0066] In addition, the information processing device 20 estimates various types of position information of the cinema camera K1, the tracking camera K2, and the IR marker M2 in the AR marker coordinate system based on the image data obtained as a result of capturing an image of the AR marker M1 by the cinema camera K1, the image data obtained as a result of capturing an image of the IR marker M2 by the tracking camera K2, and the relative position information of the cinema camera K1 and the tracking camera K2. The position information here can include information about position and direction. The detailed configuration of the information processing device 20 according to the present disclosure will be described later.

[0067] Furthermore, the information processing apparatus 20 may output various types of information regarding the calibration process to the display unit 21. For example, the user may check the calibration result displayed on the display unit 21.

[0068] The above has described the outline of the information processing system according to the present disclosure. Figure 2 A functional configuration embodiment of the information processing device 20 according to the present disclosure is described.

[0069] <1.2. Configuration Example>

[0070] Figure 2 1 is an explanatory diagram for describing an embodiment of the functional configuration of the information processing device 20 according to the present disclosure. Figure 2 As shown, the information processing device 20 according to the present disclosure includes a communication unit 210 , a storage unit 220 , and a control unit 230 .

[0071] (Communication Unit 210)

[0072] The communication unit 210 according to the present disclosure performs various types of communication with the cinema camera K1 and the tracking camera K2. For example, the communication unit 210 receives image data obtained by capturing an image of the AR marker M1 from the cinema camera K1. In addition, the communication unit 210 receives image data obtained by capturing an image of the IR marker M2 from the tracking camera K2.

[0073] Note that either the movie camera K1 or the tracking camera K2 may output image data acquired by itself to the other camera. In this case, the communication unit 210 may receive image data acquired as a result of the movie camera K1 capturing an image of the AR marker M1, and image data acquired as a result of the tracking camera K2 capturing an image of the IR marker M2, from either the movie camera K1 or the tracking camera K2.

[0074] (Storage Unit 220)

[0075] The storage unit 220 according to the present disclosure stores software and various data. For example, the storage unit 220 stores various types of image data received by the communication unit 210. In addition, the storage unit 220 can store various types of information, such as image data, an identification (ID) of the image data, the ID of each IR marker M2, the coordinate values ​​of the IR marker M2 in the image data, the ID of the AR marker M1, the coordinate values ​​of the AR marker M1, or the position information of the tracking camera K2 corresponding to the image data. In addition, the storage unit 220 can store observation values ​​of other types of sensors such as an inertial measurement unit (IMU).

[0076] (Control Unit 230)

[0077] The control unit 230 according to the present disclosure controls the overall operation of the information processing device 20. For example, the control unit 230 controls the transmission and reception of various types of information by the communication unit 210. Figure 2 As shown, the control unit 230 includes an estimation unit 231 .

[0078] {Estimation Unit 231}

[0079] The estimation unit 231 according to the present disclosure is an embodiment of an acquisition unit, and estimates relative position information of the cine camera K1 and the tracking camera K2 .

[0080] Furthermore, the estimation unit 231 is an example of a first estimation unit, and estimates position information of the cinema camera K1 in the AR marker coordinate system based on image data obtained as a result of capturing an image of the AR marker M1 displayed by the LED panel 10 by the cinema camera K1 .

[0081] Furthermore, the estimation unit 231 is an embodiment of a second estimation unit, and estimates the position information of the IR marker M2 and the tracking camera K2 in the AR marker coordinate system based on image data obtained as a result of capturing an image of the IR marker M2 arranged on the ceiling in the imaging space by the tracking camera K2, relative position information of the cinema camera K1 and the tracking camera K2, and position information of the cinema camera K1 in the AR marker coordinate system. Details of various types of processing performed by the estimation unit 231 will be described later.

[0082] The functional configuration example of the information processing device 20 according to the present embodiment has been described above. Figure 2 The described functional configuration is merely an example, and the functional configuration of the information processing apparatus 20 according to the present embodiment is not limited to such an example.

[0083] For example, the storage unit 220 according to the present disclosure may be provided in a device separate from the information processing device 20. In addition, the information processing device 20 according to the present embodiment may further include, for example, an operation unit that receives a user operation or the like.

[0084] Furthermore, some of the functions of the estimation unit 231 included in the information processing device 20 may be implemented by another device. For example, the cine camera K1 or the tracking camera K2 may have a functional configuration for estimating the relative position information of the cine camera K1 and the tracking camera K2. In this case, the communication unit 210 corresponds to an acquisition unit that acquires relative position information from the cine camera K1 or the tracking camera K2. Furthermore, another device (e.g., a server, etc.) may have a functional configuration corresponding to the first estimation unit or the second estimation unit.

[0085] The functional configuration of the information processing device 20 according to the present disclosure can be flexibly modified according to specifications, operations, etc. Figures 3 to 6 Details of various types of processing performed by the information processing system according to the present disclosure are described.

[0086] <1.3. Details>

[0087] Figure 3 This is an explanatory diagram for describing the overview of the calibration process of the information processing system. While the position and orientation of the cinema camera K1 can be expressed in the AR marker coordinate system, the position and orientation of the cinema camera K1 in the CG coordinate system can actually be used. Therefore, as described above, during VP, the LED panel 10 displays a CG image corresponding to the position and orientation of the cinema camera K1 in the AR marker coordinate system (i.e., the CG coordinate system).

[0088] In VP, CG images are rendered based on the position and orientation of the cinema camera K1 obtained by the tracking system, but in the initial state before the calibration process is performed, the AR marker coordinate system, the coordinate system of the cinema camera K1, and the coordinate system of the tracking camera K2 are independent of each other.

[0089] Therefore, in order to reflect the position and direction of the cinema camera K1 in the CG image, the AR marker coordinate system (which is the coordinate system attached to the LED panel 10 ) and the coordinate system of the tracking system used internally by the tracking system must be made common to each other.

[0090] As described above, in order to make the AR marker coordinate system and the tracking system coordinate system common to each other, for example, three independent calibration processes may be required as an initial configuration: map generation, mounting offset estimation, and LED volume alignment.

[0091] For example, the calibration of the LED volume alignment allows the coordinate system of the tracking system to be aligned to the LED panel coordinate system. Furthermore, the installation offset estimation allows the local coordinate system to be corrected using the coordinate system offset between the cine camera K1 and the tracking camera K2.

[0092] Figures 4A to 4C is an explanatory diagram for describing details related to the correction of the local coordinate system. For example, the user rotates the movie camera K1 and the tracking camera K2 around the pivot axis PP of the tracking camera K2 (e.g., the position where the tracking camera K2 is attached to the movie camera K1), as shown in FIG. Figure 4A In this case, Figure 4A As shown in the right figure of FIG, the optical center NP of the movie camera K1 moves in an arc.

[0093] Here, when the local coordinate systems of the movie camera K1 and the tracking camera K2 are different from each other, as Figure 4BAs shown, the optical center NP of the movie camera K1 is considered as a pure rotation, which is different from Figure 4A Actual rotation shown.

[0094] like Figure 4C As shown, in order to move the optical center NP of the cine camera in an arc in a manner similar to actual rotation, the local coordinate system needs to be converted by offset correction based on the relative positions of the cine camera K1 (optical center NP) and the tracking camera K2 (pivot PP).

[0095] This relative position of the cine camera K1 and the tracking camera K2 is estimated by a calibration process based on the estimation of the mounting offset.The details related to the calibration of the local coordinate system have been described above.

[0096] As described above, according to the comparative example, as an initial configuration, calibration related to map generation is first performed (first process), then calibration related to mounting offset estimation is performed (second process), and finally calibration related to LED volume alignment is performed (third process). However, this calibration process according to the comparative example has the problem of consuming a lot of time and effort.

[0097] Therefore, the information processing device 20 according to the present disclosure can reduce the time and effort required for calibration. Specifically, the information processing device 20 according to the present disclosure first performs calibration related to installation offset estimation using the AR marker M1 displayed on the LED panel 10, and then simultaneously performs the corresponding processing corresponding to calibration involving map generation and calibration corresponding to calibration involving LED volume alignment. The use of information related to this AR marker M1 makes it possible to reduce the time and effort required for calibration. Below, the details of each calibration process according to the present disclosure will be described sequentially.

[0098] (Installation offset estimate)

[0099] The estimation unit 231 according to the present disclosure estimates the relative positions of the movie camera K1 and the tracking camera K2. For example, the estimation unit 231 may estimate the relative positions of the movie camera K1 and the tracking camera K2 through hand-eye calibration.

[0100] Specifically, the estimation unit 231 can estimate the relative positions of the cine camera K1 and the tracking camera K2 using hand-eye calibration represented by the following equation (Expression 1).

[0101] AX=XB

[0102] (Expression 1)

[0103] A: Relative posture of tracking camera K2

[0104] B: Relative pose of movie camera K1

[0105] X: Installation offset

[0106] Here, the installation offset X in (Expression 1) corresponds to the relative position of cine camera K1 and tracking camera K2. That is, estimation unit 231 can estimate the relative position of cine camera K1 and tracking camera K2 by obtaining a six-degree-of-freedom coordinate transformation matrix using the relative pose of cine camera K1 and the relative pose of tracking camera K2 as input. Note that the technique used by estimation unit 231 to solve hand-eye calibration is not particularly limited, and known techniques can be used.

[0107] Note that hand-eye calibration can be considered a robotics problem. In robotics, it is primarily used to transform the coordinate system of a camera attached to the end of a movable arm included in the robot into the robot's coordinate system. Therefore, the position and orientation of the movable arm change over time, resulting in a single sample of observation points being used for coordinate transformation estimation.

[0108] On the other hand, the tracking camera K2 according to the present disclosure is attached to the cine camera K1. That is, since the positional relationship between the cine camera K1 and the tracking camera K2 is fixed, multiple samples with different times can be used in coordinate transformation estimation.

[0109] Furthermore, if the image generation of the tracking system is not completed, the six-degree-of-freedom (6DoF) pose of the tracking camera K2 is not obtained from the tracking system, making it necessary to obtain the relative pose of the tracking camera K2 from the image data obtained by the tracking camera K2. With these preconditions in mind, the calibration process related to the installation offset estimation performed by the estimation unit 231 will be described. In the following description, the image data obtained as a result of capturing an image of the AR marker M1 by the movie camera K1 may be represented as an AR marker image, and the image data obtained as a result of capturing an image of the IR marker M2 by the tracking camera K2 may be represented as an IR marker image.

[0110] Figure 5 First, at a first position X1, the cine camera K1 acquires a first AR marker image by capturing an image of the AR marker M1, and the tracking camera K2 acquires a first IR marker image by capturing an image of the IR marker M2.

[0111] The user then moves the cinema camera K1 and the tracking camera K2 from the first position X1 to the second position X2. For example, the user places the cinema camera K1 and the tracking camera K2 on a cart and moves the cart from the first position X1 to the second position X2. Note that the user can rotate the cinema camera K1 and the tracking camera K2 instead of translating them, or can translate them in addition to rotating them.

[0112] Then, at the second position X2 , the movie camera K1 acquires a second AR marker image by capturing an image of the AR marker M1 , and the tracking camera K2 acquires a second IR marker image by capturing an image of the IR marker M2 .

[0113] Here, the estimation unit 231 can estimate the position and orientation of the cinema camera K1 based on the AR marker image obtained as a result of imaging performed by the cinema camera K1. Therefore, the estimation unit 231 estimates the translational movement amount of the cinema camera K1 and the relative posture of the cinema camera K1 (the relative position and orientation of the cinema camera K1 between the first AR marker image and the second AR marker image) based on the image data pair including the first AR marker image and the second AR marker image.

[0114] Note that in order to obtain the installation offset with higher accuracy, the translational movement amount of the movie camera K1 is desirably greater than or equal to a predetermined value. Specifically, the translational movement amount of the movie camera K1 is desirably greater than or equal to 10% of the installation height of the IR marker M2.

[0115] Furthermore, the estimation unit 231 estimates three-dimensional positions (scale uncertainties) of at least five IR markers M2 included in the image data pair including the first IR marker image and the second IR marker image through a five-point algorithm based on the image data pair.

[0116] Furthermore, the user moves the movie camera K1 and the tracking camera K2 from the second position X2 to a third position (not shown).

[0117] Next, at the third position, the movie camera K1 acquires a third AR marker image by capturing an image of the AR marker M1 , and the tracking camera K2 acquires a third IR marker image by capturing an image of the IR marker M2 .

[0118] Then, the estimation unit 231 estimates the translational movement amount of the cinema camera K1 and the relative posture of the cinema camera K1 (the relative position and direction of the cinema camera K1 between the second AR marker image and the third AR marker image) based on the second AR marker image and the third AR marker image. Note that here, the translational movement amount of the cinema camera K1 is also desirably greater than or equal to a predetermined value.

[0119] In addition, the estimation unit 231 estimates the relative posture of the tracking camera K2 (the relative position and direction of the tracking camera K2 between the second IR marker image and the third IR marker image) through perspective n points (PnP) based on the image data pair of the second IR marker image and the third IR marker image and the three-dimensional position of the IR marker M2 estimated by the five-point algorithm.

[0120] Then, the estimation unit 231 can estimate the relative positions of the movie camera K1 and the tracking camera K2 by solving the hand-eye calibration based on seven-degree-of-freedom parameters, where the seven-degree-of-freedom parameters include six-degree-of-freedom parameters related to the positions and orientations of the movie camera K1 and the tracking camera K2 and a single-degree-of-freedom parameter related to scale invariance.

[0121] Note that in the above embodiment, the description has been made of an embodiment in which the cine camera K1 and the tracking camera K2 obtain three AR marker images and three IR marker images, respectively, and the estimation unit 231 estimates the relative pose of the cine camera K1 and the relative pose of the tracking camera K2 using two pairs of image data for each relative pose. However, the estimation unit 231 may estimate the relative pose of the cine camera K1 and the tracking camera K2 using four or more pieces of image data (in other words, three or more pairs of image data).

[0122] For example, the estimation unit 231 can estimate the relative pose of the tracking camera K2 through PnP based on an image data pair including a third IR marker image obtained at a third position and a fourth IR marker image obtained at a fourth position (another position after the cine camera K1 and the tracking camera K2 have moved further from the third position), and the three-dimensional position of the IR marker M2 estimated by the five-point algorithm. As described above, increasing the number of image data used to estimate the relative pose of the cine camera K1 and the relative pose of the tracking camera K2 (in other words, the number of image data pairs) allows for an increase in the estimation accuracy of the relative positions of the cine camera K1 and the tracking camera K2 estimated by hand-eye calibration.

[0123] The details of the installation offset estimation according to the present disclosure have been described above. According to the above installation offset estimation, the relative posture of the movie camera K1 can be obtained by using the AR marker M1, and the installation offset estimation can be simplified by applying the relative posture of the movie camera K1 and the tracking camera K2 to the hand-eye calibration. Therefore, the burden on the user when performing calibration can be reduced. Next, reference will be made to Figure 6 Details of the process of generating a map with pose priors including map generation and LED volume alignment are described.

[0124] (Processing of Generating a Graph with Pose Priors)

[0125] -First method

[0126] Figure 6 This is an explanatory diagram for describing an embodiment of a process for generating a graph with a pose prior. The estimation unit 231 according to the present disclosure estimates the position information (position and orientation) of the tracking camera K2 in the AR marker coordinate system and the position information (three-dimensional position) of the IR marker M2 in the AR marker coordinate system based on the AR marker image obtained as a result of capturing an image of the AR marker M1 by the cine camera K1, the IR marker image obtained as a result of capturing an image of the IR marker M2 by the tracking camera K2, and the relative position information of the cine camera K1 and the tracking camera K2.

[0127] For example, at a specific starting point location (e.g. Figure 6 The estimation unit 231 obtains the position information of the tracking camera K2 in the AR marker coordinate system at the starting point position based on the position information of the cinema camera K1 in the AR marker coordinate system estimated from the AR marker image and the relative position information of the cinema camera K1 and the tracking camera K2.

[0128] Then, the user moves the movie camera K1 and the tracking camera K2 from the starting point position to the moved position (for example, Figure 6 ). Then, at the moved position, an AR marker image is obtained as a result of imaging performed by the cinema camera K1, and an IR marker image is obtained as a result of imaging performed by the tracking camera K2. Here, the estimation unit 231 estimates the position information of the tracking camera K2 in the AR marker coordinate system at the moved position based on the position information of the cinema camera K1 in the AR marker coordinate system estimated from the AR marker image and the relative position information of the cinema camera K1 and the tracking camera K2.

[0129] Then, the estimation unit 231 estimates the relative movement amount from the starting point position to the moved position based on the estimation result of the position information of the two points of the tracking camera K2 in the AR marker coordinate system.

[0130] In addition, the estimation unit 231 can estimate the position information of the IR marker M2 included in the IR marker image pair in the AR marker coordinate system based on the two IR marker images (also called IR marker image pair) obtained at the starting point position and the moved position and the relative movement amount between the images included in the IR marker image pair.

[0131] More specifically, the estimation unit 231 can estimate the position information of the IR marker M2 included in the IR marker image pair through triangulation based on the relative movement amount between the IR marker image pair obtained at the starting point position and the moved position and the images included in the IR marker image pair.

[0132] Here, because the position and orientation of the tracking camera K2 have already been estimated in advance in the AR marker coordinate system, the position information of the IR marker M2 estimated by the estimation unit 231 is also estimated as the three-dimensional position of the IR marker M2 in the AR marker coordinate system. In other words, the above-mentioned process of generating a graph with a priori pose information makes it possible to omit the calibration process for the LED volume alignment.

[0133] Furthermore, the estimation unit 231 can estimate the relative movement of the tracking camera K2 based on the three-dimensional position (temporary position) of the IR marker M2 in the AR marker coordinate system using PnP. Furthermore, the estimation unit 231 can estimate the three-dimensional position of the IR marker M2 in the AR marker coordinate system based on the relative movement of the tracking camera K2 estimated using PnP. As described above, the estimation unit 231 alternately solves triangulation and PnP, making it possible to estimate the three-dimensional position of the IR marker M2 in the AR marker coordinate system with higher accuracy.

[0134] Note that in triangulation, the relative movement of the tracking camera K2 obtained by PnP is considered an input value, and in PnP, the three-dimensional position of the IR marker M2 obtained by triangulation is considered an input value. Therefore, there are cases where the accuracy of recovering the three-dimensional position of the IR marker M2 cannot be improved simply by alternately solving triangulation and PnP. In such cases, for example, the estimation unit 231 can treat the position and orientation of the tracking camera K2 in the AR marker coordinate system as constraints. This allows the estimation unit 231 to estimate the three-dimensional position of the IR marker M2 in the AR marker coordinate system with higher accuracy.

[0135] The user moves the movie camera K1 and the tracking camera K2 to each position in the imaging space and causes the movie camera K1 and the tracking camera K2 to capture an image of the AR marker M1 and an image of the IR marker M2 at each position. Then, when the estimation unit 231 repeatedly performs the process of estimating the three-dimensional position of the IR marker M2, the three-dimensional positions of all (or some) IR markers M2 arranged on the ceiling in the imaging space in the AR marker coordinate system can be restored.

[0136] Then, the estimation unit 231 can generate map information in the imaging space based on the three-dimensional positions of the plurality of IR markers M2 arranged on the ceiling in the imaging space in the AR marker coordinate system. For example, the estimation unit 231 can generate map information including the three-dimensional positions of all (or some) IR markers M2 arranged on the ceiling in the imaging environment in the AR marker coordinate system.

[0137] Here, since the positions and directions of the movie camera K1 and the tracking camera K2 in the AR marker coordinate system conform to the physical scale, the estimation unit 231 can generate map information with a known scale.

[0138] Furthermore, after the estimation unit 231 corrects the scale based on the image data first obtained by the tracking camera K2 and performs alignment in the AR marker coordinate system, the user can remove the tracking camera K2 from the cine camera K1. The user can then move around the entire imaging space using the removed tracking camera K2. Removing the tracking camera K2 from the cine camera K1 as described above allows for a reduction in weight compared to a case where both the cine camera K1 and the tracking camera K2 are moved, thereby also increasing user convenience.

[0139] - Second method

[0140] In the first method, an embodiment has been described in which the estimation unit 231 estimates the three-dimensional position of the IR marker M2 in the AR marker coordinate system by alternately solving triangulation and PnP, but the method in which the estimation unit 231 estimates the three-dimensional position of the IR marker M2 in the AR marker coordinate system is not limited to such an embodiment.

[0141] For example, the estimation unit 231 can perform processing related to the known corresponding point search on two pieces of image data obtained as a result of imaging performed by the tracking camera K2, and output information about the corresponding point pairs (the same IR marker M2) included in each of the two pieces of image data to the storage unit 220.

[0142] Next, the estimation unit 231 may estimate the three-dimensional position of the IR marker M2 in the AR marker coordinate system through triangulation based on the new corresponding point pair stored in the storage unit 220 , and output the estimated position to the storage unit 220 .

[0143] In addition, the estimation unit 231 can perform bundle adjustment as needed to correct the position and orientation of the tracking camera K2 or the three-dimensional position of the IR marker M2 in the AR marker coordinate system. Note that bundle adjustment here is an optimization process used to improve the accuracy of position estimation and is not necessarily performed. In addition, the estimation unit 231 can use sensing information obtained by another sensor (such as an inertial measurement unit (IMU)) as a constraint condition.

[0144] The details of various types of processing performed by the information processing system according to the present disclosure have been described above. As described above, in the information processing system according to the present disclosure, the position and direction of the movie camera K1 based on the AR marker M1 displayed on the LED panel 10 are used to estimate the position information of the tracking camera K2 and the IR marker M2 in the AR marker coordinate system, so that the work imposed on the user can be reduced and highly accurate and robust image information can be estimated. Next, reference will be made to Figure 7 and Figure 8 An embodiment of the operation process of the information processing apparatus 20 according to the present disclosure is described.

[0145] <<2. Example of Operation Process>>

[0146] (Overall processing)

[0147] Figure 7 First, the estimation unit 231 tracks the AR marker M1 based on the AR marker image obtained from the movie camera K1 and estimates the position information of the movie camera K1 in the AR marker coordinate system ( S101 ).

[0148] Next, the estimation unit 231 performs installation offset estimation based on the AR marker image obtained by the cine camera K1 and the IR marker image obtained by the tracking camera K2 , and acquires relative position information of the cine camera K1 and the tracking camera K2 ( S105 ).

[0149] Subsequently, the estimation unit 231 performs processing related to local coordinate system conversion between the cine camera K1 and the tracking camera K2 based on the relative position information to make the coordinate system of the tracking camera K2 and the coordinate system of the cine camera K1 common to each other ( S109 ).

[0150] Then, the estimation unit 231 performs a process of generating a graph with a priori poses to estimate the position information of the tracking camera K2 and the IR marker M2 in the AR marker coordinate system, and generates graph information of the imaging space in the AR marker coordinate system based on the estimation result (S113), and the estimation unit 231 according to the present disclosure ends the calibration process. Next, an embodiment of the installation offset estimation process in S105 will be described.

[0151] (Installation offset estimate)

[0152] Figure 8 is an explanatory diagram for describing an embodiment of the installation offset estimation process performed by the information processing apparatus 20 according to the present disclosure. First, the communication unit 210 receives an AR marker image from the movie camera K1 and an IR marker image from the tracking camera K2 ( S201 ).

[0153] Next, the estimation unit 231 determines whether the movie camera K1 and the tracking camera K2 have been sufficiently translated based on the AR marker image (S205). If sufficient translation has been performed (S205: Yes), the process proceeds to S209, and if sufficient translation has not been performed (S205: No), the movie camera K1 and the tracking camera K2 are moved by the user, and the process returns to S201 again.

[0154] Subsequently, the estimation unit 231 determines whether the five-point algorithm has been executed (S209). If the five-point algorithm has not been executed (S209: No), the process proceeds to S213, and if the five-point algorithm has been executed (S209: Yes), the process proceeds to S217.

[0155] In a case where the five-point algorithm has not been executed ( S209 : No), the estimation unit 231 estimates the three-dimensional position of the IR marker M2 included in the image data pair by the five-point algorithm based on the image data pair including two IR marker images obtained at two points ( S213 ).

[0156] In the case where the five-point algorithm has been executed (S209: Yes), the estimation unit 231 estimates the relative posture of the tracking camera K2 by perspective n points (PnP) based on the image data pair including two IR marker images obtained at two points and the three-dimensional position of the IR marker M2 estimated by the five-point algorithm (S217).

[0157] Then, the estimation unit 231 determines whether a predetermined amount of image data has been obtained (S221). If the predetermined amount of image data has been obtained (S221: Yes), the process proceeds to S225, and if the predetermined amount of image data has not been obtained (S221: No), the process returns to S201 again. Note that the predetermined value here is set to at least 3.

[0158] When a predetermined amount of image data has been obtained (S221: Yes), the estimation unit 231 performs hand-eye calibration based on the relative posture of the movie camera K1 and the relative posture of the tracking camera K2 to obtain the relative position information of the movie camera K1 and the tracking camera K2 (S225), and the information processing device 20 according to the present disclosure ends the installation offset estimation processing.

[0159] The embodiment of the installation offset estimation process has been described above. Note that the installation offset estimation according to the present disclosure is not limited to the technology based on hand-eye calibration. Figure 9 Modifications of the information processing system according to the present disclosure are described.

[0160] <<3. Modifications>>

[0161] Figure 9 2 is an explanatory diagram for describing a modified example of the information processing system according to the present disclosure. According to this modified example, a dedicated camera K3 for recognizing an AR marker M1 may be pre-attached to the tracking camera K2. The dedicated camera K3 is an embodiment of a third camera. For example, the relative position information of the tracking camera K2 and the dedicated camera K3 may be estimated in advance (e.g., before shipment from the factory), and the storage unit 220 may store the relative position information of the tracking camera K2 and the dedicated camera K3.

[0162] Therefore, the estimation unit 231 acquires the position information of the dedicated camera K3 in the AR marker coordinate system based on the AR marker image obtained as a result of capturing the image of the AR marker M1 by the dedicated camera K3. In addition, the estimation unit 231 can estimate the position information of the tracking camera K2 in the AR marker coordinate system based on the position information of the dedicated camera K3 in the AR marker coordinate system and the relative position information of the tracking camera K2 and the dedicated camera K3 stored in the storage unit 220.

[0163] The estimation unit 231 can then estimate the relative position information of the movie camera K1 and the tracking camera K2 based on the position information of the tracking camera K2 in the AR marker coordinate system and the position information of the movie camera K1 in the AR marker coordinate system, where the position information of the movie camera K1 in the AR marker coordinate system is estimated based on the AR marker image obtained as a result of capturing the image of the AR marker M1 by the movie camera K1.

[0164] A dedicated camera K3 for recognizing the AR marker M1 , whose position relative to the tracking camera K2 is known, is attached to the tracking camera K2 as described above, so that the installation offset estimation by hand-eye calibration as described above can be omitted.

[0165] <<4. Hardware Configuration Example>>

[0166] Next, a hardware configuration example of the information processing apparatus 90 according to an embodiment of the present disclosure will be described. Figure 10 1 is a block diagram showing an example of a hardware configuration of an information processing device 90 according to an embodiment of the present disclosure. The information processing device 90 may be a device having a hardware configuration equivalent to that of the information processing device 20.

[0167] The information processing device 90 includes, for example, Figure 10 The processor 871, read-only memory (ROM) 872, random access memory (RAM) 873, host bus 874, bridge 875, external bus 876, interface 877, input device 878, output device 879, storage device 880, driver 881, connection port 882 and communication device 883 are shown. Note that the hardware configuration shown here is an embodiment, and some components may be omitted. In addition, components other than those shown here may be included.

[0168] (Processor 871)

[0169] For example, the processor 871 functions as an arithmetic processing device or a control device, and controls the entire or partial operation of each component based on various programs recorded in the ROM 872 , the RAM 873 , the storage device 880 , or the removable storage medium 901 .

[0170] (ROM 872 and RAM 873)

[0171] The ROM 872 is a unit that stores programs read by the processor 871, data used for calculation, etc. The RAM 873 temporarily or permanently stores, for example, programs read by the processor 871, various parameters that appropriately change when the programs are executed, and the like.

[0172] (Host Bus 874, Bridge 875, External Bus 876, and Interface 877)

[0173] The processor 871, the ROM 872, and the RAM 873 are connected to each other via, for example, a host bus 874 capable of high-speed data transmission. On the other hand, for example, the host bus 874 is connected to an external bus 876 having a relatively low data transmission speed via a bridge 875. In addition, the external bus 876 is connected to various components via an interface 877.

[0174] (Input device 878)

[0175] For example, a mouse, keyboard, touch panel, button, switch, joystick, etc. are used as the input device 878. In addition, a remote controller (hereinafter, referred to as a remote controller) capable of transmitting a control signal using infrared rays or other radio waves can be used as the input device 878. In addition, the input device 878 includes a voice input device such as a microphone.

[0176] (Output device 879)

[0177] The output device 879 is a device capable of visually or auditorily notifying the user of acquired information, such as a display device (such as a cathode ray tube (CRT), LCD, or organic EL), an audio output device (such as a speaker or headphones), a printer, a mobile phone, or a fax. In addition, the output device 879 according to the present disclosure includes various vibration devices capable of outputting tactile stimulation.

[0178] (Storage device 880)

[0179] The storage device 880 is a device for storing various data and can be, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0180] (Driver 881)

[0181] For example, the drive 881 is a device that reads information recorded on the removable storage medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable storage medium 901 .

[0182] (Removable Storage Medium 901)

[0183] For example, the removable storage medium 901 is a DVD medium, a Blu-ray (registered trademark) medium, an HDDVD medium, various semiconductor storage media, etc. Needless to say, for example, the removable storage medium 901 may be an IC card having a non-contact IC chip mounted thereon, an electronic device, etc.

[0184] (Connection port 882)

[0185] The connection port 882 is a port for connecting the external connection device 902 such as a universal serial bus (USB) port, an IEEE 1394 port, a small computer system interface (SCSI), an RS-232C port, or an optical audio terminal, for example.

[0186] (External connection device 902)

[0187] For example, the externally connected device 902 is a printer, a portable music player, a digital camera, a digital video camera, an IC recorder, or the like.

[0188] (communication device 883)

[0189] The communication device 883 is a communication device for connecting to a network, such as a wired or wireless LAN, Bluetooth (registered trademark) or a communication card for wireless USB (WUSB), a router for optical communication, a router for asymmetric digital subscriber line (ADSL) or a modem for various communications, etc.

[0190] 5. Conclusion

[0191] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such embodiments. Obviously, a person skilled in the art with ordinary knowledge in the technical field of the present disclosure can design various modifications or variations within the scope of the technical concept described in the claims, and it should be naturally understood that these modifications or variations also fall within the technical scope of the present disclosure.

[0192] For example, each step related to the processing described in the present disclosure does not necessarily need to be processed in a time series in the order described in the flowchart or sequence diagram. For example, each step related to the processing of each device may be processed in an order different from the described order, or may be processed in parallel.

[0193] Furthermore, the series of processes performed by each device described in this disclosure can be implemented by a program stored in a non-transitory computer-readable storage medium. For example, when a computer executes the program, each program is read into RAM and executed by a processor such as a CPU. Examples of storage media include magnetic disks, optical disks, magneto-optical disks, flash memory, and the like. Furthermore, the program can be distributed via, for example, a network without using a storage medium.

[0194] In addition, the effects described herein are merely exemplary or illustrative, and not restrictive. That is, in addition to or in place of the above effects, the technology according to the present disclosure can provide the above other effects that are obvious to those skilled in the art from the description of this specification.

[0195] (1) An information processing device comprising:

[0196] The circuit is configured as:

[0197] Obtain at least one first image of the display screen, where the at least one first image is acquired by a first camera;

[0198] obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera;

[0199] estimating first position information of the first camera relative to the display screen based on the at least one first image;

[0200] Obtaining offset information between the first camera and the second camera; and

[0201] estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image, and the offset information,

[0202] The positional relationship between the first camera and the second camera is fixed.

[0203] (2) The information processing device according to (1), wherein the circuit is further configured to control output of a display image of the display screen according to the position of the first camera.

[0204] (3) An information processing device according to (1) or (2), wherein the circuit is further configured to control output of a display image of a display screen during virtual production, in which the display image output by the display screen is included in an image acquired by the first camera.

[0205] (4) The information processing apparatus according to any one of (1) to (3), wherein at least one first image of the display screen includes a display mark.

[0206] (5) The information processing apparatus according to any one of (1) to (4), wherein the display marker includes an augmented reality marker.

[0207] (6) The information processing apparatus according to any one of (1) to (5), wherein the display mark is displayed at known coordinates in a coordinate system of the display screen.

[0208] (7) The information processing device according to any one of (1) to (6), wherein the circuit is configured to estimate the first position information of the first camera relative to the display screen using known coordinates of the display marker.

[0209] (8) The information processing device according to any one of (1) to (7),

[0210] The circuit is further configured to control the output of the display image of the display screen according to the position and direction of the first camera in the coordinate system of the display screen.

[0211] (9) An information processing device according to any one of (1) to (8), wherein the circuit is configured to obtain a plurality of first images of a display screen, and wherein the circuit is configured to obtain a plurality of second images of at least one mark corresponding to the plurality of first images.

[0212] (10) An information processing device according to any one of (1) to (9), wherein a plurality of first images of the display screen are acquired by a first camera from a position corresponding to the first position information, and a plurality of second images are acquired by a second camera from a position corresponding to the second position information.

[0213] (11) An information processing device according to any one of (1) to (10), wherein the plurality of first images are acquired by the first camera from a plurality of positions, and the plurality of second images are acquired by the second camera from a plurality of positions corresponding to the plurality of positions of the first camera.

[0214] (12) The information processing apparatus according to any one of (1) to (11), wherein at least one marker is a retroreflective material, and wherein the second camera is an infrared camera.

[0215] (13) The information processing device according to any one of (1) to (12), wherein the circuit is configured to estimate second position information of the second camera relative to the display screen based on the first position information and the offset information.

[0216] (14) The information processing device according to any one of (1) to (13), wherein the circuit is configured to estimate third position information of at least one marker relative to the display screen based on at least one second image and the second position information.

[0217] (15) An information processing device according to any one of (1) to (14), wherein the at least one marker includes a plurality of markers arranged around the second camera in the imaging space, and wherein the circuit is further configured to generate a map of the plurality of markers based on the third position information.

[0218] (16) An information processing device according to any one of (1) to (15), wherein the obtained offset information is obtained based on predetermined offset information between the second camera and the third camera, and wherein the positional relationship between the second camera and the third camera is fixed.

[0219] (17) The information processing device according to any one of (1) to (16), wherein the circuit is configured to estimate the offset information by performing calibration based on a plurality of relative postures of the first camera and the second camera.

[0220] (18) An information processing device according to any one of (1) to (17), wherein the circuit is configured to estimate offset information by solving hand-eye calibration based on seven-degree-of-freedom parameters, the seven-degree-of-freedom parameters including six-degree-of-freedom parameters related to the relative position and orientation of the first camera and the second camera, and a single-degree-of-freedom parameter related to scale invariance.

[0221] (19) An information processing method comprising:

[0222] Obtain at least one first image of the display screen, where the at least one first image is acquired by a first camera;

[0223] obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera;

[0224] estimating first position information of the first camera relative to the display screen based on the at least one first image;

[0225] Obtaining offset information between the first camera and the second camera; and

[0226] estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image, and the offset information,

[0227] The positional relationship between the first camera and the second camera is fixed.

[0228] (20) A non-transitory computer-readable medium having a program embodied thereon, which, when executed by a computer, causes the computer to perform an information processing method, the method comprising:

[0229] Obtain at least one first image of the display screen, where the at least one first image is acquired by a first camera;

[0230] obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera;

[0231] estimating first position information of the first camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the at least one first image;

[0232] Obtaining offset information between the first camera and the second camera; and

[0233] estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image, and the offset information,

[0234] The positional relationship between the first camera and the second camera is fixed.

[0235] (21) An information processing device comprising:

[0236] an acquiring unit that acquires relative position information about the relative positions of the first camera and a second camera attached to the first camera;

[0237] a first estimating unit that estimates position information of the first camera in a coordinate system of the display device arranged in a specific space, based on image data obtained as a result of capturing an image of the first marker displayed by the display device by the first camera; and

[0238] A second estimation unit estimates position information of a second marker arranged on a wall in a space and position information of the second camera in the coordinate system of the display device based on image data obtained as a result of capturing an image of the second marker by the second camera, relative position information, and position information of the first camera in the coordinate system of the display device.

[0239] (22) The information processing device according to (21),

[0240] The acquiring unit acquires relative position information based on the relative posture of the first camera and the relative posture of the second camera.

[0241] (23) The information processing device according to (22),

[0242] wherein the relative pose of the first camera is based on at least two image data pairs obtained as a result of capturing images of the first marker at at least three positions in space by the first camera, and

[0243] The relative pose of the second camera is based on at least two image data pairs obtained as a result of capturing an image of the second marker by the second camera at a position corresponding to a time when the first camera captured the image of the first marker.

[0244] (24) The information processing device according to (23),

[0245] The acquisition unit acquires the three-dimensional positions of the at least five second markers based on an image data pair obtained as a result of capturing images of the at least five second markers at a first position and a second position in space by the second camera, and acquires the relative posture of the second camera based on the three-dimensional positions of the at least five second markers and the image data pair obtained as a result of capturing images of the at least five second markers at a second position and a third position in space by the second camera.

[0246] (25) The information processing device according to (24),

[0247] The acquisition unit further acquires the relative posture of the second camera based on at least one image data pair obtained as a result of capturing images of at least five second markers at a third position and another position or multiple other positions in space by the second camera.

[0248] (26) The information processing device according to (23),

[0249] The first marker is an AR marker whose coordinate value is defined in the coordinate system of the display device.

[0250] (27) The information processing device according to (23),

[0251] The acquisition unit acquires relative position information based on seven-degree-of-freedom parameters, where the seven-degree-of-freedom parameters include six-degree-of-freedom parameters related to position and direction and a single-degree-of-freedom parameter related to scale invariance.

[0252] (28) The information processing device according to (21),

[0253] The acquisition unit acquires relative position information about the relative position of the first camera and the second camera based on image data obtained as a result of capturing an image of the first marker by a third camera, the third camera being pre-attached to the second camera and having a relative positional relationship with the second camera pre-registered.

[0254] (29) The information processing device according to any one of (21) to (28),

[0255] Wherein, the second estimation unit estimates the position information of the second camera in the coordinate system of the display device based on the position information and relative position information of the first camera in the coordinate system of the display device, and estimates the position information of the second marker included in the image data pair in the coordinate system of the display device based on the position information of the second camera in the coordinate system of the display device and the image data pair obtained as a result of imaging performed by the second camera.

[0256] (30) The information processing device according to (29),

[0257] Here, the second estimation unit estimates position information of the second marker in the coordinate system of the display device based on a pair of image data obtained as a result of imaging performed at different positions in space by the second camera.

[0258] (31) The information processing device according to (30),

[0259] In which, the second estimation unit estimates temporary position information of the second marker in the coordinate system of the display device based on an image data pair obtained as a result of imaging performed by the second camera at different positions in space, estimates the relative movement amount of the second camera based on the temporary position information, and estimates the position information of the second marker in the coordinate system of the display device based on the relative movement amount.

[0260] (32) The information processing device according to any one of (21) to (28),

[0261] In which, the second estimation unit performs processing related to corresponding point search on an image data pair obtained as a result of imaging performed by a second camera at different positions in space, estimates a corresponding point pair indicating the same second mark included in each of the two image data included in the image data pair, and estimates position information of the second mark in the coordinate system of the display device based on the corresponding point pair.

[0262] (33) The information processing device according to any one of (21) to (32),

[0263] The second estimating unit estimates the graph information in the space based on position information of a plurality of second markers arranged in the space in a coordinate system of the display device.

[0264] (34) The information processing device according to any one of (21) to (33),

[0265] Wherein, the second mark comprises a retroreflective mark.

[0266] (35) An information processing method executed by a computer, the information processing method comprising:

[0267] acquiring relative position information about relative positions of a first camera and a second camera attached to the first camera;

[0268] estimating position information of the first camera in a coordinate system of the display device arranged in a specific space based on image data obtained as a result of capturing an image of the first marker displayed by the display device through the first camera; and

[0269] Based on the image data obtained as a result of capturing an image of the second marker by the second camera, the relative position information, and the position information of the first camera in the coordinate system of the display device, the position information of the second marker arranged on the wall in the space and the position information of the second camera in the coordinate system of the display device are estimated.

[0270] (36) A program for causing a computer to function as an information processing device, the information processing device comprising:

[0271] an acquiring unit that acquires relative position information about the relative positions of the first camera and a second camera attached to the first camera;

[0272] a first estimating unit that estimates position information of the first camera in a coordinate system of the display device arranged in a specific space, based on image data obtained as a result of capturing an image of the first marker displayed by the display device by the first camera; and

[0273] A second estimation unit estimates position information of a second marker arranged on a wall in a space and position information of the second camera in the coordinate system of the display device based on image data obtained as a result of capturing an image of the second marker by the second camera, relative position information, and position information of the first camera in the coordinate system of the display device.

[0274] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0275] [Reference Numbers List] 10 LED Panel

[0276] 20 Information processing device 21 Display unit

[0277] 210 Communication Unit

[0278] 220 storage units

[0279] 230 Control unit 231 Estimation unit K1 Cine camera K2 Tracking camera K3 Special camera M1 AR marker M2 IR marker

Claims

1. An information processing device, comprising: The circuit is configured as: obtaining at least one first image of a display screen, wherein the at least one first image is acquired by a first camera; obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera; estimating first position information of the first camera relative to the display screen based on the at least one first image; Obtaining offset information between the first camera and the second camera; and estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image, and the offset information, The positional relationship between the first camera and the second camera is fixed.

2. The information processing device according to claim 1, in, The circuit is further configured to control output of a display image of the display screen according to the position of the first camera.

3. The information processing device according to claim 2, in, The circuit is further configured to control output of the display image of the display screen during virtual production in which the display image output by the display screen is included in an image acquired by the first camera.

4. The information processing device according to claim 1, in, The at least one first image of the display screen includes a display indicia.

5. The information processing device according to claim 4, in, The display marker includes an augmented reality marker.

6. The information processing device according to claim 4, in, The display mark is displayed at known coordinates in the coordinate system of the display screen.

7. The information processing device according to claim 6, in, The circuit is configured to estimate the first position information of the first camera relative to the display screen using the known coordinates of the display marker.

8. The information processing device according to claim 6, in, The circuit is further configured to control output of a display image of the display screen according to a position and an orientation of the first camera in the coordinate system of the display screen.

9. The information processing device according to claim 1, in, The circuitry is configured to obtain a plurality of first images of the display screen, and wherein the circuitry is configured to obtain a plurality of second images of the at least one mark corresponding to the plurality of first images.

10. The information processing device according to claim 9, in, The plurality of first images of the display screen are acquired by the first camera from positions corresponding to the first position information, and the plurality of second images are acquired by the second camera from positions corresponding to the second position information.

11. The information processing device according to claim 9, in, The plurality of first images are acquired by the first camera from a plurality of positions, and the plurality of second images are acquired by the second camera from a plurality of positions corresponding to the plurality of positions of the first camera.

12. The information processing device according to claim 1, in, The at least one indicia is a retroreflective material, and Wherein, the second camera is an infrared camera.

13. The information processing device according to claim 1, in, The circuit is configured to estimate the second position information of the second camera relative to the display screen based on the first position information and the offset information.

14. The information processing device according to claim 13, in, The circuit is configured to estimate the third position information of the at least one marker relative to the display screen based on the at least one second image and the second position information.

15. The information processing device according to claim 14, in, The at least one marker includes a plurality of markers arranged around the second camera in the imaging space, and The circuit is further configured to generate a map of the plurality of markers based on the third position information.

16. The information processing device according to claim 1, in, The obtained offset information is obtained based on predetermined offset information between the second camera and the third camera, and The positional relationship between the second camera and the third camera is fixed.

17. The information processing device according to claim 1, in, The circuit is configured to estimate the offset information by performing calibration based on a plurality of relative poses of the first camera and the second camera.

18. The information processing device according to claim 1, in, The circuit is configured to estimate the offset information by solving hand-eye calibration based on seven-degree-of-freedom parameters, wherein the seven-degree-of-freedom parameters include: six degrees of freedom parameters related to the relative position and orientation of the first camera and the second camera, and Single degree of freedom parameters related to scale invariance.

19. An information processing method, comprising: obtaining at least one first image of a display screen, wherein the at least one first image is acquired by a first camera; obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera; estimating first position information of the first camera relative to the display screen based on the at least one first image; Obtaining offset information between the first camera and the second camera; and estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image, and the offset information, The positional relationship between the first camera and the second camera is fixed.

20. A non-transitory computer-readable medium having a program embodied thereon, wherein when the program is executed by a computer, the computer is caused to perform an information processing method, the method comprising: obtaining at least one first image of a display screen, wherein the at least one first image is acquired by a first camera; obtaining at least one second image of at least one marker in an imaging space around the display screen, the at least one second image being acquired by a second camera; estimating first position information of the first camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the at least one first image; Obtaining offset information between the first camera and the second camera; and estimating second position information of the second camera relative to the display screen and third position information of the at least one marker relative to the display screen based on the first position information, the at least one second image, and the offset information, The positional relationship between the first camera and the second camera is fixed.

Images