Information processing device and information processing method

Abstract

To obtain information representing a position in the virtual space of a camera for imaging a real space, to estimate a three-dimensional space from a two-dimensional image, without using an installation object such as a marker.SOLUTION: An information processing device 10 includes: a feature point extraction part 14 for extracting a feature point from a real space image obtained by imaging a real space and at least one virtual space image; a feature point matching part 15 for matching the feature point of the real space image with the feature point of the virtual space image; a parameter calculation part 16 for calculating a conversion parameter on imaging of the real space by substituting a reference space coordinate in the virtual space of a matched reference feature point and the reference image coordinate of a reference feature point in the real space image for the conversion formula representing a relation between the three-dimensional coordinate and two-dimensional coordinate of a specific point by using the conversion parameter; and an output part 17 for outputting the conversion parameter on the imaging of the real space as camera information representing a position in the virtual space of a camera imaging the real space image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] There is a demand for a technology that can estimate a three-dimensional space based on a two-dimensional image. To achieve this, it is necessary to determine the positional relationship between an image captured by a camera of a real space and a three-dimensional virtual space corresponding to the virtual space. There is a technology that determines the positional relationship between a camera image and a three-dimensional space by placing objects, such as markers, whose positions are known, in the real space and capturing an image including the markers. Furthermore, for example, Patent Document 1 listed below describes a technology related to learning a model for estimating the position, orientation, etc. of an object from an image of real space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-81081 Summary of the Invention [Problem to be solved by the invention]

[0004] As described above, to estimate a three-dimensional space from a two-dimensional image, it is necessary to determine various parameters related to the projection of the three-dimensional space onto the two-dimensional image, as well as the correspondence between coordinates in the two-dimensional image and coordinates in the three-dimensional space. However, markers or the like whose coordinates in the three-dimensional space are known must be installed in advance, and installing the markers in the target three-dimensional space in advance is time-consuming. Furthermore, it is impossible to determine the various parameters and the correspondence between coordinates in the two-dimensional image and coordinates in the three-dimensional space from past images or the like in which no markers are installed.

[0005] Therefore, the present invention has been made in consideration of the above problems, and aims to obtain information representing the position in virtual space of a camera that captures real space in order to estimate three-dimensional space from two-dimensional images without using any installed objects such as markers. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing device according to one aspect of the present disclosure includes a feature point extraction unit that extracts feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space that corresponds to the real space and is represented by a virtual space model captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching unit that matches feature points of the real space image with feature points of the virtual space image; and a feature point matching unit that matches reference space coordinates in the virtual space of a reference feature point that is a matched feature point, and a reference feature point that is the coordinate of the reference feature point in the real space image. the parameter calculation unit calculates transformation parameters for capturing the real space by substituting the quasi-image coordinates into a predetermined transformation formula that expresses, using predetermined transformation parameters, the relationship between the three-dimensional coordinates of a specific point that is a specific point in three-dimensional space and the two-dimensional coordinates of the specific point in a captured image that captures the three-dimensional space, wherein the reference space coordinates are calculated based on the transformation parameters for capturing the virtual space by the virtual camera and virtual reference image coordinates that are the coordinates of a reference feature point in the virtual space image; and an output unit that outputs the transformation parameters for capturing the real space as camera information that represents the position in the virtual space of the camera that captured the real space image.

[0007] According to the above aspect, feature points extracted from each of the real space image and the virtual space image are matched as reference feature points, and the two-dimensional reference image coordinates of the reference feature point in the real space image are associated with the three-dimensional reference space coordinates of the reference feature point in the virtual space, thereby enabling calculation of transformation parameters in the transformation formula. Since the calculated transformation parameters constitute information representing the relative relationship between a position in the real space image and a position in the virtual space, it is possible to obtain information representing the position in the virtual space of the camera that captures the real space from camera information consisting of the transformation parameters. [Effects of the Invention]

[0008] Since three-dimensional space is estimated from two-dimensional images without using any installed objects such as markers, it is possible to obtain information representing the position in virtual space of the camera capturing the real space. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the functional configuration of an information processing system and an information processing device according to an embodiment of the present invention; [Figure 2] FIG. 10 is a diagram illustrating an example of generating a virtual space model from a real space image. [Figure 3] Fig. 3(a) is a diagram showing a first example of the installation of a virtual camera in a virtual space, and Fig. 3(b) is a diagram showing a second example of the installation of a virtual camera in a virtual space. [Figure 4] 1A to 1C are diagrams illustrating an example of a virtual space image, an example of a real space image, and an example of extraction and matching processing of feature points. [Figure 5] 10A and 10B are diagrams illustrating an example of a calculation process of reference space coordinates, which are coordinates of a reference feature point in a virtual space. [Figure 6] FIG. 10 is a diagram illustrating an example of re-installation of the virtual camera. [Figure 7] 10A and 10B are diagrams illustrating an example of how to superimpose the imaging areas of a virtual space image captured by a specific virtual camera and a virtual space image captured by a virtual camera to be reinstalled. [Figure 8]10 is a flowchart showing the processing content of an information processing method in the information processing device. [Figure 9] FIG. 2 is a diagram showing a configuration of an information processing program. [Figure 10] FIG. 2 is a hardware block diagram of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of an information processing system according to the present invention will be described with reference to the drawings. Whenever possible, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] 1 is a diagram showing the functional configuration of an information processing system and an information processing device according to this embodiment. The information processing system 1 of this embodiment is a system that obtains information representing the position in a virtual space of a camera that captures an image of a real space in order to estimate a three-dimensional space from a two-dimensional image without using any installed objects such as markers, and is configured by an information processing device 10, as an example.

[0012] 1, the information processing device 10 functionally comprises a real space image acquisition unit 11, a setting unit 12, a virtual space image acquisition unit 13, a feature point extraction unit 14, a feature point matching unit 15, a parameter calculation unit 16, an output unit 17, a resetting unit 18, and a position calculation unit 19. These functional units 11 to 19 may be configured in one device as exemplified in FIG. 1, or may be configured as being distributed across multiple devices.

[0013] Each of the functional units 11 to 19 of the information processing device 10 is configured to be able to access storage means (storage) such as a real space image storage unit 21 and a virtual space model storage unit 22. Each of the storage units 21 to 22 may be provided in the information processing device 10, or may be configured in another device configured to be able to be accessed from the information processing device 10, as exemplified in FIG.

[0014] Next, a description will be given of each functional unit of information processing device 10. Real space image acquisition unit 11 acquires a real space image obtained by capturing a real space. Specifically, real space image acquisition unit 11 may acquire the real space image from a camera that captures the real space, or may acquire the real space image from real space image storage unit 21, which is storage means that stores the captured real space image.

[0015] Fig. 2 is a diagram showing an example of generating a virtual space model from a real space image. As shown in Fig. 2, a real space image acquisition unit 11 acquires a real space image rs. Also, as shown in Fig. 2, a virtual space model vm representing a virtual space corresponding to the real space is generated based on the real space image rs.

[0016] The virtual space model vm may be generated based on the real space image rs by any known method. For example, the virtual space model vm may be generated based on the distance to an object measured by LiDAR (Light Detection and Ranging) when capturing the real space image. Alternatively, the virtual space model vm may be generated by estimating the depth of each pixel of the two-dimensional real space image rs using known technology, converting the depth and color value of each pixel into point cloud data, and generating a three-dimensional virtual space by three-dimensional display based on the point cloud data.

[0017] The generated virtual space model vm may be stored in the virtual space model storage unit 22. The virtual space model storage unit 22 is a storage unit that stores the virtual space model vm that has been generated in advance.

[0018] The setting unit 12 sets up a virtual camera in the virtual space represented by the virtual space model vm. Specifically, the setting unit 12 sets the position of the virtual camera in the virtual space represented by the virtual space model vm. The position of the virtual camera is a viewpoint position for capturing a virtual space image, which is an image obtained by projecting the virtual space model vm.

[0019] 3A and 3B show examples of installation positions of virtual cameras in a virtual space, and Fig. 3A and Fig. 3B show first and second examples of installation positions of virtual cameras in the virtual space as viewed from above the virtual space vs. As shown in Fig. 3A and Fig. 3B, the setting unit 12 may set multiple virtual cameras vc in the virtual space vs.

[0020] In the first example shown in Fig. 3(a), the setting unit 12 may install multiple virtual cameras vc along the periphery of the virtual space vs, with the normal direction of the periphery as the imaging direction. Alternatively, as shown in Fig. 3(b), the setting unit 12 may install multiple virtual cameras vc within the virtual space vs (for example, around the center of the virtual space vs) with the periphery direction of the virtual space vs as the imaging direction. By installing multiple virtual cameras vc in this way, it is possible to comprehensively capture images of the virtual space corresponding to the space represented in the real-space image rp.

[0021] The virtual space image acquisition unit 13 acquires at least one virtual space image captured by the virtual camera vc of the virtual space vs. The virtual space image acquisition unit 13 may acquire the virtual space image by a known method based on the virtual space vs represented by the virtual space model vm.

[0022] 4 is a diagram showing an example of a virtual space image, an example of a real space image, and an example of feature point extraction and matching processing. Specifically, the virtual space image acquisition unit 13 acquires a virtual space image vp by projecting the virtual space vs onto a virtual screen, using the position of a virtual camera vc set in the virtual space vs represented based on the virtual space model vm as the viewpoint position.

[0023] The feature point extraction unit 14 extracts feature points from a real space image rp captured from a real space rs and at least one virtual space image vp. The feature point matching unit 15 matches the feature points of the real space image rp with the feature points of the virtual space image vp. The feature point extraction unit 14 and the feature point matching unit 15 may extract and match feature points using known methods, such as SIFT and AKAZE.

[0024] 4, the feature point extraction unit 14 extracts feature points vfp1 and vfp2 from the virtual space image vp. The feature point extraction unit 14 also extracts feature points rfp1 and rfp2 from the real space image rp. The feature points are extracted, for example, based on the difference in pixel values ​​between adjacent pixels in the image, and for example, the corners and end points of an object represented in the image are extracted as feature points.

[0025] Furthermore, the feature point matching unit 15 matches each of the feature points vfp1 and vfp2 of the virtual space image vp with each of the feature points rfp1 and rfp2 of the real space image rp, as indicated by the symbol fm, based on the feature amount of each feature point.

[0026] The parameter calculation unit 16 calculates transformation parameters related to imaging of the real space by substituting, into a predetermined transformation formula, reference space coordinates, which are the three-dimensional coordinates in the virtual space vs of the reference feature point, which is the matched feature point, and reference image coordinates, which are the coordinates of the reference feature point in the real space image rp.

[0027] The transformation formula is a formula that uses predetermined transformation parameters to express the relationship between the three-dimensional coordinates of a specific point in a three-dimensional space and the two-dimensional coordinates of the specific point in a captured image of the three-dimensional space. If the three-dimensional coordinates in the three-dimensional space are (X, Y, Z) and the two-dimensional coordinates in the two-dimensional image are (u, v), an example of the predetermined transformation formula is expressed by the following transformation formula (1):

number

[0028] Then, assuming that the three-dimensional reference space coordinate is t, the virtual reference image coordinate, which is the two-dimensional coordinate of the reference feature point in the virtual space image vp, is d', and the transformation matrix that is expressed by the transformation parameters of transformation equation (1) and transforms the reference space coordinate t into the virtual reference image coordinate d' is A, the relationship between these coordinates is expressed by the following equation (2).

[0029] d'=tA (2) Furthermore, if the two-dimensional reference image coordinates are d and the transformation matrix that transforms the reference space coordinates t, expressed by the transformation parameters of transformation equation (1), into the reference image coordinates d is B, the relationship between these coordinates is expressed as in the following equation (3).

[0030] d=tB (3) As shown in equation (2), the transformation matrix A is a transformation matrix (transformation parameter) that indicates the relationship between three-dimensional coordinates indicating a position in the virtual space vs and two-dimensional coordinates of the position in an image captured by the virtual camera vc placed in the virtual space vs, and is therefore known. Therefore, the parameter calculation unit 16 can calculate the reference space coordinate t based on the transformation matrix A and the virtual reference image coordinate d'.

[0031] Specifically, the parameter calculation unit 16 may calculate the reference space coordinate t using a technique known as hit determination. Fig. 5 is a diagram schematically showing an example of a calculation process of the reference space coordinate by hit determination.

[0032] 5, since the transformation matrix A is known and therefore the position of the virtual camera vc is also known, the parameter calculation unit 16 can calculate a three-dimensional straight line ln that passes through the virtual reference image coordinate d' of the reference feature point fp in the virtual space image vp.The parameter calculation unit 16 then calculates the point where the plane (mesh) that constitutes the virtual space model vm intersects with the straight line ln as the reference space coordinate t of the reference feature point fp.

[0033] The parameter calculation unit 16 calculates the transformation matrix B in equation (3) as a transformation parameter related to imaging of the real space, based on the reference space coordinate t and the reference image coordinate d of the reference feature point fp.

[0034] The setting unit 12 may set up multiple virtual cameras vc in the virtual space vs, as in the example described with reference to Figures 3(a) and 3(b). Then, the virtual space image acquisition unit 13 acquires multiple virtual space images vp acquired by each of the multiple virtual cameras vc.

[0035] The parameter calculation unit 16 may calculate the transformation parameters related to imaging of the real space based on at least one virtual space image vp selected from among the plurality of virtual space images vp based on the number of reference feature points fp.

[0036] Since the number of transformation parameters in transformation formula (1) constituting transformation matrix B is 15, the parameter calculation unit 16 constructs transformation formula (1) using the reference space coordinates t and reference image coordinates d of at least eight reference feature points fp, and calculates the transformation parameters by solving the constructed multiple transformation formulas as equations. Therefore, the parameter calculation unit 16 may calculate transformation parameters related to imaging of real space based on a virtual space image vp having a number of reference feature points fp equal to or greater than a predetermined threshold.

[0037] Specifically, the parameter calculation unit 16 may calculate the transformation parameters related to imaging of the real space using a virtual space image vp having eight or more reference feature points fp. Alternatively, the parameter calculation unit 16 may calculate the transformation parameters related to imaging of the real space using a plurality of virtual space images vp in which the total number of reference feature points fp is equal to or greater than a predetermined threshold.

[0038] Furthermore, the parameter calculation unit 16 may calculate transformation parameters related to capturing real space based on each of the multiple virtual space images vp, thereby calculating transformation parameters associated with each virtual space image vp.

[0039] The output unit 17 outputs the transformation parameters (corresponding to transformation matrix B) related to capturing the real space calculated by the parameter calculation unit 16 as camera information that essentially represents the position in the virtual space of the camera that captured the real space image.

[0040] Furthermore, the output unit 17 may process the calculated transformation parameters using a predetermined statistical method and output the resulting transformation parameters as the camera information. Specifically, the output unit 17 may process the calculated transformation parameters using a least squares method or the like and output the resulting transformation parameters as the camera information. In this way, the transformation parameters calculated based on the virtual space images vp acquired by each of the multiple virtual cameras vc are statistically processed to obtain the final transformation parameters that are output, thereby improving the accuracy of the transformation parameters that are output as the camera information.

[0041] The resetting unit 18 resets the virtual camera vc in the virtual space vs to which the transformation parameters for capturing images of the real space calculated by the parameter calculation unit 16 have been applied as transformation parameters for capturing images of the virtual space vs.

[0042] When setting up the initial virtual camera vc and acquiring the virtual space image vp, the transformation parameters applied to the virtual camera vc may be set arbitrarily. After the transformation parameters for capturing the real space are calculated based on the initial virtual space image vp, the calculated transformation parameters can be applied to the virtual camera vc. Once the calculated transformation parameters are applied, the virtual camera vc is likely to have a positional relationship and characteristics similar to those of the camera that captured the real space image rp.

[0043] The virtual space image acquisition unit 13 acquires a virtual space image vp captured by a virtual camera vc reinstalled in the virtual space vs. The feature point extraction unit 14 extracts feature points from the virtual space image vp captured by the reinstalled virtual camera vc, and the feature point matching unit 15 matches the feature points. The parameter calculation unit 16 then calculates transformation parameters based on the virtual space image vp captured by the reinstalled virtual camera vc. In this way, by recalculating the transformation parameters based on the virtual space image vp captured by the virtual camera vc to which the calculated transformation parameters have been applied, it is possible to improve the accuracy of the transformation parameters.

[0044] The resetting unit 18 may reset at least one virtual camera vc within a predetermined range around a specific virtual camera vc, which is the virtual camera vc that captured the virtual space image vp that has the most reference feature points fp, among the virtual cameras vc set by the setting unit 12.

[0045] Fig. 6 is a diagram showing an example of the resetting of a virtual camera. In the example shown in Fig. 6, the resetting unit 18 extracts a specific virtual camera vc1, which is the virtual camera that captured the virtual space image vp having the most reference feature points fp, from among the virtual cameras vc installed in the virtual space vs in the calculation of the initial or previous transformation parameters. The specific virtual camera vc1 is the virtual camera that captured the virtual space image vp having many feature points that match the feature points in the real space image rp, and therefore is highly likely to be a camera installed in a position in the virtual space vs that corresponds to the position of the camera rc that captured the real space image rp.

[0046] The resetter 18 resets the virtual camera vc within a predetermined range around the specific virtual camera vc1. For example, the resetter 18 may reset the virtual camera vc to a position within a predetermined distance from the position of the specific virtual camera vc1. As illustrated in Fig. 6, the resetter 18 may reset the virtual cameras vc2 and vc3 to positions adjacent to the specific virtual camera vc1 along the outer periphery of the virtual space vs.

[0047] In this way, by recalculating the transformation parameters based on the virtual space image vp based on the virtual camera vc that has been re-installed within a predetermined range around the specific virtual camera vc1, it is possible to further improve the accuracy.

[0048] Furthermore, the resetting unit 18 may reset at least one virtual camera vc so that a virtual space image vp is captured in which a partial region including the reference feature point fp overlaps with the virtual space image vp captured by the specific virtual camera vc1. Fig. 7 is a diagram showing an example of how to superimpose the captured region between the virtual space image captured by the specific virtual camera and the virtual space image captured by the reset virtual camera.

[0049] 7, when a virtual space image vp having an imaging area ts0 is captured by a specific virtual camera vc1, the resetter 18 resets the virtual camera vc so as to capture an imaging area that is superimposed on the imaging area ts0 in a partial area including the reference feature point fp. Specifically, the resetter 18 resets the virtual camera vc to a position such that imaging areas ts1, ts2, ts3, and ts4, including an area including the reference feature point fp located at the edge of the imaging area ts0, are captured as the virtual space image vp. By acquiring the virtual space image vp using the virtual camera vc reset in this way, it is possible to improve the accuracy of the transformation parameters related to lens distortion.

[0050] Referring again to Figure 1, the position calculation unit 19 calculates the position of the object in three-dimensional space using the transformation parameters included in the camera information, based on the two-dimensional coordinates indicating the position of the object shown in the real-space image rp captured from the real-space rs.

[0051] In this way, it is possible to estimate the position in three-dimensional space of an object depicted in the real-space image rp, which is a captured image of the real space, using the two-dimensional coordinates of the object in the real-space image rp and the virtual space model vm corresponding to that real space.

[0052] 8 is a flowchart showing the processing content of the information processing method in the information processing system 1. In step S1, the physical space image acquisition unit 11 acquires a physical space image rp obtained by capturing an image of the physical space.

[0053] In step S2, a virtual space model vm representing a virtual space vs corresponding to the real space rs is generated based on the real space image rp. The virtual space model vm may be generated by the processor of the information processing device 10 or by another device. The generated virtual space model vm may be stored in the virtual space model storage unit 22.

[0054] In step S3, the setting unit 12 sets a virtual camera vc in the virtual space vs represented by the virtual space model vm. In step S4, the virtual space image acquisition unit 13 acquires at least one virtual space image vp obtained by capturing the virtual space vs with the virtual camera vc.

[0055] In step S5, the feature point extraction unit 14 extracts feature points from each of the real space image rp and the virtual space image vp. In step S6, the feature point matching unit 15 matches the feature points of the real space image rp with the feature points of the virtual space image vp.

[0056] In step S7, the parameter calculation unit 16 determines whether the number of reference feature points fp, which are matched feature points, is equal to or greater than a threshold. If it is determined that the number of reference feature points fp is equal to or greater than the threshold, the process proceeds to step S8. On the other hand, if it is not determined that the number of reference feature points fp is equal to or greater than the threshold, the process returns to step S3.

[0057] In step S8, the parameter calculation unit 16 acquires the reference space coordinate t of the reference feature point fp based on the transformation parameters (transformation matrix A) that indicate the relationship between the three-dimensional coordinates indicating the position in the virtual space vs and the two-dimensional coordinates of the position in the image captured by the virtual camera vc placed in the virtual space vs, and the virtual reference image coordinate d'.

[0058] In step S9, the parameter calculation unit 16 calculates transformation parameters (transformation matrix B) related to imaging of the real space based on the reference space coordinates t and the reference image coordinates d.

[0059] In step S10, the parameter calculation unit 16 determines whether or not to end the process. If it is determined that the virtual camera vc should be reinstalled to improve the accuracy of the calculated transformation parameters, the process proceeds to step S11. If it is determined that the process should be ended, the process proceeds to step S12.

[0060] In step S11, the resetter 18 applies the transformation parameters calculated in step S9 to the virtual camera vc. Then, the process returns to step S3 to repeat the calculation of the transformation parameters based on the resetting of the virtual camera vc. When the process returns to step S3 via step S11, the resetter 18 resets the virtual camera vc.

[0061] In step S12, the output unit 17 outputs the transformation parameters (transformation matrix B) related to capturing the real space calculated by the parameter calculation unit 16 as camera information that essentially represents the position in the virtual space of the camera that captured the real space image rp.

[0062] Next, an information processing program for causing a computer to function as the information processing device 10 of this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram showing the configuration of the information processing program. The information processing program P1 is configured to include a main module m10 that comprehensively controls information processing in the information processing device 10, a real space image acquisition module m11, a setting module m12, a virtual space image acquisition module m13, a feature point extraction module m14, a feature point matching module m15, a parameter calculation module m16, an output module m17, a resetting module m18, and a position calculation module m19. Each of the modules m11 to m19 realizes a function for each of the functional units 11 to 19.

[0063] The information processing program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M1 as shown in FIG.

[0064] According to the information processing system, information processing device 10, information processing method, and information processing program P1 of the present embodiment described above, feature points extracted from each of the real space image and the virtual space image are matched as reference feature points, and the two-dimensional reference image coordinates of the reference feature point in the real space image are associated with the three-dimensional reference space coordinates of the reference feature point in the virtual space, thereby enabling calculation of transformation parameters in the transformation formula. The calculated transformation parameters constitute information representing the relative relationship between a position in the real space image and a position in the virtual space, and therefore, it is possible to obtain information representing the position in the virtual space of the camera that captures the real space from camera information consisting of the transformation parameters.

[0065] The information processing device and information processing method according to the present disclosure may have the following configurations: The actions and effects of each configuration will be described as follows.

[0066] An information processing device according to one aspect of the present disclosure includes a feature point extraction unit that extracts feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space corresponding to the real space and represented by a virtual space model captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching unit that matches feature points of the real space image with feature points of the virtual space image; and a feature point matching unit that matches reference space coordinates in the virtual space of a reference feature point that is the matched feature point, and reference image coordinates that are the coordinates of the reference feature point in the real space image. a parameter calculation unit that calculates transformation parameters related to imaging of real space by substituting the above into a predetermined transformation formula that expresses, using predetermined transformation parameters, the relationship between the three-dimensional coordinates of a specific point, which is a specific point in three-dimensional space, and the two-dimensional coordinates of the specific point in a captured image that captures the three-dimensional space, wherein the reference space coordinates are calculated based on the transformation parameters related to imaging of virtual space by a virtual camera and virtual reference image coordinates that are the coordinates of a reference feature point in the virtual space image; and an output unit that outputs the transformation parameters related to imaging of real space as camera information that represents the position in virtual space of the camera that captured the real space image.

[0067] An information processing method according to one aspect of the present disclosure includes a feature point extraction step executed by a processor and extracting feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space corresponding to the real space and represented by a virtual space model captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching step matching feature points of the real space image with feature points of the virtual space image; and a reference space coordinate in the virtual space of a reference feature point that is the matched feature point and a reference feature point that is the coordinate of the reference feature point in the real space image. the reference space coordinates are calculated based on the transformation parameters related to the imaging of the virtual space by the virtual camera and virtual reference image coordinates which are the coordinates of a reference feature point in the virtual space image; and an output step of outputting the transformation parameters related to the imaging of the real space as camera information which represents the position in the virtual space of the camera which captured the real space image.

[0068] According to the above aspect, feature points extracted from each of the real space image and the virtual space image are matched as reference feature points, and the two-dimensional reference image coordinates of the reference feature point in the real space image are associated with the three-dimensional reference space coordinates of the reference feature point in the virtual space, thereby enabling calculation of transformation parameters in the transformation formula. Since the calculated transformation parameters constitute information representing the relative relationship between a position in the real space image and a position in the virtual space, it is possible to obtain information representing the position in the virtual space of the camera that captures the real space from camera information consisting of the transformation parameters.

[0069] In addition, an information processing device according to another aspect may further include a setting unit that sets up a plurality of virtual cameras, and the parameter calculation unit may calculate transformation parameters related to imaging of real space based on at least one virtual space image selected based on the number of reference feature points from among a plurality of virtual space images acquired by each of the plurality of virtual cameras.

[0070] According to the above aspect, the virtual space image selected from the plurality of virtual space images based on the number of reference feature points is an image captured by a virtual camera that is likely to be installed in a position close to the position of the camera that captured the real space image. Therefore, by calculating the transformation parameters based on such a virtual space image, the accuracy of the calculation can be improved.

[0071] In addition, in an information processing device relating to another aspect, the setting unit may install multiple virtual cameras at least along the periphery of the virtual space, with the normal direction of the periphery as the imaging direction, or may install multiple virtual cameras within the virtual space, with the periphery direction of the virtual space as the imaging direction.

[0072] According to the above aspect, a virtual space corresponding to a space represented in a real space image can be comprehensively captured by a plurality of installed virtual cameras.

[0073] In addition, in an information processing device according to another aspect, the parameter calculation unit may calculate a transformation parameter relating to imaging of real space based on a virtual space image having a number of reference specific points equal to or greater than a threshold value relating to reference feature points.

[0074] According to the above aspect, the threshold is set according to the number of unknown transformation parameters in the transformation formula, so that the transformation parameters can be reliably found.

[0075] In addition, an information processing device according to another aspect may further include a resetting unit that resets a virtual camera in the virtual space to which the transformation parameters for capturing images of the real space calculated by the parameter calculation unit have been applied as transformation parameters for capturing images of the virtual space, and the feature point extraction unit may extract feature points from a virtual space image captured of the virtual space by the reset virtual camera.

[0076] According to the above aspect, a virtual camera to which calculated transformation parameters have been applied is likely to have a positional relationship and characteristics similar to those of a camera that captured a real-space image. By recalculating the transformation parameters based on a virtual-space image captured by the virtual camera to which the calculated transformation parameters have been applied, it is possible to improve the accuracy of the transformation parameters.

[0077] In addition, in an information processing device relating to another aspect, the resetting unit may reset at least one virtual camera within a predetermined range around a specific virtual camera, which is the virtual camera that captured the virtual space image having the most reference feature points.

[0078] According to the above aspect, the specific virtual camera is likely to be a camera installed in a position close to the camera that captured the real-space image in the virtual space. Therefore, by recalculating the transformation parameters based on the virtual-space image captured by a virtual camera installed within a predetermined range around the specific virtual camera, it is possible to further improve the accuracy of the transformation parameters.

[0079] In addition, in an information processing device relating to another aspect, the resetting unit may reset at least one virtual camera so that a virtual space image is captured in which a partial area including a reference feature point overlaps with the virtual space image captured by a specific virtual camera.

[0080] According to the above aspect, a virtual space image including a reference feature point located at an edge of an imaging area of ​​the virtual space image captured by the specific virtual camera is acquired by the relocated virtual camera, thereby improving the accuracy of the transformation parameters related to the lens distortion.

[0081] In addition, in an information processing device according to another aspect, the parameter calculation unit may calculate transformation parameters for capturing real space based on each of the plurality of virtual space images, and the output unit may output the transformation parameters obtained by processing the calculated plurality of transformation parameters using a predetermined statistical method as camera information.

[0082] According to the above aspect, the transformation parameters calculated based on the virtual space images acquired by each of the multiple virtual cameras are statistically processed to obtain the final transformation parameters to be output, thereby improving the accuracy of the transformation parameters to be output as camera information.

[0083] In another aspect of the information processing device, the transformation parameters include six predetermined camera external parameters, four camera internal parameters, and five parameters related to the lens distortion of the camera, which are related to the projection of the three-dimensional space onto the two-dimensional image. The parameter calculation unit calculates a transformation formula (1) below that expresses the relationship between three-dimensional coordinates (X, Y, Z) in the three-dimensional space and two-dimensional coordinates (u, v) in the two-dimensional image.

number

[0084] According to the above aspect, a transformation parameter consisting of 15 variables can be calculated as camera information.

[0085] The block diagram shown in FIG. 1 shows functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. Furthermore, the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using one device that is physically or logically coupled, or may be realized using two or more devices that are physically or logically separated and connected directly or indirectly (for example, by wire, wirelessly, etc.). The functional block may also be realized by combining software with the one device or the multiple devices.

[0086] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, election, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, and assignment. For example, a functional block (component) that performs transmission is called a transmitting unit or transmitter. As mentioned above, there are no particular limitations on how these functions are implemented.

[0087] For example, the information processing device 10 according to an embodiment of the present invention may function as a computer. Fig. 10 is a diagram showing an example of the hardware configuration of the information processing device 10 according to this embodiment. The information processing device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0088] In the following description, the term "apparatus" can be interpreted as a circuit, a device, a unit, etc. The hardware configuration of the information processing device 10 may be configured to include one or more of the apparatuses shown in FIG. 10, or may be configured to exclude some of the apparatuses.

[0089] Each function of the information processing device 10 is realized by loading specified software (programs) onto hardware such as the processor 1001, memory 1002, etc., so that the processor 1001 performs calculations and controls communication via the communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.

[0090] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the functional units 11 to 19 shown in FIG. 1 may be realized by the processor 1001.

[0091] Furthermore, the processor 1001 reads programs (program codes), software modules, and data from the storage 1003 and / or the communication device 1004 into the memory 1002, and executes various processes in accordance with these. The programs used are those that cause a computer to execute at least some of the operations described in the above-described embodiments. For example, the functional units 11 to 19 of the information processing device 10 may be implemented by a control program stored in the memory 1002 and executed by the processor 1001. While the above-described various processes have been described as being executed by one processor 1001, they may also be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented on one or more chips. The programs may also be transmitted from a network via a telecommunications line.

[0092] The memory 1002 is a computer-readable recording medium and may be composed of at least one of, for example, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), and a random access memory (RAM). The memory 1002 may also be called a register, a cache, a main memory (primary storage device), or the like. The memory 1002 can store executable programs (program codes), software modules, and the like for implementing an information processing method according to one embodiment of the present invention.

[0093] Storage 1003 is a computer-readable recording medium, and may be, for example, at least one of an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray disc), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, etc. Storage 1003 may also be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and / or storage 1003.

[0094] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0095] The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated into one device (for example, a touch panel).

[0096] Furthermore, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

[0097] The information processing device 10 may also be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

[0098] The notification of information is not limited to the aspects / embodiments described in the present disclosure and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI) and Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, and broadcast information (Master Information Block (MIB) and System Information Block (SIB))), other signals, or a combination thereof. Furthermore, the RRC signaling may be referred to as an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0099] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-Advanced (LTE-A), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (New Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark), IEEE 802.20, UWB (Ultra-Wideband), Bluetooth (registered trademark), or other appropriate systems, and next-generation systems extended based on these. Furthermore, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A with 5G, etc.) may also be applied.

[0100] The order of the procedures, sequences, flowcharts, etc. of each aspect / embodiment described in this disclosure may be changed unless it is consistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0101] In the present disclosure, a specific operation described as being performed by a base station may be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and another network node other than the base station (for example, but not limited to, an MME or an S-GW). Although the above example illustrates a case where there is one other network node other than the base station, a combination of multiple other network nodes (for example, an MME and an S-GW) may also be used.

[0102] Information etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer), or may be input / output via multiple network nodes.

[0103] Input and output information may be stored in a specific location (for example, memory) or managed in a management table. Input and output information may be overwritten, updated, or added to. Output information may be deleted. Input information may be sent to another device.

[0104] The determination may be made based on a value represented by one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0105] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched depending on the implementation. Furthermore, notification of predetermined information (e.g., notification that "X is true") is not limited to being done explicitly, but may be done implicitly (e.g., by not notifying the predetermined information).

[0106] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described herein. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0107] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0108] Software, instructions, etc. may also be transmitted or received over a transmission medium. For example, if software is transmitted from a website, server, or other remote source using wired technologies such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and / or wireless technologies such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of transmission media.

[0109] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0110] It should be noted that terms explained in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0111] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0112] Furthermore, the information, parameters, etc. described in the present disclosure may be expressed as absolute values, relative values ​​from a predetermined value, or other corresponding information. For example, a radio resource may be indicated by an index.

[0113] The names used for the above-described parameters are not intended to be limiting in any way. Furthermore, the mathematical expressions using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not intended to be limiting in any way.

[0114] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, searching, inquiring (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in memory), and the like. Furthermore, "judgment" and "decision" can include regarding resolving, selecting, choosing, establishing, comparing, etc. as having been "judged" or "decided." In other words, "judgment" and "decision" can include regarding some action as having been "judged" or "decided." Furthermore, "judgment (decision)" can be interpreted as "assuming," "expecting," "considering," etc.

[0115] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0116] When designations such as "first," "second," etc. are used in this disclosure, any reference to an element does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

[0117] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.

[0118] To the extent that the terms "include," "including," and variations thereof are used herein or in the claims, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or," as used herein or in the claims, is not intended to be an exclusive or.

[0119] In this disclosure, where articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0120] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0121] The information processing system 1 of the present disclosure may have the following configuration.

[0122] [1] a feature point extraction unit that extracts feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space that corresponds to the real space and is represented by a virtual space model, captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching unit that matches feature points of the real space image with feature points of the virtual space image; a parameter calculation unit that calculates transformation parameters related to imaging of the real space by substituting reference space coordinates in the virtual space of a reference feature point, which is the matched feature point, and reference image coordinates, which are coordinates of the reference feature point in the real space image, into a predetermined transformation formula that expresses, using predetermined transformation parameters, a relationship between the three-dimensional coordinates of a specific point, which is a specific point in three-dimensional space, and the two-dimensional coordinates of the specific point in a captured image obtained by capturing an image of the three-dimensional space, wherein the reference space coordinates are calculated based on the transformation parameters related to imaging of the virtual space by the virtual camera and virtual reference image coordinates, which are coordinates of the reference feature point in the virtual space image; an output unit that outputs the transformation parameters related to capturing the real space image as camera information that represents a position in the virtual space of a camera that captured the real space image; An information processing device comprising: [2] a setting unit that sets up a plurality of the virtual cameras, the parameter calculation unit calculates the transformation parameters related to imaging of the real space based on at least one virtual space image selected based on the number of reference feature points from among a plurality of virtual space images acquired by each of the plurality of virtual cameras; [1] The information processing device according to [1]. [3] The setting unit installs a plurality of the virtual cameras at least along an outer periphery of the virtual space, with an imaging direction being a normal direction of the outer periphery, or a plurality of virtual cameras are installed in the virtual space with the imaging direction directed toward the periphery of the virtual space; [2] The information processing device according to [2]. [4] the parameter calculation unit calculates the transformation parameters related to imaging of the real space based on the virtual space image having a number of reference specific points equal to or greater than a threshold value related to the reference feature points; The information processing device according to any one of [1] to [3]. [5] a resetting unit that resets the virtual camera in the virtual space to which the transformation parameters related to imaging of the real space calculated by the parameter calculation unit have been applied as transformation parameters related to imaging of the virtual space, the feature point extraction unit extracts feature points from the virtual space image captured by the reset virtual camera of the virtual space; [4] The information processing device according to [4]. [6] the resetting unit resets at least one of the virtual cameras within a predetermined range around a specific virtual camera that is the virtual camera that captured the virtual space image having the most reference feature points. [5] The information processing device according to [5]. [7] the resetting unit resets the at least one virtual camera so that a virtual space image is captured in which a partial area including the reference feature point overlaps with the virtual space image captured by the specific virtual camera. [6] The information processing device according to [6]. [8] the parameter calculation unit calculates the transformation parameters related to capturing the real space based on each of the plurality of virtual space images; the output unit processes the calculated plurality of transformation parameters using a predetermined statistical method, and outputs the transformation parameters as the camera information. The information processing device according to any one of [1] to [7]. [9] the transformation parameters include six predetermined camera extrinsic parameters related to the projection of a three-dimensional space onto a two-dimensional image, four camera intrinsic parameters, and five parameters related to the camera lens distortion; The parameter calculation unit calculates the following transformation formula (1) that represents the relationship between three-dimensional coordinates (X, Y, Z) in a three-dimensional space and two-dimensional coordinates (u, v) in a two-dimensional image:

number

[10] a feature point extraction step of extracting feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space corresponding to the real space and represented by a virtual space model captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching step of matching feature points of the real space image with feature points of the virtual space image; a parameter calculation step of calculating transformation parameters related to imaging of the real space by substituting reference space coordinates in the virtual space of a reference feature point, which is the matched feature point, and reference image coordinates, which are the coordinates of the reference feature point in the real space image, into a predetermined transformation formula that expresses, using predetermined transformation parameters, a relationship between the three-dimensional coordinates of a specific point, which is a specific point in three-dimensional space, and the two-dimensional coordinates of the specific point in a captured image obtained by capturing an image of the three-dimensional space, wherein the reference space coordinates are calculated based on the transformation parameters related to imaging of the virtual space by the virtual camera and virtual reference image coordinates, which are the coordinates of the reference feature point in the virtual space image; an output step of outputting the transformation parameters related to capturing the real space image as camera information representing a position in the virtual space of a camera that captured the real space image; An information processing method executed by a processor, comprising: [Explanation of symbols]

[0123] 1...information processing system, 10...information processing device, 11...real space image acquisition unit, 12...setting unit, 13...virtual space image acquisition unit, 14...feature point extraction unit, 15...feature point matching unit, 16...parameter calculation unit, 17...output unit, 18...resetting unit, 19...position calculation unit, 21...real space image storage unit, 22...virtual space model storage unit, M1...recording medium, m11...real space image acquisition module, m12...setting module, m13...virtual space image acquisition module, m14...feature point extraction module, m15...feature point matching module, m16...parameter calculation module, m17...output module, m18...resetting module, m19...position calculation module, P1...information processing program.

Claims

1. a feature point extraction unit that extracts feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space that corresponds to the real space and is represented by a virtual space model, captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching unit that matches feature points of the real space image with feature points of the virtual space image; a parameter calculation unit that calculates transformation parameters related to imaging of the real space by substituting reference space coordinates in the virtual space of a reference feature point that is the matched feature point and reference image coordinates that are the coordinates of the reference feature point in the real space image into a predetermined transformation formula that expresses, using predetermined transformation parameters, a relationship between the three-dimensional coordinates of a specific point that is a specific point in three-dimensional space and the two-dimensional coordinates of the specific point in a captured image that captures the three-dimensional space, wherein the reference space coordinates are calculated based on the transformation parameters related to imaging of the virtual space by the virtual camera and virtual reference image coordinates that are the coordinates of the reference feature point in the virtual space image; an output unit that outputs the transformation parameters related to capturing the real space image as camera information that represents a position in the virtual space of a camera that captured the real space image; An information processing device comprising:

2. a setting unit that sets up a plurality of the virtual cameras, the parameter calculation unit calculates the transformation parameters related to imaging of the real space based on at least one virtual space image selected based on the number of reference feature points from among a plurality of virtual space images acquired by each of the plurality of virtual cameras; The information processing device according to claim 1 .

3. The setting unit installs a plurality of the virtual cameras at least along an outer periphery of the virtual space, with an imaging direction being a normal direction of the outer periphery, or a plurality of virtual cameras are installed in the virtual space with the imaging direction directed toward the periphery of the virtual space; The information processing device according to claim 2 .

4. the parameter calculation unit calculates the transformation parameters related to imaging of the real space based on the virtual space image having a number of reference specific points equal to or greater than a threshold value related to the reference feature points; The information processing device according to claim 1 .

5. a resetting unit that resets the virtual camera in the virtual space to which the transformation parameters related to imaging of the real space calculated by the parameter calculation unit have been applied as transformation parameters related to imaging of the virtual space, the feature point extraction unit extracts feature points from the virtual space image captured by the reset virtual camera of the virtual space; The information processing device according to claim 4 .

6. the resetting unit resets at least one of the virtual cameras within a predetermined range around a specific virtual camera that is the virtual camera that captured the virtual space image having the most reference feature points. The information processing device according to claim 5 .

7. the resetting unit resets the at least one virtual camera so that a virtual space image is captured in which a partial area including the reference feature point overlaps with the virtual space image captured by the specific virtual camera. The information processing device according to claim 6 .

8. the parameter calculation unit calculates the transformation parameters related to capturing the real space based on each of the plurality of virtual space images; the output unit processes the calculated plurality of transformation parameters using a predetermined statistical method, and outputs the transformation parameters as the camera information. The information processing device according to claim 1 .

9. the transformation parameters include six predetermined camera extrinsic parameters related to the projection of a three-dimensional space onto a two-dimensional image, four camera intrinsic parameters, and five parameters related to the camera lens distortion; The parameter calculation unit calculates the three-dimensional coordinates (X, Y, Z) in the three-dimensional space and the two-dimensional coordinates (u, v) in the two-dimensional image using the following transformation formula (1): [Equation 1] In the above, by substituting the three-dimensional reference space coordinates into the three-dimensional coordinates (X, Y, Z) and the two-dimensional reference image coordinates into the two-dimensional coordinates (u, v), the camera external parameters R, t and the camera internal parameters f each having three degrees of freedom are obtained. ｘ , f ｙ , c ｘ , c ｙ , and a parameter k relating to the lens distortion of the camera １ , k ２ , k ３ , p １ , p ２ , to calculate The information processing device according to claim 1 .

10. a feature point extraction step of extracting feature points from a real space image captured of a real space and at least one virtual space image, wherein the virtual space image is an image of a virtual space corresponding to the real space and represented by a virtual space model captured by a virtual camera installed in the virtual space, and the position of the virtual camera is a viewpoint position for capturing an image projected from the virtual space model; a feature point matching step of matching feature points of the real space image with feature points of the virtual space image; a parameter calculation step of calculating transformation parameters related to imaging of the real space by substituting reference space coordinates in the virtual space of a reference feature point, which is the matched feature point, and reference image coordinates, which are coordinates of the reference feature point in the real space image, into a predetermined transformation formula that expresses, using predetermined transformation parameters, a relationship between the three-dimensional coordinates of a specific point, which is a specific point in three-dimensional space, and the two-dimensional coordinates of the specific point in a captured image obtained by capturing an image of the three-dimensional space, wherein the reference space coordinates are calculated based on the transformation parameters related to imaging of the virtual space by the virtual camera and virtual reference image coordinates, which are coordinates of the reference feature point in the virtual space image; an output step of outputting the transformation parameters related to capturing the real space image as camera information representing a position in the virtual space of a camera that captured the real space image; An information processing method executed by a processor, comprising:

Images