Information processing device and information processing method

Abstract

This invention provides an information processing device that efficiently estimates light source information to achieve optical consistency, seamlessly blending shadows from the real world with those from the virtual world. [Solution] The information processing device includes a generation means for generating map information for estimating the position and orientation of an imaging device using an image that satisfies predetermined conditions from among the images captured by the imaging device, and an acquisition means for acquiring light source information using an image that satisfies the predetermined conditions at the timing when the map information is updated.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method.

Background Art

[0002] In recent years, the development of Virtual Reality (VR) systems and Mixed Reality (MR) systems, which aim at a seamless connection between the real space and the virtual space, has been actively carried out. Among various types of technological developments, various techniques for estimating the light source information of the real world have been developed as techniques for realizing optical consistency by seamlessly integrating the shadows of the real world and the virtual world. Patent Document 1 discloses a method of generating a composite image of a photographed image of the real space and a virtual image in real time in response to changes in ambient light without using a special device such as a two-dimensional marker by arranging a spherical mirror.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The computing resources of an information processing apparatus for providing a mixed reality system are limited. Since the computing resources are mainly used for estimating the position and orientation of a display device that presents a mixed reality image and rendering a virtual space based on the estimated position and orientation information, it is preferable that the light source information be estimated by efficiently using the computing resources.

[0005] Therefore, an object of the present invention is to provide an information processing apparatus that efficiently estimates light source information for realizing optical consistency that seamlessly integrates the shadows of the real world and the virtual world.

Means for Solving the Problems

[0006] The information processing device according to the present invention is characterized by comprising: generation means for generating map information for estimating the position and orientation of an imaging device using an image that satisfies predetermined conditions from among the images captured by the imaging device; and acquisition means for acquiring light source information using an image that satisfies the predetermined conditions at the timing when the map information is updated. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an information processing device that efficiently estimates light source information in order to achieve optical consistency that seamlessly fuses shadows from the real world with shadows from the virtual world. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram illustrating the functional configuration of an image processing system. [Figure 2] This is a flowchart illustrating the processing steps of an information processing device. [Figure 3] This diagram illustrates when map information is updated. [Figure 4] This diagram illustrates the hardware configuration of an information processing device. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The embodiments described are merely examples of means for realizing the present invention and can be appropriately modified depending on the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is not limited to the following embodiments.

[0010] [First Embodiment] Figure 1 is a block diagram illustrating the functional configuration of the image processing system 10. The image processing system 10 is a system for presenting a mixed reality (MR) space, which fuses the real space and the virtual space, to the system user. The image processing system 10 can present the MR space to the user by displaying a composite image that combines an image of the real space with an image of the virtual space rendered by computer graphics (CG).

[0011] The image processing system 10 includes a display device 100 and an information processing device 110. The information processing device 110 can combine an image of the real world captured from the display device 100 with an image of a virtual world generated by the information processing device 110, and output the combined image to the display device 100 as a mixed reality image (MR image).

[0012] The image processing system 10 is a system that displays images of a virtual space. The image processing system 10 is not limited to an MR (mixed reality) system that displays an MR image which is a composite of an image of the real space and an image of the virtual space. The image processing system 10 can also be applied to a VR (virtual reality) system that presents only images of the virtual space to the user, or an AR (augmented reality) system that presents images of the virtual space to the real space through a transparent display unit.

[0013] The display device 100 includes an imaging unit 101 (imaging device). The imaging unit 101 captures images of the real space in a continuous time series and outputs the captured images of the real space to the information processing device 110. The imaging unit 101 may include a stereo camera consisting of two cameras fixed in a position capable of capturing images of the real space in the line of sight from the user's viewpoint.

[0014] The display device 100 includes a display unit 102. The display unit 102 displays MR images output from the information processing device 110. The display unit 102 may include two displays, one corresponding to the user's left eye and one corresponding to the user's right eye. In this case, the left-eye display, corresponding to the user's left eye, displays the MR image for the left eye, and the right-eye display, corresponding to the user's right eye, displays the MR image for the right eye.

[0015] The display device 100 is, for example, a head-mounted display device (HMD) worn by the user on their head. However, the display device 100 is not limited to an HMD and may be a handheld display (HHD). An HHD is a handheld display that the user holds in their hand and looks through like binoculars to observe images. Furthermore, the display device 100 may be a display terminal such as a tablet or a smartphone.

[0016] The information processing device 110 and the display device 100 are connected to each other in a manner that enables data communication. The connection between the information processing device 110 and the display device 100 may be wired or wireless. The information processing device 110 may also be located inside the housing of the display device 100.

[0017] The information processing device 110 includes a position and orientation estimation unit 111, a light source information acquisition unit 112, an image generation unit 113, an image synthesis unit 114, and a data storage unit 115.

[0018] The position and orientation estimation unit 111 estimates and obtains the position and orientation of the imaging unit 101 in the world coordinate system. The position and orientation estimation unit 111 uses images that satisfy predetermined conditions from the images captured by the imaging unit 101 (imaging device) to obtain map information for estimating the position and orientation of the imaging unit 101. It is generated. Specifically, the position and orientation estimation unit 111 executes the process of Simultaneous Localization and Mapping (SLAM) based on the feature points reflected in the image of the real space captured by the imaging unit 101. The position and orientation estimation unit 111 can estimate the position and orientation of the imaging unit 101 by executing the SLAM process. The position and orientation estimation unit 111 records the information of the estimated position and orientation of the imaging unit 101 in the data storage unit 115.

[0019] The predetermined condition is a condition indicating that the captured image captured by the imaging unit 101 is suitable for the estimation of the position and orientation. The predetermined condition may include, for example, that the acceleration of the imaging unit 101 when capturing the captured image is smaller than the predetermined acceleration, and that the angular acceleration of the imaging unit 101 when capturing the captured image is smaller than the predetermined angular acceleration. Further, the predetermined condition may include that an index representing the sharpness of the captured image is larger than the predetermined value, and that the number of feature points included in the captured image is larger than the predetermined number.

[0020] The light source information acquisition unit 112 acquires the light source information in the real space based on the image of the real space captured by the imaging unit 101. The light source information includes information such as the position of the light source, the intensity of the light source, and the type of the light source. The light source information acquisition unit 112 records the acquired light source information in the data storage unit 115. The process of acquiring the light source information is not limited to the process of estimating the light source information in the real space, and includes at least one of the process of determining the necessity of estimating the light source information and the process of applying all or part of the light source information estimated in the past.

[0021] The types of light sources are parallel light sources, point light sources, spotlights, rectangular light sources, etc. When the light source information acquisition unit 112 acquires the information of a point light source, for example, it estimates the luminance of the light source, the color of the light source, and the position of the light source. Any method may be used as long as it uses the captured image of the real space captured by the imaging unit 101 to estimate the light source information.

[0022] For example, the light source information acquisition unit 112 may estimate information about the light source using a metal sphere that reflects the surrounding environment, acquire information about the light source that is directly reflected in the captured image, or estimate information about the light source that is not directly reflected in the captured image using a light source information inference model.

[0023] Furthermore, the light source information acquisition unit 112 may estimate light source information using multiple images captured by the imaging unit 101, or it may generate an environment map (map information) from the images captured by the imaging unit 101 and estimate light source information from the generated map information.

[0024] The light source information acquisition unit 112 may estimate light source information in the real space by using information other than images of the real space. For example, if the imaging unit 101 has an auto exposure function, the light source information acquisition unit 112 can estimate light source information by determining the amount of light using the exposure value. Also, if the imaging unit 101 has an auto white balance function that automatically corrects the color of the light source, the light source information acquisition unit 112 can estimate light source information based on the white balance value.

[0025] The image generation unit 113 constructs a virtual space based on the virtual space data stored in the data storage unit 115. The virtual space data includes light source information acquired by the light source information acquisition unit 112, data relating to each virtual object that constitutes the virtual space, and data relating to real-world approximation objects that incorporate 3D shape information of real objects acquired from the real space into the virtual space.

[0026] The image generation unit 113 sets a virtual viewpoint based on the position and orientation of the imaging unit 101 estimated by the position and orientation estimation unit 111. The image generation unit 113 generates an image of the virtual space visible from the set virtual viewpoint (virtual space image). Since the technique for generating an image of the virtual space visible from a viewpoint with a predetermined position and orientation is well known, a detailed explanation of this will be omitted.

[0027] The image synthesis unit 114 generates an MR image by combining the image of the virtual space generated by the image generation unit 113 with the image of the real space captured by the imaging unit 101. The image synthesis unit 114 outputs the generated MR image to the display unit 102.

[0028] The data storage unit 115 includes RAM and a hard disk drive device. The data storage unit 115 stores various information used by each functional unit of the information processing device 110, as well as information that is described as known information.

[0029] Figure 2 is a flowchart illustrating the process by which the information processing device 110 generates an MR image and outputs it to the display device 100.

[0030] In step S201, the imaging unit 101 acquires a captured image. In step S202, the position and orientation estimation unit 111 estimates the position and orientation of the imaging unit 101 using the captured image acquired in step S201.

[0031] In step S203, the position and orientation estimation unit 111 determines whether or not to update the map information for estimating the position and orientation of the imaging unit 101. The position and orientation estimation unit 111 updates the map information if the image captured by the imaging unit 101 satisfies predetermined conditions. That is, the map information is generated using images from the images captured by the imaging unit 101 that are suitable for estimating the position and orientation of the imaging unit 101. For example, if the position and orientation of the imaging unit 101 is estimated by SLAM processing, the keyframe information generated by the SLAM processing corresponds to the map information.

[0032] If the position and orientation estimation unit 111 determines that it will update the map information, the process proceeds to step S204. If the position and orientation estimation unit 111 determines that it will not update the map information, the process proceeds to step S206.

[0033] In step S204, the position and orientation estimation unit 111 updates the map information for estimating the position and orientation of the imaging unit 101. The process of updating the map information includes at least one of the following: adding, changing, or deleting keyframe images included in the map information, and adding, changing, or deleting feature points included in the keyframe images.

[0034] In step S205, the light source information acquisition unit 112 acquires light source information in the real space. In step S204, if a keyframe image is deleted from the map information, the position and orientation estimation unit 111 may store the deleted keyframe image in the data storage unit 115. The light source information acquisition unit 112 may also store the keyframe image deleted from the map information and use it to acquire light source information. By storing the deleted keyframe image, the light source information acquisition unit 112 can effectively utilize an image suitable for acquiring light source information.

[0035] In step S206, the image generation unit 113 generates an image in the virtual space based on the position and orientation of the imaging unit 101 estimated in step S202 and the light source information acquired in step S205.

[0036] In step S207, the image synthesis unit 114 synthesizes the image of the virtual space generated in step S206 with the image of the real space captured by the imaging unit 101 to generate an MR image (composite image).

[0037] In step S208, the information processing device 110 determines whether or not the termination condition has been met. The information processing device 110 can determine, for example, that the termination condition has been met when a user inputs an instruction to terminate the process of generating MR images. If the termination condition is met, the process shown in Figure 2 terminates. If the termination condition is not met, the process returns to step S201.

[0038] Figure 3 is a diagram illustrating the timing of map information updates. Keyframe images KF1 to KF6 are images captured on the trajectory 310 of the movement of the display device 100 (imaging unit 101), and are images captured by the imaging unit 101 that satisfy predetermined conditions. Keyframe images KF1 to KF6 are images retained as map information in order to calculate the position and orientation of the display device 100.

[0039] Each keyframe image is associated with the position and orientation information of the imaging unit 101 at the time the keyframe image was captured, and is stored as keyframe information. The position and orientation estimation unit 111 can generate map information by adding keyframe information to the map information or updating the keyframe information included in the map information.

[0040] Keyframe information includes keyframe images, position and orientation information, keyframe tree structure information, and feature point information. The keyframe image is an image captured by the imaging unit 101 when creating the keyframe information. The keyframe image may also include information about the creation date and time of the keyframe image as metadata. Note that the creation date and time of the keyframe image may be stored as keyframe information. Position and orientation information is information about the position and orientation of the imaging unit 101 when the keyframe image was captured, and is expressed, for example, in an XYZ coordinate system. Keyframe tree structure information is information that shows the relationships between keyframes, such as the order of shooting and the positional relationship of the locations where the keyframe images were captured. Feature point information is information about feature points detected in the keyframe image. If multiple feature points are detected in the keyframe image, the feature point information includes information about each of the feature points.

[0041] The timing for updating keyframe information is determined by conditions such as the amount of change in the movement or rotation speed of the display device 100, the clarity of the image captured by the imaging unit 101, and the number of feature points.

[0042] The predetermined conditions for determining whether or not to update the map information (keyframe information) are not limited to conditions indicating that the captured image is suitable for acquiring the position and orientation of the imaging unit 101, but may also include conditions indicating that the captured image is suitable for acquiring light source information. For example, the predetermined conditions may include that the imaging direction of the imaging unit 101 was vertically upward when the captured image was taken. The predetermined conditions may also include that the captured image of the imaging unit 101 includes feature points located vertically above the imaging unit 101, or that the captured image includes regions with a brightness greater than a predetermined brightness. The position and orientation estimation unit 111 records the added or updated keyframe information in the data storage unit 115.

[0043] In the example in Figure 3, keyframe images KF1 to KF6 satisfy predetermined conditions, and the position and orientation estimation unit 111 updates the map information at the time keyframe images KF1 to KF6 are captured. In this case, in step S203 in Figure 2, the position and orientation estimation unit 111 determines that the map information should be updated, and the process proceeds to step S204. In step S204, the position and orientation estimation unit 111 updates the map information for estimating the position and orientation of the imaging unit 101. In step S205, the light source information acquisition unit 112 acquires light source information in real space. In this way, the light source information acquisition unit 112 acquires light source information using images that satisfy predetermined conditions at the time the map information is updated.

[0044] The keyframe images added to the map information are images that meet certain conditions. The keyframe image is a clear image with little or no image blur and contains more feature points than a predetermined number. Therefore, the position and orientation of the imaging unit 101 estimated based on the keyframe image will have a smaller error than the actual position and orientation of the imaging unit 101. By using the keyframe image, the light source information acquisition unit 112 can accurately estimate the light source information in real space based on a clear image and highly accurate position and orientation information.

[0045] According to the first embodiment described above, the information processing device 110 can acquire light source information at an effective timing by using an image (keyframe image) that satisfies predetermined conditions at the timing when the map information is updated.

[0046] (Variation 1) In the first modified example, if the keyframe information (map information) is updated frequently, the light source information acquisition unit 112 may acquire the light source information at a time when some of the keyframe information update timings are omitted. For example, if the frequency at which the keyframe information is updated is higher than the first frequency, the light source information acquisition unit 112 will not acquire light source information from images that meet predetermined conditions at some of the timings in which the keyframe information is updated. The first frequency is the number of updates per unit time, and the light source information acquisition unit 112 can limit the number of times light source information is acquired per unit time. The first frequency may be set based on the processing capacity of the information processing device 110, or it may be set based on the required accuracy of the light source information.

[0047] Conversely, if the keyframe information (map information) is not updated frequently, the light source information acquisition unit 112 may acquire light source information at additional timings separate from the keyframe information update timing. For example, if the frequency of keyframe information updates is lower than the second frequency, the light source information acquisition unit 112 acquires light source information from captured images at additional timings other than the keyframe information update timing. The second frequency may be set based on the processing capacity of the information processing device 110, or it may be set based on the required accuracy of the light source information. The additional timing should be determined, for example, so that the frequency of acquiring light source information approaches the second frequency.

[0048] According to Modification 1, even if there is variation in the timing of updating the map information, the information processing device 110 can acquire light source information at the appropriate time.

[0049] (Modification 2) The first embodiment shows an example in which the updating of map information used to estimate the position and orientation of the imaging unit 101, the acquisition of light source information in the real space, and the generation of a composite image are performed in parallel. In contrast, in Modification 2, the information processing device 110 performs the generation of a composite image separately from the updating of map information and the acquisition of light source information. According to Modification 2, the user can acquire map information and light source information in the real space in advance in order to generate a composite image.

[0050] (Variation 3) In the first embodiment, light source information is estimated by the light source information acquisition unit 112 of the information processing device 110. In contrast, in Modification 3, the light source information is estimated by an external device. The light source information acquisition unit 112 may transmit the information used to estimate the light source information to another information processing device (external device) via a network, and receive and acquire the light source information estimated by the external device. According to Modification 3, by performing the estimation of light source information on an external device, the processing load of the information processing device 110 can be reduced.

[0051] [Second Embodiment] In the first embodiment, each part constituting the information processing device 110 shown in Figure 1 is hardware It is assumed that the system will consist of the above. In the second embodiment, part of the configuration of the information processing device 110 is made up of software. The information processing device 110 according to the second embodiment is a computer that realizes part of the operation described in the first embodiment by executing software, and implements the remaining operation (function) as hardware.

[0052] Figure 4 shows an example of a computer hardware configuration applicable to the information processing device 110. The CPU 401 uses the programs and data stored in the RAM 402 and ROM 403 to control the entire computer and execute the various processes of the information processing device 110 described in the above embodiment.

[0053] RAM 402 has an area for temporarily storing programs and data loaded from an external storage device 407 or a storage medium drive 408. RAM 402 also has an area for temporarily storing data received from an external device via an interface 409. The external device is, for example, a display device 100. The data received from the external device is, for example, a real-world image.

[0054] RAM 402 also has a work area used by CPU 401 when executing various processes. In other words, RAM 402 can provide various areas as appropriate. For example, RAM 402 also functions as the data storage unit 115 shown in Figure 1. ROM 403 is a non-volatile memory that stores computer configuration data and boot programs, etc.

[0055] The keyboard 404 and mouse 405 are examples of input devices that allow the computer user to input various instructions to the CPU 401 by operating them.

[0056] The display unit 406 is a CRT display or a liquid crystal display, and can display the processing results of the CPU 401 as images and characters. For example, the display unit 406 can display messages for measuring the position and orientation of the display device 100.

[0057] The external storage device 407 is a large-capacity information storage device, such as a hard disk drive. The external storage device 407 stores the OS (operating system), programs and data for the CPU 401 to execute various processes of the information processing device 110.

[0058] The programs stored in the external storage device 407 include programs corresponding to the position and orientation estimation unit 111, the light source information acquisition unit 112, the image generation unit 113, and the image synthesis unit 114, respectively. The data stored in the external storage device 407 includes data from the virtual space as well as information described as known information. The programs and data stored in the external storage device 407 are loaded into the RAM 402 as appropriate under the control of the CPU 401. The CPU 401 executes each process of the information processing device 110 by executing processing using the programs and data loaded into the RAM 402. Note that the external storage device 407 may also be used as the data storage unit 115 shown in Figure 1.

[0059] The storage medium drive 408 reads programs and data recorded on computer-readable storage media such as CD-ROMs or DVD-ROMs, and writes programs and data to these storage media. Some or all of the programs and data stored in the external storage device 407 may also be recorded on these storage media. The programs and data read by the storage medium drive 408 from the storage media are output to the external storage device 407 or RAM 402.

[0060] I / F409 is an analog video port or a digital input / output port such as IEEE1394 for connecting the imaging unit 101 of the display device 100. I / F409 also has This may be an Ethernet® port for outputting the composite image to the display unit 102 of the display device 100. The data received via I / F409 is input to RAM402 or external storage device 407. Furthermore, when a sensor system is used to acquire position and attitude information by the position and attitude estimation unit 111, I / F409 is used as an interface for connecting the sensor system. Bus 410 connects the various parts shown in Figure 4 to each other.

[0061] The present invention includes cases in which a software program is supplied directly or remotely to a system or device, and the computer of the system or device reads and executes the supplied program code to achieve the functions of the above embodiments (including each modification). The program supplied to the system or device is a program for performing the processes corresponding to the flowchart described in Figure 2.

[0062] The functions in the above embodiments (including each modified example) may be realized by the computer executing the read program, or they may be realized in cooperation with an OS running on the computer based on the program's instructions. In this case, the functions in the embodiments are realized by the OS or other system executing some or all of the functions.

[0063] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0064] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0065] Furthermore, the embodiments described above (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0066] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0067] This embodiment includes the following configurations, methods, and programs. (Composition 1) A generation means for generating map information for estimating the position and orientation of an imaging device using images that satisfy predetermined conditions from among the images captured by the imaging device, Acquisition means for acquiring light source information using an image that satisfies the predetermined conditions at the timing when the map information is updated. An information processing device characterized by having the following features. (Configuration 2) The process of updating the map information includes at least one of the following processes: adding, modifying, or deleting keyframe images included in the map information, and adding, modifying, or deleting feature points included in the keyframe images. The information processing device according to configuration 1, characterized by the above. (Composition 3) The process by which the acquisition means acquires the light source information includes at least one of the following: a process for estimating the light source information; a process for determining whether or not it is necessary to estimate the light source information; and a process for applying previously estimated light source information. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 4) The acquisition means acquires the light source information based on the exposure value or white balance value when an image satisfying the predetermined conditions is captured. An information processing device according to any one of configurations 1 to 3, characterized by the above. (Composition 5) The predetermined conditions are those that indicate the captured image is suitable for estimating the position and orientation. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The conditions indicating that the captured image is suitable for estimating the position and orientation include at least one of the following: the acceleration of the imaging device when the image is captured is less than a predetermined acceleration; the angular acceleration of the imaging device when the image is captured is less than a predetermined angular acceleration; an index representing the clarity of the image is greater than a predetermined value; and the number of feature points included in the image is greater than a predetermined number. The information processing apparatus according to configuration 5, characterized by the features described herein. (Composition 7) The predetermined conditions include conditions indicating that the captured image is suitable for acquiring the light source information. The information processing apparatus according to configuration 5 or 6, characterized by the above. (Composition 8) The conditions indicating that the captured image is suitable for acquiring the light source information include at least one of the following: the imaging direction of the imaging device when the image was captured was vertically upward; the captured image includes feature points located vertically above the imaging device; and the captured image includes regions with a brightness greater than a predetermined brightness. The information processing apparatus according to configuration 7, characterized by the features described above. (Composition 9) If the frequency at which the map information is updated is higher than the first frequency, the acquisition means will not acquire the light source information from images that meet the predetermined conditions at some of the multiple timings in which the map information is updated. An information processing device according to any one of configurations 1 to 8, characterized by the above. (Composition 10) The acquisition means acquires the light source information from the captured image at additional timings other than the timing when the map information is updated, if the frequency at which the map information is updated is lower than the second frequency. An information processing device according to any one of configurations 1 to 9, characterized by the above. (Composition 11) The system further includes a retention means for holding information to be deleted from the aforementioned map information, The acquisition means acquires the light source information using the information held in the holding means. An information processing device according to any one of configurations 1 to 10, characterized by the above. (Composition 12) The acquisition means transmits information used to estimate the light source information to an external device, and the external device acquires the estimated light source information. An information processing device according to any one of configurations 1 to 11, characterized by the above. (method) A step of generating map information for estimating the position and orientation of the imaging device using images that satisfy predetermined conditions from among the images captured by the imaging device, A step of acquiring light source information using an image that satisfies the predetermined conditions at the timing when the map information is updated. An information processing method characterized by having the following features. (program) A program for causing a computer to function as one of the means of the information processing device described in any of configurations 1 to 12. [Explanation of Symbols]

[0068] 110: Information processing device, 111: Position and orientation estimation unit, 112: Light source information acquisition unit, 101: Imaging unit

Claims

1. A generation means for generating map information for estimating the position and orientation of an imaging device using images that satisfy predetermined conditions from among the images captured by the imaging device, Acquisition means for acquiring light source information using an image that satisfies the predetermined conditions at the timing when the map information is updated. An information processing device characterized by having the following features.

2. The process of updating the map information includes at least one of the following processes: adding, modifying, or deleting keyframe images included in the map information, and adding, modifying, or deleting feature points included in the keyframe images. The information processing apparatus according to feature 1.

3. The process by which the acquisition means acquires the light source information includes at least one of the following: a process for estimating the light source information; a process for determining whether or not it is necessary to estimate the light source information; and a process for applying previously estimated light source information. The information processing apparatus according to feature 1.

4. The acquisition means acquires the light source information based on the exposure value or white balance value when an image satisfying the predetermined conditions is captured. The information processing apparatus according to feature 1.

5. The predetermined conditions are those that indicate the captured image is suitable for estimating the position and orientation. The information processing apparatus according to feature 1.

6. The conditions indicating that the captured image is suitable for estimating the position and orientation include at least one of the following: the acceleration of the imaging device when the image is captured is less than a predetermined acceleration; the angular acceleration of the imaging device when the image is captured is less than a predetermined angular acceleration; an index representing the clarity of the image is greater than a predetermined value; and the number of feature points included in the image is greater than a predetermined number. The information processing apparatus according to feature 5.

7. The predetermined conditions include conditions indicating that the captured image is suitable for acquiring the light source information. The information processing apparatus according to feature 5.

8. The conditions indicating that the captured image is suitable for acquiring the light source information include at least one of the following: the imaging direction of the imaging device when the image was captured was vertically upward; the captured image includes feature points located vertically above the imaging device; and the captured image includes regions with a brightness greater than a predetermined brightness. The information processing apparatus according to feature 7.

9. If the frequency at which the map information is updated is higher than the first frequency, the acquisition means will not acquire the light source information from images that satisfy the predetermined conditions at some of the multiple timings in which the map information is updated. The information processing apparatus according to feature 1.

10. If the frequency at which the map information is updated is lower than the second frequency, the acquisition means will use the captured images at additional timings other than the timing at which the map information is updated to obtain the Acquire light source information The information processing apparatus according to feature 1.

11. The system further includes a retention means for holding information to be deleted from the aforementioned map information, The acquisition means acquires the light source information using the information held in the holding means. The information processing apparatus according to feature 1.

12. The acquisition means transmits information used to estimate the light source information to an external device, and the external device acquires the estimated light source information. The information processing apparatus according to feature 1.

13. A step of generating map information for estimating the position and orientation of the imaging device using images that satisfy predetermined conditions from among the images captured by the imaging device, A step of acquiring light source information using an image that satisfies the predetermined conditions at the timing when the map information is updated. An information processing method characterized by having the following features.

14. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 12.

Images