Information processing apparatus, information processing method, and program

JP2024081277A5Pending Publication Date: 2025-11-06CANON KK

View PDF 0 Cites 0 Cited by

Patent Information

Application Number: JP2022194787
Authority / Receiving Office: JP · JP
Patent Type: Applications
Current Assignee / Owner
Filing Date: 2022-12-06
Publication Date: 2025-11-06

AI Technical Summary

Benefits of technology

【0008】 本発明によれば、撮影シーンの変化による影響を抑制し、安定してカメラの位置姿勢を推定することができる。

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure 00000000_0000_ABST

Patent Text Reader

Abstract

To provide an information processing apparatus capable of stably estimating a position and attitude of a camera by suppressing the influence of changes in a shooting scene.SOLUTION: The information processing apparatus has: acquisition means for acquiring three-dimensional information of a real space from an image of the real space obtained by an imaging unit; selection means for setting a plurality of regions in the real space based on the three-dimensional information and selecting a selection region, which is a region used for setting the imaging unit, from the plurality of regions based on at least one of a subject distance, luminance and feature points of each region; and setting means for setting the imaging unit in accordance with the selection region.SELECTED DRAWING: Figure 1

Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, the introduction of mixed reality (MR) systems aimed at seamlessly combining real space and virtual space has been progressing. In MR technology, in order to accurately align the real space and virtual space, it is required to accurately estimate the position and orientation of the camera in the real space in real time and stably. Therefore, a method for determining the camera settings has been proposed so that feature points used to estimate the camera position and orientation can be appropriately extracted. For example, Non-Patent Document 1 proposes a method for optimizing the exposure time of the camera using an index that allows more feature points to be extracted. Patent Document 1 also proposes a method for setting the shooting conditions of the camera using distance information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-191624 A [Non-patent literature]

[0004] [Non-Patent Document 1] Z. Zhang.et al.,“Active Exposure Control for Robust Visual Odometry in HDR Environments”,2017 IEEE International Conference on Robotics and Automation(ICRA),2017 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Non-Patent Document 1, the camera settings are gradually optimized by repeated calculations, so when a window with a different brightness from the room comes into the imaging range, the camera settings are not changed in real time and are affected by the change in brightness until they are optimized. Also, just considering distance information as in Patent Document 1 can make it difficult to appropriately set the camera's imaging conditions in response to changes in the imaging scene and accurately estimate the camera's position and orientation.

[0006] An object of the present invention is to provide an information processing device that can suppress the influence of changes in the shooting scene and stably estimate the position and orientation of a camera. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having an acquisition means for acquiring three-dimensional information of the real space from an image of the real space obtained by an imaging unit, a selection means for setting a plurality of areas in the real space based on the three-dimensional information and selecting from the plurality of areas a selection area to be used for setting the imaging unit based on at least one of a subject distance, a brightness, and a number of feature points of each area, and a setting means for setting the imaging unit according to the selection area. Effect of the Invention

[0008] According to the present invention, it is possible to suppress the influence of changes in the captured scene and stably estimate the position and orientation of the camera. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a functional configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3] 4 is a flowchart illustrating a process of the information processing device according to the first embodiment. [Figure 4]FIG. 13 is a diagram illustrating a method for determining a reliability using a distance histogram. [Diagram 5] FIG. 11 is a diagram illustrating an example of a functional configuration of an information processing system according to a second embodiment. [Figure 6] 10 is a flowchart illustrating a process of an information processing device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> Hereinafter, embodiments of the present invention will be described with reference to the drawings. An information processing device according to a first embodiment sets a plurality of areas in a real space based on three-dimensional information, and sets a camera based on the reliability of each of the plurality of areas.

[0011] The three-dimensional information is information used to restore a three-dimensional space, and includes an image of the real space, the position and orientation of the camera when the image of the real space was acquired, and a world coordinate system. The reliability is determined for each region based on the subject distance, the brightness value, and the number of feature points. The information processing device selects a region (selected region) to be used for the camera settings based on the reliability of each region. By determining the camera settings such as exposure and white balance according to the selected region, the information processing device can appropriately extract more feature points from the image.

[0012] Feature points are points in an image that have distinctive changes in color, etc., and by matching feature points between multiple images, it is possible to estimate the position and orientation of the camera. Visual SLAM (Simultaneous 3D Modeling) is a method for estimating the position and orientation of a camera in real space. A method called SLAM (Simultaneous Localization and Mapping) is known. SLAM is used not only for mixed reality systems, but also for self-localization of robots and automated guided vehicles.

[0013] The position and orientation of the camera relative to a reference coordinate system defined in a scene can be estimated based on feature points fixed to the scene (i.e., the three-dimensional coordinates do not change). By extracting more and more feature points accurately, the position and orientation of the camera can be estimated more accurately. In order to extract more and more feature points from an image accurately, camera settings such as exposure and white balance are important. However, in an indoor scene, if a window that is brighter than the room comes into the imaging range, the exposure of the camera may be set according to the brightness of the window. As a result, the room is imaged dark, and the number of feature points extracted from the indoor area decreases. Also, even if the number of feature points behind the window increases, the overall number of feature points may decrease. The information processing device selects an area based on the reliability of each of multiple areas in the real space and sets the camera using the selected area (selected area), thereby extracting more and more feature points accurately. By extracting more and more feature points accurately, the information processing device can improve the estimation accuracy of the position and orientation of the camera by SLAM or the like.

[0014] Fig. 1 is a block diagram illustrating a functional configuration of an information processing system according to embodiment 1. The information processing system in Fig. 1 includes a camera 100 and an information processing device 200. The camera 100 and the information processing device 200 are connected to each other by at least one of a wired connection and a wireless connection so as to be able to communicate data with each other.

[0015] The camera 100 will be described as a stereo camera that captures moving images in real space, but it may be a monocular camera or two or more cameras. Images of each frame included in the moving images captured by each camera (images of real space) are transmitted to the information processing device 200. When the camera 100 includes multiple cameras such as a stereo camera, it is assumed that the image frames from the multiple cameras input to the information processing device 200 are synchronized (the image capture times are the same). In addition, the following description is given assuming that the internal and external parameters of each camera are calibrated in advance.

[0016] By specifying an area on a captured image, camera 100 performs photometry of the real space of the specified area, and adjusts settings such as aperture value, shutter speed, ISO sensitivity, and exposure compensation according to the photometry results to determine the optimal exposure. Camera 100 may adjust the white balance excluding the specified area, or may adjust the white balance using the specified area.

[0017] The method of determining the exposure and adjusting the white balance is not limited to the above. For example, by providing the camera 100 with areas suitable for setting the exposure and white balance in the captured image, the camera 100 can set the exposure and white balance using the suitable areas. Also, by providing the camera 100 with areas unsuitable for setting the exposure and white balance in the captured image, the camera 100 can set the exposure and white balance without using the unsuitable areas.

[0018] The information processing device 200 is, for example, a computer device such as a PC (personal computer), or a portable terminal device such as a smartphone or a tablet. The information processing device 200 may be configured to include a camera 100. The information processing device 200 has, as functional units, an estimation unit 210, a three-dimensional information acquisition unit 211, a feature point number acquisition unit 212, a determination unit 213, and a setting unit 214. The processing of each functional unit will be described in detail in the description of the flowchart shown in FIG. 3.

[0019] 2 is a diagram illustrating an example of the hardware configuration of an information processing device. The information processing device 200 has, as the basic components of a computer, a processor 201, a memory 202, a storage medium 203, an input I / F 205, and an output I / F 206. The processor 201, the memory 202, the storage medium 203, the input I / F 205, and the output I / F 206 can exchange data with each other via a bus 204.

[0020] The processor 201 is, for example, a CPU, and controls the operation of the entire computer. The memory 202 is, for example, a RAM, and temporarily stores programs, data, etc. The computer-readable storage medium 203 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. on a long-term basis.

[0021] The input I / F 205 includes an operation input interface connected to an operation input device that inputs data in response to a user's operation. The operation input device is, for example, a pointing device or a keyboard. The input I / F 205 also includes an external device interface that acquires various data such as image data from the camera 100. The output I / F 206 includes an external device interface that supplies image display data stored in the memory 202 or the storage medium 203 to a display device or the like.

[0022] Fig. 3 is a flowchart illustrating the processing of the information processing device 200 according to embodiment 1. The processing shown in Fig. 3 is processing for setting the camera 100 suitable for estimating the position and orientation of the camera 100 using feature points extracted from an image in real space. The processing in Fig. 3 is realized by the processor 201 reading into the memory 202 a program that realizes the functions of each functional unit of the information processing device 200 stored in the storage medium 203 and executing the program.

[0023] In step S301, the setting unit 214 sets the camera 100, such as exposure, in accordance with a predetermined area. In step S301, since three-dimensional information for determining the settings of the camera 100 is not acquired, the predetermined area may be, for example, an area including the center of the shooting range, or an area designated by the user and recorded in advance in the storage medium 203, etc. Also, the white balance setting may be determined for the entire image without designating an area.

[0024] In step S302, the estimation unit 210 estimates the position and orientation of the camera 100 from images in real space acquired by the camera 100. First, the estimation unit 210 extracts feature points from each of the images successively sent from the camera 100.

[0025] The feature points are extracted by determining the image coordinates of the feature points in the image. For example, the estimation unit 210 can extract points where the luminance gradient between neighboring pixels is equal to or greater than a threshold as feature points. The luminance gradient is the amount of change in density values (pixel values) of adjacent pixels on an image. For example, the estimation unit 210 can apply a Sobel filter operator or a Prewitt filter operator to the horizontal or vertical direction of the image and obtain the output value as the luminance gradient. The estimation unit 210 can extract feature points based on the luminance gradient.

[0026] The feature points may be feature points (natural feature points) each having a different texture feature. The estimation unit 210 can detect features from an image by performing template matching on the image using a template image of each feature that is stored in advance as known information. The estimation unit 210 may also use a feature that has an identifier and is identifiable, such as a SIFT feature. However, the estimation unit 210 may use any feature that is fixed in space and can be detected from a captured image.

[0027] Next, the estimation unit 210 estimates the position and orientation of the camera 100 using a map corresponding to the features. The map corresponding to the features is held as three-dimensional coordinates in a world coordinate system of feature points extracted from a plurality of images. Information on feature points in a three-dimensional space can be estimated, for example, by associating coordinates on the images. The estimation unit 210 tracks feature points from images captured at different positions in a time series, for example, by a feature tracking method called KLT (Kanade-Lucas-Tomasi Feature Tracker), and associates feature points between a plurality of images. The estimation unit 210 estimates a transformation matrix called an E matrix (fundamental matrix) from the correspondence of feature points between a plurality of images, and obtains the position and orientation of the camera 100 from the E matrix. The estimation unit 210 estimates three-dimensional coordinates of the positions of feature points by a stereo method based on the relative positions and orientations between a plurality of images.

[0028] When the camera 100 is a stereo camera capable of measuring the depth of pixels, the estimation unit 210 may estimate the three-dimensional coordinates of the feature points with respect to the image coordinates. It is assumed that the parameters of the stereo camera are calibrated in advance. The estimation unit 210 acquires the three-dimensional coordinates of the feature points in the world coordinate system using the estimated position and orientation of the camera 100 in the world coordinate system, and adds the feature points to the map.

[0029] The method of estimating the position and orientation of the camera 100 using the feature points and the map may be any known method. For example, the estimation unit 210 projects the feature points included in the map onto the input image. The projected feature points may be, for example, feature points associated with the position and orientation of the camera 100 that is closest to the current position and orientation of the camera 100 among the positions and orientations of the camera 100 when the most recent image was captured in time series. The estimation unit 210 estimates the position and orientation of the camera 100 in the world coordinate system based on the luminance values around the position of the projected feature point in the image so that the luminance values at the positions of the projected feature points match.

[0030] Also, if the map does not hold feature points, the estimation unit 210 estimates the position and posture of the camera 100. As the orientation, a predetermined value (e.g., the origin of the world coordinate system) may be used, or a position and orientation calculated by another method may be used. Other methods include, for example, a method of calculating the position and orientation of the camera 100 by detecting a marker in an image, or a method of using a sensor capable of tracking the position and orientation of an object, such as a motion capture technique. The estimation unit 210 sends the estimated position and orientation of the camera 100 to the setting unit 214.

[0031] In step S303, the three-dimensional information acquisition unit 211 acquires three-dimensional information of the real space. The three-dimensional information is information used to restore the three-dimensional space, and includes an image of the real space, the position and orientation of the camera 100 when the image of the real space was acquired, and a world coordinate system. The three-dimensional information acquisition unit 211 records the acquired three-dimensional information in a storage unit such as the memory 202 or the storage medium 203. By recording the acquired three-dimensional information in the storage unit with the same world coordinate system, in step S301, the setting unit 214 can read out the recorded results and start the flowchart with the three-dimensional information from the beginning.

[0032] The three-dimensional information may include a point cloud or polygon having three-dimensional position information in a world coordinate system. For example, if the camera 100 is a stereo camera capable of measuring the depth of pixels, the three-dimensional information may be depth information in the real space and information on the position and orientation of the camera 100 in the world coordinate system. The three-dimensional information may also include information on a three-dimensional point cloud in the world coordinate system and a polygon formed by connecting the point clouds, which are created from the depth information and the position and orientation of the camera 100. The three-dimensional information may also be information on a feature point whose position in three dimensions is known and extracted by the estimation unit 210, and information on a map generated by the estimation unit 210. The three-dimensional information acquisition unit 211 sends the acquired three-dimensional information of the real space to the feature point number acquisition unit 212.

[0033] In addition, when past three-dimensional information acquired at substantially the same position and orientation as the current position and orientation of the camera 100 is recorded in the storage unit, the three-dimensional information acquisition unit 211 may acquire the past three-dimensional information from the storage unit and reuse the past three-dimensional information. By reusing the past three-dimensional information, the three-dimensional information acquisition unit 211 can reduce the processing load of acquiring the three-dimensional information. When the past three-dimensional information is recorded together with the information on the brightness value and the number of feature points for each region previously acquired in step S304, the processing of step S304 can be omitted. Furthermore, when the past three-dimensional information is recorded together with the information on the reliability for each region previously acquired in step S305, the processing of step S305 can be omitted.

[0034] When the current three-dimensional information has changed from the past state recorded in the storage unit due to the movement of an object within the imaging range (for example, furniture has moved during indoor imaging), the current three-dimensional information may be used. When past three-dimensional information acquired at approximately the same position and orientation exists, the three-dimensional information acquisition unit 211 may proceed to the process of step S304 using the past three-dimensional information regardless of whether the current three-dimensional information has changed. In this case, the three-dimensional information acquisition unit 211 executes a process of acquiring the current three-dimensional information in parallel, and when the current three-dimensional information differs from the past three-dimensional information, updates the past three-dimensional information recorded in the storage unit. The updated three-dimensional information is used when setting the camera 100 from the next time onwards.

[0035] In step S304, feature point number acquisition unit 212 acquires information on the luminance value and the number of feature points from the three-dimensional information acquired in step S303. The acquired luminance value and the number of feature points are the luminance value and the number of feature points of the image acquired by camera 100. The feature points are the feature points extracted in step S302 to estimate the position and orientation of camera 100. If the image is a color image expressed in RGB, the luminance value of the image may be the average value of the RGB values, or may be the luminance value obtained by adding the RGB values with weighting according to human visual characteristics.

[0036] The brightness value of an image may take into consideration the exposure at the time of shooting. The exposure is indicated by an EV value, and is determined based on the shutter speed, the lens aperture value, the ISO sensitivity, and the like. The amount of change in brightness value due to differences in exposure (for example, the amount of change in the average brightness value) can be obtained using images of the same scene captured at different EV values. For example, the amount of change in brightness value when the setting is changed from EV value 0 to EV value 1 can be obtained using an image captured at EV value 0 and an image captured at EV value 1. The feature point number acquisition unit 212 can acquire a brightness value taking exposure into consideration by increasing or decreasing the amount of change obtained from the brightness value of the image.

[0037] By acquiring in advance the amount of change in luminance value from an image captured at a reference EV value for each EV value that can be set by the camera 100, the feature point number acquisition unit 212 can acquire a luminance value that takes into account the exposure at the time of capture. Note that the method of acquiring the luminance value that takes into account the exposure is not limited to the above method. For example, the feature point number acquisition unit 212 may acquire the luminance value that takes into account the exposure by multiplying a predetermined coefficient according to the EV value.

[0038] Feature point number acquisition section 212 projects the image of camera 100 onto the three-dimensional information in order to associate the brightness value and the number of feature points of the acquired image with the three-dimensional information. When the three-dimensional information is a three-dimensional position in a world coordinate system, feature point number acquisition section 212 may project the image onto the three-dimensional information using the position and orientation of camera 100 and internal parameters of camera 100. Feature point number acquisition section 212 sets a plurality of regions (two-dimensional regions) in real space, and associates the number of feature points and the average brightness value of the image with each region.

[0039] A method of setting a plurality of regions in real space includes, for example, a method of setting a region by a cube of a predetermined size (a region obtained by projecting a cube onto an image in real space) and a method of setting a region using a method such as a mean-shift method. The feature point number acquisition unit 212 may set a region using a known method, such as setting a range in which a subject such as a desk and a chair recognized using machine learning exists as a region. The feature point number acquisition unit 212 sends the three-dimensional information acquired in step S303 and the information on the brightness value and the number of feature points acquired in step S304 to the determination unit 213. The feature point number acquisition unit 212 may record the information on the brightness value and the number of feature points together with the three-dimensional information in a storage unit such as the memory 202.

[0040] In step S305, the determination unit 213 determines the reliability for each area set in the real space using the three-dimensional information, the brightness value, and the number of feature points. The reliability is used to determine the setting of the camera 100. Specifically, the reliability of each area is determined based on the subject distance (the distance from the camera 100 to the area), the brightness, and the number of feature points.

[0041] Among the regions set in step S304, the reliability of the distance (hereinafter referred to as distance reliability) is set lower in the regions where the subject distance is equal to or greater than the distance threshold than in the regions where the subject distance is less than the distance threshold. The distance from the camera 100 to the region can be, for example, the average value, mode, or median of the distances corresponding to each pixel in the region.

[0042] Depth information of feature points extracted in an area where the subject distance is equal to or greater than a distance threshold (hereinafter referred to as distance threshold) may have low accuracy, so the distance reliability is set lower in the area where the subject distance is equal to or greater than the distance threshold. Also, regarding white balance, the distance reliability is set lower in the area where the subject distance is equal to or greater than the distance threshold, because the light source may be different. Multiple distance thresholds may be set, and the distance reliability is determined to be lower, for example, as the distance becomes longer. In this way, the distance reliability is determined based on the relationship between the subject distance of the area and the distance threshold.

[0043] The distance threshold may be set using a histogram of object distances. For example, the distance threshold may be , can be obtained from a histogram using Otsu's method or other known methods. The number of distance thresholds may be specified by the user or may be set in advance in the information processing device 200. Note that the distance confidence is not limited to being determined based on the relationship between the subject distance of the region and the distance threshold, and may be determined by a function or table for obtaining the distance confidence corresponding to the subject distance of the region.

[0044] The reliability of the area is not limited to the distance reliability, but may be the reliability of the brightness value (hereinafter referred to as brightness reliability). For example, if the exposure is adjusted to the area outside the window, which has a different brightness from the indoor area, the number of feature points that have been extracted indoors may decrease. In addition, since the light source outside the window is different from the indoor area, the area outside the window, where the brightness is too high or too low, may not be appropriate for adjusting the white balance.

[0045] In order to lower the luminance reliability of an area where the luminance is too high or too low, for example, a first luminance threshold and a second luminance threshold lower than the first luminance threshold are set as luminance value thresholds (hereinafter also referred to as luminance thresholds). The luminance reliability of an area where the luminance value is equal to or greater than the first luminance threshold or equal to or less than the second luminance threshold is determined to be lower. The luminance value of an area can be, for example, the average value, mode, or median of the luminance values of each pixel in the area.

[0046] The first luminance threshold and the second luminance threshold may be set using a luminance histogram. The luminance reliability may be determined to be highest in an area having a luminance value that is the most frequent value in the luminance histogram, for example. In an area where the luminance value is equal to or higher than the first luminance threshold or equal to or lower than the second luminance threshold, the luminance reliability is determined to be lower. The number of luminance thresholds is not limited to two, and one or more than three luminance thresholds may be provided. The luminance reliability may be determined to be highest in an area having a luminance value that is the most frequent value in the luminance histogram, and may be determined to be lower each time the threshold is exceeded. In this way, the luminance reliability is determined based on the relationship between the luminance value of the area and the luminance threshold. Note that the luminance reliability is not limited to being determined based on the relationship between the luminance value of the area and the luminance threshold, and may be determined by a function or table for determining the luminance reliability corresponding to the luminance value of the area.

[0047] The reliability of an area is not limited to the distance reliability and the brightness reliability, but may be the reliability of the number of feature points (hereinafter referred to as the feature point reliability). A threshold value for the number of feature points is set, and an area having a number of feature points less than the threshold value for the number of feature points (hereinafter referred to as the feature point threshold) is determined to have a lower feature point reliability. For example, even if the camera 100 is set to match an area having a number of feature points less than the feature point threshold, the number of feature points may not increase, so the feature point reliability of the area is determined to be lower. The feature point threshold may be set by the user, or may be set based on the total number of feature points extracted from each area. In this way, the feature point reliability is determined based on the relationship between the number of feature points of the area and the feature point threshold. Note that the feature point reliability is not limited to the case where it is determined based on the relationship between the number of feature points of the area and the feature point threshold, and may be determined by a function or a table for obtaining the feature point reliability corresponding to the number of feature points of the area.

[0048] The distance reliability, brightness reliability, and feature point reliability are expressed, for example, by integer values, and are set to larger values as the reliability increases. The reliability of an area can be any one of the distance reliability, brightness reliability, and feature point reliability. In addition, the reliability values that the distance reliability, brightness reliability, and feature point reliability can take may be set to a common value (for example, expressed by three integer levels from 1 to 3), and the reliability of an area may be set to the lowest or highest reliability of the three reliability levels. In other words, which reliability of the distance reliability, brightness reliability, and feature point reliability is adopted may differ for each area.

[0049] The reliability of an area is calculated by adding the distance reliability, brightness reliability, and feature point reliability of the area. The reliability may be a value obtained by weighting and adding up the reliability of each reliability. The weighting for each reliability may be set by the user depending on the environment. The reliability for white balance and the reliability for exposure may be set separately.

[0050] Since the three-dimensional information is based on the world coordinate system, the determination unit 213 may add newly acquired information of the three-dimensional information for which the reliability of the region has been determined to the storage unit. The newly acquired information is, for example, a point group in real space, a polygon, depth information, the position and orientation of the camera when the depth information was acquired, and feature points extracted to obtain the position and orientation. In addition, the determination unit 213 may update three-dimensional information previously acquired with approximately the same position and orientation, using the three-dimensional information for which the reliability of the region has been determined. For example, the determination unit 213 may record information on the determined reliability in a storage unit such as the memory 202 together with the three-dimensional information. The determination unit 213 sends the three-dimensional information for which the reliability has been determined to the setting unit 214.

[0051] In step S306, the setting unit 214 selects an area (selection area) for adjusting the settings of the camera 100, using the three-dimensional information for which the reliability has been determined and the position and orientation of the camera 100. The setting unit 214 projects the three-dimensional information for which the reliability has been determined onto an image using the position and orientation of the camera 100, and selects an area for adjusting the settings of the camera 100 from the image onto which the reliability information has been projected.

[0052] The setting unit 214 may select an area with the highest reliability as an area for adjusting the exposure. Alternatively, the setting unit 214 may select an area with a reliability equal to or greater than a predetermined threshold as an area for adjusting the exposure.

[0053] Furthermore, the setting unit 214 may select an area closest to the area most recently used for setting the camera 100 from among areas whose reliability is equal to or greater than a predetermined threshold. Also, the setting unit 214 may select an area with the highest reliability from among areas within a predetermined range from the area most recently used for setting the camera 100. If the camera 100 is set to an area far from the area most recently used for setting the camera 100 from among a plurality of areas set in the image in real space, the appearance of the image may change. For this reason, by selecting an area (selected area) based on the area most recently used for setting the camera 100 from among a plurality of areas set in the image in real space, changes in the image due to changes in the settings of the camera 100 are suppressed. The predetermined threshold and the predetermined range may be set by the user or may be preset values.

[0054] Regarding the white balance, the setting unit 214 may adjust the white balance in a region other than the region where the reliability is equal to or lower than the predetermined threshold, or may adjust the white balance using a region where the reliability is equal to or higher than the predetermined threshold. When setting the white balance, the setting unit 214 may use the same region as that used for setting the exposure, or may use a different region. The predetermined threshold may be set by the user, or may be a value set in advance.

[0055] In step S307, the processor 201 determines whether or not the user has input an instruction to end the process shown in Fig. 3. If an instruction to end the process has been input, the process shown in Fig. 3 ends. If an instruction to end the process has not been input, the process returns to S302.

[0056] In the above-mentioned first embodiment, the information processing device 200 determines the reliability of each of a plurality of areas set in real space based on the subject distance, brightness, and number of feature points of the area, and sets the camera 100 according to the selected area selected based on the reliability. When a window that is brighter than the room comes into the imaging range during imaging indoors, the information processing device 200 determines the reliability of the window area to be lower than that of the area inside the room. By setting the camera 100 according to a selected area with a higher reliability, the information processing device 200 can reduce the influence of the window area. Without this, it is possible to stably extract feature points suitable for estimating the position and orientation of the camera 100 from the indoor area. By stably extracting feature points suitable for estimating the position and orientation of the camera 100, the information processing device 200 can improve the accuracy of alignment using SLAM or the like and estimate the position and orientation of the camera 100 more accurately.

[0057] <Modification of the first embodiment> In the above-mentioned embodiment 1, the information processing device 200 determines the reliability of the region using distance information, brightness information, information on the number of feature points, etc. In a modification of the embodiment 1, the information processing device 200 uses a histogram of distances (distance histogram) when determining the reliability of the region. In the following, the description of the same parts as in the embodiment 1 will be omitted, and the description will focus on the different parts.

[0058] In the process of the information processing device 200 according to the modification, the process in which the determination unit 213 determines the reliability is different from that in the embodiment 1. The determination unit 213 according to the modification determines the reliability of the region by using a distance histogram in addition to the method of determining the reliability of the region in the embodiment 1.

[0059] FIG. 4 is a diagram for explaining a method of determining reliability using a distance histogram. In the distance histogram shown in FIG. 4, the distance (subject distance) indicated on the horizontal axis is divided into three ranges, a first range, a second range, and a third range, in order of distance from the camera 100. The threshold A dividing the first range and the second range, and the threshold B dividing the second range and the third range, may be, for example, a distance at which the frequency of the histogram is minimal. The threshold A and the threshold B may be set by a user. The number of thresholds is not limited to two, and may be one or three or more. The thresholds such as the threshold A and the threshold B may be set by Otsu's method. When using the Otsu's method, the number of thresholds is determined in advance.

[0060] In the first embodiment, since the first range is closer to the subject than the second and third ranges, the reliability of the first range is determined to be higher than the reliability of the second and third ranges. In contrast, in the modified example, the reliability of the area is determined by further considering the frequency in the distance histogram. Specifically, since the frequency of the first range is lower than that of the second range, the reliability of the first range is determined to be relatively lower than the reliability of the second range. Also, since the frequency of the second range is higher than that of the first range, the reliability of the second range is determined to be relatively higher than the reliability of the first range.

[0061] A specific example of a method for determining reliability based on the subject distance and the frequency of the distance histogram will be described with reference to Fig. 4. In the example of Fig. 4, the reliability of an area is calculated by adding the reliability based on the frequency to the distance reliability. When two thresholds, threshold A and threshold B, are set, the distance reliability of the area in the first range, second range, and third range is set to 3, 2, and 1, respectively.

[0062] The confidence scores based on the frequency are determined so that the sum of the distance confidence scores (3+2+1=6) is 6. The confidence scores based on the frequency are determined based on the ratio of the sum of the frequencies in each range to the sum of the frequencies in the distance histogram.

[0063] For example, the total frequency of the distance histogram in FIG. 4 is 1000, and the frequency of pixels included in the second range is 800 (80% of the total). The frequencies of pixels included in the first range, which is closer to camera 100 than the second range, and the third range, which is farther from camera 100 than the second range, are each 100 (10% of the total). In this case, in the area corresponding to the second range, the reliability based on the frequency is 4.8, which is 80% of the total reliability of 6. In the areas corresponding to the first and third ranges, the reliability based on the frequency is 0.6, which is 10% of the total reliability of 6. By adding the distance reliability and the reliability based on the frequency, the reliability of the first range, The confidence levels of the regions corresponding to the second and third ranges are determined to be 3.6, 6.8 and 1.6, respectively.

[0064] As shown in the example of FIG. 4, even if the second range is farther from the camera 100 than the first range, the frequency is higher than that of the first range, so that the feature points may be extracted more in the second range than in the first range. Therefore, as in the first embodiment, when the distance confidence is determined using a distance threshold, the number of feature points extracted in the first range may be reduced. The information processing device 200 can perform alignment by SLAM or the like without reducing the number of extracted feature points by determining the confidence of the region using a distance histogram in addition to the distance confidence. In the modified example, an example in which the distance confidence and the distance histogram are combined is shown, but the information processing device 200 may determine the confidence of the region by combining the distance confidence, the brightness confidence, the feature point confidence, or a combination of these with the distance histogram.

[0065] <Embodiment 2> In the first embodiment, it is assumed that the information processing device 200 is used for self-position estimation of a robot or an automatic guided vehicle. In contrast, in the second embodiment, it is assumed that a virtual space (VR: Virtual Reality) system or an MR system is used, and the reliability of a plurality of regions set in a real space is determined.

[0066] In the following, differences from the first embodiment will be mainly described, and descriptions of similarities to the first embodiment will be omitted. Fig. 5 is a diagram illustrating a functional configuration of an information processing system according to the second embodiment. The information processing system according to the second embodiment has an HMD 500 and an information processing device 510. The HMD 500 and the information processing device 510 are connected to each other by at least one of wired and wireless communication so that data can be communicated with each other. The information processing device 510 may be configured as the same device as the HMD 500.

[0067] The HMD 500 is an image display device equipped with an MR system and capable of allowing a user to experience an MR space. The image display device used in the MR system is of a video see-through type or an optical see-through type. The video see-through type image display device displays a composite image in which an image of a virtual space (such as a virtual object or text information drawn by computer graphics) generated according to the position and orientation of the camera 100 is superimposed on an image of a real space captured by the camera 100. The optical see-through type image display device displays an image of a virtual space generated according to the position and orientation of the user's viewpoint on an optical see-through display worn on the user's head.

[0068] By wearing the HMD500 on the head, the user can experience the MR space through the HMD500. In the second embodiment, the image display device is described as the HMD500, which is a head-mounted display device, but is not limited to this. The image display device may be another type of head-mounted display device. In addition, the image display device is not limited to a head-mounted display device, and may be any display device that allows the user to view images in the VR space or the MR space, for example, a handheld display device.

[0069] The HMD500 has a display unit 501 and a camera 100. The display unit 501 has a display screen for the right eye and a display screen for the left eye. The display screen for the right eye displays images (images and text) based on a video signal for the right eye sent from the information processing device 510. The display screen for the left eye displays images (images and text) based on a video signal for the left eye sent from the information processing device 510. The display screen for the right eye is provided so as to be positioned in front of the right eye of a user wearing the HMD500 on their head, and the display screen for the left eye is provided so as to be positioned in front of the left eye of a user wearing the HMD500 on their head. An image based on a video signal for the right eye is presented to the right eye of the user wearing the glasses, and an image based on a video signal for the left eye is presented to the left eye of the user. Note that the display screen for the right eye and the display screen for the left eye of the display unit 501 are not limited to being composed of one display screen each, and may be composed of multiple display screens. The display unit 501 may have multiple display screens observed by the right eye and multiple display screens observed by the left eye.

[0070] The camera 100 is a stereo camera (camera for acquiring position and orientation) capable of determining the position and orientation of the camera 100, as in the first embodiment. The HMD 500 may further include a stereo camera for a user that captures a moving image of the real space, in addition to the camera for acquiring position and orientation. The image of the real space captured by the stereo camera for the user is used to generate an image of the MR space by compositing it with a VR image. The stereo camera has an imaging unit (right camera) that captures an image to be provided to the right eye of the user, and an imaging unit (left camera) that captures an image to be provided to the left eye of the user. The internal parameters and external parameters of the right camera and the left camera are assumed to be calibrated in advance. The HMD 500 is additionally provided with a camera for the user that captures a moving image of the real space, so that the camera for acquiring position and orientation can be set separately from the camera for the user. As a result, when the user experiences the MR system, the user can view an image captured by setting the camera for the user to a normal camera setting. On the other hand, the camera for acquiring position and orientation can be set to a camera suitable for acquiring the position and orientation, so that the position and orientation can be stably acquired without impeding the experience of the MR system.

[0071] The information processing device 510 according to the second embodiment includes an estimation unit 210, a three-dimensional information acquisition unit 211, a feature point number acquisition unit 212, a determination unit 513, a setting unit 214, a generation unit 511, and a storage unit 512. The processes of the estimation unit 210, the three-dimensional information acquisition unit 211, the feature point number acquisition unit 212, and the setting unit 214 are similar to those of the information processing device 200 according to the first embodiment, and therefore will not be described below. The generation unit 511, the storage unit 512, and the determination unit 513 will be described in detail in the description of the flowchart shown in FIG.

[0072] Fig. 6 is a flowchart illustrating the processing of the information processing device 510 according to the second embodiment. The processing shown in Fig. 6 is processing for sending a video signal of a VR image or an MR image to the display unit 502 of the HMD 500 and setting the camera 100 suitable for estimating the position and orientation of the camera 100 using feature points. The processing of steps S301 to S304, S306, and S307 is similar to the processing in the first embodiment shown in Fig. 3, and therefore detailed description thereof will be omitted.

[0073] When the position and orientation of the camera 100 is estimated in step S302, in step S601, the generation unit 511 constructs a virtual space in which each virtual object is arranged, using information on the virtual object stored in the storage unit 512. The storage unit 512 stores information used to draw the virtual object, such as shape information of the virtual object in the world coordinate system, color information, information representing the texture, information defining the texture, and texture. The generation unit 511 may be capable of arranging the virtual object set by the user in the virtual space.

[0074] The generation unit 511 generates an image of the VR space seen from the position and orientation of the right camera and the left camera, using the position and orientation of the camera 100 for acquiring the position and orientation acquired in step S302 and the external parameters. The image of the VR space seen from the position and orientation of the camera 100 can be generated using a known technique.

[0075] The generation unit 511 can generate an image in the MR space (a composite image) by synthesizing the generated image in the VR space with the images in the real space acquired from the right and left cameras. The generation unit 511 performs the synthesis process by overlaying the image in the VR space on the image in the real space. In the composite image, pixels in the image in the real space are displayed in pixels other than the area of the virtual object. will be done.

[0076] When observing a VR space, the generation unit 511 sends a video signal of a VR space image for the right eye to a display for the right eye, and sends a video signal of a VR space image for the left eye to a display for the left eye. When observing an MR space, the generation unit 511 sends a video signal of an MR space image for the right eye to a display for the right eye, and sends a video signal of an MR space image for the left eye to a display for the left eye. This allows the user to observe a VR space or an MR space.

[0077] When the brightness value and the number of feature points for each region are acquired in step S304, in step S602, the determination unit 513 uses the brightness value and the number of feature points for each region to determine the reliability for setting the camera 100 for each region.

[0078] A part of the process of the determination unit 513 of the second embodiment is different from that of the determination unit 213 of the first embodiment. The determination unit 503 differs from the determination unit 213 of the first embodiment in the method of setting the distance threshold when determining the distance confidence. In the first embodiment, the determination unit 213 determines the distance threshold based on a distance histogram or a user setting. In contrast, in the second embodiment, the determination unit 503 sets the distance threshold based on information of a virtual object. Since it is expected that the user will focus on the virtual object, it is preferable that the camera 100 is set according to the area including the virtual object, and more feature points are appropriately extracted in the area including the virtual object.

[0079] The determination unit 503 sets the distance threshold based on information about the position of a virtual object placed in a virtual space. For example, the determination unit 503 can set the distance from the camera 100 to the center of gravity of the virtual object, or the distance from the camera 100 to the closest position or the farthest position on the virtual object as the distance threshold.

[0080] When feature points extracted from an area farther away than the virtual object are used, the error in the display position of the virtual object increases due to the effect of measurement errors of the feature points. The increase in the error in the display position of the virtual object hinders the user's experience of the VR space or MR space. In this case, it is preferable to determine the distance threshold based on the information on the position of the virtual object so that the reliability of the area including the virtual object is determined to be higher.

[0081] On the other hand, even if the virtual object is far from the camera 100 and there is a measurement error in the feature points in the area including the virtual object, the user's experience of the VR space or MR space is less likely to be hindered by extracting more feature points in the area including the virtual object. In this case as well, the distance threshold is determined based on the position information of the virtual object so that the reliability of the area including the virtual object is determined to be higher, so that more feature points are extracted from the area including the virtual object.

[0082] In the above-mentioned second embodiment, when a user experiences an MR system or a VR system, the information processing device 510 determines the reliability of an area set in a real space based on the position of a virtual object. Specifically, the information processing device 510 determines the reliability of an area based on the relationship between the subject distance and a distance threshold set based on the position of the virtual object. The information processing device 510 determines the reliability of an area including a virtual object so as to be higher, thereby making it possible to set the camera 100 suitable for estimating the position and orientation of the camera 100 by SLAM or the like according to the area including the virtual object. The information processing device 510 can appropriately extract more feature points from the area including the virtual object, and therefore can more accurately estimate the position and orientation of the camera 100 relative to the virtual object. This allows the user to observe the MR image and the VR image without feeling uncomfortable.

[0083] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-mentioned embodiments merely shows one embodiment of the present invention, and each embodiment can be appropriately combined.

[0084] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0085] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) an acquisition means for acquiring three-dimensional information of the real space from an image of the real space acquired by an imaging unit; a selection means for setting a plurality of regions in the real space based on the three-dimensional information, and selecting a selection region to be used for setting the imaging unit from the plurality of regions based on at least one of a subject distance, a luminance, and a number of feature points of each region; a setting means for setting the imaging unit in accordance with the selected area; 13. An information processing device comprising: (Configuration 2) The method further includes a determination unit for determining a reliability of each of the plurality of regions based on at least one of a subject distance, a luminance, and a number of feature points of the region, The selection means selects the selected region based on the reliability of each of the plurality of regions. 2. The information processing device according to configuration 1. (Configuration 3) The determining means determines the distance reliability determined based on the relationship between the subject distance of the region and a distance threshold value as the reliability of the region. 3. The information processing device according to configuration 2. (Configuration 4) The distance threshold is set based on a distance histogram of the image in real space. 4. The information processing device according to configuration 3. (Configuration 5) The determining means determines the luminance reliability determined based on the relationship between the luminance value of the region and a luminance threshold value as the reliability of the region. 3. The information processing device according to configuration 2. (Configuration 6) 6. The information processing device according to configuration 5, wherein the brightness value threshold is set based on a brightness histogram of the image in the real space. (Configuration 7) The determining means determines the feature point reliability determined based on the relationship between the number of feature points included in the region and a threshold value for the number of feature points as the reliability of the region. 3. The information processing device according to configuration 2. (Configuration 8) The determining means determines, as the reliability of the region, a value obtained by adding together a distance reliability determined based on a relationship between a subject distance of the region and a distance threshold, a luminance reliability determined based on a relationship between a luminance value of the region and a luminance threshold, and a feature point reliability determined based on a relationship between the number of feature points included in the region and a feature point number threshold, or a value obtained by weighting and adding each of the above. Decide 3. The information processing device according to configuration 2. (Configuration 9) The determining means determines, as the reliability of the region, the lowest reliability or the highest reliability among a distance reliability determined based on a relationship between a subject distance of the region and a distance threshold, a luminance reliability determined based on a relationship between a luminance value of the region and a luminance threshold, and a feature point reliability determined based on a relationship between the number of feature points included in the region and a feature point number threshold. 3. The information processing device according to configuration 2. (Configuration 10) The determining means generates a distance histogram of the image in real space, and determines the reliability of the region based on the distance and frequency of the distance histogram. 3. The information processing device according to configuration 2. (Configuration 11) The selection means projects information on the reliability of each of the plurality of regions onto an image using the position and orientation of the imaging unit, and selects, as the selected region, a region having the highest reliability or a region having a reliability equal to or greater than a predetermined threshold. 11. The information processing device according to any one of configurations 2 to 10. (Configuration 12) The selection means selects the selected area based on the area most recently used for setting the imaging unit among the plurality of areas. 12. The information processing device according to configuration 11. (Configuration 13) a recording means for recording the three-dimensional information acquired from the real space in a storage unit; an updating means for adding newly acquired information from the three-dimensional information for which the reliability of the region has been determined to the storage unit, or updating previous three-dimensional information acquired at approximately the same position and orientation and recorded in the storage unit with the three-dimensional information for which the reliability of the region has been determined; and When past three-dimensional information acquired in substantially the same position and orientation as the image capturing unit is recorded in the storage unit, the acquisition unit acquires the past three-dimensional information. 13. The information processing device according to any one of configurations 2 to 12. (Configuration 14) an imaging unit that captures an image of the real space; a generating unit that generates an image of an MR space by synthesizing an image of a VR space in which a virtual object is placed and an image of the real space; 14. The information processing device according to any one of configurations 2 to 13, further comprising: (Configuration 15) The imaging unit includes a camera for acquiring a position and orientation of the imaging unit, and a camera for capturing an image of the real space to generate an image of the MR space. 15. The information processing device according to configuration 14. (Configuration 16) the determining means determines a distance reliability determined based on a relationship between a subject distance of the region and a distance threshold as a reliability of the region; The distance threshold is set based on the position of the virtual object. 16. The information processing device according to configuration 14 or 15. (Configuration 17) The setting of the imaging unit is exposure or white balance setting. 17. The information processing device according to any one of configurations 1 to 16. (Configuration 18) The three-dimensional information is information used to reconstruct a three-dimensional space. 18. The information processing device according to any one of configurations 1 to 17. (Configuration 19) The three-dimensional information includes an image of the real space, a position and orientation of the imaging unit when the image of the real space was acquired, and a world coordinate system. 19. The information processing device according to configuration 18. (method) an acquisition step of acquiring three-dimensional information of the real space from an image of the real space acquired by an imaging unit; a selection step of setting a plurality of regions in the real space based on the three-dimensional information, and selecting a selection region to be used for setting the imaging unit from the plurality of regions based on at least one of a subject distance, a luminance, and a number of feature points of each region; a setting step of setting the imaging unit in accordance with the selected area; An information processing method comprising causing a computer to execute the steps of: (program) 20. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 19. [Explanation of symbols]

[0086] 200: information processing device, 201: processor

Claims

1. an acquisition means for acquiring three-dimensional information of the real space from an image of the real space acquired by an imaging unit; a selection means for setting a plurality of regions in the real space based on the three-dimensional information, and selecting a selection region to be used for setting the imaging unit from the plurality of regions based on at least one of a subject distance, a luminance, and a number of feature points of each region; a setting means for setting the imaging unit in accordance with the selected area; 13. An information processing device comprising:

2. The method further includes a determination unit for determining a reliability of each of the plurality of regions based on at least one of a subject distance, a luminance, and a number of feature points of the region, The selection means selects the selected region based on the reliability of each of the plurality of regions.

2. The information processing apparatus according to claim 1,

3. The determining means determines the distance reliability determined based on the relationship between the subject distance of the region and a distance threshold value as the reliability of the region.