Information processing device, information processing method, and information processing program
The phase modulation technology generates adaptive light projection patterns and depth estimation, which solves the problem of low measurement accuracy in the prior art, and achieves high accuracy estimation and reliability improvement of object distance.
Patent Information
- Application Number
- CN202380028091.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-30
- Filing Date
- 2023-02-24
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, when measuring the distance of an object, the measurement accuracy is not high and is easily affected by the object reflectivity and external light, resulting in matching errors.
The phase modulation method is used to generate an adaptive light projection pattern, adjust the light quantity distribution and density according to the object's reflectivity and shape, capture the image through the stereo camera device for depth estimation, generate a depth image and calculate a reliability map, and dynamically adjust the light quantity to improve measurement Accuracy.
Highly accurate estimate of the distance from the imaging device to the object is achieved, errors due to light quantity distribution and reflectivity are reduced, and measurement reliability is improved.
Smart Images

Figure CN118922685A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an information processing device, an information processing method and an information processing program. Background Art
[0002] Conventionally, a distance measurement method for measuring the distance of an object is known. For example, as a distance measurement method, an active stereo method (also referred to as an active stereo method) is known. The active stereo method is a method in which a light pattern is projected onto an object, the object on which the pattern is projected is imaged by two left and right camera devices, and the distance from the camera device to the object is measured based on two images obtained by imaging the object on which the pattern is projected. For example, as a technique related to the active stereo method, a technique of projecting a light pattern (such as polka dots) with different intensity distributions on an object is known.
[0003] Reference List
[0004] Patent Literature
[0005] Patent document 1: US2018 / 0260623 A Summary of the invention
[0006] Technical issues
[0007] However, in the above-mentioned conventional technology, light patterns having different intensity distributions are simply projected onto the object, and the distance from the imaging device to the object cannot always be estimated with high accuracy.
[0008] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that are capable of estimating the distance from an imaging device to a subject with high accuracy.
[0009] Solution to the problem
[0010] In order to solve the above-mentioned problem, an information processing device includes: a generating unit which generates a projection pattern of light to be projected onto an object using a phase modulation method; and an estimating unit which estimates distance information indicating the distance from an imaging device to the object based on a captured image obtained by imaging the object onto which the projection pattern generated by the generating unit is projected, by an imaging device. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 is a diagram showing an overview of information processing according to an embodiment of the present disclosure.
[0012] Figure 2 is a diagram showing a configuration example of an information processing device according to an embodiment.
[0013] Figure 3is a diagram showing an example of a projection pattern according to an embodiment.
[0014] Figure 4 is a flowchart illustrating an information processing procedure according to an embodiment.
[0015] Figure 5 is a diagram showing an overview of information processing according to a first modification example of the embodiment.
[0016] Figure 6 is a diagram showing an overview of information processing according to a second modification example of the embodiment.
[0017] Figure 7 is a hardware configuration diagram showing an example of a computer that realizes the function of the information processing device. DETAILED DESCRIPTION
[0018] Hereinafter, embodiments of the present disclosure will be described in detail based on the accompanying drawings. Note that in the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions will be omitted.
[0019] (Implementation Method)
[0020] [1. Preface]
[0021] In a conventional active stereo method, the pattern of light (hereinafter also referred to as a projection pattern) projected onto an object to be measured (hereinafter referred to as an object) is fixed. Therefore, the distance to the object to be measured and the measurement accuracy may differ depending on how the projection pattern reaches the object. In addition, when matching is performed on each part between two images obtained by imaging each object on which the pattern is projected, there are cases of matching errors due to the reflectivity of the object and the influence of external light. Therefore, the conventional active stereo method has certain limitations in measurement accuracy.
[0022] On the other hand, the information processing device according to the embodiment of the present disclosure freely changes the light quantity distribution in the projection pattern and the density of the projection pattern using a phase modulation method. Therefore, the information processing device can generate an adaptive projection pattern according to the reflectivity of the object and the shape of the object. In addition, the information processing device estimates the distance information indicating the distance from the imaging device to the object based on the image obtained by imaging the object on which the adaptive projection pattern is projected by the imaging device. Therefore, the information processing device can prevent the distance to the object to be measured and the measurement accuracy from changing due to the way the projection pattern reaches the object. In addition, the information processing device can prevent mismatching due to the reflectivity of the object and the influence of external light. Therefore, the information processing device can estimate the distance from the imaging device to the object with high accuracy.
[0023] [2. Overview of information processing]
[0024] Figure 1 is a diagram showing an overview of information processing according to an embodiment of the present disclosure. Figure 1 In the process 1, the pattern generation unit of the information processing device 100 generates an initial projection pattern using, for example, a phase modulation method. For example, the pattern generation unit can generate a random pattern as the initial projection pattern. In addition, the projector included in the information processing device 100 projects the initial projection pattern generated by the pattern generation unit onto the object O1 (process 1).
[0025] In addition, the stereo camera included in the information processing device 100 images the object O1 on which the initial projection pattern is projected through the left camera and the right camera. The stereo camera corrects the distortion of each of the captured images captured by the left camera and the right camera. In addition, the stereo camera matches the two captured images after the distortion is corrected (process 2).
[0026] In addition, the depth estimation unit of the information processing device 100 estimates distance information (also referred to as depth information or depth value) indicating the distance from the stereo camera to the object O1 based on the two captured images after the distortion is corrected. For example, the depth estimation unit estimates the distance information based on the distance between the left and right cameras included in the stereo camera (the distance between the focal points of the two cameras), the focal length of the stereo camera, and the parallax between the two captured images. Subsequently, the depth estimation unit generates a depth image (also referred to as a distance measurement image or a depth map) in which the distance information (depth value) is stored as a pixel value of each pixel in the captured image (process 3).
[0027] In addition, the reliability map generation unit of the information processing device 100 calculates the reliability of the distance information (depth value) estimated by the depth estimation unit. The reliability is an index indicating the certainty of the distance information (depth value) in each pixel of the captured image. Subsequently, the reliability map generation unit generates a reliability map that stores the reliability as a pixel value of each pixel in the captured image (process 4).
[0028] In addition, the pattern generation unit of the information processing device 100 obtains a reliability map generated by the reliability map generation unit. The pattern generation unit specifies low-reliability pixels, i.e., pixels with low reliability, based on the reliability map. For example, the pattern generation unit specifies low-reliability pixels whose reliability is equal to or less than a first threshold value (e.g., 60%, etc.). Here, it is considered that the depth value is correctly estimated in the portion with high reliability, while the depth value cannot be correctly estimated in the portion with low reliability for some reason. In addition, one of the reasons why the active stereo method cannot correctly estimate the depth value is that the amount of light of the projector is insufficient (or the amount of light is too large). In particular, in the case where there is a low-reflectivity object with low reflectivity (or a high-reflectivity object with high reflectivity) in the scene, it is well known that when the entire light amount is uniform, the depth value of the part cannot be correctly estimated.
[0029] Therefore, in the case where a low-reliability pixel is specified, the pattern generation unit determines whether the brightness value of the low-reliability pixel is equal to or less than the first brightness value. When it is determined that the brightness value of the low-reliability pixel is equal to or less than the first brightness value, the pattern generation unit determines that the object imaged with the low-reliability pixel (which may be a local area of the object O1) is a low-reflectivity object with low reflectivity. In addition, when it is determined that the object imaged with the low-reliability pixel is a low-reflectivity object, the pattern generation unit generates a new projection pattern in which a phase modulation method is used to make the amount of light projected onto the object imaged with the low-reliability pixel (low-reflectivity object) higher than the amount of light projected onto the object imaged with pixels different from the low-reliability pixel (process 5).
[0030] Furthermore, the projector included in the information processing device 100 projects the new projection pattern generated by the pattern generation unit onto the object O1 (process 6). For example, the information processing device 100 repeats the above-mentioned processes 2 to 6 until low reliability pixels are no longer detected from the reliability map.
[0031] As described above, for example, the information processing device 100 can generate a new projection pattern in which the amount of light projected onto a low-reflectivity object imaged with low-reliability pixels having low brightness is increased, and the new projection pattern is projected. Therefore, for example, the information processing device 100 can estimate the distance from the stereo camera to the low-reflectivity object with high accuracy.
[0032] [3. Configuration of information processing device]
[0033] Figure 2 1 is a diagram showing a configuration example of an information processing device 100 according to an embodiment of the present disclosure. Figure 2 As shown, the information processing device 100 includes a communication unit 110 , a storage unit 120 , a sensor unit 130 , and a control unit 140 .
[0034] For example, the communication unit 110 is implemented using a network interface card (NIC), etc. Then, the communication unit 110 can be connected to a network in a wired or wireless manner, and, for example, transmit and receive information to and from another information processing device.
[0035] For example, the storage unit 120 is implemented using a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 120 stores information about the projection pattern generated by the generation unit 141.
[0036] The sensor unit 130 includes various sensors. Specifically, the sensor unit 130 may include a projector. For example, the sensor unit 130 may include a projection device (e.g., a phase modulation device), which, as an example of a projector, can freely change the focal length and the focusing characteristics of light using a phase modulation method. In addition, the sensor unit 130 may include an imaging device. For example, as an example of an imaging device, the sensor unit 130 may include a stereo camera. Note that, as an example of an imaging device, the sensor unit 130 may include a monocular camera instead of a stereo camera.
[0037] The control unit 140 is a controller and is implemented, for example, by executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the information processing device 100 using a storage area such as a RAM as a work area by a central processing unit (CPU), a microprocessing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. Figure 2 In the illustrated example, the control unit 140 includes a generation unit 141 , an output control unit 142 , and an estimation unit 143 .
[0038] The generating unit 141 includes Figure 1 The pattern generating unit shown in . The generating unit 141 generates a projection pattern of light to be projected onto the object using a phase modulation method. For example, the generating unit 141 generates an initial projection pattern using a phase modulation method. Figure 3 1 is a diagram showing an example of a projection pattern according to an embodiment of the present disclosure. The generation unit 141 may generate a projection pattern as follows: Figure 3 The random pattern P1 shown on the left is used as the initial projection pattern.
[0039] The output control unit 142 controls the projector included in the sensor unit 130 so as to project the projection pattern generated by the generation unit 141 onto the object. For example, the output control unit 142 controls the projector included in the sensor unit 130 so as to project the random pattern P1 generated by the generation unit 141 onto the object. For example, the output control unit 142 controls the projector included in the sensor unit 130 so as to project the random pattern P1 onto the object. Figure 3 The center of FIG. 1 is shown on two objects O11 and O12 in the actual scene G1.
[0040] The estimation unit 143 includes Figure 1 The depth estimation unit shown. The estimation unit 143 obtains a captured image obtained by imaging the object on which the projection pattern generated by the generation unit 141 is projected by the imaging device. For example, the stereo camera of the sensor unit 130 images the object on which the initial projection pattern is projected by the left camera and the right camera. The stereo camera corrects the distortion of each of the images captured by the left camera and the right camera. In addition, the stereo camera performs matching on the two captured images after the distortion is corrected. The estimation unit 143 obtains the two captured images after the distortion is corrected from the stereo camera. For example, the estimation unit 143 obtains two captured images obtained by imaging the two objects O11 and O12 on which the random pattern P1 generated by the generation unit 141 is projected by the stereo camera.
[0041] Then, the estimation unit 143 estimates distance information indicating the distance from the imaging device to the object based on the acquired captured image. For example, the estimation unit 143 estimates the distance information based on the distance between the left and right cameras included in the stereo camera (the distance between the focal points of the two cameras), the focal length of the stereo camera, and the parallax between the two captured images. Then, the estimation unit 143 generates a depth image based on the estimated distance information, in which the distance information (depth value) is stored as a pixel value of each pixel in the captured image.
[0042] Furthermore, the generation unit 141 includes a reliability map generation unit. For example, the generation unit 141 calculates the reliability of the distance information (depth value) estimated by the estimation unit 143. Then, the generation unit 141 generates a reliability map that stores the reliability as a pixel value of each pixel in the captured image.
[0043] In addition, the generation unit 141 specifies low-reliability pixels, i.e., pixels with low reliability, based on the reliability map. For example, the generation unit 141 specifies low-reliability pixels whose reliability is equal to or less than a first threshold value (e.g., 60%, etc.). Subsequently, in the case where the low-reliability pixels are specified, the generation unit 141 determines whether the brightness value of the low-reliability pixels is equal to or less than the first brightness value. When it is determined that the brightness value of the low-reliability pixels is equal to or less than the first brightness value, the generation unit 141 determines that the object imaged with the low-reliability pixels (which may be a partial area of the object) is a low-reflectivity object with low reflectivity. In addition, when it is determined that the object imaged with the low-reliability pixels is a low-reflectivity object, the generation unit 141 generates a new projection pattern using a phase modulation method, in which the amount of light projected onto the object imaged with the low-reliability pixels (low-reflectivity object) is higher than the amount of light projected onto the object imaged with pixels other than the low-reliability pixels. For example, in Figure 3 In the example shown in the center of FIG. , the generation unit 141 determines that the object O11 is a low reflectivity object. Figure 3 In the actual scene G1 shown in the center of , the generation unit 141 generates a new projection pattern P2 using a phase modulation method, in which the amount of light projected onto the object O11 as a low-reflectivity object is higher than the amount of light projected onto the area other than the object O11 as a low-reflectivity object. As described above, in the case where the brightness value of the pixel corresponding to the low-reliability pixel is equal to or less than the first brightness value, the generation unit 141 generates a new projection pattern in which the amount of light projected onto the object imaged with the low-reliability pixel is higher than the amount of light projected onto the object imaged with the high-reliability pixel (non-low-reliability pixel).
[0044] In addition, when a low-reliability pixel is specified, the generation unit 141 determines whether the brightness value of the low-reliability pixel is equal to or greater than a second brightness value. For example, the second brightness value may be a value greater than the first brightness value. When it is determined that the brightness value of the low-reliability pixel is equal to or greater than the second brightness value, the generation unit 141 determines that the object imaged with the low-reliability pixel (which may be a partial area of the object) is a high-reflectivity object with high reflectivity. In addition, when it is determined that the object imaged with the low-reliability pixel is a high-reflectivity object, the generation unit 141 generates a new projection pattern using a phase modulation method, in which the amount of light projected onto the object imaged with the low-reliability pixel (high-reflectivity object) is lower than the amount of light projected onto the object imaged with pixels different from the low-reliability pixel. For example, in Figure 3 In the example shown in the center of FIG. , the generation unit 141 determines that the object O12 is a high reflectivity object. Figure 3In the actual scene G1 shown in the center of , the generation unit 141 generates a new projection pattern P2 using a phase modulation method, in which the amount of light projected onto the object O12 as a high reflectivity object is lower than the amount of light projected onto the area other than the object O12 as a high reflectivity object. As described above, in the case where the brightness value of the pixel corresponding to the low reliability pixel is equal to or greater than the second brightness value, the generation unit 141 generates a new projection pattern in which the amount of light projected onto the object imaged with the low reliability pixel is lower than the amount of light projected onto the object imaged with the high reliability pixel.
[0045] In addition, the estimation unit 143 estimates distance information based on a new captured image obtained by imaging the object on which the new projection pattern generated by the generation unit 141 is projected by the imaging device. For example, the estimation unit 143 acquires two captured images obtained by imaging the two objects O11 and O12 on which the new projection pattern P2 generated by the generation unit 141 is projected by the stereo camera device. Subsequently, the estimation unit 143 estimates distance information indicating the distance from the imaging device to the object based on the acquired captured images. For example, the estimation unit 143 estimates distance information indicating the respective distances from the stereo camera device to the objects O11 and O12 based on the captured images obtained by imaging the two objects O11 and O12 on which the new projection pattern P2 is projected.
[0046] [4. Information processing process]
[0047] Figure 41 is a flowchart illustrating an information processing process according to an embodiment of the present disclosure. The generation unit 141 of the information processing device 100 generates a projection pattern of light to be projected onto an object using a phase modulation method (step S1). The projector included in the sensor unit 130 of the information processing device 100 projects the projection pattern generated by the generation unit 141 onto the object under the control of the output control unit 142 (step S2). The estimation unit 143 of the information processing device 100 acquires a stereoscopic image from the stereo camera device included in the sensor unit 130 (step S3). Subsequently, the estimation unit 143 estimates distance information indicating the distance from the stereo camera device to the object based on the acquired stereo image. Subsequently, the estimation unit 143 generates a depth image based on the estimated distance information (step S4). Subsequently, the estimation unit 143 generates a reliability map based on the generated depth map (step S5). The generation unit 141 determines whether the reliability of each pixel in the reliability map generated by the estimation unit 143 exceeds a threshold value (step S6). When it is determined that the reliability of each pixel in the reliability map exceeds the threshold value (step S6; yes), the generation unit 141 ends the processing. On the other hand, when it is determined that the reliability in a predetermined pixel of the reliability map is equal to or less than the threshold value (step S6; no), the generation unit 141 generates a new projection pattern based on the reliability in the reliability map (step S1).
[0048] [5. Modifications]
[0049] The information processing device 100 according to the above-described embodiment can be implemented in various different embodiments other than the above-described embodiment. Therefore, another embodiment of the information processing device 100 will be described below. Note that the same components as those in this embodiment are denoted by the same reference numerals, and their description will be omitted.
[0050] [5-1. First Modification Example]
[0051] In the above-described embodiment, a case has been described in which the information processing device 100 generates a reliability map and generates a new projection pattern based on the generated reliability map. The first modified example is different from the embodiment in that the information processing device 100 estimates the three-dimensional shape of the object instead of generating a reliability map. In addition, the information processing device 100 generates a new projection pattern based on the estimated three-dimensional shape of the object. Note that the description of the content overlapping with the present embodiment will be omitted.
[0052] Figure 5 : is a diagram showing an overview of information processing according to a first modification example of an embodiment of the present disclosure. Figure 5 and Figure 1 The difference is that the information processing device 100 includes a three-dimensional information analysis unit instead of a reliability map generation unit. Figure 5In the process 1 to 3 and 6 with Figure 1 The processing process is the same as in , so its description is omitted.
[0053] exist Figure 5 , the three-dimensional information analysis unit of the information processing device 100 estimates the three-dimensional shape of the object O1 based on the distance information (depth value) estimated by the depth estimation unit (processing procedure 4A). Figure 2 The generation unit 141 shown includes Figure 5 The three-dimensional information analysis unit shown. For example, the object O1 is an artificial object such as a chair or a table. Since the object O1 is an artificial object, the object O1 has a flat surface or a curved surface of a certain area, so the three-dimensional shape can be easily estimated. The generation unit 141 estimates the three-dimensional shape of the object O1 based on the distance information (depth value) estimated by the estimation unit 143. For example, the generation unit 141 estimates the flat surface or the curved surface that fits the point group in the depth image generated by the estimation unit 143 using the least squares method.
[0054] Here, generally speaking, it is well known that depending on the three-dimensional shape of the object, a phenomenon occurs in which the focus partially deviates or the density of the projection pattern is uneven. For example, when a sphere is irradiated with a uniform projection pattern, since the vicinity of the center of the sphere is located close to the camera device, the density of the projection pattern becomes low and the focus position becomes close. On the other hand, since the end of the sphere is located far from the camera device, the density of the projection pattern becomes high and the focus position becomes far.
[0055] Therefore, the generation unit 141 estimates the distance information indicating the distance from the imaging device to each position of the object based on the three-dimensional shape of the object. Subsequently, using the phase modulation method, the generation unit 141 generates a new projection pattern in which the density of the projection pattern projected onto each position of the object at a distance from the imaging device to the object equal to or less than the first distance (for example, a position near the center of the sphere) is higher than the density of the projection pattern projected onto each position of the object at a distance from the imaging device to the object exceeding the first distance (for example, a position other than the center of the sphere) (process 5A).
[0056] In addition, using the phase modulation method, the generation unit 141 generates a new projection pattern in which the density of the projection pattern projected onto each position of the object at a distance from the imaging device to the object equal to or greater than the second distance (e.g., the position of the end of the sphere) is lower than the density of the projection pattern projected onto each position of the object at a distance from the imaging device to the object less than the second distance (e.g., the position other than the end of the sphere) (process 5A). For example, the second distance may be a distance greater than the first distance.
[0057] As described above, for example, the information processing device 100 can estimate the three-dimensional shape of the object based on the depth value, generate a new projection pattern in which the density of the projection pattern is changed according to the estimated three-dimensional shape, and project the new projection pattern. Therefore, the information processing device 100 can estimate the distance from the stereo camera device to the object with high accuracy.
[0058] [5-2. Second Modification Example]
[0059] In the above-described embodiment, a case has been described in which the information processing device 100 estimates the distance information indicating the distance from the imaging device to the object based on a captured image obtained by imaging the object on which a projection pattern including a specific pattern is projected by the imaging device. In the second modified example, the information processing device 100 generates a flat pattern and a specific pattern, the flat pattern being a uniform projection pattern that does not include a pattern, and the specific pattern being a projection pattern that includes a specific pattern. Then, a case will be described in which the information processing device 100 estimates the distance information indicating the distance from the imaging device to the object based on a captured image obtained by imaging the object on which a flat pattern and a specific pattern are alternately projected in a time-division manner. Note that the description of the content overlapping with the present embodiment will be omitted.
[0060] Figure 6 : is a diagram showing an overview of information processing according to a second modification example of an embodiment of the present disclosure. Figure 6 and Figure 1 The difference is that the information processing device 100 includes a scene analysis unit instead of a reliability map generation unit. Here, Figure 6 The scene analysis unit in corresponds to Figure 1 The reliability diagram generation unit in Figure 5 In addition, Figure 6 and Figure 1 The difference is that the information processing device 100 includes a pattern storage memory. Figure 6 As shown, the projector of the information processing device 100 can project the calculated phase modulation pattern pre-stored in the pattern storage memory.
[0061] Here, generally speaking, as a feature of a specific pattern (e.g., a random pattern, etc.) of a projection pattern including a specific pattern, there is a point where it is difficult to estimate the depth value near the edge of an object, but it is easy to estimate the depth value of the surface of the object. On the other hand, flat lighting (hereinafter also referred to as a flat pattern) (i.e., uniform light not including a pattern) is characterized in that it is easy to estimate the depth near the edge of an object, but it is difficult to estimate the depth of the surface of the object.
[0062] Therefore, for example, the generation unit 141 generates a flat pattern (i.e., a uniform projection pattern that does not include a pattern) and a specific pattern (e.g., an adaptive projection pattern described in the present embodiment, etc.) using a phase modulation method, and the specific pattern is a projection pattern that includes a specific pattern. In addition, the projector included in the information processing device 100 alternately projects the flat pattern and the specific pattern onto the object O1 in a time-division manner. The estimation unit 143 estimates distance information indicating the distance from the imaging device to the object based on a captured image obtained by imaging the object by the imaging device, and the flat pattern and the specific pattern generated by the generation unit 141 are alternately projected onto the object in a time-division manner.
[0063] As described above, for example, the information processing device 100 can estimate the distance from the imaging device to the object with higher accuracy by illuminating the object in a time-division manner through the adaptive projection pattern and flat lighting described in this embodiment.
[0064] [6. Effect]
[0065] As described above, the information processing device 100 according to the embodiment or modification of the present disclosure includes a generation unit 141 and an estimation unit 143. The generation unit 141 generates a projection pattern of light to be projected onto an object using a phase modulation method. The estimation unit 143 estimates distance information indicating the distance from the imaging device to the object based on a captured image obtained by imaging the object by the imaging device, and the projection pattern generated by the generation unit 141 is alternately projected onto the object in a time-division manner.
[0066] Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0067] In addition, the generation unit 141 generates a new projection pattern based on the distance information in each pixel of the imaging area in which the object is imaged in the captured image. The estimation unit 143 estimates the distance information based on a new captured image obtained by imaging the object onto which the new projection pattern generated by the generation unit 141 is projected by the imaging device.
[0068] Therefore, the information processing device 100 can generate an adaptive projection pattern according to the reflectivity of the object and the shape of the object. In addition, the information processing device 100 estimates the distance information indicating the distance from the imaging device to the object based on the image obtained by imaging the object on which the adaptive projection pattern is projected by the imaging device. Therefore, the information processing device 100 can prevent the distance to the object to be measured and the measurement accuracy from changing due to the way the projection pattern reaches the object. In addition, the information processing device can prevent mismatching due to the reflectivity of the object and the influence of external light. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0069] Furthermore, the generation unit 141 generates a new projection pattern based on the reliability, which is an index indicating the certainty of the distance information in each pixel of the imaging area.
[0070] Therefore, for example, the information processing device 100 can generate an adaptive projection pattern according to the reflectivity of the object based on the reliability. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0071] In addition, the generation unit 141 generates a new projection pattern in which the amount of light projected onto an object imaged with high-reliability pixels having a reliability exceeding a first reliability is different from the amount of light projected onto an object imaged with low-reliability pixels having a reliability equal to or less than the first reliability.
[0072] Therefore, for example, the information processing device 100 can generate an adaptive projection pattern according to the reflectivity of the object based on the degree of reliability. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0073] Furthermore, the generation unit 141 generates a new projection pattern in which the amount of light projected onto an object imaged with pixels of low reliability is higher than the amount of light projected onto an object imaged with pixels of high reliability.
[0074] Therefore, for example, the information processing device 100 can generate an adaptive projection pattern according to the reflectivity of the object based on the degree of reliability. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0075] In addition, when the brightness value of the pixel corresponding to the low-reliability pixel is equal to or less than the first brightness value, the generation unit 141 generates a new projection pattern in which the amount of light projected onto the object imaged with the low-reliability pixel is higher than the amount of light projected onto the object imaged with the high-reliability pixel.
[0076] Thus, for example, the information processing device 100 can generate a new projection pattern in which the amount of light projected onto a low-reflectivity object with low brightness imaged with low-reliability pixels is increased, and project the new projection pattern. Thus, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0077] Furthermore, the generation unit 141 generates a new projection pattern in which the amount of light projected onto an object imaged with pixels of low reliability is lower than the amount of light projected onto an object imaged with pixels of high reliability.
[0078] Therefore, for example, the information processing device 100 can generate an adaptive projection pattern according to the reflectivity of the object based on the degree of reliability. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0079] In addition, when the brightness value of the pixel corresponding to the low-reliability pixel is equal to or greater than the second brightness value, the generation unit 141 generates a new projection pattern in which the amount of light projected onto the object imaged with the low-reliability pixel is lower than the amount of light projected onto the object imaged with the high-reliability pixel.
[0080] Thus, for example, the information processing device 100 can generate a new projection pattern in which the amount of light projected onto a high-reflectivity object with high brightness imaged with low-reliability pixels is reduced, and project the new projection pattern. Thus, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0081] Furthermore, the generation unit 141 generates a new projection pattern based on the three-dimensional shape of the object estimated from the distance information in each pixel of the imaging area.
[0082] Therefore, for example, the information processing device 100 can estimate the three-dimensional shape of the object according to the depth value, and generate an adaptive projection pattern according to the estimated three-dimensional shape. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0083] Furthermore, the generation unit 141 generates a new projection pattern in which the density of the projection pattern projected onto each position of the object is made different based on distance information indicating the distance from the imaging device to each position of the object estimated based on the three-dimensional shape of the object.
[0084] Therefore, for example, the information processing device 100 can estimate the three-dimensional shape of the object based on the depth value, generate a new projection pattern that changes the density of the projection pattern according to the estimated three-dimensional shape, and project the new projection pattern. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0085] In addition, the generation unit 141 generates a new projection pattern in which a density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object is equal to or less than the first distance is higher than a density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object exceeds the first distance.
[0086] Therefore, for example, the information processing device 100 can estimate the three-dimensional shape of the object based on the depth value, generate a new projection pattern that changes the density of the projection pattern according to the estimated three-dimensional shape, and project the new projection pattern. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0087] In addition, the generation unit 141 generates a new projection pattern in which the density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object is equal to or greater than the second distance is lower than the density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object is less than the second distance.
[0088] Therefore, for example, the information processing device 100 can estimate the three-dimensional shape of the object based on the depth value, generate a new projection pattern that changes the projection pattern density according to the estimated three-dimensional shape, and project the new projection pattern. Therefore, the information processing device 100 can estimate the distance from the imaging device to the object with high accuracy.
[0089] In addition, the generating unit 141 generates a flat pattern which is a uniform projection pattern not including a pattern and a specific pattern which is a projection pattern including a specific pattern. The estimating unit 143 estimates distance information indicating a distance from the imaging device to the object based on a captured image obtained by imaging the object by the imaging device, and the flat pattern and the specific pattern generated by the generating unit 141 are alternately projected onto the object in a time-division manner.
[0090] Therefore, for example, the information processing device 100 can estimate the distance from the imaging device to the object with higher accuracy by irradiating the object using an adaptive projection pattern and a flat pattern in a time-division manner.
[0091] [7. Hardware Configuration]
[0092] For example, an information device such as the information processing device 100 according to the above-described embodiment is composed of a Figure 7 The configuration shown is reproduced by computer 1000. Figure 7 1000 is a hardware configuration diagram showing an example of a computer 1000 that reproduces the functions of an information processing device such as the information processing device 100. Hereinafter, the information processing device 100 according to the embodiment will be described as an example. The computer 1000 includes a CPU 1100, a RAM 1200, a read-only memory (ROM) 1300, a hard disk drive (HDD) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.
[0093] The CPU 1100 operates based on the program stored in the ROM 1300 or the HDD 1400, and controls each unit. For example, the CPU 1100 develops the program stored in the ROM 1300 or the HDD 1400 in the RAM 1200, and executes processing corresponding to the various programs.
[0094] The ROM 1300 stores programs such as a boot program of a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, programs depending on the hardware of the computer 1000 , and the like.
[0095] The HDD 1400 is a computer-readable recording medium in which a program executed by the CPU 1100, data used by the program, etc. are non-transitorily recorded. Specifically, the hard disk 1400 is a recording medium in which a program according to the present disclosure as an example of program data 1450 is recorded.
[0096] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (eg, the Internet). For example, the CPU 1100 receives data from another device via the communication interface 1500 or transmits data generated by the CPU 1100 to another device.
[0097] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard and a mouse via the input / output interface 1600. In addition, the CPU 1100 sends data to output devices such as a display, a speaker, or a printer via the input / output interface 1600. In addition, the input / output interface 1600 can be used as a medium interface for reading a program recorded in a predetermined recording medium (medium), etc. For example, the medium is an optical recording medium such as a digital versatile disk (DVD) or a phase change rewritable disk (PD), a magneto-optical recording medium such as a magneto-optical disk (MO), a magnetic tape medium, a magnetic recording medium, a semiconductor memory, etc.
[0098] For example, in the case where the computer 1000 is used as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 reproduces the functions of the control unit 140 and the like by executing the program loaded on the RAM 1200. In addition, the hard disk 1400 stores the program and various data according to the present disclosure. Note that the CPU 1100 reads the program data 1450 from the HDD 1400 and executes the program data, but as another example, these programs may be acquired from another device via the external network 1550.
[0099] In addition, the effects described in this specification are only illustrative or exemplary, not restrictive. That is, together with or in place of the above effects, according to the description of this specification, the technology according to the present disclosure can show other effects that are obvious to those skilled in the art.
[0100] Note that the present technology may also have the following configurations.
[0101] (1) An information processing device comprising:
[0102] a generating unit that generates a projection pattern of light to be projected onto an object using a phase modulation method; and
[0103] An estimating unit estimates distance information indicating a distance from the imaging device to the object, based on a captured image obtained by imaging, by the imaging device, the object on which the projection pattern generated by the generating unit is projected.
[0104] (2) The information processing device according to (1),
[0105] wherein the generating unit generates a new projection pattern based on distance information in each pixel of an imaging area in which the object is imaged in the captured image, and
[0106] The estimation unit estimates the distance information based on a new captured image obtained by imaging, by the imaging device, an object onto which the new projection pattern generated by the generation unit is projected.
[0107] (3) The information processing device according to (2),
[0108] The generating unit generates the new projection pattern based on reliability, which is an indicator indicating certainty of distance information in each pixel of the imaging area.
[0109] (4) The information processing device according to (3),
[0110] In which, the generation unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto an object imaged by high-reliability pixels whose reliability exceeds a first reliability is different from the amount of light projected onto an object imaged by low-reliability pixels whose reliability is equal to or less than the first reliability.
[0111] (5) The information processing device according to (4),
[0112] The generating unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto the object imaged by the low-reliability pixels is higher than the amount of light projected onto the object imaged by the high-reliability pixels.
[0113] (6) The information processing device according to (4) or (5),
[0114] In which, when the brightness value of the pixel corresponding to the low-reliability pixel is equal to or less than the first brightness value, the generation unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto the object imaged by the low-reliability pixel is higher than the amount of light projected onto the object imaged by the high-reliability pixel.
[0115] (7) The information processing device according to any one of (4) to (6),
[0116] The generating unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto the object imaged by the low-reliability pixels is lower than the amount of light projected onto the object imaged by the high-reliability pixels.
[0117] (8) The information processing device according to any one of (4) to (7),
[0118] In which, when the brightness value of the pixel corresponding to the low-reliability pixel is equal to or greater than the second brightness value, the generation unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto the object imaged by the low-reliability pixel is lower than the amount of light projected onto the object imaged by the high-reliability pixel.
[0119] (9) The information processing device according to (2),
[0120] The generating unit generates a new projection pattern based on the three-dimensional shape of the object estimated according to the distance information in each pixel of the imaging area.
[0121] (10) The information processing device according to (9),
[0122] The generating unit generates a new projection pattern in which a density of the projection pattern projected onto each position of the object is different based on distance information indicating a distance from the imaging device to each position of the object estimated based on a three-dimensional shape of the object.
[0123] (11) The information processing device according to (10),
[0124] In which, the generating unit generates the following new projection pattern: in the new projection pattern, the density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object is equal to or less than a first distance is higher than the density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object exceeds the first distance.
[0125] (12) The information processing device according to (10) or (11),
[0126] In which, the generating unit generates the following new projection pattern: in the new projection pattern, the density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object is equal to or greater than a second distance is lower than the density of the projection pattern projected onto each position of the object whose distance from the imaging device to the object is less than the second distance.
[0127] (13) The information processing device according to any one of (1) to (12),
[0128] The generating unit generates a flat pattern and a specific pattern, wherein the flat pattern is a uniform projection pattern without a pattern, and the specific pattern is a projection pattern including a specific pattern, and
[0129] The estimation unit estimates distance information indicating a distance from the imaging device to the object based on a captured image obtained by imaging, by the imaging device, the object to which the flat pattern and the specific pattern generated by the generation unit are alternately projected in a time-division manner.
[0130] (14) An information processing method executed by a computer, the method comprising:
[0131] A generating step of generating a projection pattern of light to be projected onto the object by using a phase modulation method; and
[0132] An estimating step of estimating distance information indicating a distance from the imaging device to the object based on a captured image obtained by imaging, by the imaging device, the object onto which the projection pattern generated in the generating step is projected.
[0133] (15) An information processing program for causing a computer to perform the following operations:
[0134] a generating process, using phase modulation to generate a projection pattern of light to be projected onto an object; and
[0135] The estimation process estimates distance information indicating the distance from the imaging device to the object based on a captured image obtained by imaging, by the imaging device, the object on which the projection pattern generated in the generating process is projected.
[0136] Reference numerals list
[0137] 100 Information processing device
[0138] 110 communication device
[0139] 120 storage units
[0140] 130 sensor units
[0141] 140 control unit
[0142] 141Generation Unit
[0143] 142 Output control unit
[0144] 143 Estimation Unit
Claims
1. An information processing device, include: a generating unit, wherein the generating unit generates a projection pattern of light to be projected onto the object using a phase modulation method; as well as An estimating unit estimates distance information indicating a distance from the imaging device to the object based on a captured image obtained by imaging, by the imaging device, the object onto which the projection pattern generated by the generating unit is projected.
2. The information processing device according to claim 1, in, The generation unit generates a new projection pattern based on distance information in each pixel of an imaging area in which a subject is imaged in the captured image, and The estimation unit estimates the distance information based on a new captured image obtained by imaging, by the imaging device, an object onto which the new projection pattern generated by the generation unit is projected.
3. The information processing device according to claim 2, in, The generation unit generates a new projection pattern based on reliability, which is an index indicating certainty of distance information in each pixel of the imaging area.
4. The information processing device according to claim 3, in, The generation unit generates a new projection pattern in which the amount of light projected onto an object imaged by high-reliability pixels having a reliability exceeding a first reliability is different from the amount of light projected onto an object imaged by low-reliability pixels having a reliability equal to or less than the first reliability.
5. The information processing device according to claim 4, in, The generation unit generates a new projection pattern in which an amount of light projected onto an object imaged with the low-reliability pixels is higher than an amount of light projected onto an object imaged with the high-reliability pixels.
6. The information processing device according to claim 5, in, When the brightness value of the pixel corresponding to the low-reliability pixel is equal to or less than the first brightness value, the generation unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto the object imaged with the low-reliability pixel is higher than the amount of light projected onto the object imaged with the high-reliability pixel.
7. The information processing device according to claim 4, in, The generation unit generates a new projection pattern in which an amount of light projected onto an object imaged with the low-reliability pixels is lower than an amount of light projected onto an object imaged with the high-reliability pixels.
8. The information processing device according to claim 7, in, When the brightness value of the pixel corresponding to the low-reliability pixel is equal to or greater than the second brightness value, the generation unit generates the following new projection pattern: in the new projection pattern, the amount of light projected onto the object imaged with the low-reliability pixel is lower than the amount of light projected onto the object imaged with the high-reliability pixel.
9. The information processing device according to claim 2, in, The generation unit generates a new projection pattern based on a three-dimensional shape of the object estimated from distance information in each pixel of the imaging area.
10. The information processing device according to claim 9, in, The generating unit generates a new projection pattern in which a density of the projection pattern projected onto each position of the object is made different based on distance information indicating a distance from the imaging device to each position of the object estimated based on a three-dimensional shape of the object.
11. The information processing device according to claim 10, in, The generating unit generates a new projection pattern in which a density of the projection pattern projected onto each position of an object whose distance from the imaging device to the object is equal to or less than a first distance is higher than a density of the projection pattern projected onto each position of an object whose distance from the imaging device to the object exceeds the first distance.
12. The information processing device according to claim 10, in, The generating unit generates a new projection pattern in which a density of a projection pattern projected onto each position of an object whose distance from the imaging device to the object is equal to or greater than a second distance is lower than a density of a projection pattern projected onto each position of an object whose distance from the imaging device to the object is less than the second distance.
13. The information processing device according to claim 1, in, The generating unit generates a flat pattern and a specific pattern, the flat pattern being a uniform projection pattern including no pattern, the specific pattern being a projection pattern including a specific pattern, and The estimation unit estimates distance information indicating a distance from the imaging device to the object based on a captured image obtained by imaging, by the imaging device, the object to which the flat pattern and the specific pattern generated by the generation unit are alternately projected in a time-division manner.
14. An information processing method executed by a computer, the method include: a generating step of generating a projection pattern of light to be projected onto the object by using a phase modulation method; as well as An estimating step of estimating distance information indicating a distance from the imaging device to the object based on a captured image obtained by imaging, by the imaging device, the object onto which the projection pattern generated in the generating step is projected.
15. An information processing program for causing a computer to execute: a generating process, using phase modulation to generate a projection pattern of light to be projected onto an object; and The estimation process estimates distance information indicating the distance from the imaging device to the object based on a captured image obtained by imaging the object on which the projection pattern generated in the generation process is projected by the imaging device.
