Photographing system, information processing device, method and program

Abstract

To ensure that a highly accurate depth value can be obtained when performing imaging with an imaging part that performs imaging from multiple directions.SOLUTION: An information processor 101 comprises: an image acquisition part 110 that acquires an image imaged by an imaging part 100 that performs imaging from multiple directions; an index detection part 150 that detects information on an index shown in the image acquired by the image acquisition part 110; and a depth correction part 160 that corrects a parallax or a depth value obtained from the parallax in the image acquired by the image acquisition part 110 based on the information of the index detected by index detection part 150.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention provides Photography system, The present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] Conventionally, depth measurements of scenes captured with stereo cameras have been performed, but the camera parameters can fluctuate during the measurement. These camera parameters include external camera parameters such as the relative position and orientation between the stereo cameras, and internal camera parameters such as the focal length and principal point position of each stereo camera. The influence of camera parameter fluctuations can cause errors in the calculated depth values ​​of the scene, making it difficult to obtain highly accurate depth values. Patent Document 1 discloses a technique for calibrating deviations that occur due to the installation state of a stereoscopic imaging device. Patent Document 2 discloses a technique for calibrating parameters relating to the positions of two image capturing devices that are set in a distance measurement system that is mounted on a moving body and performs distance measurement using the image capturing devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-113434 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-058188 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, fluctuations in camera parameters of a stereo camera are corrected by calculating the number of effective pixels of the parallax (intermediate data before calculating the depth value) for each acquired pixel, and if calibration is necessary, the camera parameters are corrected and recalculated. It is determined whether the number of effective pixels increases as a result of the recalculation, and the camera parameters are adjusted until they exceed a predetermined threshold, thereby correcting dynamically fluctuating camera parameters. However, in Patent Document 1, calculations are repeated until the number of effective pixels exceeds a predetermined threshold, and there is a problem in that output of depth values ​​stops during recalculation.

[0005] In Patent Document 2, a stereo camera mounted on a moving object captures images at two points, and calculates a correction value for a parameter related to the position of the imaging device from the parallax and the amount of change in parallax. However, Patent Document 2 has a problem in that it is necessary to take images at two locations, and correction values ​​cannot be calculated in a stationary state.

[0006] The present invention has been made in consideration of the above-mentioned points, and aims to make it possible to obtain highly accurate depth values ​​when photographing with an imaging unit that photographs from multiple directions. [Means for solving the problem]

[0007] The information processing device of the present invention includes an image acquisition unit that acquires a first image and a second image captured from different directions using an imaging unit, and a camera that captures the first image and the second image captured by the image acquisition unit. to an index detection means for detecting a distance in three-dimensional space between two points included in the imaged index; and an image acquisition means for acquiring the distance detected by the index detection means based on a distance between two points included in the imaged index that has been prepared in advance. The first image and the second image image required from a correction unit that corrects the parallax or a depth value calculated from the parallax; and a correction unit that corrects the first image and the second image acquired by the image acquisition unit. toThe image capturing device is characterized by comprising a determination means for determining whether or not to perform the correction depending on the state of the index that is displayed or the relative stationary state between the imaging unit and the index. [Effects of the Invention]

[0008] According to the present invention, when an image is captured by an image capturing unit that captures images from a plurality of directions, highly accurate depth values ​​can be obtained. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an imaging system according to a first embodiment. [Figure 2] 4 is a flowchart showing processing executed by the information processing apparatus according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing a state in which an index is captured in a corrected camera image. [Figure 4] FIG. 2 is a diagram for explaining three-dimensional points in a camera coordinate system. [Figure 5] 10A and 10B are diagrams for explaining a calculation process of a depth correction value. [Figure 6] 10A and 10B are characteristic diagrams showing examples of depth correction values ​​and depth values; [Figure 7] FIG. 1 is a diagram illustrating an example of a hardware configuration for realizing an information processing device to which the present invention is applied. [Figure 8] FIG. 10 is a diagram illustrating a functional configuration of a depth correction unit of an information processing device according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of an image display in the second embodiment. [Figure 10] 10 is a flowchart showing a process executed by an information processing apparatus according to a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating a functional configuration of a depth correction unit of an information processing device according to a third embodiment. [Figure 12] FIG. 11 is a diagram showing an example of an image display in the third embodiment. [Figure 13]10 is a flowchart showing a process executed by an information processing apparatus according to a third embodiment. [Figure 14] FIG. 10 is a diagram showing the functional configuration of an imaging system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. First Embodiment 1A and 1B are diagrams for explaining the configuration of an imaging system according to the first embodiment, in which FIG. 1A is a diagram showing the functional configuration of the imaging system, and FIG. 1B is a diagram showing the functional configuration of a depth correction unit 160 of an information processing device 101. As shown in FIG. 1(a), the photography system includes an imaging unit 100, an information processing device 101, an image generation unit 170, and a display 180. Note that the configuration shown in FIG. 1(a) is an example and is not limited to this. For example, the imaging unit 100 may be configured as an imaging device independent of the information processing device 101, or the imaging unit 100 and the information processing device 101 may be integrated into one device. Furthermore, the functions of the information processing device 101 may be realized by multiple devices, or the information processing device 101 may include the image generation unit 170.

[0011] The imaging unit 100 is an imaging unit that captures images from multiple directions, and for example, a stereo camera with two fixed cameras that capture a scene is used. Note that, as long as the scene to be captured does not vary spatially, the present invention can also be applied to a method in which the position and orientation of one camera is measured and stereo measurement is performed from images captured from multiple directions.

[0012] The information processing device 101 includes an image acquisition unit 110 , an image correction unit 120 , a storage unit 130 , a parallax image generation unit 140 , an index detection unit 150 , and a depth correction unit 160 . The image acquisition unit 110 acquires the image captured by the imaging unit 100 and outputs the acquired image to the image correction unit 120. The image acquisition unit 110 is an example of the image acquisition means referred to in the present invention. The image correction unit 120 performs, as correction processing on the image acquired by the image acquisition unit 110, lens distortion correction and rectification for high-speed processing of stereo camera images. The image correction unit 120 reads camera parameters necessary for the correction processing from the storage unit 130. For lens distortion correction and rectification, the methods described in Patent Document 1 and Patent Document 2, or other known techniques may be used. The image correction unit 120 stores the corrected image (referred to as the post-correction camera image in this embodiment) in the storage unit 130.

[0013] The storage unit 130 is a module for storing and managing various types of information. The storage unit 130 stores, for example, the following information. · Post-correction camera images (left and right images in the case of a binocular stereo camera) · Camera parameters (external camera parameters such as the relative position and orientation between stereo cameras, and internal camera parameters such as the focal length and principal point position in each stereo camera) · Disparity images generated by the disparity image generation unit 140 · Index information (information such as the ID which is the identification information of the index and the length of the sides of the rectangular region, input in advance by the user) · Index detection information detected by the index detection unit 150 (the ID of the index, the coordinate values <X value, Y value> of the vertices of the rectangular region for each index) · Past index detection information (for example, index detection information for the past 10 frames) Regarding the camera parameters, the storage unit 130 stores the camera parameters stored in the imaging unit 100 that are loaded at the time of initialization. If there are no camera parameters in the imaging unit 100, the imaging unit 100 may capture a calibration pattern for camera calibration in advance and calculate the camera parameters from the image data. Note that the information stored in the storage unit 130 is not limited to having the above data structure, and it is sufficient that the information necessary for performing the processing in each block is stored.

[0014] The parallax image generation unit 140 calculates the parallax in the corrected camera image based on the corrected camera image and camera parameters stored in the storage unit 130, and generates a parallax image. To generate the parallax image, as described in Patent Document 1 and Patent Document 2, a method (such as the Sum of Absolute Difference method) of matching images acquired by a stereo camera block by block to obtain the parallax for each pixel may be used. The parallax image generation unit 140 stores the generated parallax image in the storage unit 130.

[0015] The index detection unit 150 detects information on an index (referred to as index detection information in this embodiment) reflected in the corrected camera image stored in the storage unit 130, and stores it in the storage unit 130. The index detection unit 150 is an example of the index detection means referred to in the present invention. As the index, as shown in FIG. 3, a marker having a rectangular area including a square is used. For the detection of the index, a rectangular marker detection process having an ID such as ArUco may be used. Specifically, a rectangular area is detected from the corrected camera image, the bit pattern arranged in the rectangular area is subjected to a homography transformation and identified, and the ID and the coordinate values of the four vertices of the rectangular area are output. Referring to FIG. 3, an example of the index detection process will be described. The index 300 is reflected in the left and right images 151L and 151R which are the corrected camera images, and the index detection unit 150 specifies the coordinate values of the four vertices of the rectangular area of the index 300. Specifically, the index detection unit 150 specifies the coordinate values <X value, Y value> of the four vertices 310A to 310D of the rectangular area of the index 300 detected in the left image 151L. Further, the index detection unit 150 specifies the coordinate values <X value, Y value> of the four vertices 310E to 310H of the rectangular area of the index 300 detected in the right image 151R. Then, the index detection unit 150 stores the coordinate values of the four vertices 310A to 310D and the coordinate values of the four vertices 310E to 310H in the storage unit 130. The index detection unit 150 sequentially determines whether or not an index appears in the corrected camera image acquired at regular intervals by the image acquisition unit 110 and corrected by the image correction unit 120. If an index appears in the corrected camera image, the index detection unit 150 outputs index detection information to the storage unit 130 and updates the index detection information for the current frame. If an index does not appear in the corrected camera image, the index detection unit 150 outputs information indicating that there is no index as index detection information. If multiple indices are detected in the same frame, the coordinate values ​​of the four vertices of a rectangular area are linked to the ID of each detected index and stored in the storage unit 130.

[0016] The depth correction unit 160 reads out the parallax images stored in the storage unit 130 and corrects the depth values ​​calculated from the parallax images. As shown in Fig. 1(b), the depth correction unit 160 includes a correction amount calculation unit 161 and a depth image generation unit 165. The depth correction unit 160 is an example of the correction means referred to in the present invention.

[0017] The correction amount calculation unit 161 calculates the amount of correction for correcting the depth value. The processing of the correction amount calculation unit 161 will be described with reference to FIGS. First, the correction amount calculation unit 161 references the index detection information stored in the storage unit 130 and acquires the coordinate values ​​of the vertices of the rectangular area of ​​the index 300 in the corrected camera image. When the left and right images 151L and 151R shown in FIG. 3 are acquired, the depth correction unit 160 associates the vertices 310A-310D and 310E-310H of the rectangular area of ​​the index 300 having the same ID in the left and right images 151L and 151R. Specifically, the vertex 310A in the left image 151L and the vertex 310E in the right image 151R are associated as if they were the same point in three-dimensional space. Similarly, the vertex 310D in the left image 151L and the vertex 310H in the right image 151R are associated as if they were the same point in three-dimensional space. For simplicity, the following description will be limited to the vertices 310A and 310D. When multiple indices are detected in the same frame, the association is performed for each index.

[0018] Once the corresponding points (vertices 310A and 310E, and vertices 310D and 310H in FIG. 3) in the left and right images 151L and 151R have been associated with each other, three-dimensional points 410A and 410D in the camera coordinate system can be calculated using a known triangulation calculation method, as shown in FIG. 4. The camera coordinate system is a coordinate system based on the camera, and the depth direction with respect to the imaging unit 100 is defined as the Z direction. The correction amount calculation unit 161 calculates the distance between points 410A and 410D (the length of line segment 510 connecting points 410A and 410D). Furthermore, the correction amount calculation unit 161 refers to the index information stored in the storage unit 130 and obtains the length of the side of the rectangular area of ​​the index having the corresponding ID. Furthermore, as shown in FIG. 5, correction amount calculation unit 161 sets straight lines 540 and 550 passing through points 410A and 410D from camera origin O in the camera coordinate system, and sets the length of line segment 510 connecting points 410A and 410D. Next, correction amount calculation unit 161 estimates three-dimensional points 500A and 500D that are output when there is no error in the camera parameters. Three-dimensional points 500A and 500D are determined so as to satisfy the following conditions. (1) Point 500A is on the line 540. (2) Point 500D is on the line 550. (3) The slope of the line segment 520 connecting points 500A and 500D is the same as the slope of the line segment 510. (4) The distance between points 500A and 500D (the length of line segment 520 connecting points 500A and 500D) is the same as the length of the side of the index information. Then, the correction amount calculation unit 161 sets, for example, difference 530 between the Z values ​​of point 500A and point 410A as the correction amount for correcting the depth value (called the depth correction value in this embodiment). In this way, the depth correction value is calculated using the index information and the known side length included therein as the correct value.

[0019] The depth image generation unit 165 calculates a depth value for each pixel from the parallax image stored in the storage unit 130, and adds the depth correction value calculated by the correction amount calculation unit 161 to this depth value. Then, the depth image generation unit 165 generates a depth image with the corrected depth value, and outputs it to the image generation unit 170. In this way, the image captured by the imaging unit 100 can be displayed on the display 180 as a depth image with the corrected depth value.

[0020] The depth correction unit 160 may perform processing immediately after an image acquired by the image acquisition unit 110 is updated and a parallax image is generated, or may perform processing at regular intervals. For example, the depth correction value may be calculated every time 10 frames of parallax images are generated. Referring to FIG. 6, a time series change when the depth correction value is calculated at regular intervals will be described. In FIG. 6(a), the horizontal axis represents time t and the vertical axis represents depth correction value f. A depth correction value 601 is calculated at the interval from time t0 to time t1, and then depth correction values ​​602, 603, ... are calculated at regular intervals. In FIG. 6(b), the horizontal axis represents time t and the vertical axis represents depth value d. A dotted line 650 indicates the uncorrected depth value, and a solid line 655 indicates the corrected depth value. As shown in FIG. 6(b), when the depth correction value f is updated, the depth correction value f is added to the depth value, and the same depth correction value f is reflected in subsequent frames. Note that instead of simply reflecting the updated depth correction value f, a weight may be assigned to the depth correction value f. In Fig. 6(c), the horizontal axis represents time t and the vertical axis represents the depth value d. As shown in Fig. 6(c), the weight may be increased over time, so that the change in the corrected depth value becomes smoother.

[0021] Next, the processing executed by the information processing device 101 will be described with reference to Fig. 2. Fig. 2(a) is a flowchart showing the processing executed by the information processing device 101. Details of the processing executed by each unit have been described above, and redundant explanations will be omitted in each step described below. In step S200, the information processing device 101 acquires the camera parameters from the imaging unit 100 and stores them in the storage unit 130. Note that the information processing device 101 is not limited to acquiring the camera parameters from the imaging unit 100, and the results of calibrating the camera parameters in advance may be stored in the storage unit 130.

[0022] In step S210, the image acquisition unit 110 acquires the image captured by the imaging unit 100. In step S220, image correction unit 120 performs correction processing on the image acquired in step S210 using the camera parameters stored in storage unit 130, and stores the corrected camera image in storage unit 130. In step S230, the disparity image generation unit 140 calculates the disparity in the corrected camera image based on the corrected camera image and camera parameters stored in the storage unit 130, generates a disparity image, and stores the disparity image in the storage unit 130. In step S240, the index detection unit 150 detects index detection information of the indexes that appear in the corrected camera image stored in the storage unit 130, and stores the information in the storage unit 130.

[0023] In step S250, the depth correction unit 160 executes a depth correction process. Details of the depth correction process in step S250 will be described later with reference to FIG. In step S260, the information processing device 101 determines whether or not the termination condition is met. For example, if a termination instruction is input from the user, it determines that the termination condition is met. If the termination condition is met, this flowchart ends. If the termination condition is not met, the process returns to step S210.

[0024] FIG. 2B is a flowchart showing the details of the depth correction process in step S250. In step S251, the depth correction unit 160 acquires camera parameters, parallax images, index detection information, and index information from the storage unit 130 as information necessary for the depth correction process. In step S253, the depth correction unit 160 calculates the depth correction value as described above based on the information acquired in step S251. In step S255, the depth correction unit 160 corrects the depth value using the depth correction value calculated in step S253, and generates a depth image.

[0025] As described above, the depth value is corrected based on the index detection information of the indexes that appear in the images captured by the imaging unit 100. As a result, even if the camera parameters fluctuate when measuring the depth of the scene, if the indexes can be captured, the depth value is corrected at that point, and depth values ​​with high reproducibility can be obtained even in continuous measurements. Furthermore, three-dimensional points 500A and 500D in the camera coordinate system can be calculated in one go, eliminating the need for repeated correction calculations as in Patent Document 1. The correction calculations can be completed within the update time of one image frame, and the correction process does not stop the output of depth values. Furthermore, unlike Patent Document 2, the present invention does not require a moving object, but can perform depth correction even when the object is stationary. Furthermore, unlike Patent Document 2, it is not necessary to use images captured at two locations, and the depth correction value can be obtained from an image captured at one location. Therefore, the present invention can also be applied to a stereo camera with a fixed viewpoint. Furthermore, since the three-dimensional position of the index is not required as index information to be prepared in advance, the index does not need to be fixed in space and may move in the observation space. For example, when a user wants to correct the depth value, depth correction can be completed simply by capturing an image of the index held in the user's hand with the imaging unit 100. This simplifies the configuration for depth correction and eliminates the need for maintenance knowledge, contributing to reduced maintenance costs.

[0026] In this embodiment, an example has been described in which the depth correction value is calculated from information on adjacent vertices 310A and 310D (vertices 310E and 310H) in a rectangular area, but the present invention is not limited to this. For example, any two vertices out of the four vertices of the rectangular area may be selected. Also, the depth correction value may be calculated using the distance between two vertices of multiple combinations selected from the four vertices of the rectangular area. In this case, multiple depth correction values ​​are calculated, and the average value of these values ​​may be used, for example. Furthermore, when selecting two vertices, it is preferable to select those with a long distance between them on the corrected camera image 151L or 151R. If the distance between the two vertices on the image is short, sampling errors will occur in the line fitting process used to identify the vertex information, and the estimated vertices will likely contain errors.

[0027] In addition, in the present embodiment, an example has been described in which the depth value is corrected by calculating the depth correction value, but the parallax used to calculate the depth value may be corrected. In this case, the depth value can be calculated from the corrected parallax, and there is no need to correct the depth value. The parallax correction can be performed by calculating the difference in parallax values ​​obtained from the difference 530 of the Z values ​​and adding it to the parallax value.

[0028] FIG. 7 is a diagram showing an example of a hardware configuration for realizing an information processing device to which the present invention is applied. The CPU 701 controls the entire computer using computer programs and data stored in the RAM 707 and the ROM 702. The CPU 701 also executes the processes described below as being performed by the information processing device in each embodiment. The RAM 707 temporarily stores computer programs and data loaded from the external storage device 706 or the storage medium drive 705. The RAM 707 also has an area for temporarily storing data received from an external device. The RAM 707 also has a work area used by the CPU 701 when executing each process. In other words, the RAM 707 can provide various areas as needed. The ROM 702 also stores computer setting data, boot programs, etc. The keyboard 709 and mouse 708 are examples of input devices, and can be operated by a user to input various instructions to the CPU 701. The display unit 704 is composed of a CRT, LCD screen, etc., and displays the results of processing by the CPU 701 as images, text, etc. For example, the display unit 704 can display an image captured by the imaging unit 100. The external storage device 706 is a large-capacity information storage device, typified by a hard disk drive. The external storage device 706 stores an operating system (OS), as well as programs and data for causing the CPU 701 to execute various processes performed by the information processing device. The computer programs and data stored in the external storage device 706 are loaded into the RAM 707 as appropriate under control of the CPU 701. The CPU 701 executes processes using the loaded programs and data, thereby realizing the functions of the information processing device to which the present invention is applied. The storage medium drive 705 reads programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and writes computer programs and data to the storage medium. Note that some or all of the programs and data described as being stored in the external storage device 706 may be recorded on this storage medium. The computer programs and data read from the storage medium by the storage medium drive 705 are output to the external storage device 706 or the RAM 707.The interface (I / F) 703 is configured by an analog video port or a digital input / output port such as IEEE1394 for connecting the imaging unit 100. Data received via the I / F 703 is input to a RAM 707 or an external storage device 706. A bus 710 connects the above-mentioned components via bus signals.

[0029] <Second embodiment> In the second embodiment, an example will be described in which it is determined whether or not to perform depth value correction depending on the state of an index appearing in an image captured by the imaging unit 100. Note that the basic configuration and processing operation of the imaging system are similar to those of the first embodiment, and therefore the following description will omit descriptions of those common to the first embodiment and will focus on differences from the first embodiment. When correcting the depth value, the accuracy of index intersection detection is higher if the index is photographed facing the imaging unit 100. If there is a difference in the lengths of two adjacent sides (vertical and horizontal sides) of the rectangular area of ​​the index, the number of samples taken in the line fitting process used to detect the rectangular area will not be uniform, which may result in variations in the detection of vertices. Therefore, by photographing the index facing the imaging unit 100, the difference in the lengths of the two adjacent sides of the rectangular area is minimized, thereby reducing variations in the detection of vertices.

[0030] 8 shows the functional configuration of the depth correction unit 160 of the information processing device 101 according to the second embodiment. In the second embodiment, the depth correction unit 160 includes an index state determination unit 810, a correction amount calculation unit 161, and a depth image generation unit 165. The index state determination unit 810 determines whether or not to correct the depth value depending on the state of the index shown in the image captured by the imaging unit 100.

[0031] Next, the processing executed by the information processing device 101 will be described with reference to Fig. 2(a) and Fig. 10. The flowchart in Fig. 2(a) is common, but in this embodiment, the depth correction processing in step S250 is changed compared to the first embodiment. Fig. 10 is a flowchart showing the details of the depth correction in step S250. In step S1010, the index state determination unit 810 acquires the coordinate values ​​of the four vertices of the rectangular area of ​​the index having the ID of interest in the corrected camera image. Then, the index state determination unit 810 calculates the lengths of the four sides of the rectangular area from the acquired coordinate values ​​of the four vertices, and calculates the difference in length between two adjacent sides.

[0032] In step S1020, the index state determination unit 810 determines whether the difference in length between the two adjacent sides calculated in step S1010 is equal to or less than a predetermined threshold. That is, the unit determines whether the difference in length between the two adjacent sides is minimized to reduce variation in vertex detection. If the unit 810 determines that the difference is equal to or less than the predetermined threshold, the unit 810 proceeds to step S251. On the other hand, if the unit 810 determines that the difference exceeds the predetermined threshold, the unit 810 proceeds to step S1030.

[0033] In step S1030, the index state determination unit 810 acquires the corrected camera image stored in the memory unit 130 and combines a notification message urging the user to orient the index with a guide frame indicating the appropriate position of the index on the image. In step S1040, the information processing device 101 outputs the corrected camera image onto which the notification message and guide frame have been combined in step S1030 to the image generation unit 170. As a result, as shown in FIG. 9 , the notification message 900 and guide frame 910 can be combined with the image captured by the imaging unit 100 and displayed on the display 180.

[0034] As described above, depth value correction is performed when the state of the index shown in the image captured by the imaging unit 100 is favorable, thereby reducing the variation in the detection of the vertices of the rectangular area and preventing a decrease in the accuracy of depth measurement.

[0035] <Third embodiment> In the third embodiment, an example will be described in which it is determined whether or not to perform depth value correction depending on the relative stationary state between the imaging unit 100 and the index. Note that the basic configuration and processing operation of the imaging system are the same as those of the first embodiment, and therefore the following description will omit the description of the same parts as the first embodiment and will focus on the differences from the first embodiment. In the third embodiment, in order to detect the vertices of the rectangular area of ​​the index, the relative stationary state between the imaging unit 100 and the index is determined, and the median of the coordinate values ​​of the same vertex across multiple frames from the corrected camera image when in a stationary state is used as the coordinate value of the vertex.

[0036] 11 shows the functional configuration of a depth correction unit 160 of an information processing device 101 according to the third embodiment. In the third embodiment, the depth correction unit 160 includes a still state determination unit 1110, a correction amount calculation unit 161, and a depth image generation unit 165. The still state determination unit 1110 determines whether or not to correct the depth value depending on the relative still state between the imaging unit 100 and the index.

[0037] Next, the processing executed by the information processing device 101 will be described with reference to Fig. 2(a) and Fig. 13. The flowchart in Fig. 2(a) is common, but in this embodiment, the depth correction processing in step S250 is changed compared to the first embodiment. Fig. 13 is a flowchart showing the details of the depth correction in step S250. In step S1310, the still state determination unit 1110 determines the relative still state between the imaging unit 100 and the index. Specifically, the still state determination unit 1110 acquires past index detection information stored in the storage unit 130 and acquires the coordinate values ​​of two vertices of a rectangular area of ​​an index having a target ID in each frame. The still state determination unit 1110 then calculates the amount of movement of the two vertices between frames. The amount of movement of one vertex in each frame is found by calculating differential values ​​of the X and Y coordinates between frames, and if the amount of movement exceeds a predetermined threshold in all frames, it is determined that movement has occurred.

[0038] If the stationary state determination unit 1110 determines in step S1310 that the object is in a stationary state, the process proceeds to step S1325. On the other hand, if the stationary state determination unit 1110 determines that the object is not in a stationary state, the process proceeds to step S1330.

[0039] In step S1325, the stationary state determination unit 1110 changes the coordinate values ​​of the index detection information acquired from the storage unit 130 in step S251 to use the median coordinate values ​​of the two vertices of interest in the past index detection information over multiple frames.

[0040] In step S1330, stationary state determination unit 1110 acquires the corrected camera image stored in storage unit 130 and synthesizes a notification message urging the user to enter a stationary state. In step S1340, the information processing device 101 outputs the corrected camera image with the notification message composited in step S1330 to the image generation unit 170. As a result, the notification message 1200 can be composited with the image captured by the imaging unit 100 and displayed on the display 180, as shown in FIG.

[0041] As described above, depth correction is performed when the image capturing unit 100 and the index are relatively stationary, and depth correction can be performed with high detection accuracy of the vertices of the rectangular area. Although the present embodiment has been described based on the first embodiment, it may also be combined with the second embodiment. In this case, the facing determination is performed in steps S1010 and S1020, and if it is determined that the devices are facing each other, the stationary state determination is performed in step S1310. Furthermore, if the determination is not OK, a notification message corresponding to each determination may be displayed (S1030 / S1330).

[0042] <Fourth embodiment> In the fourth embodiment, a first imaging unit that captures images from multiple directions and a second imaging unit that captures images from multiple directions and has smaller time fluctuations in camera parameters than the first imaging unit are fixed to each other, and index information is detected from images captured by the second imaging unit. Note that the basic configuration and processing operation of the imaging system are the same as those of the first embodiment, and the following description will focus on the differences from the first embodiment, omitting descriptions of those common to the first embodiment.

[0043] 14 is a diagram showing the functional configuration of an imaging system according to embodiment 4. The imaging system includes an imaging unit 100 that is a first imaging unit, a second imaging unit 1400, an information processing device 101, an image generation unit 170, and a display 180, and the imaging unit 100 and the second imaging unit 1400 are fixed to each other. Here, one possible cause of time fluctuations in camera parameters is minute fluctuations in the relative position and orientation of the image receiving sensor and lens due to temperature, impact, etc. In particular, cameras with small image receiving sensor areas and small lens areas are more susceptible to the effect of lens changes per pixel on the image receiving sensor, resulting in larger time fluctuations in camera parameters. On the other hand, image receiving sensors with the same number of pixels but large areas and large lens areas are less susceptible to minute fluctuations and result in smaller time fluctuations in camera parameters. In this embodiment, the second imaging unit 1400 is composed of a large image sensor and a lens, and the temporal variation of camera parameters is smaller compared to the imaging unit 100. For example, the imaging unit 100 is a small color camera for presenting video to the user. Also, the second imaging unit 1400 is a grayscale camera that estimates the position and orientation of the imaging unit 100 by SLAM (Simultaneous Localization and Mapping). The second imaging unit 1400 has a large image sensor area to stably estimate the position and orientation even in a dark place, and the variation of camera parameters over time is small. The imaging unit 100 and the second imaging unit 1400 are fixedly arranged with respect to each other, and a device for realizing AR (Augmented Reality) that synthesizes and displays a CG image on the video of the imaging unit 100 while measuring the position and orientation of the head by wearing it on the user's head or the like is assumed. In the image acquisition unit 110, image correction unit 120, storage unit 130, and index detection unit 150 of the information processing apparatus 101, the image captured by the second imaging unit 1400 is processed in the same manner as the image captured by the imaging unit 100. Each block may be configured to be able to process the imaging unit 100 and the second imaging unit 1400 in parallel, or may be configured to process them sequentially in a time-sharing manner.

[0044] In this embodiment, compared with the first embodiment, the information stored in the storage unit 130 is different in the following points. Information added to the storage unit 130: · Corrected camera image of the second imaging unit 1400 (left and right images in the case of a two-eye stereo camera) · Camera parameters of the second imaging unit 1400 (external camera parameters such as the relative position and orientation between stereo cameras, internal camera parameters such as the focal length and principal point position in each stereo camera) · Index detection information detected by the index detection unit 150 from the corrected camera image of the second imaging unit 1400 (index ID, coordinate values <X value, Y value> of the vertices of the rectangular region for each index) · Past index detection information detected from the corrected camera image of the second imaging unit 1400 (for example, index detection information for the past 10 frames)

[0045] Information in the storage unit 130 changed from the first embodiment: Index information (information generated based on index detection information detected from the corrected camera image of the second imaging unit 1400, such as an ID that is identification information of the index, the length of the side of the rectangular area, etc.)

[0046] In this way, since the index information is generated based on the index detection information of the second imaging unit 1400, there is an advantage that the index information does not need to be prepared in advance by the user, for example by inputting it. It also indicates that the index detection information of the second imaging unit 1400 is used as the correct value (the length of the side of the rectangular area) when the depth correction unit 160 calculates the depth correction value. The calculation process of the depth correction value in the depth correction unit 160 can be realized by the same process as in the first embodiment, except that index information generated from index detection information of the second imaging unit 1400 is referenced. Furthermore, the processing executed by the information processing device 101 is the same as that in the first embodiment, and therefore a description thereof will be omitted. As described above, even if index information is not prepared in advance, the depth correction value can be calculated by using the camera parameters obtained from an imaging unit with small time fluctuations as the correct value.

[0047] Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of specific examples of how the present invention can be implemented, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0048] 100: imaging unit, 101: information processing device, 110: image acquisition unit, 120: image correction unit, 130: storage unit, 140: parallax image generation unit, 150: index detection unit, 160: depth correction unit, 161: correction amount calculation unit, 165: depth image generation unit, 810: index state determination unit, 1110: still state determination unit, 1400: second imaging unit

Claims

1. an image acquisition means for acquiring a first image and a second image captured from different directions using an imaging unit; an index detection means for detecting a distance in a three-dimensional space between two points included in an index shown in the first image and the second image acquired by the image acquisition means; a correction means for correcting a parallax calculated from the first image and the second image acquired by the image acquisition means or a depth value calculated from the parallax, based on the distance detected by the index detection means and a distance between two points included in the index that is prepared in advance; An information processing device characterized by comprising a determination means for determining whether or not to perform the correction depending on the state of the index appearing in the first image and the second image acquired by the image acquisition means, or the relative stationary state between the imaging unit and the index.

2. an image acquisition means for acquiring a first image and a second image captured from different directions using an imaging unit; an index detection means for detecting a distance in a three-dimensional space between two points included in an index shown in the first image and the second image acquired by the image acquisition means; a correction means for correcting a parallax calculated from the first image and the second image acquired by the image acquisition means or a depth value calculated from the parallax, based on the distance detected by the index detection means and a distance between two points included in the index that is prepared in advance, The information processing device is characterized in that the index includes identification information of the index.

3. A first stereo camera; a second stereo camera having camera parameters whose time fluctuation is smaller than that of the first stereo camera; an image acquisition means for acquiring a first image captured by the first stereo camera and a second image captured by the second stereo camera; an index detection means for detecting first information of an index shown in the first image and second information of the index shown in the second image; and a correction means for correcting the parallax obtained from the first image acquired by the image acquisition means or the depth value obtained from the parallax based on the first information and the second information.

4. 4. The photographing system according to claim 3, wherein the correction means uses the second information as a correct value.

5. 5. The photographing system according to claim 3, wherein the first stereo camera is a camera for presenting an image to a user, and the second stereo camera is a camera for estimating a position and orientation by SLAM.

6. The information processing device an image acquisition step of acquiring a first image and a second image captured from different directions using an imaging unit; an index detection step of detecting a distance in a three-dimensional space between two points included in an index shown in the first image and the second image acquired in the image acquisition step; a determination step of determining whether or not to correct the parallax calculated from the first image and the second image acquired in the image acquisition step or the depth value calculated from the parallax, depending on the state of the indices shown in the first image and the second image acquired in the image acquisition step or the relative stationary state between the imaging unit and the indices; An information processing method characterized by performing a correction step in which, if it is determined in the determination step that correction is to be performed, a correction step is performed in which the parallax obtained from the first image and the second image acquired in the image acquisition step or the depth value obtained from the parallax is corrected based on the distance detected in the index detection step and the distance between two points included in the index that has been prepared in advance.

7. The information processing device an image acquisition step of acquiring a first image and a second image captured from different directions using an imaging unit; an index detection step of detecting a distance in a three-dimensional space between two points included in an index shown in the first image and the second image acquired in the image acquisition step; a correction step of correcting a parallax calculated from the first image and the second image acquired in the image acquisition step or a depth value calculated from the parallax, based on the distance detected in the index detection step and a distance between two points included in the index that has been prepared in advance; An information processing method, wherein the index includes identification information of the index.

8. 1. An information processing method for an imaging system including a first stereo camera and a second stereo camera having camera parameters with smaller time fluctuations than the first stereo camera, The information processing device an image acquisition step of acquiring a first image captured by the first stereo camera and a second image captured by the second stereo camera; an index detection step of detecting first information of an index shown in the first image and second information of the index shown in the second image; and a correction step of correcting the parallax obtained from the first image acquired in the image acquisition step or the depth value obtained from the parallax based on the first information and the second information.

9. A program for causing a computer to execute each step of the information processing method according to any one of claims 6 to 8.

Images