Information processor, method, and program

JP2023087845A5Active Publication Date: 2024-11-22CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2021202355
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2021-12-14
Publication Date
2024-11-22
Estimated Expiration
2041-12-14

AI Technical Summary

Technical Problem

Existing methods for calculating depth values from stereo cameras suffer from inaccuracies due to fluctuations in camera parameters, leading to unreliable depth measurements, especially when cameras are not stationary or require multiple shots at different points.

Method used

An information processing apparatus that captures images from multiple directions, uses index detection to correct parallax and depth values, and calculates correction amounts based on detected indices to stabilize depth measurements.

Benefits of technology

Enables highly accurate and reproducible depth values even with fluctuating camera parameters, eliminating the need for repeated recalculations and multiple shots, and simplifying the depth correction process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

To ensure that a highly accurate depth value can be obtained when performing imaging with an imaging part that performs imaging from multiple directions.SOLUTION: An information processor 101 comprises: an image acquisition part 110 that acquires an image imaged by an imaging part 100 that performs imaging from multiple directions; an index detection part 150 that detects information on an index shown in the image acquired by the image acquisition part 110; and a depth correction part 160 that corrects a parallax or a depth value obtained from the parallax in the image acquired by the image acquisition part 110 based on the information of the index detected by index detection part 150.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, method, and program suitable for use when performing shooting with an imaging unit that performs shooting from a plurality of directions.

Background Art

[0002] Conventionally, it has been possible to measure the depth of a scene photographed with a stereo camera, but the camera parameters may vary in that case. Camera parameters include external camera parameters such as the relative position and orientation between stereo cameras, and internal camera parameters such as the focal length and principal point position in each stereo camera. Due to the influence of fluctuations in camera parameters, there has been a problem that an error occurs in the calculated depth value of the scene, and it becomes impossible to obtain a highly accurate depth value. Patent Document 1 discloses a technique for calibrating misalignment caused by the installation state of a stereo image capturing device. Patent Document 2 discloses a technique for calibrating parameters related to the positions of imaging devices set in a distance measurement system mounted on a moving body and performing distance measurement using two imaging devices.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 describes how fluctuations in the camera parameters of a stereo camera are corrected by determining the number of effective pixels of the parallax (intermediate data before calculating the depth value) at each acquired pixel, and correcting the camera parameters and recalculating if calibration is necessary. It determines whether the number of effective pixels increases or not through recalculation, and corrects dynamically fluctuating camera parameters by adjusting the camera parameters until they exceed a predetermined threshold. However, Patent Document 1 had the problem that the calculation was repeated until the number of effective pixels exceeded a predetermined threshold, and the output of the depth value stopped during the recalculation.

[0005] Patent Document 2 describes a method in which a stereo camera mounted on a mobile device takes images at two locations, and a correction value for the position parameter of the imaging device is calculated from the parallax and the amount of parallax change. However, Patent Document 2 had the problem that it required taking pictures at two locations, and correction values ​​could not be calculated when the subject was stationary.

[0006] This invention has been made in view of the above-mentioned points, and aims to enable the acquisition of highly accurate depth values ​​when taking images with an imaging unit that takes images from multiple directions. [Means for solving the problem]

[0007] The information processing apparatus of the present invention is characterized by comprising: an image acquisition means for acquiring an image captured by an imaging unit that captures images from multiple directions; an indicator detection means for detecting information of an indicator appearing in the image acquired by the image acquisition means; and a correction means for correcting the parallax in the image acquired by the image acquisition means or the depth value obtained from the parallax based on the information of the indicator detected by the indicator detection means. [Effects of the Invention]

[0008] According to the present invention, when imaging is performed by an imaging unit that captures images from multiple directions, highly accurate depth values ​​can be obtained. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram illustrating the configuration of the imaging system according to the first embodiment. [Figure 2] This is a flowchart showing the process performed by the information processing device according to the first embodiment. [Figure 3] This diagram shows the corrected camera image with the indicator visible. [Figure 4] This is a diagram illustrating a three-dimensional point in the camera coordinate system. [Figure 5] This diagram illustrates the process for calculating depth correction values. [Figure 6] This is a characteristic diagram showing examples of depth correction values ​​and depth values. [Figure 7] This figure shows an example of a hardware configuration for realizing an information processing device to which the present invention is applied. [Figure 8] This diagram shows the functional configuration of the depth correction unit of the information processing device according to the second embodiment. [Figure 9] This figure shows an example of image display in the second embodiment. [Figure 10] This is a flowchart showing the process performed by the information processing device according to the second embodiment. [Figure 11] This diagram shows the functional configuration of the depth correction unit of the information processing device according to the third embodiment. [Figure 12] This figure shows an example of image display in the third embodiment. [Figure 13] This is a flowchart showing the process performed by the information processing device according to the third embodiment. [Figure 14] This figure shows the functional configuration of the imaging system according to the fourth embodiment. [Modes for carrying out the invention]

[0010] Preferred embodiments of the present invention will be described below with reference to the attached drawings. <First Embodiment> FIG. 1 is a diagram for explaining the configuration of a photographing system according to a first embodiment, where (a) is a diagram showing the functional configuration of the photographing system, and (b) is a diagram showing the functional configuration of the depth correction unit 160 of the information processing apparatus 101. As shown in FIG. 1(a), the photographing system includes an imaging unit 100, an information processing apparatus 101, an image generation unit 170, and a display 180. Note that the configuration shown in FIG. 1(a) is an example and is not limited thereto. For example, the imaging unit 100 may be configured as an imaging device independent of the information processing apparatus 101, or the imaging unit 100 and the information processing apparatus 101 may be integrated. Also, the functions of the information processing apparatus 101 may be realized by a plurality of devices, or the information processing apparatus 101 may include the image generation unit 170.

[0011] The imaging unit 100 is an imaging unit that performs photographing from a plurality of directions. For example, a stereo camera in which two cameras for photographing a scene are fixed is used. If the scene to be photographed is under conditions where it does not vary spatially, the present invention is also applicable to a method of measuring the position and orientation of one camera and performing stereo measurement from images taken from a plurality of directions.

[0012] The information processing apparatus 101 includes an image acquisition unit 110, an image correction unit 120, a storage unit 130, a disparity image generation unit 140, an index detection unit 150, and a depth correction unit 160. The image acquisition unit 110 acquires the image photographed by the imaging unit 100 and outputs the acquired image to the image correction unit 120. The image acquisition unit 110 is an example of the image acquisition means in the present invention. The image correction unit 120 performs correction processing, such as lens distortion correction and rectification for high-speed processing of stereo camera images, on the image acquired by the image acquisition unit 110. The image correction unit 120 reads out camera parameters necessary for the correction processing from the storage unit 130. For lens distortion correction and rectification, methods described in Patent Document 1 and Patent Document 2, or other known techniques may be used. The image correction unit 120 stores the corrected image (referred to as the post-correction camera image in the present embodiment) in the storage unit 130.

[0013] The storage unit 130 is a module for storing and managing various types of information. The storage unit 130 stores, for example, the following information. · Corrected camera image (in the case of a binocular stereo camera, the left and right images) · Camera parameters (external camera parameters such as the relative position and orientation between stereo cameras, and internal camera parameters such as the focal length and principal point position in each stereo camera) · Disparity image generated by the disparity image generation unit 140 · Index information (ID which is the identification information of the index, information pre-input by the user such as the length of the sides of the rectangular area) · Index detection information detected by the index detection unit 150 (ID of the index, coordinate values <X value, Y value> of the vertices of the rectangular area for each index) · Past index detection information (for example, index detection information for the past 10 frames) Regarding the camera parameters, the storage unit 130 stores the camera parameters stored in the imaging unit 100 that are read at the time of initialization. If there are no camera parameters in the imaging unit 100, the imaging unit 100 may previously capture a calibration pattern for camera calibration and calculate the camera parameters from the image data. Note that the information stored in the storage unit 130 is not limited to having the above-described data structure, and it is sufficient that the information necessary for performing the processing in each block is stored.

[0014] The disparity image generation unit 140 calculates the disparity in the corrected camera image based on the corrected camera image and the camera parameters stored in the storage unit 130, and generates a disparity image. For the generation of the disparity image, as described in Patent Document 1 and Patent Document, a method of matching the images acquired by the stereo camera block by block to obtain the disparity for each pixel (Sum of Absolute Difference method, etc.) may be used. The disparity image generation unit 140 stores the generated disparity image in the storage unit 130.

[0015] The index detection unit 150 detects information on the index (referred to as index detection information in this embodiment) reflected in the corrected camera image stored in the storage unit 130 and stores it in the storage unit 130. The index detection unit 150 is an example of the index detection means referred to in the present invention. As the index, as shown in FIG. 3, a marker having a rectangular area including a square is used. For index detection, a rectangular marker detection process with an ID such as ArUco may be used. Specifically, a rectangular area is detected from the corrected camera image, and the bit pattern arranged in the rectangular area is subjected to a homography transformation and identified to output the ID and the coordinate values of the four vertices of the rectangular area. Referring to FIG. 3, an example of the index detection process will be described. The index 300 is reflected in the left and right images 151L and 151R which are corrected camera images, and the index detection unit 150 specifies the coordinate values of the four vertices of the rectangular area of the index 300. Specifically, the index detection unit 150 specifies the coordinate values <X value, Y value> of the four vertices 310A to 310D of the rectangular area of the index 300 detected in the left image 151L. Further, the index detection unit 150 specifies the coordinate values <X value, Y value> of the four vertices 310E to 310H of the rectangular area of the index 300 detected in the right image 151R. Then, the index detection unit 150 stores the coordinate values of the four vertices 310A to 310D and the coordinate values of the four vertices 310E to 310H in the storage unit 130. The index detection unit 150 sequentially determines whether an index is reflected in the corrected camera image acquired at regular intervals by the image acquisition unit 110 and corrected by the image correction unit 120. Then, when an index is reflected in the corrected camera image, the index detection unit 150 outputs the index detection information to the storage unit 130 and updates the index detection information in the current frame. When no index is reflected in the corrected camera image, information indicating that there is no index is output as the index detection information. When a plurality of indices are detected in the same frame, the coordinate values of the four vertices of the rectangular area are associated with each detected index ID and stored in the storage unit 130.

[0016] The depth correction unit 160 reads the disparity image stored in the memory unit 130 and corrects the depth value obtained from the disparity image. As shown in Figure 1(b), the depth correction unit 160 comprises a correction amount calculation unit 161 and a depth image generation unit 165. The depth correction unit 160 is an example of a correction means as defined in the present invention.

[0017] The correction amount calculation unit 161 calculates a correction amount for correcting the depth value. The process of the correction amount calculation unit 161 will be explained with reference to Figures 3 to 5. First, the correction amount calculation unit 161 refers to the index detection information stored in the memory unit 130 to obtain the coordinate values ​​of the vertices of the rectangular area of ​​the index 300 in the corrected camera image. When left and right images 151L and 151R, as shown in Figure 3, are obtained, the depth correction unit 160 associates the vertices 310A~310D and 310E~310H of the rectangular area of ​​the index 300 with the same ID in the left and right images 151L and 151R. Specifically, vertex 310A in the left image 151L and vertex 310E in the right image 151R are associated as being the same point in 3D space. Similarly, vertex 310D in the left image 151L and vertex 310H in the right image 151R are associated as being the same point in 3D space. Here, for the sake of simplicity, we will focus on vertices 310A and 310D. If multiple indexes are detected in the same frame, the association is made for each index.

[0018] Once the corresponding points (vertices 310A and 310E, and vertices 310D and 310H in Figure 3) in the left and right images 151L and 151R can be mapped, the three-dimensional points 410A and 410D in the camera coordinate system can be calculated using a known triangulation calculation method, as shown in Figure 4. The camera coordinate system is a coordinate system based on the camera, with the depth direction relative to the imaging unit 100 being the Z direction. The correction amount calculation unit 161 determines the distance between points 410A and 410D (the length of the line segment 510 connecting points 410A and 410D). Furthermore, the correction amount calculation unit 161 refers to the index information stored in the storage unit 130 and obtains the side lengths of the rectangular area of ​​the index with the corresponding ID. Furthermore, as shown in Figure 5, the correction amount calculation unit 161 sets straight lines 540 and 550 passing through points 410A and 410D from the camera origin O in the camera coordinate system, and sets the length of the line segment 510 connecting points 410A and 410D. Next, the correction amount calculation unit 161 estimates the three-dimensional points 500A and 500D that are output when there is no error in the camera parameters. The three-dimensional points 500A and 500D are determined to satisfy the following conditions. (1) Point 500A lies on the straight line 540. (2) Point 500D lies on the straight line 550. (3) The slope of line segment 520 connecting points 500A and 500D is the same as the slope of line segment 510. (4) The distance between points 500A and 500D (the length of the line segment 520 connecting points 500A and 500D) is the same as the length of the side in the index information. The correction amount calculation unit 161 then sets, for example, the difference 530 in the Z values ​​of point 500A and point 410A as the correction amount for correcting the depth value (referred to as the depth correction value in this embodiment). In this way, the depth correction value is calculated using the index information and the known side lengths included therein as the correct values.

[0019] The depth image generation unit 165 calculates a depth value for each pixel from the disparity image stored in the memory unit 130, and adds the depth correction value calculated by the correction amount calculation unit 161 to this depth value. Then, the depth image generation unit 165 generates a depth image with corrected depth values ​​and outputs it to the image generation unit 170. In this way, the image captured by the imaging unit 100 can be displayed on the display 180 as a depth image with corrected depth values.

[0020] The processing of the depth correction unit 160 may be performed immediately after the image acquired by the image acquisition unit 110 is updated and a disparity image is generated, or it may be performed at a certain interval. For example, the depth correction value may be calculated each time 10 frames of disparity images are generated. Referring to Figure 6, the time-series change when the depth correction value is calculated at a certain interval will be explained. In Figure 6(a), the horizontal axis is time t and the vertical axis is the depth correction value f. The depth correction value 601 is calculated at the interval from time t0 to time t1, and thereafter, depth correction values ​​602, 603, ... are calculated at a certain interval. In Figure 6(b), the horizontal axis is time t and the vertical axis is the depth value d. The dotted line 650 shows the uncorrected depth value, and the solid line 655 shows the corrected depth value. As shown in Figure 6(b), when the depth correction value f is updated, the depth correction value f is added to the depth value, and the same depth correction value f is reflected in subsequent frames. Furthermore, instead of simply reflecting the updated depth correction value f, it is also possible to assign weights to the depth correction value f. Figure 6(c) shows time t on the horizontal axis and the depth value d on the vertical axis. As shown in Figure 6(c), the weights may be increased over time to control the smooth change in the corrected depth value.

[0021] Next, with reference to Figure 2, the processes executed by the information processing device 101 will be explained. Figure 2(a) is a flowchart showing the processes executed by the information processing device 101. The details of the processes executed by each part have already been described, and redundant explanations will be omitted in the following steps. In step S200, the information processing device 101 acquires camera parameters from the imaging unit 100 and stores them in the storage unit 130. Note that the process is not limited to acquiring camera parameters from the imaging unit 100; the storage unit 130 may also store the results of prior calibration of the camera parameters.

[0022] In step S210, the image acquisition unit 110 acquires the image captured by the imaging unit 100. In step S220, the image correction unit 120 performs correction processing on the image acquired in step S210 using camera parameters stored in the storage unit 130, and stores the corrected camera image in the storage unit 130. In step S230, the disparity image generation unit 140 calculates the disparity in the corrected camera image based on the corrected camera image and camera parameters stored in the storage unit 130, generates a disparity image, and stores the disparity image in the storage unit 130. In step S240, the indicator detection unit 150 detects indicator detection information of the indicators shown in the corrected camera image stored in the storage unit 130 and stores it in the storage unit 130.

[0023] In step S250, the depth correction unit 160 performs depth correction processing. Details of the depth correction processing in step S250 will be described later with reference to Figure 2(b). In step S260, the information processing device 101 determines whether the termination condition is met. For example, if a termination instruction is entered by the user, it determines that the termination condition is met. If the termination condition is met, this flowchart is terminated. If the termination condition is not met, the process returns to step S210.

[0024] Figure 2(b) is a flowchart showing the details of the depth correction process in step S250. In step S251, the depth correction unit 160 acquires camera parameters, disparity images, indicator detection information, and indicator information from the storage unit 130 as information necessary for depth correction processing. In step S253, the depth correction unit 160 calculates the depth correction value as described above based on the information acquired in step S251. In step S255, the depth correction unit 160 corrects the depth value using the depth correction value calculated in step S253 and generates a depth image.

[0025] As described above, the depth value is corrected based on the indicator detection information of the indicators captured in the image taken by the imaging unit 100. This means that even if the camera parameters fluctuate when measuring the depth of the scene, if the indicators can be captured, the depth value is corrected at that point, and a depth value with high reproducibility can be obtained even with continuous measurements. Furthermore, the three-dimensional points 500A and 500D in the camera coordinate system can be calculated in a single step, eliminating the need for repeated correction calculations as described in Patent Document 1. The correction calculation can be completed within the update time of one image frame, and the output of depth values ​​is not interrupted by the correction process. Furthermore, unlike Patent Document 2, which assumes the subject is moving, depth correction can be performed even when the subject is stationary. Also, unlike Patent Document 2, it is not necessary to use images taken at two locations; the depth correction value can be determined from an image taken at one location. Therefore, the present invention can also be applied to stereo cameras with a fixed viewpoint. Furthermore, since the system does not require the 3D position of the indicator as pre-prepared indicator information, there is no need to fix the indicator in space, and the indicator may move in the space being observed. For example, when a user wants to correct the depth value, depth correction can be completed simply by having the imaging unit 100 photograph the indicator held in the user's hand. Therefore, the configuration for depth correction can be simplified, and maintenance knowledge is not required, contributing to a reduction in maintenance costs.

[0026] In this embodiment, an example was described in which the depth correction value is obtained from the information of adjacent vertices 310A and 310D (vertices 310E and 310H) in a rectangular region, but the method is not limited to this. For example, any two vertices out of the four vertices of the rectangular region may be selected. Alternatively, the depth correction value may be obtained using the distance between two vertices of multiple combinations selected from the four vertices of the rectangular region. In this case, multiple depth correction values ​​will be obtained, but for example, the average value may be used. Furthermore, when selecting two vertices, it is preferable to select those with a long distance between them on the corrected camera image 151L or 151R. This is because if the distance between two vertices on the image is short, sampling errors occur in the linear fitting process used to identify the vertex information, and the estimated vertex tends to contain errors.

[0027] Furthermore, although this embodiment describes an example of calculating a depth correction value and correcting the depth value, it is also possible to correct the parallax used to determine the depth value. In this case, the depth value can be determined from the corrected parallax, and there is no need to correct the depth value. Parallax correction can be done by calculating the difference in parallax values ​​obtained from the difference of Z values ​​530 and adding it to the parallax value.

[0028] Figure 7 shows an example of a hardware configuration for realizing an information processing device to which the present invention is applied. The CPU 701 controls the entire computer using computer programs and data stored in the RAM 707 and ROM 702. The CPU 701 also performs the processes described below as being performed by the information processing device in each embodiment. The RAM 707 temporarily stores computer programs and data loaded from the external storage device 706 and the storage medium drive 705. The RAM 707 also has an area for temporarily storing data received from the outside. Furthermore, the RAM 707 also has a work area used by the CPU 701 when executing each process. In other words, the RAM 707 can provide various areas as appropriate. The ROM 702 stores computer setting data, boot programs, etc. The keyboard 709 and mouse 708 are examples of input devices, and various instructions can be input to the CPU 701 by the user's operation. The display unit 704 is composed of a CRT or liquid crystal screen, etc., and displays the processing results by the CPU 701 as images, characters, etc. For example, the display unit 704 can display an image captured by the imaging unit 100. The external storage device 706 is a large-capacity information storage device, such as a hard disk drive. The external storage device 706 stores the OS (operating system) and programs and data that cause the CPU 701 to execute various processes performed by the information processing device. The computer programs and data stored in the external storage device 706 are loaded into the RAM 707 as appropriate according to the control of the CPU 701. The CPU 701 executes processing using these loaded programs and data, thereby realizing the functions of the information processing device to which the present invention is applied. The storage medium drive 705 reads programs and data recorded on storage media such as CD-ROMs and DVD-ROMs, and writes computer programs and data to storage media. Note that some or all of the programs and data described as being stored in the external storage device 706 may be recorded on this storage medium. The computer programs and data read from the storage medium by the storage medium drive 705 are output to the external storage device 706 and the RAM 707.Interface (I / F) 703 consists of an analog video port or a digital input / output port such as IEEE1394 for connecting the imaging unit 100. Data received via I / F 703 is input to RAM 707 or external storage device 706. Bus 710 connects each of the above components by bus signals.

[0029] <Second Embodiment> In the second embodiment, an example is described in which it is determined whether or not to perform depth value correction according to the state of an indicator shown in the image captured by the imaging unit 100. The basic configuration and processing operation of the imaging system are the same as in the first embodiment, so the following description will omit explanations that are common to the first embodiment and will focus on the differences from the first embodiment. When correcting the depth value, the accuracy of intersection detection of the indicator is improved if the indicator is directly facing the imaging unit 100 during the image capture. If there is a difference in the lengths of two adjacent sides (vertical and horizontal) of the rectangular area of ​​the indicator, the number of samples used in the linear fitting process when detecting the rectangular area will not be uniform, which may cause variability in vertex detection. Therefore, by directly facing the indicator during the image capture, the difference in the lengths of two adjacent sides of the rectangular area is minimized, thereby reducing variability in vertex detection.

[0030] Figure 8 shows the functional configuration of the depth correction unit 160 of the information processing device 101 according to the second embodiment. In the second embodiment, the depth correction unit 160 comprises an indicator state determination unit 810, a correction amount calculation unit 161, and a depth image generation unit 165. The indicator state determination unit 810 determines whether or not to perform depth value correction according to the state of the indicator shown in the image captured by the imaging unit 100.

[0031] Next, the processes executed by the information processing device 101 will be described with reference to Figures 2(a) and 10. Note that the flowchart in Figure 2(a) is common to all embodiments, and in this embodiment, the depth correction process in step S250 is modified compared to the first embodiment. Figure 10 is a flowchart detailing the depth correction in step S250. In step S1010, the indicator state determination unit 810 obtains the coordinate values ​​of the four vertices of the rectangular region of the indicator with the ID of interest in the corrected camera image. Then, the indicator state determination unit 810 calculates the lengths of the four sides of the rectangular region from the obtained coordinate values ​​of the four vertices and calculates the difference between the lengths of two adjacent sides.

[0032] In step S1020, the index state determination unit 810 determines whether the difference in the lengths of two adjacent sides calculated in step S1010 is less than or equal to a predetermined threshold. That is, it determines whether the variation in vertex detection is suppressed by minimizing the difference in the lengths of two adjacent sides. If the index state determination unit 810 determines that it is less than or equal to the predetermined threshold, it moves the process to step S251. On the other hand, if the index state determination unit 810 determines that it exceeds the predetermined threshold, it moves the process to step S1030.

[0033] In step S1030, the indicator state determination unit 810 acquires the corrected camera image stored in the memory unit 130 and combines a notification message prompting the user to align the indicator directly with a guide frame representing the appropriate position of the indicator on the image. In step S1040, the information processing device 101 outputs the corrected camera image, which has the notification message and guide frame combined in step S1030, to the image generation unit 170. As a result, as shown in Figure 9, the notification message 900 and guide frame 910 can be combined with the image captured by the imaging unit 100 and displayed on the display 180.

[0034] As described above, by performing depth value correction when the state of the indicators in the image captured by the imaging unit 100 is favorable, it is possible to reduce the variation in the detection of vertices in the rectangular area and prevent a decrease in the accuracy of depth measurement.

[0035] <Third Embodiment> In the third embodiment, an example is described in which it is determined whether or not to perform depth value correction depending on the relative stationary state of the imaging unit 100 and the indicator. The basic configuration and processing operation of the imaging system are the same as in the first embodiment, so the following description will omit explanations that are common to the first embodiment and will focus on the differences from the first embodiment. In the third embodiment, in order to detect the vertices of the rectangular region of the indicator, the relative stationary state between the imaging unit 100 and the indicator is determined, and the median value of the coordinate values ​​of the same vertex across multiple frames from the corrected camera image when it is stationary is used as the coordinate value of that vertex.

[0036] Figure 11 shows the functional configuration of the depth correction unit 160 of the information processing device 101 according to the third embodiment. In the third embodiment, the depth correction unit 160 comprises a stationary state determination unit 1110, a correction amount calculation unit 161, and a depth image generation unit 165. The stationary state determination unit 1110 determines whether or not to perform depth value correction according to the relative stationary state between the imaging unit 100 and the indicator.

[0037] Next, the processes executed by the information processing device 101 will be described with reference to Figures 2(a) and 13. Note that the flowchart in Figure 2(a) is common to all embodiments, and in this embodiment, the depth correction process in step S250 is modified compared to the first embodiment. Figure 13 is a flowchart showing the details of the depth correction in step S250. In step S1310, the stationary state determination unit 1110 determines the relative stationary state of the indicator between the imaging unit 100 and the indicator. Specifically, the stationary state determination unit 1110 obtains past indicator detection information stored in the memory unit 130 and obtains the coordinate values ​​of the two vertices of the rectangular area of ​​the indicator with the ID of interest for each frame. Then, the stationary state determination unit 1110 calculates the amount of movement of the two vertices between frames. The amount of movement of one vertex in each frame is obtained by calculating the differential values ​​of the X and Y coordinates between frames, and if it exceeds a predetermined threshold in all frames, it is determined that it is moving.

[0038] In step S1320, if the stationary state determination unit 1110 determines that the vehicle is in a stationary state in step S1310, it moves the process to step S1325. On the other hand, if the stationary state determination unit 1110 determines that the vehicle is not in a stationary state, it moves the process to step S1330.

[0039] In step S1325, the stationary state determination unit 1110 changes the coordinate values ​​of the index detection information obtained from the storage unit 130 in step S251 to use the median of the coordinate values ​​of the two vertices of interest from past index detection information across multiple frames.

[0040] In step S1330, the stationary state determination unit 1110 acquires the corrected camera image stored in the memory unit 130 and synthesizes a notification message prompting the user to enter a stationary state. In step S1340, the information processing device 101 outputs the corrected camera image, into which the notification message was synthesized in step S1330, to the image generation unit 170. As a result, as shown in Figure 12, the notification message 1200 can be synthesized with the image captured by the imaging unit 100 and displayed on the display 180.

[0041] As described above, depth correction is performed when the imaging unit 100 and the indicator are in a relative stationary state, allowing depth correction to be performed with high detection accuracy for the vertices of the rectangular area. Although this embodiment has been described based on the first embodiment, the second embodiment may also be combined. In this case, the alignment determination in steps S1010 and S1020 should be performed, and if it is determined that the objects are aligned, the stationary state determination in step S1310 should be performed. If the determination is not OK in any of the steps, a notification message corresponding to the determination should be displayed (S1030 / S1330).

[0042] <Fourth Embodiment> In the fourth embodiment, a first imaging unit that captures images from multiple directions and a second imaging unit that also captures images from multiple directions and has smaller time fluctuations in camera parameters than the first imaging unit are fixed to each other, and an example is described in which index information is detected from the image captured by the second imaging unit. The basic configuration and processing operation of the imaging system are the same as in the first embodiment, so the following description will omit explanations that are common to the first embodiment and will focus on the differences from the first embodiment.

[0043] Figure 14 shows the functional configuration of the imaging system according to the fourth embodiment. The imaging system comprises an imaging unit 100 which is a first imaging unit, a second imaging unit 1400, an information processing device 101, an image generation unit 170, and a display 180, with the imaging unit 100 and the second imaging unit 1400 being fixed to each other. Here, one possible factor causing camera parameters to vary over time is the minute fluctuations in the relative position and orientation of the image sensor and lens due to temperature, shock, etc. In particular, cameras with small image sensor areas and small lens areas are more susceptible to the influence of changes in the lens relative to each pixel of the image sensor, resulting in larger temporal fluctuations in camera parameters. On the other hand, even with the same number of pixels, if the image sensor area is large and the lens area is also large, it is less susceptible to minute fluctuations, resulting in smaller temporal fluctuations in camera parameters. In this embodiment, the second imaging unit 1400 is composed of a large imaging sensor and a lens, and the temporal variation of camera parameters is smaller compared to the imaging unit 100. For example, the imaging unit 100 is a small color camera for presenting video to the user. Also, the second imaging unit 1400 is a grayscale camera that estimates the position and orientation of the imaging unit 100 by SLAM (Simultaneous Localization and Mapping). The second imaging unit 1400 has a large imaging sensor area to stably estimate the position and orientation even in a dark place, and the variation of camera parameters over time is small. The imaging unit 100 and the second imaging unit 1400 are fixedly arranged with each other, and it is assumed to be a device for realizing AR (Augmented Reality) that synthesizes and displays a CG image on the video of the imaging unit 100 while measuring the position and orientation of the head by wearing it on the user's head or the like. In the image acquisition unit 110, image correction unit 120, storage unit 130, and index detection unit 150 of the information processing device 101, the images captured by the second imaging unit 1400 are processed in the same manner as the images captured by the imaging unit 100. Each block may be configured to be able to process the imaging unit 100 and the second imaging unit 1400 in parallel, or may be configured to process them sequentially in a time-sharing manner.

[0044] In this embodiment, compared with the first embodiment, the information stored in the storage unit 130 is different in the following points. Information added to the storage unit 130: · Corrected camera image of the second imaging unit 1400 (left and right images in the case of a two-eye stereo camera) · Camera parameters of the second imaging unit 1400 (camera external parameters such as the relative position and orientation between stereo cameras, and camera internal parameters such as the focal length and principal point position in each stereo camera) · Index detection information detected by the index detection unit 150 from the corrected camera image of the second imaging unit 1400 (index ID, coordinate values <X value, Y value> of the vertices of the rectangular region for each index) · Past index detection information detected from the corrected camera image of the second imaging unit 1400 (for example, index detection information for the past 10 frames)

[0045] Information of the storage unit 130 that is modified from the first embodiment: • Indicator information (information generated based on indicator detection information detected from the corrected camera image of the second imaging unit 1400, such as the ID, which is the identification information of the indicator, and the length of the sides of the rectangular area.)

[0046] Thus, since the index information is generated based on the index detection information of the second imaging unit 1400, there is the advantage that the user does not need to prepare the index information in advance by inputting it or otherwise. Furthermore, it indicates that the index detection information from the second imaging unit 1400 is used as the correct value (side length of the rectangular area) when the depth correction unit 160 determines the depth correction value. Furthermore, the calculation of the depth correction value in the depth correction unit 160 can be implemented using the same process as in the first embodiment, except that it refers to the index information generated from the index detection information of the second imaging unit 1400. Furthermore, the processing performed by the information processing device 101 is the same as that in the first embodiment, so its explanation will be omitted. As described above, even without preparing index information in advance, depth correction values ​​can be calculated using data obtained from the imaging unit, where the time variation of camera parameters is small, as the correct values.

[0047] Although the present invention has been described above along with its embodiments, these embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or its main features. (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0048] 100: Imaging unit, 101: Information processing unit, 110: Image acquisition unit, 120: Image correction unit, 130: Storage unit, 140: Parallax image generation unit, 150: Indicator detection unit, 160: Depth correction unit, 161: Correction amount calculation unit, 165: Depth image generation unit, 810: Indicator state determination unit, 1110: Stationary state determination unit, 1400: Second imaging unit

Claims

1. an image acquisition means for acquiring images captured by an imaging unit that captures images from a plurality of directions; an index detection means for detecting information of an index appearing in the image acquired by the image acquisition means; a correction means for correcting a parallax in the image acquired by the image acquisition means or a depth value calculated from the parallax based on information of the index detected by the index detection means; an image acquisition unit that acquires an image of an image corresponding to the image of the image acquisition unit, and an image acquisition unit that acquires an image of the image of the image acquisition unit;

2. 2 . The information processing apparatus according to claim 1 , wherein the correction means calculates a correction amount for the correction by using index information related to the index.

3. The information processing apparatus according to claim 2 , wherein the index information includes a distance between two points included in the index.

4. 4. The information processing apparatus according to claim 2, wherein the index information includes identification information of the index.

5. 5. The information processing apparatus according to claim 1, further comprising a determination unit that determines whether or not to execute the correction depending on a relative stationary state between the imaging unit and the index.

6. 5. The information processing apparatus according to claim 2, wherein the index information is prepared in advance.

7. a first imaging unit that captures images from a plurality of directions and a second imaging unit that captures images from a plurality of directions and has smaller time fluctuations in camera parameters than the first imaging unit, the first imaging unit and the second imaging unit being fixed to each other; 5. The information processing apparatus according to claim 2, wherein the index information is detected from an image captured by the second imaging section.

8. An image acquisition step of acquiring images captured by an imaging unit that captures images from a plurality of directions; an index detection step of detecting information of an index appearing in the image acquired in the image acquisition step; a determination step of determining whether or not to perform a correction on the image acquired in the image acquisition step, depending on a state of an index shown in the image acquired in the image acquisition step; An information processing method characterized by having a correction step of correcting the parallax in the image acquired in the image acquisition step or a depth value calculated from the parallax based on information of the index detected in the index detection step, if it is determined in the determination step that correction should be performed.

9. an image acquisition means for acquiring images captured by an imaging unit that captures images from a plurality of directions; an index detection means for detecting information of an index appearing in the image acquired by the image acquisition means; a correction means for correcting a parallax in the image acquired by the image acquisition means or a depth value calculated from the parallax based on information of the index detected by the index detection means; A program for causing a computer to function as a determination unit that determines whether or not to execute the correction depending on a state of an index shown in the image acquired by the image acquisition unit.