Depth estimation device, calibration method, calibration program

Abstract

To provide a technique capable of realizing highly accurate depth estimation without requiring manual work or an additional apparatus, in a technique of estimating a depth in an image captured by a camera.SOLUTION: A depth estimation apparatus according to the present disclosure executes the processing of: estimating a depth in an image; acquiring an installation position of a camera with respect to a horizontal plane or a vertical plane; specifying a plane region in the image in which the horizontal plane or the vertical plane is reflected; setting a plurality of partial regions in the image; calculating a regression plane based on the estimated depth in the plane region for each partial region; calculating a calibration value for each partial region by comparing a position of the camera with respect to the regression plane with the installation position of the camera with respect to the horizontal plane or the vertical plane; and performing calibration based on the corresponding calibration value in each partial region.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present disclosure relates to a technique for analyzing an image captured by a camera and estimating the depth in the image.

Background Art

[0002] Patent Document 1 discloses a calibration method for estimating camera parameters representing the characteristics of each imaging system in a distance measuring device including a plurality of imaging systems. In the calibration method disclosed in Patent Document 1, the camera parameters are estimated using information obtained by imaging a reference chart arranged so as to have a predetermined positional relationship with the distance measuring device.

[0003] In addition, there are the following Patent Documents 2 and Patent Documents 3 as documents showing the technical level of this technical field.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technique disclosed in Patent Document 1 is applied to a device (for example, a device using a stereo camera) that recognizes the depth in an image by detecting the parallax between a plurality of images.

[0006] On the other hand, techniques are being considered to estimate depth in an image by analyzing a single image captured by a monocular camera or similar device. Such techniques have advantages such as low cost and space saving because they require less equipment. They also have the advantage of being applicable to cameras used for other purposes. In particular, in recent years, there has been a growing interest in using pre-trained machine learning models to analyze images.

[0007] Incidentally, the appearance of the image being analyzed varies depending on the characteristics of the camera used to capture it. Therefore, in order to ensure estimation accuracy, it is necessary to perform calibration according to the image appearance and camera characteristics.

[0008] However, traditionally, calibration has required manual work or additional equipment, resulting in considerable time and cost. For example, when analyzing images using a pre-trained machine learning model, calibration is performed by retraining the parameters.

[0009] One objective of this disclosure is to provide a technology that enables highly accurate depth estimation without requiring manual work or additional equipment, in relation to a technology for analyzing images and estimating depth in those images. [Means for solving the problem]

[0010] The first aspect of this disclosure relates to a depth estimation device.

[0011] The depth estimation device according to the first aspect comprises one or more processors that analyze images captured by a camera to estimate the depth in the image, and a storage device that stores information on the camera's installation position relative to a horizontal or vertical plane. The one or more processors are further configured to perform a first process of calculating a calibration value for the estimated depth, and a second process of performing calibration of the estimated depth based on the calculated calibration value. The first process includes identifying a planar region in the image in which the horizontal or vertical plane is reflected, setting a plurality of sub-regions in the image, calculating a regression plane representing the horizontal or vertical plane for each sub-region based on the depth estimated in the planar region included in each sub-region, and calculating a calibration value for each sub-region by comparing the camera's position relative to the regression plane with the installation position. The second process performs calibration of the depth estimated for each sub-region based on the corresponding calibration value.

[0012] A second aspect of this disclosure relates to a calibration method for performing a calibration of the depth estimated in an image captured by a camera using a computer.

[0013] The calibration method relating to the second aspect involves a computer performing the following steps: acquiring information on the camera's installation position relative to a horizontal or vertical plane; identifying a planar region in the image in which the horizontal or vertical plane is reflected; setting multiple sub-regions in the image; calculating a regression plane representing the horizontal or vertical plane for each of the multiple sub-regions based on the depth estimated in the planar region included in each sub-region; calculating a calibration value for each sub-region by comparing the camera's position relative to the regression plane with the installation position; and performing a calibration of the depth estimated based on the corresponding calibration value in each sub-region.

[0014] A third aspect of this disclosure relates to a calibration program that causes a computer to perform depth calibration in images captured by a camera.

[0015] The calibration program relating to the third aspect causes the computer to perform the following processes: acquiring information on the camera's installation position relative to a horizontal or vertical plane; identifying a planar region in the image in which the horizontal or vertical plane is reflected; setting multiple sub-regions in the image; calculating a regression plane representing the horizontal or vertical plane for each of the multiple sub-regions based on the depth estimated in the planar region included in each sub-region; calculating a calibration value for each sub-region by comparing the camera's position relative to the regression plane with the installation position; and performing a calibration of the depth estimated based on the corresponding calibration value in each sub-region. [Effects of the Invention]

[0016] According to this disclosure, appropriate calibration values ​​can be calculated for each sub-region set in an image without requiring manual work or additional equipment. Then, in each sub-region, depth calibration is performed based on the estimated depth value. This enables highly accurate depth estimation without requiring manual work or additional equipment. [Brief explanation of the drawing]

[0017] [Figure 1] This figure illustrates the function of the depth estimation device according to this embodiment. [Figure 2] This is a diagram illustrating the process of calculating calibration values. [Figure 3] This figure shows an example of the results of various processes. [Figure 4] This diagram illustrates the results of various processes. [Figure 5]This is a diagram showing an example of the configuration of the depth estimation device according to the present embodiment. [Figure 6] This is a diagram showing an example of the processing executed by the depth estimation device according to the present embodiment regarding the calibration of the estimated depth. MODE FOR CARRYING OUT THE INVENTION

[0018] Hereinafter, the present embodiment will be described with reference to the drawings.

[0019] 1. Depth Estimation Device FIG. 1 is a diagram for explaining the basic functions of the depth estimation device 10 according to the present embodiment. The depth estimation device 10 analyzes the image captured by the camera 200 and estimates the depth in the image.

[0020] The camera 200 is installed to capture an arbitrary area for measuring depth. The camera 200 outputs the data of the captured image. The camera 200 may output the data of a video composed of continuously captured images. In the present embodiment, the configuration of the camera 200 is not particularly limited.

[0021] The depth estimation device 10 executes a depth estimation process P10 and a calibration process P20 on the image captured by the camera 200.

[0022] The depth estimation process P10 analyzes the image and estimates the depth in the image. The depth in the image is given, for example, for each pixel of the image. In this case, when each pixel is represented by coordinates (M, N) on the image, the depth in the image can be represented by D(M, N). D(M, N) indicates the magnitude of the depth given to the pixel corresponding to the coordinates (M, N) on the image. D(M, N) can also be called a "depth map".

[0023] Figures 3(A) and 3(B) show an example of an image captured by camera 200 and an example of the depth estimated by depth estimation processing P10, respectively. In Figure 3(B), the magnitude of the estimated depth relative to the image shown in Figure 3(A) is indicated by the intensity of the intensity. In particular, the intensity increases as the estimated depth increases.

[0024] The depth estimation process P10 may employ known and suitable techniques. For example, the depth estimation process P10 may be configured to estimate the depth in an image using a pre-trained machine learning model that takes an image as input data.

[0025] Incidentally, in the depth estimation device 10, it is conceivable that a camera 200 with suitable characteristics is selected depending on the object and environment being imaged. If the characteristics of the camera 200 are different, the image will change. On the other hand, the depth estimation process P10 is usually optimized for images captured by a specific camera. For example, when using a pre-trained machine learning model, the depth estimation process P10 is optimized for the images used as training data. Therefore, the depth estimated by the depth estimation process P10 may have errors depending on the image.

[0026] In particular, depth estimation is significantly affected by distortion, which distorts the image. The image shown in Figure 3(A) exhibits barrel distortion. As shown in Figure 3(A), distortion increases with distance from the center of the image. Therefore, the error associated with distortion is thought to differ across different parts of the image and vary depending on the magnitude of the distortion.

[0027] Refer to Figure 1 again. Calibration process P20 performs calibration of the estimated depth to reduce such errors. In calibration process P20, calibration is performed based on the calibration value managed by the depth estimation device 10. The depth estimation device 10 outputs the result of calibration process P20 as the estimation result.

[0028] The depth estimation device 10 according to this embodiment achieves highly accurate depth estimation by calculating appropriate calibration values ​​according to the image quality. In particular, the depth estimation device 10 according to this embodiment makes it possible to calculate calibration values ​​without requiring manual work or additional equipment. The following describes the process for calculating calibration values ​​and the details of the calibration process P20 based on the calculated calibration values.

[0029] 2. Calculation of Calibration Values The depth estimation device 10 performs a process to calculate a calibration value at a predetermined timing. The predetermined timing may be set as preferred. For example, the predetermined timing may be when the camera 200 is activated. Alternatively, the predetermined timing may be set periodically at regular intervals. Another example is when the device receives an instruction from the user to update the calibration value.

[0030] In the process of calculating the calibration value, the calibration value is calculated from the image captured by the camera 200 at a predetermined timing, the depth estimated by the depth estimation process P10, and the installation position of the camera 200 relative to a predetermined horizontal or vertical plane. Information on the installation position of the camera 200 relative to the horizontal or vertical plane may be managed by the depth estimation device 10. The horizontal or vertical plane for defining the installation position may be suitably selected depending on the environment to which this embodiment is applied. Typically, the horizontal plane is the floor or ground, and the vertical plane is a wall. The horizontal or vertical plane may also be one surface of an object placed around the camera 200. For example, the horizontal plane may be the top surface of a table. However, it is desirable that the horizontal or vertical plane has a certain extent of spread. The installation position of the camera 200 relative to the horizontal plane is, for example, the height of the camera 200 relative to the horizontal plane. The installation position of the camera 200 relative to the vertical plane is, for example, the horizontal distance of the camera 200 relative to the vertical plane. The depth estimation device 10 may also be configured to manage information on multiple installation locations for each of multiple horizontal or vertical planes.

[0031] The process for calculating calibration values ​​will be explained below with reference to Figure 2. The process for calculating calibration values ​​consists of a planar region identification process P31, a partial region setting process P32, a depth extraction process P33, a regression plane calculation process P34, and a calibration value calculation process P35.

[0032] The planar region identification process P31 identifies planar regions in an image where a horizontal or vertical plane is reflected. Figure 3(C) shows an example of a planar region 21 identified by the planar region identification process P31. In Figure 3(C), the planar region 21 is the region where the floor is reflected in the image shown in Figure 3(A). The planar region identification process P31 can be implemented, for example, by performing semantic segmentation on the image. In this case, the region of pixels labeled as a horizontal or vertical plane can be defined as the planar region 21. The planar region identification process P31 may identify multiple planar regions 21 for multiple horizontal or vertical planes. For example, in the image shown in Figure 3(A), the planar region identification process P31 may further identify the region where the wall is reflected as another planar region 21.

[0033] The partial region setting process P32 sets multiple partial regions in the image. Figure 3(D) shows an example of multiple partial regions 22 set by the partial region setting process P32. As mentioned above, the error associated with distortion is thought to differ in each part of the image and change depending on the magnitude of the distortion. Therefore, by setting multiple partial regions 22 in this way, it can be expected that the error associated with distortion will be of a similar magnitude in each partial region 22.

[0034] There are various ways to define multiple sub-regions 22. For example, multiple sub-regions 22 can be defined by dividing the image into a grid. Alternatively, multiple sub-regions 22 can be defined by placing regions of a predetermined shape on the image.

[0035] In Figure 3(D), each subregion 22 is a region enclosed by edges with curvature. In particular, the curvature is greater for subregions 22 that are farther from the center of the image. Also, the size of each subregion 22 is smaller for subregions 22 that are farther from the center of the image.

[0036] As mentioned above, distortion increases with distance from the center of the image. Therefore, by increasing the curvature of the edges in the sub-regions 22 that are farther from the center of the image, each sub-region 22 can be shaped to correspond to the change in distortion. Furthermore, by decreasing the size of each sub-region 22 in the sub-regions 22 that are farther from the center of the image, the sub-regions 22 can be set more precisely in areas where the distortion is greater. In other words, by providing the curvature of the edges and the size of each sub-region 22 in this way, it is possible to set multiple sub-regions 22 that are more appropriate in order to make the error associated with distortion aberration similar in each sub-region 22. In this case, each sub-region 22 can be defined, for example, by the curvature of the edges and the width and height of the region. At this time, the curvature of the edges and the width and height of the region may be optimally determined by testing.

[0037] Refer to Figure 2 again. The depth extraction process P33 extracts the estimated depth for each sub-region 22 in the planar region 21 contained within the sub-region 22. The depth extracted for each sub-region 22 is associated with the corresponding sub-region 22. The depth extraction process P33 can be implemented, for example, by calculating the area of ​​the common part between the target sub-region 22 and the planar region 21, and then using the calculated common area to reference the estimated depth.

[0038] Figure 4(A) shows the planar region 21 and multiple subregions 22 superimposed on Figure 3(C) and Figure 3(D), respectively. Figure 4(B) shows an example of the depth extracted by the depth extraction process P33 for the subregion 22A shown in Figure 4(A). In Figure 4(B), it can be seen that the depth in the area where subregion 22A and planar region 21 intersect has been extracted from the depth shown in Figure 3(C).

[0039] Incidentally, the subregion 22B shown in Figure 4(A) has no common area with the planar region 21. Such a subregion 22 can be dealt with by performing the above process on other planar regions 21 that reflect other horizontal or vertical planes. For example, for subregion 22B, the above process can be performed on other planar regions 21 that have been identified as areas where walls are reflected.

[0040] Refer to Figure 2 again. The regression plane calculation process P34 calculates a regression plane representing a horizontal or vertical plane for each sub-region 22 based on the extracted depth. The regression plane calculated for each sub-region 22 is associated with the corresponding sub-region 22.

[0041] The regression plane calculation process P34 performs the following steps, for example:

[0042] First, the regression plane calculation process P34 obtains the position in world coordinates (X,Y,Z) of the planar region 21 contained within the target subregion 22 from the intrinsic parameters of the camera 200 and the depth extracted for the target subregion 22. In other words, when the extracted depth is represented by D(M,N), the image coordinates (M,N) of the planar region 21 contained within the target subregion 22 are converted to world coordinates (X,Y,Z) using the intrinsic parameters of the camera 200 and the extracted depth D(M,N).

[0043] Then, the regression plane calculation process P34 calculates a regression plane representing a horizontal or vertical plane for the target sub-region 22 by performing regression analysis using the position of the planar region 21 in the acquired world coordinates (X,Y,Z) as the explanatory variable. In this case, the regression plane is expressed as the equation of a plane in world coordinates (X,Y,Z) with the camera 200's position as the origin.

[0044] The calibration value calculation process P35 calculates a calibration value for each sub-region 22 by comparing the position of the camera 200 relative to the calculated regression plane with the installation position of the camera 200 relative to the horizontal or vertical plane managed by the depth estimation device 10. In other words, in the depth estimation device 10 according to this embodiment, the calibration value is managed for each sub-region 22.

[0045] As described above, the regression plane is calculated for each sub-region 22 from the estimated depth. Furthermore, it is expected that the error occurring in each sub-region 22 will be of a similar magnitude. Therefore, the difference between the position of the camera 200 relative to the regression plane and the installation position of the camera 200 relative to the horizontal or vertical plane is considered to be proportional to the error occurring in each sub-region 22. Accordingly, by comparing the position of the camera 200 relative to the regression plane and the installation position of the camera 200 relative to the horizontal or vertical plane, an appropriate calibration value can be calculated for each sub-region 22.

[0046] For example, the calibration value calculation process P35 can calculate the calibration value as follows. Let's assume that the installation position of the camera 200 relative to the horizontal plane 1 is given by the height h of the camera 200 relative to the horizontal plane 1, as shown in Figure 4(C). Let's also assume that a regression plane 2 has been calculated for a certain sub-region 22 by the regression plane calculation process P34. Then, let's assume that the height of the camera 200 relative to the regression plane 2 is expressed as c from the calculated equation for the regression plane 2. In this case, the calibration value calculation process P35 can calculate the calibration value for this sub-region 22 using the scale factor h / c, which is the estimated depth.

[0047] Depending on the pattern of the multiple sub-regions 22 that are set, there may be areas in the image that are not included in any of the sub-regions 22. For example, in the pattern shown in Figure 4(A), each of the four corner regions is not included in any of the sub-regions 22. For such regions, a calibration value may be given by referring to the calibration value calculated for a neighboring sub-region 22. For example, the calibration value for the lower right corner region shown in Figure 4(A) may be the average of the calibration value calculated for sub-region 22C and the calibration value calculated for sub-region 22D. The degree of error due to distortion aberration is considered to be similar in neighboring sub-regions 22. Therefore, by performing this processing, an appropriate calibration value can be given even to regions that are not included in any of the sub-regions 22.

[0048] 3. Calibration process Next, the details of the calibration process P20 will be described. As described above, the depth estimation device 10 according to this embodiment manages calibration values ​​for each sub-region 22. Therefore, the calibration process P20 performs calibration of the depth estimated in each sub-region 22 based on the corresponding calibration value.

[0049] For example, the calibration process P20 performs the following steps. Suppose the depth estimated by the depth estimation process P10 is D(M,N). Also, assume that a scale factor α is managed as a calibration value for each sub-region 22. In this case, the calibration process P20 performs calibration by calculating α × D(M,N) using the corresponding scale factor α for each sub-region 22. In this case, the depth estimation device 10 outputs α × D(M,N) as the estimation result.

[0050] 4. Structure The configuration of the depth estimation device 10 according to this embodiment will be described below. Figure 5 is a diagram showing an example of the configuration of the depth estimation device 10 according to this embodiment.

[0051] The depth estimation device 10 comprises a processing unit 100, a camera 200, and a user interface 300.

[0052] The processing unit 100 is a computer that includes one or more processors 110 (hereinafter simply referred to as "processor 110") and one or more storage devices 120 (hereinafter simply referred to as "storage devices 120"). The processing unit 100 is configured to communicate with the camera 200 and the user interface 300.

[0053] The processor 110 performs various processes. The processor 110 can be composed of, for example, a CPU (Central Processing Unit) including an arithmetic unit and registers. The storage device 120 is connected to the processor 110 and stores various information necessary for the execution of the processor 110's processes. The storage device 120 can be composed of, for example, a recording medium such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive).

[0054] The storage device 120 stores a computer program 121, calibration values ​​122, and installation location information 123.

[0055] The computer program 121 consists of multiple instructions that cause the processor 110 to perform various processes. The computer program 121 may be stored on a computer-readable recording medium included in the storage device 120. The processor 110 performs various processes by operating according to the computer program 121. In particular, as described above, the processor 110 performs the process of calculating the calibration value (first process), the depth estimation process P10, and the calibration process P20 (second process).

[0056] A calibration value 122 is assigned to each sub-region 22. The processor 110 stores the calibration value 122 calculated for each sub-region 22 by executing the process into the storage device 120.

[0057] The installation position information 123 provides the installation position of the camera 200 relative to a horizontal or vertical plane. The installation position information 123 may be pre-stored in the storage device 120. Alternatively, the installation position information 123 may be set by the user via the user interface 300.

[0058] The user interface 300 is provided for the user to utilize the functions of the depth estimation device 10. The user interface 300 consists of, for example, input devices such as a keyboard and a touch panel, and output devices such as a display and a speaker. The user performs actions such as inputting setting information, checking estimation results, and acquiring estimation result data via the user interface 300.

[0059] As described above, the depth estimation device 10 according to this embodiment is configured. Furthermore, the calibration method according to this embodiment is realized when the processor 110 performs processing.

[0060] Figure 6 is a flowchart showing an example of the process that processor 110 performs regarding the calibration of the estimated depth.

[0061] In step S100, the processor 110 refers to the installation position information 123 to obtain the installation position of the camera 200 relative to a horizontal or vertical plane.

[0062] Next, in step S110, the processor 110 identifies a planar region 21 in the image that reflects a horizontal or vertical plane.

[0063] Next, in step S120, the processor 110 sets multiple sub-regions 22 in the image.

[0064] Next, in step S130, the processor 110 calculates a regression plane for each sub-region 22 based on the depth estimated in the planar region 21 contained within each sub-region 22.

[0065] Next, in step S140, the processor 110 calculates calibration values ​​for each sub-region 22 by comparing the position of the camera 200 with the installation position acquired in step S100.

[0066] Next, in step S150, the processor 110 performs calibration of the estimated depth in each sub-region 22 based on the corresponding calibration value.

[0067] Thus, the calibration method according to this embodiment is realized. Furthermore, the computer program 121 that causes the processor 110 to execute processing realizes the calibration program according to this embodiment.

[0068] 5. Effects As described above, according to this embodiment, for each sub-region 22 set in the image, a regression plane 2 representing a horizontal or vertical plane is calculated based on the depth estimated in the planar region 21 included in each sub-region 22. Then, a calibration value is calculated for each sub-region 22 by comparing the position of the camera 200 with the installation position of the camera 200 with respect to the horizontal or vertical plane. This makes it possible to calculate an appropriate calibration value for each sub-region 22. Then, calibration of the depth estimated based on the corresponding calibration value is performed in each sub-region 22. This makes it possible to achieve highly accurate depth estimation without requiring manual work or additional equipment. [Explanation of Symbols]

[0069] 10 Depth estimation device 21 Planar area 22 Partial area 100 Processing Unit 110 processors 120 Storage device 121 Computer Programs 122 Calibration Value 123 Installation location information 200 Cameras 300 User Interfaces

Claims

1. One or more processors that analyze images captured by a camera and estimate the depth in the images, A storage device that stores information about the installation position of the camera relative to a horizontal or vertical plane, Equipped with, The one or more processors further include: A first process to calculate a calibration value for the estimated depth, A second process involves performing calibration of the estimated depth based on the calibration value, It is configured to perform, The first process is, To identify the planar region in the aforementioned image in which the horizontal or vertical plane is reflected, Setting multiple sub-regions in the aforementioned image, For each of the aforementioned sub-regions, the region of the common area between each sub-region and the planar region is calculated, and the depth of each common region is extracted by referring to the estimated depth. Based on the depth of each of the extracted common regions, a regression plane representing the horizontal or vertical plane in each of the subregions is calculated, The calibration value is calculated for each sub-region by comparing the position of the camera with respect to the regression plane and the installation position. Includes, The second process performs calibration of the estimated depth in each of the sub-regions based on the corresponding calibration value. Depth estimation device.

2. A depth estimation device according to claim 1, Each of the aforementioned subregions is a region enclosed by edges having curvature, The curvature is greater in the subregions further from the center of the image. Depth estimation device.

3. A depth estimation device according to claim 1 or claim 2, The size of each subregion is smaller the further away it is from the center of the image. Depth estimation device.

4. A depth estimation device according to claim 1 or 2, Calculating the aforementioned regression plane is Based on the depth of each of the extracted common regions and the intrinsic parameters of the camera, the coordinates of each of the common regions on the image are converted to world coordinates. Perform a regression analysis using the regions of each common area in the aforementioned world coordinates as explanatory variables. This further includes Depth estimation device.

5. A calibration method that uses a computer to perform calibration of the depth estimated in an image captured by a camera, A process for acquiring information on the installation position of the camera relative to a horizontal or vertical plane, A process to identify the planar region in the aforementioned image in which the horizontal or vertical plane is reflected, The process of setting multiple sub-regions in the aforementioned image, For each of the aforementioned sub-regions, the region of the common area between each sub-region and the planar region is calculated, and the depth of each common region is extracted by referring to the estimated depth. A process to calculate a regression plane representing the horizontal or vertical plane in each of the extracted common regions, based on the depth of each common region, A process to calculate calibration values ​​for each sub-region by comparing the position of the camera with the installation position relative to the regression plane, In each of the aforementioned sub-regions, a process is performed to perform calibration of the estimated depth based on the corresponding calibration value. The computer executes the above. Calibration method.

6. A calibration program that causes a computer to perform depth calibration in an image captured by a camera, A process for acquiring information on the installation position of the camera relative to a horizontal or vertical plane, A process to identify the planar region in the aforementioned image in which the horizontal or vertical plane is reflected, The process of setting multiple sub-regions in the aforementioned image, For each of the aforementioned sub-regions, the region of the common area between each sub-region and the planar region is calculated, and the depth of the common region is extracted by referring to the estimated depth. A process to calculate a regression plane representing the horizontal or vertical plane in each of the extracted common regions, based on the depth of each common region, A process to calculate calibration values ​​for each sub-region by comparing the position of the camera with the installation position relative to the regression plane, In each of the aforementioned sub-regions, a process is performed to perform calibration of the estimated depth based on the corresponding calibration value. To cause the computer to execute Calibration program.

Images