Three-dimensional information correction device and three-dimensional information correction method

Abstract

To provide a three-dimensional information correction apparatus and a three-dimensional information correction method capable of faithfully reproducing three-dimensional information of an object.SOLUTION: A three-dimensional information correction apparatus comprises: an image acquisition unit that acquires an image obtained by imaging an object; a straight-line detection unit that detects a straight line included in the acquired image; a Depth image acquisition unit that acquires a Depth image that includes a plurality of Depth values which each represent information on distance to the object and which are on coordinates in a two-dimensional coordinate system; a correction function calculation unit that calculates a correction function which is a function for correcting a Depth value at a point on the detected straight line, out of the acquired Depth values, and which is a function indicating relationship between the Depth value at the point on the straight line and the coordinates of the image; and a correction processing unit that corrects, using the calculated correction function, the Depth value at the point on the detected straight line, out of the acquired Depth values.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional information correction apparatus and a three-dimensional information correction method. [Background Art]

[0002] Conventionally, there has been a method of calculating a distance to an object by irradiating distance measurement light toward the object, receiving reflected light reflected from the measurement object, and measuring the time from when the light is irradiated to when it is received. These distance measurement methods are widely known as ToF (Time of Flight) methods. As an example of an apparatus for acquiring three-dimensional information of an object, there has been an apparatus in which, in addition to RGB pixels, light-receiving elements (ToF sensors) for performing distance measurement by the ToF method are two-dimensionally arranged. By using such an apparatus, image information and distance measurement information of an object can be acquired at the same time, and three-dimensional information of the object can be generated by combining the acquired information (see, for example, Patent Document 1). [Prior Art Literature] [Patent Literature]

[0003] [Patent Document 1] Japanese Unexamined Patent Publication No. 2021-26236 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] However, due to the performance of ToF sensors, distortion may exist in the distance information obtained by the ToF method. For example, even when an object has a linear portion, if three-dimensional information of the linear portion is acquired, the three-dimensional information may become a wavy shape. That is, when three-dimensional information (specifically, point cloud data) is generated from visible light image information and distance information, there has been a problem that the shape of a subject that should originally be a straight line or a plane is distorted. Due to this distortion, there has been a problem that the three-dimensional information of the object cannot be reproduced faithfully.

[0005] This invention has been made in view of the above circumstances, and aims to provide a three-dimensional information correction device and a three-dimensional information correction method that can faithfully reproduce the three-dimensional information of an object. [Means for solving the problem]

[0006] [1] One aspect of the present invention includes: an image acquisition unit that acquires an image of an object; a line detection unit that detects a line included in the acquired image; a depth image acquisition unit that acquires a depth image which includes a plurality of depth values, which are distance information to the object, at each coordinate of a two-dimensional coordinate system; a correction function calculation unit that calculates a correction function which is a function for correcting the depth value at a point on the detected line from among the acquired depth values, and which is a function that shows the relationship between the depth value at a point on the line and the coordinate of the image; and a correction processing unit that corrects the depth value at a point on the detected line from among the acquired depth values ​​using the calculated correction function. Furthermore, a surface detection unit detects a surface included in the image based on the intersection points of the multiple lines that have been detected, Equipped with The line detection unit detects a plurality of lines included in the acquired image; the correction function calculation unit calculates a function as the correction function for correcting the Depth value at a detected point on the surface from among the acquired Depth values, which is a function that shows the relationship between the Depth value at a point on a line that includes pixels constituting the surface and is aligned in a specific direction, and the coordinates of the image; the correction processing unit corrects the Depth value using the correction function for each coordinate on the line aligned in the specific direction, starting from a specific point on the plurality of detected lines; the correction processing unit corrects the Depth value when the adjacent pixel values ​​on the detected surface are within a predetermined range, and does not correct the Depth value when they are outside the predetermined range. This is a three-dimensional information correction device.

[0007] [2] In addition, in one aspect of the present invention, in the three-dimensional information correction device described in [1] above, the correction function calculation unit determines two points for calculating the correction function based on the Depth values ​​at a plurality of points on the detected straight line from among the acquired Depth values, and uses a linear function passing through the two determined points as the correction function.

[0008] [3] Another aspect of the present invention is a three-dimensional information correction device described in [1] or [2] above, wherein the image is an image colored according to the degree of deflection acquired by the deflection sensor.

[0011] [ 4Furthermore, one aspect of the present invention includes: an image acquisition step of acquiring an image of an object; a line detection step of detecting a line included in the acquired image; a depth image acquisition step of acquiring a depth image which includes a plurality of depth values, which are distance information to the object, at each coordinate of a two-dimensional coordinate system; a correction function calculation step of calculating a correction function which is a function for correcting the depth value at a point on the detected line from among the acquired depth values, and which is a function that shows the relationship between the depth value at a point on the line and the coordinate of the image; and a correction processing step of correcting the depth value at a point on the detected line from among the acquired depth values ​​using the calculated correction function. The process includes a surface detection step that detects a surface included in the image based on the intersection points of the multiple lines that have been detected, to have The line detection step detects a plurality of lines included in the acquired image; the correction function calculation step calculates a function as the correction function for correcting the Depth values ​​at points on the detected surface from among the acquired Depth values, which is a function that shows the relationship between the Depth values ​​at points on a line that includes pixels constituting the surface and is aligned in a specific direction, and the coordinates of the image; the correction processing step corrects the Depth values ​​using the correction function for each coordinate on the line aligned in the specific direction, starting from a specific point on the plurality of detected lines; the correction processing step corrects the Depth values ​​based on the image if the adjacent pixel values ​​on the detected surface are within a predetermined range, and does not correct the Depth values ​​if they are outside the predetermined range. Three-dimensional information correction method. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a three-dimensional information correction device and a three-dimensional information correction method that can faithfully reproduce the three-dimensional information of an object. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram illustrating the outline of the three-dimensional information acquisition system according to Embodiment 1. [Figure 2] This is a schematic diagram showing an example of a cross-section of a three-dimensional information acquisition device according to Embodiment 1. [Figure 3] This is a functional configuration diagram showing an example of the functional configuration of the three-dimensional information correction device according to Embodiment 1. [Figure 4] This is a diagram illustrating an example of the linear detection process according to Embodiment 1. [Figure 5] This figure shows an example of a linear function for identifying a straight line detected by the straight line detection process according to Embodiment 1. [Figure 6] This figure shows an example of a depth value on a straight line detected by the straight line detection process according to Embodiment 1. [Figure 7]FIG. 1 is a diagram illustrating an example of a correction function according to Embodiment 1. [Figure 8] FIG. 2 is a flowchart illustrating a series of processes up to calculation of a correction function in the three-dimensional information correction method according to Embodiment 1. [Figure 9] FIG. 3 is a flowchart illustrating a series of correction processes in the three-dimensional information correction method according to Embodiment 1. [Figure 10] FIG. 4 is a diagram illustrating an example of a deflected image acquired by the three-dimensional information correction apparatus according to a modification of Embodiment 1. [Figure 11] FIG. 5 is a functional block diagram illustrating an example of a functional configuration of the three-dimensional information correction apparatus according to Embodiment 2. [Figure 12] FIG. 6 is a diagram for explaining an example of surface detection processing according to Embodiment 2. [Figure 13] FIG. 7 is a diagram for explaining processing for avoiding unevenness existing on a surface in the three-dimensional information correction according to Embodiment 2. [Figure 14] FIG. 8 is a flowchart illustrating an example of processing for avoiding unevenness existing on a surface in the three-dimensional information correction according to Embodiment 2. [Figure 15] FIG. 9 is a diagram for explaining an example of horizontal correction in the three-dimensional information correction according to Embodiment 2. [Figure 16] FIG. 10 is a diagram for explaining an example of vertical correction in the three-dimensional information correction according to Embodiment 2. DETAILED DESCRIPTION OF EMBODIMENTS

[0014] A distance information acquiring apparatus according to an aspect of the present invention will be described in detail below with reference to the accompanying drawings by presenting preferred embodiments. It should be noted that the embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the following embodiments. In addition, the expression "based on XX" as used in the present application means "based at least on XX", and includes cases where it is based on another element in addition to XX. Furthermore, "based on XX" is not limited to the case where XX is directly used, but also includes cases where it is based on a result obtained by performing an operation or processing on XX. "XX" is any element (e.g., any information). In addition, in the following drawings, in order to make each configuration easy to understand, the scale, number, and the like of each structure may be different from those in the actual structure.

[0015] [PRIOR ART] First, the problem to be solved in the present embodiment will be described. According to a conventional ToF camera, depth information measured by a ToF method is collected in a bitmap format for each pixel arranged in a two-dimensional array, and a distance image is generated. In addition, an image such as an RGB image is captured by an imaging device. Three-dimensional point cloud data can be generated by combining depth information of the distance image with an image such as an RGB image. However, distance images often contain a large amount of noise. Various problems may occur due to the noise contained in the distance image. For example, an object from which three-dimensional information is acquired may include linear portions and planar portions. If noise occurs in linear portions or planar portions, an originally flat portion becomes a shape having irregularities, which may result in unnatural three-dimensional information. The present embodiment aims to eliminate such unnatural distortion and faithfully reproduce the three-dimensional information of an object.

[0016] [Embodiment 1] Embodiment 1 will be described below with reference to FIGS. 1 to 10.

[0017] Figure 1 is a diagram illustrating the outline of a three-dimensional information acquisition system according to Embodiment 1. The outline of the three-dimensional information acquisition system 1 will be explained with reference to this figure. The three-dimensional information acquisition system 1 acquires three-dimensional information of an object T existing in three-dimensional space. The object T may be one or multiple objects. The three-dimensional information acquired by the three-dimensional information acquisition system 1 includes at least information about the three-dimensional shape of the object T.

[0018] The three-dimensional information acquisition system 1 measures the distance L1 from the three-dimensional information acquisition device 10 to the object T. The three-dimensional information acquisition device 10 acquires the three-dimensional shape of the object T by measuring the distance L1 to the object T at each coordinate in the two-dimensional coordinate system. The object T targeted by the three-dimensional information acquisition device 10 includes anything that can be used to acquire three-dimensional information, such as animals and objects. In the following explanation, the case where the object T is a person will be described as an example. Behind the object T, there may be a background BG. The background BG may be a screen such as a green screen used for filming, or it may be a simple wall, floor, building, natural object, or other background that is captured during normal imaging. In addition, there may be objects O that have a three-dimensional shape in the vicinity of the object T, although they are not the target of three-dimensional information acquisition. There may be multiple objects O. The three-dimensional information acquisition system 1 may be used, for example, by a professional photographer in an indoor photography studio, or by a general user in the same circumstances as normal photography indoors or outdoors.

[0019] The three-dimensional information acquisition system 1 comprises a three-dimensional information acquisition device 10 and a three-dimensional information correction device 20. The three-dimensional information acquisition device 10 and the three-dimensional information correction device 20 may exist as a single edge device, or they may be connected to each other via a predetermined communication network. Alternatively, the three-dimensional information correction device 20 may exist as a server device (not shown) connected to the three-dimensional information acquisition device 10 via a predetermined communication network. If the three-dimensional information correction device 20 exists as a server device, multiple three-dimensional information acquisition devices 10 and one three-dimensional information correction device 20 may be connected via a predetermined communication network.

[0020] The three-dimensional information acquisition device 10 comprises a light receiving unit 110 and an illumination unit 120. The light receiving unit 110 may include, for example, an optical lens such as an objective lens. The illumination unit 120 irradiates the target object T with illumination light. The illumination unit 120 may be a light source such as a laser diode. More specifically, the illumination unit 120 may be a VCSEL (Vertical Cavity Surface Emitting Laser) capable of emitting a laser beam in the vertical direction. The light irradiated by the illumination unit 120 (first illumination light BM1) is reflected by the target object T, and the light reflected by the target object T (second illumination light BM2) is incident on the light receiving unit 110. The three-dimensional information acquisition device 10 measures the distance L1 from the three-dimensional information acquisition device 10 to the target object T according to the time (time of flight) from the time of illumination to the time of reception.

[0021] The light-receiving unit 110 comprises a plurality of light-receiving elements arranged in a two-dimensional array. The three-dimensional information acquisition device 10 measures the three-dimensional shape of the object T according to the distance measurement information from each of the plurality of light-receiving elements. In the illustrated example, a pair of light-receiving units 110 and illumination units 120 are shown, but a configuration in which a plurality of illumination units 120 are provided for a single light-receiving unit 110 is also possible. In this case, it is preferable that the plurality of illumination units 120 be provided in the vicinity of the light-receiving unit 110.

[0022] The three-dimensional information acquisition device 10 includes an image sensor (not shown). The image sensor has multiple pixels arranged in a two-dimensional array. These multiple pixels receive visible light formed by the optical lens of the light receiving unit 110, and generate a visible light image (RGB image) based on the received information. The visible light image format is not limited to the RGB format; the YCbCr format or grayscale (monochrome) format may also be used.

[0023] The three-dimensional information correction device 20 extracts the three-dimensional information of a specific object from the three-dimensional information acquired by the three-dimensional information acquisition device 10. As shown in the figure, the three-dimensional information acquired by the three-dimensional information acquisition device 10 includes not only the target object T that is the target of three-dimensional information generation, but also the three-dimensional information of multiple objects O and the background BG. Therefore, the three-dimensional information correction device 20 extracts the three-dimensional information of the target object T that is the target of three-dimensional information generation from the three-dimensional information acquired by the three-dimensional information acquisition device 10. The target object T that the three-dimensional information correction device 20 targets for three-dimensional information generation may be automatically determined by the three-dimensional information correction device 20, or it may be determined by selection by a user or the like.

[0024] Figure 2 is a schematic diagram showing an example of a cross-section of a three-dimensional information acquisition device according to Embodiment 1. An example of the arrangement of the image sensor and ToF sensor in the three-dimensional information acquisition device 10 will be described with reference to this figure. The three-dimensional information acquisition device 10 further comprises a visible light reflective dichroic film 112, an image sensor 113, and a ToF sensor 114.

[0025] Light emitted from the irradiation unit 120 and reflected from the object T enters the light receiving unit 110. In the figure, the optical axis of the incident light is indicated as optical axis OA. The light that enters the light receiving unit 110 enters the visible light reflective dichroic film 112. Upstream of the visible light reflective dichroic film 112, optical lenses or the like (not shown) may be provided.

[0026] The visible light reflective dichroic film 112 is provided on the optical path between the light receiving unit 110 and the ToF sensor 114. The visible light reflective dichroic film 112 transmits some of the incident light (specifically near-infrared light) and reflects the other light (specifically visible light). The visible light reflective dichroic film 112 guides the light to the ToF sensor 114 by transmitting some of the light reflected from the object T by the light irradiated by the irradiation unit 120. The light transmitted by the visible light reflective dichroic film 112 is referred to as infrared light IL. The light reflected by the visible light reflective dichroic film 112 is referred to as visible light VL. Infrared light IL and visible light VL pass through substantially the same optical axis upstream of the visible light reflective dichroic film 112. Substantially the same range may be, for example, a range in which the optical path is formed by a common lens.

[0027] The light, which is spectrally separated into two optical paths by the visible light reflective dichroic film 112, is received by sensors positioned in each optical path. Specifically, infrared light IL transmitted through the visible light reflective dichroic film 112 is received by the ToF sensor 114. In addition, the light reflected by the visible light reflective dichroic film 112 is received by the image sensor 113.

[0028] The image sensor 113 comprises a plurality of pixels arranged in a two-dimensional array. The image sensor 113 may also comprise pixels of each RGB color arranged in a Bayer array. Each of these plurality of pixels receives visible light VL and acquires the information necessary to generate a visible light image.

[0029] The ToF sensor 114 comprises multiple pixels arranged in a two-dimensional array. Each of these pixels receives infrared light (IL) and acquires the information necessary for distance conversion.

[0030] Figure 3 is a functional configuration diagram showing an example of the functional configuration of a three-dimensional information correction device according to Embodiment 1. An example of the functional configuration of the three-dimensional information correction device 20 will be described with reference to this figure. The three-dimensional information correction device 20 is composed of an image acquisition unit 21, a depth image acquisition unit 22, a line detection unit 23, a correction function calculation unit 24, a correction processing unit 25, and an output unit 26. Each of these functional units is implemented, for example, using an electronic circuit. In addition, each functional unit may be equipped with internal storage means such as semiconductor memory or a magnetic hard disk drive, as needed. Furthermore, each function may be implemented by a computer and software.

[0031] The image acquisition unit 21 acquires image information II from the image sensor 113, which includes information about the image (visible light image such as an RGB image) of the object T. Image information II includes brightness information for each coordinate in the two-dimensional coordinate system. This brightness information may correspond to each RGB color. The image acquisition unit 21 outputs the acquired image information II to the line detection unit 23.

[0032] The Depth image acquisition unit 22 acquires Depth image information DI, which contains information about the Depth image. The Depth image contains multiple Depth values ​​at each coordinate in the two-dimensional coordinate system. A Depth value is distance information to an object T measured by the ToF method. The Depth image acquisition unit 22 outputs the acquired Depth image information DI to the correction function calculation unit 24.

[0033] Here, it is preferable that the image acquired by the image acquisition unit 21 and the depth image are images of the same object T at the same field of view. In other words, it is preferable that the coordinates of the image acquired by the image acquisition unit 21 and the coordinates of the depth image correspond to each other. In order to make the coordinates of each other correspond, in this embodiment, a configuration is used in which a visible light reflective dichroic film 112 is used to spectrally separate the light into infrared light IL and visible light VL, as described with reference to Figure 2.

[0034] The straight line detection unit 23 detects straight line portions included in the image acquired by the image acquisition unit 21. The straight line detection unit 23 may, for example, use the functions of the OpenCV library to detect straight line portions included in the image acquired by the image acquisition unit 21. The straight line detection unit 23 outputs information identifying the detected straight line portion as straight line information LI to the correction function calculation unit 24. Note that the straight line information LI may include coordinate information of the start and end points of the detected straight line. Furthermore, if the image acquired by the image acquisition unit 21 contains multiple straight line portions, the straight line detection unit 23 may detect these multiple straight line portions. In that case, the straight line information LI may include coordinate information of the start and end points for each of the detected multiple straight line portions. Note that in the following description, the processing performed by the straight line detection unit 23 may be referred to as the straight line detection processing.

[0035] Figure 4 is a diagram illustrating an example of the straight line detection process according to Embodiment 1. An example of the straight line detection process performed by the straight line detection unit 23 will be explained with reference to this figure. Figure 4(A) is an example of an image captured by the image sensor 113. The straight line detection unit 23 detects the straight line portions included in the image shown in Figure 4(A). Figure 4(B) shows an example of the result of the straight line detection process performed by the straight line detection unit 23. Reference numerals are used in this figure for explanatory purposes. As shown, 10 straight line portions, represented by straight lines L1 to L10, are detected by the straight line detection process. The straight line detection unit 23 outputs information identifying the detected straight lines as straight line information LI to the correction function calculation unit 24. As an example, the starting point SP and ending point EP for identifying straight line L1 are shown. The straight line information LI includes, for example, the coordinate information of the starting point SP and ending point EP for identifying straight line L1. In the example shown, 10 straight line portions are detected, but more than 10 or fewer straight line portions may be detected from the figure. The conditions for detection as a straight line may be determined appropriately depending on the image, or they may be set by the user.

[0036] Returning to Figure 3, the correction function calculation unit 24 acquires line information LI from the line detection unit 23 and depth image information DI from the depth image acquisition unit 22. Based on the acquired line information LI and depth image information DI, the correction function calculation unit 24 calculates the correction function CF. The correction function CF is a function for correcting the depth value at a point on a line detected by the line detection unit 23, among the multiple two-dimensionally arranged depth values ​​included in the acquired depth image information DI. The correction function CF is also a function that shows the relationship between the depth value at a point on a line detected by the line detection unit 23 and the coordinates of the image acquired by the image acquisition unit 21. In the following explanation, we will describe the case where the correction function CF is a linear function.

[0037] First, the correction function calculation unit 24 calculates a linear function to identify the line based on the coordinates of the start and end points included in the line information LI obtained from the line detection unit 23. Figure 5 is a diagram showing an example of a linear function for identifying a line detected by the line detection process according to Embodiment 1. In this figure, the horizontal direction of the image acquired by the image acquisition unit 21 is shown as the x-axis, and the vertical direction of the image acquired by the image acquisition unit 21 is shown as the y-axis. The figure also shows an example of a linear function for identifying the line L1 shown in Figure 4(B). Since the line information LI includes coordinate information of the start point SP and the end point EP, the correction function calculation unit 24 calculates a linear function connecting the points indicated in the coordinate information.

[0038] Next, the correction function calculation unit 24 identifies the depth value on the straight line of the calculated linear function based on the depth image information DI. Figure 6 is a diagram showing an example of the depth value on the straight line detected by the straight line detection process according to Embodiment 1. In this figure, the horizontal direction of the image acquired by the image acquisition unit 21 is used as the x-axis, and the detected depth value on the straight line is used as the y-axis. The depth value of the point corresponding to the starting point SP shown in Figure 4(B) is shown as starting point SP1, and the depth value of the point corresponding to the ending point EP shown in Figure 4(B) is shown as ending point EP1. The depth value between the starting point SP1 and the ending point EP1 should ideally be a straight line, as is clear from Figure 4(B). However, the depth value acquired by the depth image acquisition unit 22 is a wavy line, as shown in the figure. This is because noise is superimposed on the depth value detected by the ToF sensor 114, and the depth value is not accurately detected.

[0039] Therefore, the correction function calculation unit 24 calculates a correction function CF to correct such a wavy line into a straight line. First, the correction function calculation unit 24 identifies the start and end points of the correction function CF. The start and end points of the correction function CF are common in that the x-coordinate corresponds to the x-axis direction of the image, but the y-coordinate is different from the start and end points shown with reference to Figure 5, where the y-coordinate is the Depth value. Hereafter, the start and end points of the correction function CF will be referred to as start point SP2 and end point EP2.

[0040] The correction function calculation unit 24 may, for example, sample Depth values ​​for several points adjacent to the start and end points and determine the Depth values ​​of the start and end points. In other words, the correction function calculation unit 24 may determine two points for calculating the correction function CF based on the Depth values ​​at multiple points on a straight line detected by the straight line detection unit 23 from among the multiple Depth values ​​included in the acquired Depth image information DI. The number of Depth values ​​to sample and the sampling interval may be determined based on an interval based on the pixel values ​​of the image in the x-axis direction or the pixel values ​​of the Depth image. The correction function calculation unit 24 calculates a linear function passing through the two points determined based on the multiple Depth values ​​as the correction function CF. The correction function CF is a function that has the x-coordinate and the Depth value as variables.

[0041] Figure 7 shows an example of a correction function according to Embodiment 1. The figure shows an example of a correction function CF calculated from the Depth value shown in Figure 6, superimposed on the Depth value shown in Figure 6. As shown in the figure, both ends of the line segment of the correction function CF pass through the starting point SP2 and the ending point EP2. The starting point SP2 may be determined, for example, based on the average value of the Depth values ​​of three points adjacent to the starting point SP1 shown in Figure 6, and the ending point EP2 may be determined, for example, based on the average value of the Depth values ​​of three points adjacent to the ending point EP1 shown in Figure 6. The correction function calculation unit 24 calculates a linear function connecting the starting point SP2 and the ending point EP2 as the correction function CF. Returning to Figure 3, the correction function calculation unit 24 outputs information about the correction function CF calculated in this way to the correction processing unit 25.

[0042] The correction processing unit 25 acquires information about the correction function CF calculated by the correction function calculation unit 24. The correction processing unit 25 uses the acquired correction function CF to correct the depth values ​​at points on the line detected by the line detection unit 23, among the depth values ​​acquired by the depth image acquisition unit 22. Here, the correction function CF is a linear function with x-coordinate as a variable. Therefore, for coordinates on the line L1 shown in Figure 4(B), the correction processing unit 25 can obtain the corrected depth value by substituting the x-coordinate value into the correction function CF. The correction processing unit 25 performs the correction process by replacing the depth value on the line L1 with the corrected depth value. Returning to Figure 3, the correction processing unit 25 outputs the three-dimensional information obtained, including the result of the correction process, as three-dimensional information 3DI to the output unit 26.

[0043] The output unit 26 acquires three-dimensional information 3DI from the correction processing unit 25. Based on the acquired three-dimensional information 3DI, the output unit 26 outputs the three-dimensional information of the object. The three-dimensional information of the object output by the output unit 26 may be information that associates image information acquired by the image acquisition unit 21 with a depth image that includes the corrected depth value, and more specifically, it may be three-dimensional point cloud data of the extracted object. Note that the information output by the output unit 26 is not limited to this example and may include the corrected depth value.

[0044] Next, with reference to Figures 8 and 9, a series of steps in the three-dimensional information correction method according to this embodiment will be described.

[0045] Figure 8 is a flowchart showing the sequence of steps up to the calculation of the correction function in the three-dimensional information correction method according to Embodiment 1. First, the sequence of steps up to the calculation of the correction function will be explained with reference to Figure 8.

[0046] (Step S11) First, the image acquisition unit 21 acquires a visible light image from the image sensor 113. This step may be described as the image acquisition step or image acquisition process. Also, the ToF sensor 114 acquires a depth image (ToF image) from the ToF sensor 114. This step may be described as the depth image acquisition step or depth image acquisition process.

[0047] (Step S12) Next, the line detection unit 23 detects one or more lines included in the image acquired by the image acquisition unit 21. This step may be described as the line detection step or line detection process.

[0048] (Step S13) Next, in steps S13 to S17, the correction function calculation unit 24 calculates a correction function. These steps may be described as the correction function calculation step or the correction function calculation process. First, in step S13, the correction function calculation unit 24 calculates a linear function equation that connects the coordinates of the start and end points of the straight line detected by the straight line detection unit 23. This linear function has the x-coordinate and y-coordinate of the image acquired by the image acquisition unit 21 as variables.

[0049] (Step S14) Next, the correction function calculation unit 24 samples depth values ​​for several points including the starting point of the straight line detected by the straight line detection unit 23, and determines the depth value that will be the starting point of the correction function CF. The several points may be, for example, about 3 points. In addition, a predetermined statistical calculation may be used to determine the depth value, for example, it may be determined by the average value.

[0050] (Step S15) Furthermore, the correction function calculation unit 24 samples depth values ​​for several points including the endpoint of the straight line detected by the straight line detection unit 23 (straight line termination point), and determines the depth value that will be the endpoint of the correction function CF. The several points may be, for example, about 3 points. In addition, a predetermined statistical calculation may be used to determine the depth value, for example, it may be determined by the average value.

[0051] (Step S16) Next, the correction function calculation unit 24 calculates a function that connects the starting point of the correction function CF determined in step S14 and the ending point of the correction function CF determined in step S15. The calculated function is, for example, a linear function and is the correction function CF. The correction function CF has the x-coordinate of the image acquired by the image acquisition unit 21 and the depth value as variables.

[0052] (Step S17) The correction function CF for one straight line can be calculated by steps S13 to S16 described above. If multiple straight lines are detected in step S12, the correction function calculation unit 24 calculates the correction function CF for all of the detected multiple straight lines. If the correction function calculation unit 24 has completed the calculation for all straight lines (i.e., step S17; YES), it proceeds to the correction process described later, referring to Figure 9. If the correction function calculation unit 24 has not completed the calculation for all straight lines (i.e., step S17; NO), it proceeds to step S13 and calculates the correction function CF for the next straight line. Note that it is not necessary to correct all straight lines detected by the straight line detection unit 23, and the correction process described later may be performed based on one or more straight lines identified by a predetermined method from among the straight lines detected by the straight line detection unit 23.

[0053] Figure 9 is a flowchart showing a series of steps in the correction process of the three-dimensional information correction method according to Embodiment 1. Next, the series of steps in the correction process performed by the correction processing unit 25 will be described with reference to Figure 9. The series of steps described with reference to Figure 9 may be referred to as correction processing steps or correction processing steps.

[0054] (Step S31) First, the correction processing unit 25 determines whether the x-coordinates of the coordinates to be corrected satisfy the condition x1 ≤ x ≤ x2. x1 and x2 are the x-coordinates of the endpoints of the line segment of the correction function CF, respectively. If the condition x1 ≤ x ≤ x2 is met (i.e., Step S31; YES), the correction processing unit 25 proceeds to Step S32. If the condition x1 ≤ x ≤ x2 is not met (i.e., Step S31; NO), the correction processing unit 25 proceeds to Step S33.

[0055] (Step S32) Next, the correction processing unit 25 rewrites the Depth value at the target coordinate. Specifically, the correction processing unit 25 obtains the Depth value by substituting the x-coordinate of the target coordinate into the correction function CF, and replaces the Depth value at the target coordinate with the obtained Depth value.

[0056] (Step S33) In this case, the correction processing unit 25 does not rewrite the Depth value at the target coordinate and proceeds to step S34.

[0057] (Step S34) Next, the correction processing unit 25 determines whether or not the correction has been completed for all Depth values ​​on the straight line detected by the straight line detection unit 23. If the correction has been completed for all Depth values ​​(i.e., Step S34; YES), the correction processing unit 25 terminates the process. If the correction has not been completed for all Depth values ​​(i.e., Step S34; NO), the correction processing unit 25 proceeds to Step S31 and continues the process.

[0058] If multiple straight lines are detected by the straight line detection unit 23, and correction processing is performed based on each of the multiple straight lines, steps S31 to S34 may be repeated according to the number of straight lines.

[0059] Next, a modified example of Embodiment 1 will be described. The modified example of Embodiment 1 differs from the above-described embodiment in that a deflection image is used in addition to, or instead of, the image acquired by the image acquisition unit 21. The three-dimensional information correction device 20 according to the modified example of Embodiment 1 acquires a deflection image in addition to, or instead of, the image acquired by the image acquisition unit 21. The deflection image acquired by the three-dimensional information correction device 20 can also be described as an image colored according to the degree of deflection acquired by the deflection sensor. In other words, the image acquired by the three-dimensional information correction device 20 according to the modified example of Embodiment 1 can also be described as an image colored according to the degree of deflection acquired by the deflection sensor.

[0060] Figure 10 shows an example of a deflected image acquired by a three-dimensional information correction device according to a modified example of Embodiment 1. As shown in the figure, it is easy to detect straight lines contained in an image using a deflected image. The straight line detection unit 23 can more accurately detect straight line portions contained in an image by detecting straight lines based on such a deflected image.

[0061] [Summary of Embodiment 1] According to the embodiment described above, the three-dimensional information correction device 20 includes an image acquisition unit 21 to acquire an image of an object, a line detection unit 23 to detect straight lines included in the acquired image, a depth image acquisition unit 22 to acquire a depth image which is distance information to the object and includes multiple depth values ​​at each coordinate in the two-dimensional coordinate system, a correction function CF which is a function for correcting the depth values ​​at points on the line detected by the line detection unit 23 among the acquired depth values ​​and shows the relationship between the depth value at the point on the line and the coordinate of the image, a correction function calculation unit 24 to calculate a correction function CF which is a function that shows the relationship between the depth value at the point on the line and the coordinate of the image, and a correction processing unit 25 to correct the depth values ​​at points on the line detected by the line detection unit 23 among the acquired depth values ​​using the calculated correction function CF. In other words, according to this embodiment, the straight lines included in the image are detected and the depth values ​​on the detected lines are corrected. Therefore, according to this embodiment, the depth values ​​of parts that are originally straight lines have noise removed and become straight lines, so natural three-dimensional information can be generated. Therefore, according to this embodiment, the three-dimensional information of the object can be faithfully reproduced.

[0062] Furthermore, according to the embodiment described above, the correction function calculation unit 24 determines two points for calculating the correction function CF based on the Depth values ​​at multiple points (for example, three points) on a straight line detected by the straight line detection unit 23 from among the Depth values ​​acquired by the Depth image acquisition unit 22, and the correction function CF is a linear function passing through the two determined points. In other words, the correction function calculation unit 24 uses multiple Depth values ​​to determine the start and end points of the correction function CF. However, if the correction function CF is calculated based on the Depth value of one point at each end, the correction function CF may deviate significantly from the actual Depth value if noise is present at the points at both ends. Therefore, according to this embodiment, by determining the start and end points of the correction function CF based on multiple points, the correction function CF can be prevented from deviating significantly from the actual Depth value.

[0063] Furthermore, according to the embodiment described above, the image acquired by the image acquisition unit 21 is a visible light image. Therefore, according to this embodiment, three-dimensional information of the object (for example, three-dimensional point cloud data) can be generated based on the visible light image and the depth value. In addition, in this case, since an image for detecting straight lines (for example, a deflection image, etc.) is not required, a configuration for acquiring an image for detecting straight lines (for example, a deflection camera, etc.) is not used, and the device can be miniaturized.

[0064] Furthermore, according to the embodiment described above, the image acquired by the image acquisition unit 21 includes an image colored according to the degree of deflection acquired by the deflection sensor (i.e., a deflected image). The linear detection unit 23 also detects linear portions based on the deflected image. Linear portions can be detected more easily using a deflected image compared to a visible light image. Therefore, according to this embodiment, linear portions can be detected easily and accurately.

[0065] [Embodiment 2] Next, Embodiment 2 will be described with reference to Figures 11 to 16. Embodiment 1 described an example of correcting the depth value of linear portions included in an image. By eliminating noise in linear portions included in an image and correcting the linear portions to natural depth values, natural three-dimensional information can be generated, and the three-dimensional information of the object can be faithfully reproduced. However, even when the linear portions have natural depth values, noise is superimposed on the shape of the surface portions enclosed by the linear portions, and when observing the overall three-dimensional shape of the object, it may appear as an unnatural shape. Therefore, Embodiment 2 differs from the first embodiment in that it corrects the depth values ​​of the surface portions included in the image.

[0066] Figure 11 is a functional configuration diagram showing an example of the functional configuration of the three-dimensional information correction device according to Embodiment 2. The three-dimensional information correction device 20A according to Embodiment 2 will be described with reference to this figure. The three-dimensional information correction device 20A differs from the three-dimensional information correction device 20 in that it includes a surface detection unit 27. In the description of the three-dimensional information correction device 20A, components common to both the three-dimensional information correction device 20 and the three-dimensional information correction device 20 may be denoted by the same reference numerals, and their explanation may be omitted.

[0067] The surface detection unit 27 acquires line information LI from the line detection unit 23. The line detection unit 23 according to Embodiment 2 detects multiple line portions included in the image acquired by the image acquisition unit 21, and the line information LI includes information about the detected multiple line portions. First, the surface detection unit 27 detects the intersection points of the multiple line portions included in the acquired line information LI. Next, the surface detection unit 27 detects a surface included in the image based on the coordinates of the detected intersection points. For example, the surface detection unit 27 detects the existence of a quadrilateral surface if four lines intersect to form a closed surface. Note that not all four sides constituting the quadrilateral need to intersect; at least two lines must intersect. The surface detection unit 27 may also detect a surface if the extensions of lines intersect. The surface detection unit 27 outputs information about the detected surface as surface information SI to the correction function calculation unit 24. Information about the detected surface may be, for example, information about multiple lines constituting the surface. The surface detection unit 27 may detect multiple surfaces and output information about multiple surfaces as surface information SI. Figure 10 shows an example of a deflected image acquired by a three-dimensional information correction device according to a modified example of Embodiment 1. As shown in the figure, it is easy to detect surfaces included in an image using a deflected image. By detecting surfaces based on such a deflected image, the surface detection unit 27 can more accurately detect the surface portion included in the image.

[0068] Figure 12 is a diagram illustrating an example of surface detection processing according to Embodiment 2. An example of surface detection processing performed by the surface detection unit 27 will be explained with reference to the figure. As shown in the figure, the line L1 and the line L2 intersect at intersection point P. Therefore, the surface detection unit 27 detects surface SF, which is the surface enclosed by the line L1 and the line L2, as a surface. The surface detection unit 27 may further use the intersection point of the line L1 and the line L4, or the intersection point of the line L3 and the line L4, etc., to detect surface SF. The surface detection unit 27 may also further use the intersection point of the extension of the line L2 and the line L3 to detect surface SF. In the illustrated example, the surface detection unit 27 detects one surface from the image, but it is not limited to this example, and the surface detection unit 27 may detect multiple surfaces from an image as shown in Figure 12.

[0069] Returning to Figure 11, the correction function calculation unit 24 acquires line information LI from the line detection unit 23 and surface information SI from the surface detection unit 27. The correction function calculation unit 24 according to Embodiment 2 calculates multiple correction functions CF that extend in a specific direction, starting from one line that constitutes the surface. The specific direction may be, for example, the direction of a line that intersects with the said line. Here, the multiple correction functions CF calculated by the correction function calculation unit 24 are used to cover all pixels included in the surface. For example, in the example shown in Figure 12, the correction function CF is defined as a line that extends in a direction parallel to line L2, which is a line that intersects with line L1, starting from a point on line L1. The correction function calculation unit 24 shifts the point on line L1 to an adjacent point and detects multiple correction functions CF. In other words, the correction function CF according to Embodiment 2 is a function for correcting the Depth value at a detected point on the surface, among the Depth values ​​acquired by the Depth image acquisition unit 22. Furthermore, the correction function CF according to Embodiment 2 is a function that shows the relationship between the Depth value at a point on a straight line containing the pixels constituting the surface and aligned in a specific direction, and the coordinates of the image. The correction function calculation unit 24 outputs information about the calculated multiple correction functions CF to the correction processing unit 25.

[0070] The correction processing unit 25 obtains information on multiple correction functions CF from the correction function calculation unit 24. The correction processing unit 25 uses the correction functions CF obtained from the correction function calculation unit 24 to correct the depth value for each coordinate on a line along a specific direction, starting from a point on a specific line among the multiple lines detected by the line detection unit 23. The correction processing unit 25 corrects the entire surface by correcting the depth value based on each of the multiple correction functions. The correction processing unit 25 outputs the three-dimensional information obtained, including the results of the correction process, as three-dimensional information 3DI to the output unit 26.

[0071] Here, the surfaces included in the image are not necessarily flat; for example, there may be irregularities on the surface. If irregularities are present on the surface and the correction process ignores these irregularities, those irregularities may be lost, making it impossible to generate accurate three-dimensional information. Therefore, in this embodiment, the loss of irregularities is prevented by performing the correction process while determining the presence or absence of irregularities based on the image's brightness information, etc.

[0072] Figure 13 is a diagram illustrating the process for avoiding irregularities on a surface in the three-dimensional information correction according to Embodiment 2. The process for avoiding irregularities on a surface will be explained with reference to this figure. The direction of correction is indicated by arrows in the figure. Each arrow can be said to represent an example of the correction function CF. As shown in the figure, there is an L-shaped protrusion in the center of the surface. If the correction process includes this protrusion, the protrusion will be lost, and accurate three-dimensional information cannot be generated. Therefore, in the three-dimensional information correction method according to Embodiment 2, in the correction process along the correction function CF, the brightness information of adjacent pixels is compared, and if the difference in brightness information is greater than or equal to a predetermined value, it is determined that an irregularity exists, and the process based on the correction function CF is stopped. After the process is stopped, the correction process is performed again based on the correction function CF starting from the next point on the straight line detected by the straight line detection unit 23.

[0073] It should be noted that by performing processing to avoid uneven areas in this way, it is expected that areas that were not corrected will remain as shadows. However, by calculating a new correction function CF starting from a different straight line among the straight lines that make up the surface and performing the correction process again, it is possible to perform correction processing from a different direction to the location where the protrusion exists. By performing correction processing from a different direction to the location where the protrusion exists, even if a protrusion exists as shown in Figure 13, it is possible to perform correction processing on all surface areas surrounding the protrusion. In the illustrated example, the surface is composed of four straight lines, so by calculating multiple correction functions CF starting from each of the four straight lines and performing correction processing starting from each of the four straight lines, it is possible to perform correction processing on all areas surrounding the protrusion. Alternatively, instead of performing correction processing starting from each of the four straight lines, the processing time can be reduced by performing correction processing starting from two opposing straight lines.

[0074] Figure 14 is a flowchart showing an example of a process for avoiding surface irregularities in the three-dimensional information correction according to Embodiment 2. An example of a process for avoiding surface irregularities will be explained with reference to this figure.

[0075] (Step S51) First, the preprocessing unit (not shown) flattens the entire depth image acquired by the depth image acquisition unit 22 as a preprocessing step. This flattening process may be a process that removes depth values ​​considered to be noise based on the average value of the depth values, etc.

[0076] (Step S52) Next, the correction processing unit 25 compares the brightness levels of adjacent pixels. Adjacent pixels may be a pixel at the coordinate to be corrected and a pixel at the coordinate adjacent to that pixel. The direction of adjacency may be the linear direction of the correction function CF.

[0077] (Step S53) Next, the correction processing unit 25 determines whether the difference is within a predetermined range based on the comparison of brightness levels. The predetermined range may be determined based on the results of statistical calculation of the brightness values ​​of the pixels included in the surface. If the difference is within the predetermined range (i.e., Step S53; YES), the correction processing unit 25 proceeds to step S54. If the difference is not within the predetermined range (i.e., Step S53; NO), the correction processing unit 25 proceeds to step S55.

[0078] (Step S54) When the difference is within a predetermined range, it means that there are no protrusions or other uneven parts. Therefore, the correction processing unit 25 corrects the depth value by substituting the x coordinate of the target coordinate into the correction function CF. That is, the correction processing unit 25 corrects the depth value based on the image acquired by the image acquisition unit 21 when the adjacent pixel values ​​on the detected surface are within a predetermined range.

[0079] (Step S55) If the difference is not within a predetermined range, it means that there are protrusions or other uneven parts, so the correction processing unit 25 does not perform correction processing on the coordinates targeted by the correction function CF. That is, based on the image acquired by the image acquisition unit 21, the correction processing unit 25 does not correct the depth value if the adjacent pixel values ​​on the detected surface are not within a predetermined range. If the correction processing unit 25 does not correct the depth value, it terminates the processing based on the correction function CF and resumes processing based on a new correction function CF that starts from a point adjacent to the starting point of the correction function CF.

[0080] (Step S56) Next, the correction processing unit 25 determines whether or not correction has been performed for all points on the correction function CF. If the correction processing unit 25 has performed correction for all points on the correction function CF (i.e., Step S56; YES), it proceeds to Step S57. If the correction processing unit 25 has not performed correction for all points on the correction function CF (i.e., Step S56; NO), it proceeds to Step S52 and repeats the processes from Step S52 to Step S54 until correction has been performed for all points on the correction function CF.

[0081] (Step S57) When correction has been completed for all points on the correction function CF, the correction processing unit 25 determines whether or not it has performed correction processing based on each correction function CF, starting from all points on the straight line detected by the straight line detection unit 23. If the correction processing unit 25 has performed correction processing based on each correction function CF, starting from all points on the straight line detected by the straight line detection unit 23 (i.e., Step S57; YES), it terminates the process. If the correction processing unit 25 has not performed correction processing based on each correction function CF, starting from all points on the straight line detected by the straight line detection unit 23 (i.e., Step S57; NO), it proceeds to Step S52 and performs processing again based on the correction function CF starting from an adjacent point on the straight line.

[0082] In the example described above, we explained an example in which a correction function CF extending in the direction of a line intersecting a specific line detected by the line detection unit 23 is used. Since the correction function CF extending in such a direction is aligned with the direction of the surface, it can correct with high accuracy. However, there was a drawback in that the computational load increased because the direction of the pixel arrangement and the direction of the correction function CF were different. Therefore, with reference to Figures 15 and 16, we will explain an example of a method that can reduce the computational load.

[0083] Figure 15 is a diagram illustrating an example of three-dimensional information correction according to Embodiment 2, specifically when correction is performed in the horizontal direction. In this case, the specific direction in which the correction processing unit 25 performs correction is the coordinate axis direction of the image acquired by the image acquisition unit 21. Figure 15(A) shows an example of correction in the horizontal direction, starting from line L1. By performing correction in the direction of the arrow shown in the figure, the amount of computation can be reduced. However, when correcting in the horizontal direction starting from line L1, only about half of the surface can be corrected. Therefore, the correction processing unit 25 further corrects the entire surface by starting from line L4 and performing correction in the horizontal direction, as shown in Figure 15(B). Even when correcting using lines L1 and L4 as starting points, if there are irregularities on the surface, areas that are not corrected will remain, like shadows. Therefore, the correction processing unit 25 further corrects the entire surface even if there are irregularities on the surface by starting from lines L2 and L3 respectively and performing correction in the horizontal direction. When using lines L1 and L4 as the starting point, the correction is performed from left to right, and when using lines L2 and L3 as the starting point, the correction is performed from right to left.

[0084] Figure 16 is a diagram illustrating an example of vertical correction in three-dimensional information correction according to Embodiment 2. Figure 16(A) shows an example of vertical correction starting from line L1. By performing correction in the direction of the arrow shown in the figure, the amount of computation can be reduced. However, when correcting vertically starting from line L1, only about three-quarters of the surface can be corrected. Therefore, the correction processing unit 25 further corrects the entire surface by starting from line L2 and correcting vertically, as shown in Figure 16(B). Even when correcting starting from lines L1 and L2, if there are irregularities on the surface, areas that are not corrected will remain, like shadows. Therefore, the correction processing unit 25 further corrects the entire surface even if there are irregularities on the surface by starting from lines L3 and L4 respectively and correcting vertically. When starting from lines L1 and L2, the correction is performed from top to bottom, and when starting from lines L3 and L4, the correction is performed from bottom to top.

[0085] [Summary of Embodiment 2] According to the embodiment described above, the line detection unit 23 detects multiple lines included in the image acquired by the image acquisition unit 21. Furthermore, the three-dimensional information correction device 20A further includes a surface detection unit 27 to detect surfaces included in the image based on the intersection points of the detected multiple lines. In the three-dimensional information correction device 20A, the correction function calculation unit 24 calculates a correction function that is a function for correcting the depth value at a point on the detected surface from among the acquired depth values, and shows the relationship between the depth value at a point on a line that includes pixels constituting the surface and is aligned in a specific direction, and the coordinates of the image. Furthermore, in the three-dimensional information correction device 20A, the correction processing unit 25 corrects the depth value for each coordinate on the line aligned in a specific direction, starting from a point on a specific line among the detected multiple lines, using the correction function. In other words, the three-dimensional information correction device 20A detects surfaces included in the image, calculates multiple correction functions that constitute the detected surface, and corrects the depth value of the surface to a flat one using the correction function. Therefore, according to this embodiment, the depth value of a portion that is originally a surface becomes flat, thus enabling the generation of natural three-dimensional information. Thus, according to this embodiment, the three-dimensional information of the object can be faithfully reproduced.

[0086] Furthermore, according to the embodiment described above, the correction processing unit 25 corrects the depth value based on the image acquired by the image acquisition unit 21 when the adjacent pixel values ​​on the surface detected by the surface detection unit 27 are within a predetermined range, and does not correct the depth value when they are outside the predetermined range. In other words, the three-dimensional information correction device 20A detects the presence or absence of irregularities on the surface, and does not perform correction if irregularities exist. Therefore, according to this embodiment, if irregularities exist, the surface can be corrected to be flat while leaving the irregularities intact, without flattening the irregular parts. Thus, according to this embodiment, the three-dimensional information of the object can be faithfully reproduced.

[0087] Furthermore, according to the embodiment described above, the specific direction in which the correction is performed by the correction processing unit 25 is the direction of the line that intersects with the specific line detected by the line detection unit 23. In other words, the correction processing unit 25 performs the correction along the direction of the detected surface. According to this embodiment, since the correction is performed along the direction of the surface, it is possible to correct to a more natural Depth value. Therefore, according to this embodiment, the three-dimensional information of the object can be faithfully reproduced.

[0088] Furthermore, according to the embodiment described above, the specific direction in which correction is performed by the correction processing unit 25 is the coordinate axis direction of the image acquired by the image acquisition unit 21. The coordinate axis direction may be the horizontal direction, as explained with reference to Figure 15, or the vertical direction, as explained with reference to Figure 16. By performing correction along the coordinate direction of the image, correction can be performed along the pixel direction, thus reducing the amount of computation. Therefore, according to this embodiment, high-speed processing can be achieved, and even when the three-dimensional information correction device 20A is used in an edge device, the processing time can be shortened.

[0089] In the embodiments described above, the explanation was based on the premise that three-dimensional information extraction processing is performed on still images. However, these embodiments are not limited to still images and may also be applied to videos. When these embodiments are applied to videos, the three-dimensional information extraction processing described above may be performed for each frame, or, to reduce the processing load, the three-dimensional information extraction processing described above may be performed every few frames. By applying these embodiments to videos, it becomes possible to highlight specific objects in videos and easily distinguish them from other subjects.

[0090] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the invention. Furthermore, the embodiments described above may be combined as appropriate. [Explanation of Symbols]

[0091] 1...Three-dimensional information acquisition system, 10...Three-dimensional information acquisition device, 110...Light receiving unit, 112...Visible light reflective dichroic film, 113...Image sensor, 114...ToF sensor, 120...Irradiation unit, 20...Three-dimensional information correction device, 21...Image acquisition unit, 22...Depth image acquisition unit, 23...Linear detection unit, 24...Correction function calculation unit, 25...Correction processing unit, 26...Output unit, 27...Surface detection unit, BM...Irradiation light, VL...Visible light, IL...Infrared light, OA...Optical axis, II...Image information, DI...Depth image information, LI...Linear information, CF...Correction function, 3DI...Three-dimensional information, PI...Attitude information

Claims

1. An image acquisition unit that acquires an image of the target object, A line detection unit that detects straight lines included in the acquired image, A Depth image acquisition unit acquires a Depth image that includes a plurality of Depth values, which are distance information to the object, at each coordinate in a two-dimensional coordinate system. A correction function calculation unit calculates a correction function which is a function for correcting the Depth value at a detected point on the line among the acquired Depth values, and which is a function that shows the relationship between the Depth value at a point on the line and the coordinates of the image. A correction processing unit that uses the calculated correction function to correct the Depth value at a detected point on the straight line among the acquired Depth values, A surface detection unit detects a surface included in the image based on the intersection points of a plurality of detected lines, Equipped with, The line detection unit detects a plurality of lines included in the acquired image, The correction function calculation unit calculates a function as the correction function for correcting the Depth value at a detected point on the surface from among the acquired Depth values, which is a function that shows the relationship between the Depth value at a point on a straight line that includes pixels constituting the surface and is aligned in a specific direction, and the coordinates of the image. The correction processing unit corrects the Depth value using the correction function for each coordinate on the line along the specific direction, starting from a specific point on one of the detected lines. The correction processing unit corrects the Depth value based on the image if the adjacent pixel values ​​on the detected surface are within a predetermined range, and does not correct the Depth value if they are outside the predetermined range. Three-dimensional information correction device.

2. The correction function calculation unit determines two points for calculating the correction function based on the Depth values ​​at multiple points on the detected straight line from the acquired Depth values, and uses a linear function passing through the two determined points as the correction function. The three-dimensional information correction device according to claim 1.

3. The aforementioned image is an image colored according to the degree of deflection acquired by the deflection sensor. The three-dimensional information correction device according to claim 1.

4. The image acquisition process involves obtaining an image of the target object, A line detection step for detecting straight lines contained in the acquired image, Depth image acquisition step: Acquires a depth image that includes a plurality of depth values, which are distance information to the object, at each coordinate in a two-dimensional coordinate system. A correction function calculation step involves calculating a correction function that is a function for correcting the Depth value at a detected point on the straight line from among the acquired Depth values, and which is a function that shows the relationship between the Depth value at a point on the straight line and the coordinates of the image. A correction process step in which the Depth values ​​at the detected points on the straight line are corrected using the calculated correction function, A surface detection step that detects a surface included in the image based on the intersection points of a plurality of detected lines, It has, The line detection step involves detecting a plurality of lines included in the acquired image, The correction function calculation step involves calculating a function for correcting the Depth values ​​at detected points on the surface from among the acquired Depth values, which is a function that shows the relationship between the Depth values ​​at points on a straight line that includes pixels constituting the surface and is aligned in a specific direction, and the coordinates of the image, as the correction function. The correction process involves taking a specific point on one of the detected lines as the starting point and correcting the Depth value for each coordinate on the line along the specific direction using the correction function. The correction process, based on the image, corrects the Depth value if the adjacent pixel values ​​on the detected surface are within a predetermined range, and does not correct the Depth value if they are outside the predetermined range. Three-dimensional information correction method.

Images