Image processing device, image processing method, and program

Abstract

To enable the distortion of a captured image that corresponds to the depth of water to be correctly corrected.SOLUTION: An image processing device according to the present disclosure comprises: an image sensor for imaging a measurement object in the water; a first laser light source for emitting a first laser light in the water; a second laser light source for emitting a second laser light in the water from a position different from the first laser light source; and a correction data generation unit for generating data for correction that is used for correcting the distortion of a captured image by the image sensor in a second depth of water which is the depth of water at which the measurement object is imaged by the image sensor, on the basis of first information that is acquired by imaging the first and second laser lights by the image sensor in a first depth of water and second information that is acquired by imaging the first and second laser lights by the image sensor in the second depth of water.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]

[0002] For example, there is a demand for technology that uses stereo cameras in autonomous underwater vehicles (AUVs) to perform underwater 3D measurements. However, the deeper the water, the greater the pressure on the robot and camera housing, which distorts the captured images, making it difficult to perform accurate 3D measurements. Patent Document 1 proposes a method for calibrating captured images using correction data acquired in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-248414 Summary of the Invention [Problem to be solved by the invention]

[0004] In the method of calibrating captured images using correction data acquired in advance, there is a possibility that the causal relationship between the pre-assumed pressure resistance and the pressure resistance that actually occurs when performing 3D measurement may not be correct, making it difficult to correctly correct distortion in the captured images.

[0005] It is desirable to provide an image processing device, an image processing method, and a program that can correctly correct distortion of a captured image according to water depth. [Means for solving the problem]

[0006] An image processing device according to one embodiment of the present disclosure includes an image sensor that captures an image of a measurement object underwater, a first laser light source that emits a first laser light underwater, a second laser light source that emits a second laser light underwater from a position different from the first laser light source, and a correction data generation unit that generates correction data used to correct distortion of an image captured by the image sensor at a second water depth based on first information acquired by capturing the first laser light and the second laser light with the image sensor at a first water depth and second information acquired by capturing the first laser light and the second laser light with the image sensor at a second water depth that is the water depth when capturing the image of the measurement object with the image sensor.

[0007] An image processing method according to an embodiment of the present disclosure includes capturing an image of an underwater measurement object by an image sensor, and From the first laser source The method includes emitting a first laser light, emitting a second laser light underwater from a position different from the first laser light source, and generating correction data used to correct distortion of an image captured by the image sensor at the second water depth based on first information acquired by capturing the first laser light and the second laser light with the image sensor at the first water depth and second information acquired by capturing the first laser light and the second laser light with the image sensor at the second water depth, which is the water depth when capturing an image of the object to be measured with the image sensor.

[0008] A program according to an embodiment of the present disclosure is From the first laser sourceThe method causes a computer to execute a process including emitting a first laser light, emitting a second laser light underwater from a position different from the first laser light source, and generating correction data used to correct distortion of an image captured by the image sensor at a second water depth based on first information acquired by capturing the first laser light and the second laser light with an image sensor at a first water depth and second information acquired by capturing the first laser light and the second laser light with an image sensor at a second water depth, which is the water depth when capturing an image of a measurement object with the image sensor. Let's say.

[0009] In an image processing device, image processing method, or program according to one embodiment of the present disclosure, correction data used to correct distortion of an image captured by the image sensor at a second water depth is generated based on first information acquired by capturing a first laser light and a second laser light by the image sensor at a first water depth, and second information acquired by capturing a first laser light and a second laser light by the image sensor at a second water depth, which is the water depth when capturing an image of a measurement object by the image sensor. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram schematically illustrating a configuration example of an image processing device according to a first embodiment of the present disclosure. [Figure 2] 10 is an explanatory diagram showing an example of the irradiation state of two line laser beams from a laser and the field of view range of a stereo camera. FIG. [Figure 3] FIG. 10 is an explanatory diagram showing an example of an image captured by a stereo camera using two line laser beams. [Figure 4] FIG. 2 is an explanatory diagram showing an example of scanning with two line laser beams by a scanning mechanism. [Figure 5] FIG. 10 is an explanatory diagram showing an example of an intersection point of two line laser beams when the water depth is shallow. [Figure 6]FIG. 10 is an explanatory diagram showing an example of an intersection point of two line laser beams when the water depth is deep. [Figure 7] FIG. 3 is an explanatory diagram illustrating an example of information stored in a correction data storage unit in the image processing device according to the first embodiment. [Figure 8] 5 is a flowchart showing an example of a process for creating reference data (reference data) performed when the water depth is shallow in the image processing device according to the first embodiment. [Figure 9] 5 is a flowchart showing an example of calibration processing performed when the water depth is deep in the image processing device according to the first embodiment. [Figure 10] FIG. 10 is a block diagram schematically illustrating an example of the configuration of an image processing device according to a second embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of the configuration of an autonomous underwater robot to which an image processing device according to a second embodiment is applied. [Figure 12] FIG. 10 is a block diagram schematically illustrating an example of the configuration of an image processing device according to a third embodiment. [Figure 13] FIG. 13 is an explanatory diagram illustrating an example of information stored in a correction data storage unit in an image processing device according to a fourth embodiment. [Figure 14] FIG. 10 is a block diagram schematically illustrating an example of the configuration of an image processing device according to a fifth embodiment. [Figure 15] FIG. 10 is a diagram illustrating an example of the configuration of an autonomous underwater robot to which an image processing device according to a fifth embodiment is applied. [Figure 16] 13 is a flowchart showing an example of a process for generating a depth image in an image processing device according to a fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order. 0. Comparative Example 1. First embodiment (FIGS. 1 to 9) 1.1 Configuration 1.2 Operation 1.3 Effects 2. Second embodiment (FIGS. 10 and 11) 3. Third embodiment (FIG. 12) 4. Fourth embodiment (FIG. 13) 5. Fifth embodiment (FIGS. 14 to 16) 6. Other embodiments

[0012] <0. Comparative Examples> For example, underwater, there is a demand for 3D measurement technology for inspections and the autonomous movement of autonomous underwater robots. Inspections, in particular, require high-resolution and high-precision 3D measurement technology. The stereo camera method, which is widely used on land, is a promising technology for achieving high resolution and high precision. However, the deeper the water, the greater the pressure exerted on the robot and camera housing, distorting the captured images, making it more difficult to perform accurate 3D measurements.

[0013] Patent Document 1 (JP 2011-248414 A) proposes a method for acquiring correction data for each pressure resistance in advance as an image processing system that can obtain images equivalent to those captured under normal pressure. However, because calibration is performed in advance, there is a possibility that the causal relationship between the assumed pressure resistance and the actual pressure resistance may be incorrect. The technology proposed in Patent Document 1 acquires correction information for each pressure in advance and calculates differential parameters (movement vectors) from those under normal pressure. However, this depends on the accuracy of the pressure and temperature sensors used to measure the environment, and the mechanical shape changes that occur when actually submerged in a high-pressure environment may not necessarily match the acquired parameters, which may result in inaccurate 3D measurements.

[0014] <1. First embodiment> [1.1 Configuration] FIG. 1 schematically illustrates an example configuration of an image processing device according to a first embodiment of the present disclosure.

[0015] The image processing device according to the first embodiment is mounted on, for example, an autonomous underwater robot and used for three-dimensional measurement underwater, such as in the sea.

[0016] The image processing device according to the first embodiment includes a stereo camera 10, a distortion correction unit 20, a parameter correction unit 21, a stereo matching unit 22, and a generated calibration parameter storage unit 23. The image processing device according to the first embodiment further includes a laser 30, a scanning mechanism 31, a line laser calibration unit 32, and a pressure sensor 50.

[0017] The stereo matching unit 22 corresponds to a specific example of a "depth image generating unit" in the technology of the present disclosure. The line laser calibration unit 32 corresponds to a specific example of a "correction data generating unit" in the technology of the present disclosure.

[0018] The distortion correction unit 20, the parameter correction unit 21, the stereo matching unit 22, and the line laser calibration unit 32 may be configured by a computer including, for example, one or more central processing units (CPUs), one or more read-only memories (ROMs), and one or more random access memories (RAMs). In this case, the processing of each unit by the distortion correction unit 20, the parameter correction unit 21, the stereo matching unit 22, and the line laser calibration unit 32 may be realized by one or more CPUs executing processing based on programs stored in one or more ROMs or RAMs. Furthermore, the processing of each unit by the distortion correction unit 20, the parameter correction unit 21, the stereo matching unit 22, and the line laser calibration unit 32 may be realized by one or more CPUs executing processing based on programs supplied from an external source via, for example, a wired or wireless network.

[0019] FIG. 2 shows an example of the irradiation state of two line laser beams L1 and L2 emitted by the laser 30 and the field of view of the stereo camera 10. As shown in FIG.

[0020] The stereo camera 10 is an image sensor that captures images of an underwater measurement object. As shown in Fig. 2, the stereo camera 10 has a left camera 10L and a right camera 10R. Fig. 2 shows an example of a field of view 11L of the left camera 10L and a field of view 11R of the right camera 10R.

[0021] The generated calibration parameter storage unit 23 stores generated calibration parameters that are generated in advance on land, for example, and serve as a reference for calibration.

[0022] The parameter correction unit 21 corrects the calibration parameters (generated calibration parameters) used to correct distortion of the images captured by the stereo camera 10, based on the correction data (correction parameters) generated by the line laser calibration unit 32.

[0023] The distortion correction unit 20 includes a distortion correction unit 20L for the left camera and a distortion correction unit 20R for the right camera. The distortion correction unit 20 corrects distortion in the captured image of the measurement object captured by the stereo camera 10 based on the corrected calibration parameters corrected by the parameter correction unit 21.

[0024] The stereo matching unit 22 is a depth image generation unit that generates a depth image based on the captured image of the measurement object corrected by the distortion correction unit 20. The stereo matching unit 22 performs stereo matching processing on the image captured by the left camera 10L after distortion correction corrected by the distortion correction unit 20L and the image captured by the right camera 10R after distortion correction corrected by the distortion correction unit 20R, and generates a depth image including three-dimensional measurement information.

[0025] As shown in Fig. 2, the laser 30 has a left laser 30L and a right laser 30R. The left laser 30L is a first laser light source that emits a line laser light L1 as a first laser light underwater. The right laser 30R is a second laser light source that emits a line laser light L2 as a second laser light from a position different from that of the left laser 30L underwater. Fig. 2 shows an example of the optical path of the line laser light L1 emitted from the left laser 30L and the optical path of the line laser light L2 emitted from the right laser 30R.

[0026] The left laser 30L is disposed, for example, to the left of the left camera 10L. The right laser 30R is disposed, for example, to the right of the right camera 10R. The number of lasers 30 is not limited to two. For example, in order to improve the processing speed of the calibration, two more lasers may be installed in the vertical direction in addition to the left laser 30L and the right laser 30R.

[0027] As shown in FIGS. 4 to 6, which will be described later, the scan mechanism 31 has a left scan mirror 31L and a right scan mirror 31R. The left scan mirror 31L is a first scan mechanism that scans the line laser light L1 emitted from the left laser 30L. The right scan mirror 31R is a second scan mechanism that scans the line laser light L2 emitted from the right laser 30R. The left scan mirror 31L and the right scan mirror 31R may each be, for example, a MEMS (Micro Electro Mechanical Systems) mirror.

[0028] The line laser calibration unit 32 includes a correction data storage unit 33. The correction data storage unit 33 may be provided outside the line laser calibration unit 32.

[0029] For example, when laser light is irradiated underwater, backscattered light is generated by plankton and the like. When this backscattered light is viewed with the stereo camera 10, the optical path of the laser light becomes visible. The line laser calibration unit 32 acquires information on the intersections of the two line laser light beams L1 and L2 for all pixels of the stereo camera 10 by controlling the scanning mechanism 31 to scan the two line laser light beams L1 and L2. The line laser calibration unit 32 previously stores information on the intersections of the two line laser light beams L1 and L2 as seen by the stereo camera 10 at a shallower water depth (first water depth) as reference data. The image processing device according to the first embodiment compares this reference data with information on the intersections of the two line laser light beams L1 and L2 acquired at a water depth higher than the first water depth (second water depth), thereby calculating the amount of distortion of the image captured by the stereo camera 10 and correcting the image distortion.

[0030] The line laser calibration unit 32 generates correction data (correction parameters) used to correct distortion of the image captured by the stereo camera 10 at the second water depth, based on first information acquired by capturing the line laser light L1 and the line laser light L2 with the stereo camera 10 at the first water depth (shallow water depth) and second information acquired by capturing the line laser light L1 and the line laser light L2 with the stereo camera 10 at the second water depth (deep water depth), which is the water depth when the stereo camera 10 captures the image of the measurement object.

[0031] The line laser calibration unit 32 acquires, as first information, information on a first intersection point between the line laser light L1 and the line laser light L2 captured by the stereo camera 10 at a first water depth, and acquires, as second information, information on a second intersection point corresponding to the first intersection point between the line laser light L1 and the line laser light L2 captured by the stereo camera 10 at a second water depth, and generates correction data (correction parameters) based on information on the amount of movement between the first intersection point and the second intersection point on the image captured by the stereo camera 10.

[0032] Line laser calibration unit 32 controls the scanning of left scan mirror 31L and right scan mirror 31R so that information on the first intersection point and information on the second intersection point can be obtained for all pixels on the image captured by stereo camera 10.

[0033] The correction data storage unit 33 stores information on the first intersection point and information on the second intersection point in association with information on the first scan angle by the left scan mirror 31L and information on the second scan angle by the right scan mirror 31R, respectively.

[0034] The pressure sensor 50 is a water pressure gauge that measures pressure in water and is capable of estimating depth.

[0035] FIG. 3 shows an example of an image captured by the stereo camera 10 using two line laser beams L1 and L2.

[0036] The upper part of Figure 3 shows a case where the water depth is shallow, i.e., the calibration parameters match the reference generated calibration parameters, so the intersection of the two line laser beams L1 and L2 is on the epipolar line. This allows for correct distortion correction by the distortion correction unit 20. The lower part of Figure 3 shows a case where the water depth is deep, and the image is distorted due to the high withstand voltage of the equipment containing the image processing device. In this case, the intersection of the two line laser beams L1 and L2 is not on the epipolar line. Therefore, if the reference generated calibration parameters are used as is, the distortion correction unit 20 will not be able to create a correctly corrected image, and the distance image (depth image) calculated by the stereo matching unit 22 will not be correct.

[0037] FIG. 4 shows an example of scanning by the scanning mechanism 31 with two line laser beams L1 and L2.

[0038] As an example, Figure 4 shows an example in which two line laser beams L1 and L2 are scanned using the scanning mechanism 31 so that the intersection of the two line laser beams L1 and L2 can be captured with all pixels of the left camera 10L. The example in Figure 4 shows the left line laser beam L1 being scanned. At a shallow water depth (first water depth), the scan angles for all pixel positions are stored in the correction data storage unit 33, and when the water depth changes (second water depth), the intersection at the same scan angle is compared with the stored pixel position, thereby obtaining the amount of movement of the intersection due to water depth.

[0039] Fig. 5 shows an example of the intersection of two line laser beams L1 and L2 when the water depth is shallow (first water depth). Fig. 6 shows an example of the intersection of two line laser beams L1 and L2 when the water depth is deep (second water depth). Fig. 7 shows an example of information stored in the correction data storage unit 33.

[0040] As shown in FIG. 5, when the water depth is shallow, the intersection of the two line laser beams L1 and L2 is at coordinates (x L ,y L ), and the coordinates (x R ,y R ), and the angle of the left scan mirror 31L at this time is (θx, θy), and the angle of the right scan mirror 31R at this time is (Φx, Φy). Also, as shown in FIG. 6, when the water is deep and the angles of the scan mechanism 31 are set to the same angles (θx, θy), (Φx, Φy) as shown in FIG. 5, the intersection of the two line laser beams L1 and L2 is at coordinates (X L ,Y L ), and the coordinates on the image captured by the right camera 10R are (X R ,Y R ) for the left camera 10L as information on the amount of movement of these coordinates. L -x L ,Y L -y L ) and the movement amount for the right camera 10R (X R -x R ,Y R-y R ) and performs the same process on the coordinates of all pixels on the image captured by the stereo camera 10, and stores the obtained information (FIG. 7) in the correction data storage unit 33 as correction data (correction parameters).

[0041] [1.2 Operation] In the image processing device according to the first embodiment, when performing calibration according to pressure resistance, the line laser calibration unit 32 controls the scan mechanism 31 to output two line laser beams L1 and L2, and performs line laser calibration using the captured image obtained from the stereo camera 10, thereby obtaining correction parameters using the above method. The parameter correction unit 21 and distortion correction unit 20 correct distortion in the captured image obtained from the stereo camera 10 based on the captured image obtained from the stereo camera 10, pre-generated calibration parameters that serve as a calibration reference, and the above correction parameters. The stereo matching unit 22 performs stereo matching on the distortion-corrected captured image to generate a depth image. Line laser calibration does not have to be performed for every frame captured by the stereo camera 10, but may be performed periodically when water pressure fluctuates or when a device such as a robot equipped with the image processing device descends.

[0042] FIG. 8 is a flowchart showing an example of a process for creating reference data (reference data) performed when the water depth is shallow in the image processing device according to the first embodiment.

[0043] First, two line laser beams L1 and L2 are emitted by the laser 30 and the scanning mechanism 31 (step S101). Next, the optical paths of the two line laser beams L1 and L2 are captured by the stereo camera 10 (step S102). Next, the line laser calibration unit 32 stores data of the intersection of the two line laser beams L1 and L2 as reference data in the correction data storage unit 33 (step S103).

[0044] Next, the line laser calibration unit 32 determines whether scanning has been completed for all pixels of the stereo camera 10 (step S104). If it is determined that scanning has not been completed for all pixels (step S104; N), the line laser calibration unit 32 then changes the scan angle of the scanning mechanism 31 (step S105) and returns to the processing of step S102.

[0045] On the other hand, if it is determined that scanning of all pixels has been completed (step S104; Y), the line laser calibration unit 32 stops the laser 30 (step S106).

[0046] FIG. 9 is a flowchart illustrating an example of calibration processing performed when the water depth is deep in the image processing device according to the first embodiment.

[0047] First, two line laser beams L1 and L2 are emitted by the laser 30 and the scanning mechanism 31 (step S201). Next, the optical paths of the two line laser beams L1 and L2 are captured by the stereo camera 10 (step S202). Next, the line laser calibration unit 32 stores data of the intersection of the two line laser beams L1 and L2 as current data in the correction data storage unit 33 (step S203).

[0048] Next, the line laser calibration unit 32 determines whether scanning has been completed for all pixels of the stereo camera 10 (step S204). If it is determined that scanning has not been completed for all pixels (step S204; N), the line laser calibration unit 32 then changes the scan angle of the scanning mechanism 31 (step S205) and returns to the processing of step S202.

[0049] On the other hand, if it is determined that scanning of all pixels has been completed (step S204; Y), the line laser calibration unit 32 stops the laser 30 (step S206). Next, the line laser calibration unit 32 generates correction parameters from the reference data and current data stored in the correction data storage unit 33 (step S207). Next, the stereo camera 10 captures an image of the measurement object (step S208). Next, the parameter correction unit 21 uses the correction parameters to correct the generated calibration parameters (step S209). Next, the distortion correction unit 20 uses the corrected calibration parameters to correct distortion in the image (stereo camera image) captured by the stereo camera 10 (step S210). Next, the stereo matching unit 22 generates a depth image from the distortion-corrected stereo image (step S211).

[0050] Next, the line laser calibration unit 32 determines whether the depth has changed based on the information from the pressure sensor 50 (step S212). If the line laser calibration unit 32 determines that the depth has not changed (step S212; N), the process returns to step S208. On the other hand, if the line laser calibration unit 32 determines that the depth has changed (step S212; Y), the process returns to step S201.

[0051] [1.3 Effects] As described above, the image processing device according to the first embodiment makes it possible to correctly correct distortion of a captured image according to water depth.

[0052] According to the image processing device of the first embodiment, changes in distortion of captured images are measured in real time and fed back to the distortion correction unit 20, so that 3D measurement can always be performed using correctly corrected images. At that time, calibration of the captured image can be easily performed at the location where the 3D measurement is performed. Even if the distortion that actually occurs differs from that measured in advance, it can be correctly corrected because calibration can be performed on the spot (in real time).

[0053] The image processing device according to the first embodiment makes it possible to calibrate captured images based on actual data even under multiple pressures, thereby enabling correct image correction and ultimately calculating a correct depth image through stereo matching. When using a stereo camera system as a high-resolution, high-precision 3D measurement technique in the sea, there is a problem of ranging errors caused by image distortion due to the influence of pressure. By performing calibration using a laser 30 and a scanning mechanism 31, this can be easily and accurately corrected in real time.

[0054] The effects described in this specification are merely examples and are not limiting, and other effects may also be achieved. The same applies to the effects of other embodiments described below.

[0055] <2. Second embodiment> Next, an image processing device and an image processing method according to a second embodiment of the present disclosure will be described. Note that, in the following, parts that are substantially the same as the components of the image processing device and the image processing method according to the first embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0056] Fig. 10 shows a schematic configuration example of an image processing device according to the second embodiment. Fig. 11 shows a schematic configuration example of an autonomous underwater robot 1 to which the image processing device according to the second embodiment is applied.

[0057] The image processing device according to the second embodiment has a configuration in which a pattern projector 40 is added to the configuration of the image processing device according to the first embodiment (FIG. 1).

[0058] The pattern projector 40 projects a predetermined pattern onto the measurement object (right side of FIG. 11). As a result, even if the measurement object does not have a texture that is difficult to stereo match, accurate 3D measurement becomes possible by projecting a texture suitable for stereo matching. As shown on the left side of FIG. 11, when performing calibration, a laser 30 is used, as in the image processing device according to the first embodiment.

[0059] Other configurations, operations, and effects may be substantially the same as those of the image processing device and image processing method according to the first embodiment.

[0060] <3. Third Embodiment> Next, an image processing device and an image processing method according to a third embodiment of the present disclosure will be described. Note that, in the following, parts that are substantially the same as the components of the image processing device and the image processing method according to the first or second embodiment will be assigned the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0061] FIG. 12 shows an outline of a configuration example of an image processing device according to the third embodiment.

[0062] The image processing device according to the third embodiment has a configuration in which two pattern projectors 41 and 42 are added to the configuration of the image processing device according to the first embodiment (FIG. 1). The two pattern projectors 41 and 42 are capable of projecting predetermined patterns of different wavelength bands onto the measurement object. For example, the pattern projector 41 may be capable of projecting a predetermined pattern in the 530 nm band, and the pattern projector 42 may be capable of projecting a predetermined pattern in the 850 nm band.

[0063] It is also possible to provide three or more pattern projectors and to configure the device so as to be able to irradiate three or more predetermined patterns with different wavelength bands.

[0064] The line laser calibration unit 32 estimates the turbidity in the water based on images of the line laser light L1 and the line laser light L2 captured by the stereo camera 10 when pressure is applied (second water depth), and selects one of the two pattern projectors 41, 42 as the pattern projector that will project a predetermined pattern based on the estimated turbidity in the water.

[0065] The image processing device according to the third embodiment utilizes the phenomenon that the closer the wavelength of light is to blue (440 nm band), the lower the absorption rate in water, resulting in a longer light reach, and the fact that the amount of backscattered light varies depending on the wavelength of the light in turbid water environments (the amount of backscattered light in the near-infrared band is less than that in the visible band). The line laser calibration unit 32 estimates the reach of the light when two line laser beams L1 and L2 are irradiated from images captured by the stereo camera 10, and further estimates the turbidity of the water. When the turbidity is low, the 530 nm band pattern projector 41, which has low absorption rate in water, is used. When the turbidity is high, the 850 nm band pattern projector 42, which has low backscattered light, is used instead, because the 530 nm band causes a lot of backscattered light, making the pattern unclear. This enables highly robust 3D measurement.

[0066] Other configurations, operations, and effects may be substantially the same as those of the image processing device and image processing method according to the first embodiment.

[0067] <4. Fourth embodiment> Next, an image processing device and an image processing method according to a fourth embodiment of the present disclosure will be described. Note that, in the following, parts that are substantially the same as the components of the image processing device and the image processing method according to any one of the first to third embodiments will be given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0068] FIG. 13 shows an example of information stored in the correction data storage unit 33 in the image processing device according to the fourth embodiment.

[0069] The basic configuration of the image processing device according to the fourth embodiment may be substantially the same as the configuration of the image processing device according to the third embodiment (FIG. 12). However, in the image processing device according to the fourth embodiment, as shown in FIG. 13, the correction data storage unit 33 stores depth (water depth) information in addition to the information shown in FIG. 7. The correction data storage unit 33 stores the correction data (correction parameters) generated by the line laser calibration unit 32 in association with the depth information of the second water depth (deep water depth).

[0070] If line laser calibration is performed every time the depth changes, it will affect the activity time of equipment such as a robot equipped with an image processing device. For this reason, it may be possible not to perform line laser calibration again at a depth (water depth) where line laser calibration has been performed once. The correction data storage unit 33 stores correction parameters together with depth information estimated from the pressure sensor 50. At a depth (water depth) where line laser calibration has been performed once, distortion of the captured image may be corrected based on the correction parameters already stored in the correction data storage unit 33.

[0071] Other configurations, operations, and effects may be substantially the same as those of the image processing device and image processing method according to the first or fourth embodiment.

[0072] <5. Fifth Embodiment> Next, an image processing device and an image processing method according to a fifth embodiment of the present disclosure will be described. Note that, in the following, parts that are substantially the same as the components of the image processing device and the image processing method according to any one of the first to fourth embodiments will be given the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0073] Fig. 14 shows a schematic configuration example of an image processing device according to the fifth embodiment. Fig. 15 shows a schematic configuration example of an autonomous underwater robot 1 to which the image processing device according to the fifth embodiment is applied.

[0074] The image processing device according to the fifth embodiment has a configuration in which a pattern projector 40 and a Z distance estimation unit 60 are added to the configuration of the image processing device according to the first embodiment (FIG. 1).

[0075] The Z distance estimation unit 60 estimates the distance Z to the measurement object based on the images captured by the stereo camera 10 using the line laser light L1 and the line laser light L2 in a pressure-resistant state (second water depth). The stereo matching unit 22 sets a search range for the measurement object for generating a depth image based on the distance Z estimated by the Z distance estimation unit 60.

[0076] As shown in FIG. 15 , the image processing device according to the fifth embodiment is suitable for measuring the distance to a planar object such as a dam wall 70. By irradiating the wall 70 with two line laser beams L1 and L2 and capturing the images using the stereo camera 10, it is possible to estimate the distance Z to the wall 70 by triangulation based on the multiple points on the wall 70 illuminated by the two line laser beams L1 and L2. Generally, in stereo ranging using block matching, the measurable range is limited due to limited memory resources and high speed. Therefore, by further limiting the search range of the stereo matching unit 22 using the estimated distance to the wall 70, it is possible to further reduce resources and increase speed. Another advantage of limiting the search range of the stereo matching unit 22 is that it is expected to improve robustness, such as preventing distance estimation errors due to incorrect matching.

[0077] FIG. 16 is a flowchart illustrating an example of a process for generating a depth image in the image processing device according to the fifth embodiment.

[0078] First, two line laser beams L1 and L2 are emitted by the laser 30 and the scanning mechanism 31 (step S301). Next, the Z distance estimation unit 60 estimates the distance Z to the wall 70 from the captured images of the two line laser beams L1 and L2 (step S302). Next, the search range of the stereo matching unit 22 is optimized to the distance Z (step S303).

[0079] Next, the pattern projector 40 projects a predetermined pattern onto the wall 70 (step S304). Next, the stereo matching unit 22 performs stereo distance measurement (step S305). Next, the stereo matching unit 22 generates a robust depth image (step S306).

[0080] Other configurations, operations, and effects may be substantially the same as those of the image processing device and image processing method according to the first or second embodiment.

[0081] <6. Other embodiments> The technology according to the present disclosure is not limited to the above-described embodiments, and various modifications are possible.

[0082] For example, the present technology can be configured as follows. According to the present technology having the following configuration, correction data used to correct distortion of an image captured by the image sensor at a second water depth is generated based on first information acquired by capturing an image of a first laser beam and a second laser beam by the image sensor at a first water depth and second information acquired by capturing an image of a first laser beam and a second laser beam by the image sensor at a second water depth, which is the water depth at which the image sensor captures an object to be measured. This makes it possible to correctly correct distortion of the captured image according to the water depth.

[0083] (1) an image sensor that captures an image of an underwater measurement object; a first laser light source that emits a first laser light in water; a second laser light source that emits a second laser light from a position different from the first laser light source in water; a correction data generation unit that generates correction data used to correct distortion of an image captured by the image sensor at a second water depth, based on first information acquired by the image sensor at a first water depth by capturing an image of the first laser light and the second laser light and second information acquired by the image sensor at a second water depth that is the water depth when the image sensor captures an image of the measurement object; Equipped with Image processing device. (2) a parameter correction unit that corrects calibration parameters used to correct distortion of the captured image by the image sensor, based on the correction data generated by the correction data generation unit. The image processing device according to (1) above. (3) a distortion correction unit that corrects distortion of an image of the object to be measured captured by the image sensor based on the calibration parameters corrected by the parameter correction unit. The image processing device according to (2) above. (4) a depth image generating unit that generates a depth image based on the captured image of the measurement object corrected by the distortion correcting unit. The image processing device according to (3) above. (5) a distance estimation unit that estimates a distance to the measurement object based on an image captured by the image sensor at the second water depth using the first laser light and the second laser light, The depth image generation unit sets a search range of the measurement object for generating the depth image based on the distance estimated by the distance estimation unit. The image processing device according to (4) above. (6) The correction data generation unit acquires, as the first information, information on a first intersection point between the first laser light and the second laser light captured by the image sensor at the first water depth, and acquires, as the second information, information on a second intersection point corresponding to the first intersection point between the first laser light and the second laser light captured by the image sensor at the second water depth, and generates the correction data based on information on the amount of movement between the first intersection point and the second intersection point on the image captured by the image sensor. The image processing device according to any one of (1) to (5) above. (7) a first scanning mechanism that scans the first laser light emitted from the first laser light source; a second scanning mechanism that scans the second laser light emitted from the second laser light source; Furthermore, The correction data generation unit controls the scan of the first scanning mechanism and the second scanning mechanism so that information on the first intersection point and information on the second intersection point are obtained for all pixels on the image captured by the image sensor. The image processing device according to (6) above. (8) a correction data storage unit that stores information on the first intersection point and information on the second intersection point in association with information on a first scan angle by the first scanning mechanism and information on a second scan angle by the second scanning mechanism, respectively. The image processing device according to (7) above. (9) The correction data storage unit stores the correction data in association with depth information of the second water depth. The image processing device according to (8) above. (10) a pattern projector that projects a predetermined pattern onto the measurement object. The image processing device according to any one of (1) to (9) above. (11) a plurality of pattern projectors that project predetermined patterns having different wavelength bands onto the measurement object; The image processing device according to any one of (1) to (10) above. (12) the correction data generation unit estimates turbidity in the water based on images captured by the image sensor at the second water depth using the first laser light and the second laser light, and selects one of the plurality of pattern projectors as a pattern projector that projects the predetermined pattern based on the estimated turbidity in the water. The image processing device according to (11) above. (13) capturing an image of an underwater measurement object using an image sensor; In water From the first laser source emitting a first laser beam; emitting a second laser light from a position different from the first laser light source in water; generating correction data used to correct distortion of an image captured by the image sensor at a first water depth based on first information acquired by the image sensor capturing an image of the first laser light and the second laser light at a first water depth and second information acquired by the image sensor capturing an image of the first laser light and the second laser light at a second water depth that is the water depth at which the image sensor captures an image of the measurement object; Contains Image processing methods. (14) In water From the first laser source emitting a first laser beam; emitting a second laser light from a position different from the first laser light source in water; generating correction data used to correct distortion of an image captured by the image sensor at a first water depth based on first information acquired by capturing an image of the first laser light and the second laser light by the image sensor at a first water depth and second information acquired by capturing an image of the first laser light and the second laser light by the image sensor at a second water depth that is a water depth when capturing an image of an object to be measured by the image sensor; A program for causing a computer to execute a process including the above. [Explanation of symbols]

[0084] 1...Autonomous Underwater Vehicle (AUV), 10...Stereo camera (image sensor), 10L...Left camera (image sensor), 10R...Right camera (image sensor), 11L...Field of view range of left camera 10L, 11R...Field of view range of right camera 10R, 20...Distortion correction unit, 20L...Distortion correction unit (distortion correction unit for left camera), 20R...Distortion correction unit (distortion correction unit for right camera), 21...Parameter correction unit, 22...Stereo matching unit (depth image generation unit), 23...Generated (reference) calibration parameter storage unit, 30...Laser (laser light source), 30L...Left laser (first laser light source), 30R...Right laser laser (second laser light source), 31...scanning mechanism, 31L...left scan mirror (first scan mechanism), 31R...right scan mirror (second scan mechanism), 32...line laser calibration unit (correction data generation unit), 33...correction data storage unit, 40...pattern projector, 41...pattern projector (530 nm), 42...pattern projector (850 nm), 50...pressure sensor (water pressure gauge), 60...Z distance estimation unit, 70...wall (measurement object, wall of dam, etc.), L1...line laser light (first laser light), L2...line laser light (second laser light).

Claims

1. an image sensor that captures an image of an underwater measurement object; a first laser light source that emits a first laser light in water; a second laser light source that emits a second laser light from a position different from the first laser light source in water; a correction data generating unit that generates correction data used to correct distortion of an image captured by the image sensor at a second water depth, based on first information acquired by the image sensor at a first water depth by capturing an image of the first laser light and the second laser light and second information acquired by the image sensor at a second water depth that is a water depth when the image sensor captures an image of the measurement object; Equipped with Image processing device.

2. a parameter correction unit that corrects calibration parameters used to correct distortion of the captured image by the image sensor, based on the correction data generated by the correction data generation unit. The image processing device according to claim 1 .

3. a distortion correction unit that corrects distortion of an image of the object to be measured captured by the image sensor based on the calibration parameters corrected by the parameter correction unit. The image processing device according to claim 2 .

4. a depth image generating unit that generates a depth image based on the captured image of the measurement object corrected by the distortion correcting unit. The image processing device according to claim 3 .

5. a distance estimation unit that estimates a distance to the measurement object based on an image captured by the image sensor at the second water depth using the first laser light and the second laser light, The depth image generation unit sets a search range of the measurement object for generating the depth image based on the distance estimated by the distance estimation unit. The image processing device according to claim 4 .

6. The correction data generation unit acquires, as the first information, information on a first intersection point between the first laser light and the second laser light captured by the image sensor at the first water depth, and acquires, as the second information, information on a second intersection point corresponding to the first intersection point between the first laser light and the second laser light captured by the image sensor at the second water depth, and generates the correction data based on information on the amount of movement between the first intersection point and the second intersection point on the image captured by the image sensor. The image processing device according to claim 1 .

7. a first scanning mechanism that scans the first laser light emitted from the first laser light source; a second scanning mechanism that scans the second laser light emitted from the second laser light source; Furthermore, The correction data generation unit controls the scan of the first scanning mechanism and the second scanning mechanism so that information on the first intersection point and information on the second intersection point are obtained for all pixels on the image captured by the image sensor. The image processing device according to claim 6 .

8. a correction data storage unit that stores information on the first intersection point and information on the second intersection point in association with information on a first scan angle by the first scanning mechanism and information on a second scan angle by the second scanning mechanism, respectively. The image processing device according to claim 7 .

9. The correction data storage unit stores the correction data in association with depth information of the second water depth. The image processing device according to claim 8 .

10. a pattern projector that projects a predetermined pattern onto the measurement object. The image processing device according to claim 1 .

11. a plurality of pattern projectors that project predetermined patterns having different wavelength bands onto the measurement object; The image processing device according to claim 1 .

12. the correction data generation unit estimates turbidity in the water based on images captured by the image sensor at the second water depth using the first laser light and the second laser light, and selects one of the plurality of pattern projectors as a pattern projector that projects the predetermined pattern based on the estimated turbidity in the water. The image processing device according to claim 11 .

13. capturing an image of an underwater measurement object using an image sensor; Emitting a first laser beam from a first laser light source underwater; emitting a second laser light from a position different from the first laser light source in water; generating correction data used to correct distortion of an image captured by the image sensor at the second water depth based on first information acquired by the image sensor capturing an image of the first laser light and the second laser light at the first water depth and second information acquired by the image sensor capturing an image of the first laser light and the second laser light at the second water depth, which is the water depth at which the image sensor captures an image of the measurement object; Contains Image processing methods.

14. Emitting a first laser beam from a first laser light source underwater; emitting a second laser light from a position different from the first laser light source in water; generating correction data used to correct distortion of an image captured by the image sensor at a second water depth based on first information acquired by capturing an image of the first laser light and the second laser light by the image sensor at a first water depth and second information acquired by capturing an image of the first laser light and the second laser light by the image sensor at a second water depth that is a water depth when capturing an image of an object to be measured by the image sensor; A program for causing a computer to execute a process including the above.

Images