Distance measuring device, distance measuring method, distance measuring program, and distance measuring system

Abstract

[PROBLEM] To provide a distance measuring device, a distance measuring method, a distance measuring program, and a distance measuring system enabling accurate distance measurement over a wide range. [SOLUTION] A distance measuring device 10 of the present embodiment comprises: an image acquisition unit 131 that acquires at least two images which are captured by a stereo camera disposed on a platform and which include a water surface; an object detection unit 132 that detects whether or not an object is present on the water surface in the at least two images; a 3D point group acquisition unit 133 that acquires a three-dimensional point group of the water surface on the basis of the at least two images in which an object was detected on the water surface; a compensation surface calculation unit 134 that calculates a compensation surface of the water surface on the basis of the three-dimensional point group; and a distance calculation unit 135 that calculates the distance between the platform and the object, on the basis of the image in which the object was detected and the compensation surface.

Description

Distance measuring device, distance measuring method, distance measuring program, and distance measuring system

[0001] The present disclosure relates to a distance measuring device, a distance measuring method, a distance measuring program, and a distance measuring system, and more particularly to a distance measuring device, a distance measuring method, a distance measuring program, and a distance measuring system for measuring the distance from a platform such as a ship to an object on the water surface.

[0002] When a ship is navigating the sea, the crew needs to keep a proper watch to find static or dynamic obstacles that may pose a collision risk. There are various methods to measure the distance to a target object in the maritime environment, and each method has its advantages and disadvantages, as follows:

[0003] Radar-based methods are applicable over long distances, work in all weather conditions, and can identify object locations, but typically have low spatial resolution and are unable to classify objects. Light Detection and Ranging (LIDAR) has high resolution and accuracy, but only works over short distances and in good weather conditions.

[0004] The Automatic Identification System (AIS) relies on the vessel being equipped with a transponder, is typically only applicable to relatively large vessels (e.g., vessels over 50 meters in length), and updates are sent infrequently, subject to risk of tampering.

[0005] Camera systems provide detailed images of objects and are good for object classification. However, they cannot accurately pinpoint an object's location because they can only calculate direction, not distance. Thermal imaging cameras can see objects even at night, but they have low resolution and are expensive. Stereo cameras can detect distance, but the range at which they can accurately calculate the distance to an object is very limited.

[0006] Japanese Patent Application Laid-Open No. 2018-036117

[0007] The problem to be solved by the present disclosure is to provide a distance measuring device, a distance measuring method, a distance measuring program, and a distance measuring system that are capable of accurate distance measurement over a wide range. However, the problem is not limited to this, and the problem to be solved by the present disclosure may also be a problem corresponding to each effect of the configuration of each embodiment described below.

[0008] The distance measuring device according to the present disclosure comprises an image acquisition unit that acquires at least two images including the water surface taken by a stereo camera mounted on a platform; a 3D point cloud acquisition unit that acquires a three-dimensional point cloud of the water surface based on the at least two images in which an object on the water surface is detected; a compensation surface calculation unit that calculates a compensation surface of the water surface based on the three-dimensional point cloud; and a distance calculation unit that calculates the distance between the platform and the object based on the image in which the object is detected and the compensation surface.

[0009] In addition, in the distance measuring device, the 3D point cloud acquisition unit may generate a disparity map based on the at least two images, perform segmentation processing on one of the at least two images to extract pixels of the water surface area that constitutes the water surface, perform binary processing on the image from which the water surface area has been extracted to generate a mask, apply the mask to the disparity map to exclude pixels that are not related to the water surface, and perform three-dimensional reprojection to acquire a three-dimensional point cloud of the water surface.

[0010] In the distance measuring device, the compensation surface calculation unit may calculate the compensation surface, which is planar or curved, by fitting a compensation function to the three-dimensional point group.

[0011] In addition, in the distance measuring device, the distance calculation unit may determine an intersection between the compensation surface and a camera ray passing through a point of the stereo camera and a water line point of the object in the image, and determine the shortest line between the reference point of the platform and the intersection.

[0012] In addition, in the distance measuring device, when the compensation surface is a plane, the distance calculation unit may correct the distance based on the calculated distance, the mean radius of the Earth, and the length of a water line drawn from the point of the stereo camera to the plane.

[0013] The distance measurement method disclosed herein includes: acquiring at least two images including the water surface captured by a stereo camera mounted on a platform; acquiring a three-dimensional point cloud of the water surface based on the at least two images in which an object on the water surface is detected; calculating a compensation surface for the water surface based on the three-dimensional point cloud; and calculating the distance between the platform and the object based on the image in which the object is detected and the compensation surface.

[0014] The distance measurement program of the present disclosure causes a computer to perform the following processes: acquire at least two images including the water surface captured by a stereo camera mounted on a platform; acquire a three-dimensional point cloud of the water surface based on the at least two images in which an object on the water surface is detected; calculate a compensation surface for the water surface based on the three-dimensional point cloud; and calculate the distance between the platform and the object based on the image in which the object is detected and the compensation surface.

[0015] The ranging system of the present disclosure comprises: a stereo camera mounted on a platform so as to be able to capture images including the water surface; and a ranging device that calculates the distance from the platform to an object on the water surface based on at least two images captured by the stereo camera, wherein the ranging device has an image acquisition unit that acquires the at least two images; a 3D point cloud acquisition unit that acquires a three-dimensional point cloud of the water surface based on the at least two images in which the object is detected; a compensation surface calculation unit that calculates a compensation surface of the water surface based on the three-dimensional point cloud; and a distance calculation unit that calculates the distance between the platform and the object based on the images in which the object is detected and the compensation surface.

[0016] 1 is a plan view of a ship according to an embodiment. FIG. 2 is a view of a stereo camera mounted on a ship according to an embodiment, viewed from an oblique front. FIG. 3 is a view of a stereo camera mounted on a ship according to an embodiment, viewed from a side. FIG. 4 is a view of a stereo camera mounted on a ship according to an embodiment, viewed from an oblique rear. FIG. 5 is a schematic configuration diagram of a distance measuring system according to an embodiment. FIG. 6 is a functional block diagram of a distance measuring device according to an embodiment. FIG. 7 is a flowchart for explaining an example of a distance measuring method according to an embodiment. FIG. 8 is a diagram for explaining an example of an image captured by a camera. FIG. 9 is a diagram for explaining an example of an object detection processing result for an image captured by a camera. FIG. 10 is a diagram for explaining an example of an object detection processing result by panoptic segmentation. FIG. 11 is a flowchart for explaining an example of a method for acquiring a 3D point cloud in the distance measuring method according to an embodiment. FIG. 12 is a diagram for explaining an example of an image captured by a stereo camera. FIG. 13 is a diagram for explaining an example of a disparity map generated from a pair of images. FIG. 14 is a diagram for explaining an example of an image from which a water surface region has been extracted. FIG. 15 is a diagram for explaining an example of a binary mask obtained from an image from which a water surface region has been extracted. FIG. 16 is a diagram for explaining a method for calculating a water surface compensation surface from a 3D point cloud. FIG. 17 is a flowchart for explaining an example of a method for calculating a distance from a ship to an object in the distance measuring method according to an embodiment. It is an enlarged view of an image in which an object is detected. It is an explanatory diagram using a pinhole camera model. It is an example of a screen displayed on a display device. It is another example of a screen displayed on a display device.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0018] In addition, terms such as "parallel" and "orthogonal" that specify shapes, geometric conditions, physical characteristics, and their degrees, as well as dimensions and values ​​of physical characteristics, used in this specification, are not bound by strict meanings but are interpreted to include the range within which similar functions can be expected.

[0019] <Stereo Camera System> A stereo camera system including multiple stereo cameras according to an embodiment will be described with reference to Fig. 1 and Fig. 2A to Fig. 2C. Fig. 1 is a plan view of a ship 1 equipped with a stereo camera system according to an embodiment. Fig. 2A, Fig. 2B, and Fig. 2C are views of the stereo camera system mounted on a mast 2 of the ship 1 as seen obliquely from the front, side, and rear, respectively.

[0020] The ship 1 is provided with a stereo camera system having four sets of stereo cameras (first to fourth stereo cameras) for observing in all directions (forward, backward, port side, and starboard side) of the ship 1. That is, the stereo camera system of this embodiment has a first stereo camera, a second stereo camera, a third stereo camera, and a fourth stereo camera.

[0021] The first to fourth stereo cameras may be time-synchronized so that the camera frames are recorded synchronously or nearly synchronously. All the cameras in the stereo camera system may be time-synchronized.

[0022] The first stereo camera has cameras 3R and 3L and observes the front of the vessel 1. The second stereo camera has cameras 3RL and 3LL and observes the port side of the vessel 1. The third stereo camera has cameras 3RR and 3LR and observes the starboard side of the vessel 1. The fourth stereo camera has cameras 3RB and 3LB and observes the rear of the vessel 1.

[0023] It should be noted that each camera may be any type of camera operating in the visible spectrum (such as monochrome or RGB), near infrared (NIR), far infrared (FIR), multispectral, hyperspectral, or other light spectrum. Each camera may also be a residual light amplifier or low-light camera capable of operating in near darkness.

[0024] To simplify processing, the cameras in the stereo camera system may be of the same type, but this is not limiting and the cameras in the stereo camera system may be of different types.

[0025] Additionally, a polarizing filter may be applied to the camera to improve detection performance under certain lighting conditions.

[0026] Furthermore, the orientation of each camera may be stabilized, for example, by a gimbal mechanism, so that the orientation of the camera is maintained even when the vessel 1 is moving.

[0027] If desired, one or more light sources (not shown) may be provided on the vessel 1 to illuminate the water surface in low light conditions. The spectrum of the light source should at least partially match the spectrum that the camera can detect. For example, a near-infrared (NIR) light source should be used for a near-infrared (NIR) camera. The illumination area of ​​the light source should at least partially overlap with the field of view of the camera.

[0028] The light source may illuminate the water surface continuously over an extended period of time, or it may illuminate the water surface for a short period of time while an image (frame) is being captured. Structured light may also be used to project a pattern onto the water surface or to continuously illuminate an area. The light may also be polarized, for example by applying a polarizing filter to the light source.

[0029] The first to fourth stereo cameras are mounted on the mast 2 of the vessel 1 so as to be able to capture images including the water surface (water surface area). Each stereo camera is mounted higher than the water surface, preferably at a position at least three meters above the water surface. The first to fourth stereo cameras may be mounted on a structure other than the mast 2. The first to fourth stereo cameras do not necessarily have to be mounted on the same structure, but may be mounted on different structures. For example, if the vessel 1 has multiple masts, the first to fourth stereo cameras may be mounted on different masts.

[0030] It is advantageous for the two cameras (e.g., camera 3R and camera 3L) of the stereo camera to be positioned far apart (i.e., have a long baseline) to increase the range over which the distance to an object can be accurately calculated. For example, it is desirable to mount each camera at least one meter apart. In this embodiment, a large water surface area must be captured and reconstructed in three dimensions. By positioning each camera far enough apart, it is possible to reliably reconstruct a wide water surface area. Note that natural water surfaces are not flat, and waves of various wavelengths appear. As a rule of thumb, at least 8 to 10 complete wavelengths are required in each dimension of the field of view to reliably and accurately interpolate the average water surface.

[0031] In this embodiment, the base lengths of the first and fourth stereo cameras are relatively long (e.g., 2 to 3 meters) so as to enable long-distance measurement, whereas the base lengths of the second and third stereo cameras are relatively short (e.g., about 1 meter) so as to enable medium-distance measurement.

[0032] The number of stereo cameras in the stereo camera system is not limited to four, but may be one, two, three, or five or more. Furthermore, the vessel 1 may be provided with two or more stereo camera systems. Each stereo camera system may be configured with a separate camera set, or some cameras may be shared among the stereo camera systems as long as the fields of view of at least two of the cameras assigned to each stereo camera system overlap.

[0033] Furthermore, the stereo camera is not limited to being installed on the ship 1, but may be installed on fixed or floating facilities or objects on land or water (hereinafter collectively referred to as "platforms"). For example, the stereo camera may be installed on a lighthouse on land or a buoy on water.

[0034] In this embodiment, four sets of stereo cameras are provided, but since the configuration and processing contents of each stereo camera are the same, only the first stereo camera will be described below.

[0035] The cameras 3L and 3R are provided so as to have at least partially overlapping fields of view (FOV). In this embodiment, the optical axes of the cameras 3L and 3R are parallel, but this is not limiting.

[0036] The height and / or angle of the cameras 3L and 3R are adjusted so that the captured images include the water surface. For example, the cameras 3L and 3R are installed so that their optical axes point downward. This allows objects close to the ship 1 (e.g., other ships) and the horizon to always be within the field of view, even when the ship 1 is moving with the waves.

[0037] The camera 3L and the camera 3R are not limited to being arranged side by side in the horizontal direction, but may be arranged side by side in the vertical direction (up and down), for example.

[0038] <Range Measuring System> Next, a distance measuring system 1000 according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a schematic configuration diagram of the distance measuring system 1000. The distance measuring system 1000 includes a stereo camera 3 having a camera 3L and a camera 3R, a display device 4, and a distance measuring device 10.

[0039] The distance measurement system 1000 may include not only the stereo camera 3 (first stereo camera) but also second to fourth stereo cameras.

[0040] The ranging system 1000 may also include five or more stereo cameras. For example, the ranging system 1000 may include six stereo cameras, each having a vertical baseline (including multiple cameras arranged in the vertical direction) and a horizontal field of view of 60°. With this configuration, it is possible to observe the entire 360° area around the ship 1.

[0041] Alternatively, the stereo camera 3 may be a multi-view stereo camera having three or more cameras, each of which has at least a partially overlapping field of view (FOV). The optical axes of the cameras do not have to be parallel to each other.

[0042] The display device 4 displays images captured by the stereo camera 3, processing results (distance information) by the distance measuring device 10, etc. The display device 4 is, for example, a personal computer, tablet terminal, or smartphone provided on the ship 1. The display device 4 may also be provided outside the ship 1 (on land, for example). In this case, the display device 4 is communicatively connected to the distance measuring device 10 of the ship 1 via a wireless communication network (not shown). The display device 4 may also be provided both on the ship 1 and outside the ship 1.

[0043] As will be explained in detail below, the distance measuring device 10 is configured to calculate the distance from the ship 1 to an object on the water surface (a ship, a buoy, etc.) based on a pair of images captured by the stereo camera 3.

[0044] <Range Measuring Device> Next, the distance measuring device 10 according to this embodiment will be described in detail with reference to Fig. 4. Fig. 4 shows a functional block diagram of the distance measuring device 10.

[0045] The distance measuring device 10 includes a communication unit 11, a storage unit 12, and a processing unit 13. The distance measuring device 10 may also include sensors such as a GPS receiver, a gyroscope, an inertial measurement unit (IMU), and a compass.

[0046] The communication unit 11 is one or more communication interfaces for transmitting and receiving information to and from external devices such as the stereo camera 3 and the display device 4. Note that the communication method (wireless or wired) and the communication protocol are not particularly limited.

[0047] The storage unit 12 stores programs executed by the processing unit 13, data used to execute the programs, image data received from the stereo camera, etc. The storage unit 12 is configured from a semiconductor memory and / or a hard disk drive, etc.

[0048] The processing unit 13 is configured with processors such as an ECU (Electronic Control Unit), a CPU (Central Processing Unit), and a GPU (Graphics Processing Unit), and executes the programs stored in the storage unit 12 .

[0049] The processing unit 13 includes an image acquisition unit 131, an object detection unit 132, a 3D point cloud acquisition unit 133, a compensation surface calculation unit 134, and a distance calculation unit 135. Each of these functional units is stored in the storage unit 12 in the form of a program executable by a computer. For example, the processing unit 13 is configured with one or more processors, and realizes the function corresponding to each program by reading and executing the program from the storage unit 12.

[0050] Each functional unit of the processing unit 13 may be realized by a processor in the distance measuring device 10 executing a predetermined program and using hardware resources to perform software processing, or may be realized by the installed hardware itself. Furthermore, one or more of the functions of the processing unit 13 may be realized by another information processing device (not shown) connected to the distance measuring device 10.

[0051] Next, each functional unit of the processing unit 13 will be described in detail.

[0052] The image acquisition unit 131 acquires a pair of images captured by the stereo camera 3. Each image includes the water surface. The image acquisition unit 131 acquires image data transmitted by the stereo camera 3 via the communication unit 11.

[0053] The object detection unit 132 detects whether or not an object exists on the water surface in the pair of images acquired by the image acquisition unit 131. Note that there are no particular limitations on the object detection algorithm executed by the object detection unit 132. Machine learning such as a convolutional neural network (CNN), an image transformer, or a combination thereof may be used to detect objects in the images.

[0054] FIG. 6A shows an image I acquired by the image acquisition unit 131. １In this example image, the target object T １ , T ２ , T ３ , T ４ Object T is shown. １ , T ２ is a small vessel, and object T ３ is a buoy, and object T ４ is a large ship (LNG tanker).

[0055] FIG. 6B shows an image I of the object detection processing result for the image of FIG. 6A. ２ In this example, the object T １ , T ２ , T ３ , T ４ are detected and are surrounded by bounding boxes B1, B2, B3, and B4, respectively. The word "boat" on each bounding box indicates the type of object that is estimated, and the numerical value indicates the probability that it is that type.

[0056] Note that object detection and classification may also be performed via an instance segmentation model. In this method, the location of each detected object is returned as a segmentation mask consisting of only pixels that are part of the object. The lowest pixel of the segmentation mask (typically the pixel with the highest vertical pixel coordinate) or the pixel immediately below it is then determined to be the waterline point of the object (e.g., point P in Figures 14 and 15, described below). ２ ) as the waterline point. Alternatively, a pixel interpolated between these two pixels may be selected as the waterline point. Alternatively, a pixel on the object mask adjacent to the water surface pixel or its adjacent pixel may be selected as the waterline point. Alternatively, multiple pixels as described above may be selected and the distance may be calculated by some averaging, filtering, or other processing.

[0057] Also, pixel masks of regions of interest other than objects, such as water surfaces and sky, may be obtained using a semantic segmentation model.

[0058] Object detection may also be performed using a panoptic segmentation model. Figure 6C shows an example of the object detection results from this process. This provides the same benefits as instance segmentation, but also returns pixel masks of other regions of interest within the image. In particular, this model can be trained to identify and locate areas such as water, sky, piers, and bridges. Panoptic segmentation simplifies computer code and saves computing resources by returning all the necessary information from a single model with a single inference.

[0059] Additionally, the bounding box or segmentation-defined object regions may be further processed using additional models and methods to obtain more detailed classifications or additional information. For example, objects may be classified as ships, buoys, markers, and other related objects according to the Collision Regulations at Sea (COLREG). Such classifications are useful for further processing by regulation-compliant collision avoidance algorithms. Furthermore, the relative orientation and direction of objects can be determined, for example, by training a model to detect the heading direction of a ship.

[0060] Additionally, an instance or panoptic segmentation model may be trained to directly classify detected objects according to the COLREG category and / or the target object orientation.

[0061] The 3D point cloud acquisition unit 133 acquires a three-dimensional point cloud of the water surface based on the pair of images acquired by the image acquisition unit 131. The three-dimensional point cloud of the water surface is a set of points distributed on the water surface in three-dimensional space. Details of the processing by the 3D point cloud acquisition unit 133 will be described later with reference to FIG. 7.

[0062] The compensation surface calculation unit 134 calculates a compensation surface for the water surface based on the three-dimensional point cloud acquired by the 3D point cloud acquisition unit 133. The compensation surface is a flat or curved surface that approximates the water surface. Details of the processing by the compensation surface calculation unit 134 will be explained in detail later.

[0063] In the case of a stereo system equipped with multiple stereo cameras, the compensation surface calculation unit 134 may calculate a compensation surface (surface equation) for the water surface in each of the forward, backward, left, and right directions of the ship 1. In this case, the compensation surfaces for each direction are converted into a common coordinate system, and then these equations are fused using other techniques such as parameter averaging, weighted averaging, or a Kalman filter or its variant. Alternatively, the surface equations for each direction may be fused into one using a neural network. Alternatively, as a method that does not use fusion, all points in a three-dimensional point cloud obtained based on images captured by stereo cameras in each direction may be converted into a common coordinate system, and then a water surface compensation surface may be fitted to all the converted points simultaneously.

[0064] The distance calculation unit 135 calculates the distance between the ship 1 (in this embodiment, the reference point of the ship 1) and the detected object, based on the image in which the object is detected and the compensation plane calculated by the compensation plane calculation unit 134. Details of the processing by the distance calculation unit 135 will be described later in detail with reference to FIGS. 13 to 16.

[0065] <Distance Measuring Method> An example of a distance measuring method using the distance measuring device 10 will be described with reference to the flowchart of FIG.

[0066] Step S1: The image acquisition unit 131 acquires a pair of images captured by the stereo camera 3 and including the water surface.

[0067] Step S2: The object detection unit 132 detects whether or not an object exists on the water surface in the pair of images acquired in step S1. The object detection algorithm used in this step is not particularly limited. For example, in the image I of FIG. 6A, １ In this case, object T １ , T ２ , T ３ , T ４When an object is detected, the object detection unit 132 may identify the type of the object (such as a ship or a buoy). The object detection process may be performed on both images of the pair of images acquired in step S1, or on only one of the images.

[0068] In step S2, if an object on the water surface is detected (S2: Yes), the process proceeds to step S3. On the other hand, if an object on the water surface is not detected (S2: No), the process returns to step S1. Note that the images used for this determination may be both or either one of the pair of images.

[0069] Step S3: The 3D point cloud acquisition unit 133 acquires a three-dimensional point cloud of the water surface based on the pair of images acquired in step S1. Details of this step will be described with reference to the flowchart in FIG.

[0070] Step S31: The 3D point cloud acquisition unit 133 generates a disparity map based on a pair of images. Specifically, the disparity map is generated by processing such as stereo reconstruction (preferably high-density stereo reconstruction). The disparity map is expressed in coordinates of a corrected image (e.g., an image captured and corrected by the camera 3L) that serves as a reference for the stereo processing. Alternatively, the disparity map may be generated using AI.

[0071] Figure 8 shows a pair of images I Ｕ , I Ｌ These images are taken by cameras arranged one above the other. That is, image I Ｕ is the image captured by the upper camera, and I Ｌ are images taken by the lower camera. The bow of the ship 1 is visible near the bottom of each image. １ , TO ２ , TO ３ , TO ４ (All are ships) are shown.

[0072] FIG. 9 shows image I Ｕ , I Ｌ1 shows a disparity map DM generated based on the above. Areas with low disparity are displayed dark, and areas with high disparity are displayed bright.

[0073] Step S32: The 3D point cloud acquisition unit 133 performs segmentation processing on the image acquired in step S1 to extract pixels in the water surface region. The image used in this step is image I Ｕ and Image I Ｌ However, it is necessary to use a corrected image that serves as a reference for stereo processing to generate the disparity map DM. In this embodiment, an image I having a relatively wide water surface area is used. Ｕ FIG. 10A shows the image I Ｕ Image I obtained by performing segmentation processing on ｗ Image I ｗ So, Object TO １ , TO ２ , TO ３ , TO ４ In addition, water surface area A ｗ and Water Area A ｓ are displayed separately.

[0074] Step S33: The 3D point cloud acquisition unit 133 performs binary processing (binarization processing) on ​​the image obtained in step S32 to generate a mask (binary mask). ｗ In the mask M, the water surface area A ｗ The areas are white and the other areas are black.

[0075] Step S34: The 3D point cloud acquisition unit 133 applies the mask M generated in step S33 to the parallax map DM generated in step S31 to remove pixels that are not related to the water surface. That is, by applying the mask M to the parallax map DM, the water surface area A ｗ The area outside is masked.

[0076] Step S35: The 3D point cloud acquisition unit 133 performs 3D reprojection by triangulation to acquire a 3D point cloud of the water surface. That is, the 3D point cloud of the water surface is acquired by performing 3D reprojection processing on the portion of the parallax map DM related to the water surface. Fig. 11 is an example of a 3D point cloud map PM including a 3D point cloud of the water surface. Fig. 11 shows the 3D point cloud as viewed from above.

[0077] In this manner, a 3D point cloud of the water surface is acquired. Note that between steps S34 and S35, a process may be performed to remove invalid data or unnecessary data (e.g., negative values, values ​​outside the allowable range) from the parallax map. Alternatively, step S34 may be omitted, and after a 3D point cloud of the entire image region is obtained by 3D reprojection in step S35, only 3D points in the water surface region may be extracted.

[0078] In the above explanation, a 3D point cloud of the water surface was obtained using a pair of images (two images) taken by a stereo camera having two cameras, but even if the stereo camera 3 is a multi-view stereo camera having three or more cameras, a 3D point cloud can be obtained in a similar manner.

[0079] Returning to the flowchart of FIG. 5, the explanation of step S4 and subsequent steps will be continued.

[0080] Step S4: The compensation surface calculation unit 134 calculates a compensation surface (water surface compensation function) for the water surface based on the three-dimensional point cloud acquired in step S3. For example, as shown in FIG. 12 , the compensation surface CP for the water surface is calculated by fitting the compensation function to the three-dimensional point cloud (plurality of points P) using the least squares method or the like. The vector n in FIG. 12 is the normal vector of the compensation surface CP. In this example, the compensation surface is a plane, but the compensation surface may also be a curved surface such as an ellipsoid or sphere. Note that an algorithm such as RANSAC (Random Sample Consensus) may be used to perform fitting that is robust against outliers. The compensation surface may also be calculated using a trained model constructed by machine learning.

[0081] When a compensation surface is obtained using a method that is robust against outliers, such as RANSAC, the compensation function may be fitted to the complete 3D point cloud including the water surface region and the non-water surface region. In this case, steps S32 to S34 described above can be omitted.

[0082] The water surface compensation function determined in step S4 is, for example, an ellipse with two axes defined by a geodetic document or information source. The unknown parameters to be fitted are three translation coordinates and three rotation coordinates that define the orientation of the ellipse relative to the reference coordinate system.

[0083] The water surface compensation function is, for example, a function that represents a sphere with the mean Earth radius (approximately 6371 km) or other approximate Earth radius as defined in the geodetic literature. Such a sphere compensation function provides a good compromise between high accuracy of distance calculation and simplicity of the method. The equation of the sphere is given by Equation 1.

[0084]

[0085] Since the mean radius r of the Earth is known in advance, the position of the center of the sphere relative to the reference coordinate system [x ０ , y ０ , z ０ ] are the parameters found by fitting.

[0086] The water surface compensation function may be a function that provides an nth-order approximation of the Earth's surface. Such a function can be obtained, for example, by expanding an ellipsoidal or spherical function into a Taylor series around a reference point on the water surface (e.g., the center of the ship projected onto the water surface) and ignoring higher-order terms greater than n. For example, a quadratic function is a paraboloid, and a linear function is a plane. The equation of a plane is given by Equation 2. In this case, the four parameters to be determined by fitting are [a, b, c, d].

[0087]

[0088] The above approximation may be used for objects close to the ship 1, since it will produce large errors when calculating distances over a wide range.

[0089] It should be noted that the accuracy of the water surface compensation function may be improved by fusing it with other sources of information about the geometric characteristics of the water surface. For example, information from LIDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), IMU (Inertial Measurement Unit), INS (Inertial Navigation System), GPS (Global Positioning System), or other navigation satellite positioning systems may be used. For example, IMU measurements may reveal the direction of the gravity vector coaxial with the surface normal of the water surface compensation function at the point where the gravity vector intersects the water surface. GPS data (especially using RTK) may be used to correct for the vertical distance from the water surface. Similarly, the position and orientation of the horizon as seen in one or more camera images may be used to correct the water surface compensation function. The horizon is easily defined from a semantic or panoptic segmentation mask when the water surface and the sky are adjacent. Kalman filtering or other methods may be used to fuse various data sources.

[0090] Other 3D representation methods such as Neural Radiance Fields (NeRF) and Gaussian Splatting may also be used to represent the compensation surface.

[0091] Step S5: The distance calculation unit 135 measures the distance to the object based on the image used in step S2 and the compensation surface obtained in step S4. １ Object T １ An example of calculating the distance to will be described. Details of this step will be described with reference to the flowchart in FIG. 13 and FIGS.

[0092] Step S51: The distance calculation unit 135 finds the intersection of the camera ray and the compensation surface for the water surface found in step S4.

[0093] First, the camera ray will be described with reference to Fig. 14. In Fig. 14, the image I ２ is located at the focal length of the stereo camera 3. ２ corresponds to the image plane IP in Figure 16. As shown in Figure 14, the camera ray Rｃ is the point P of the stereo camera 3 １ and the waterline point P ２ The line connecting point P １ is the camera center (optical center) of the reference camera (for example, camera 3L of the stereo camera 3), and is the midpoint of the line connecting the centers of the lenses of camera 3L and camera 3R. ２ is image I ２ Object T １ Point P is a point on the waterline. ３ is the camera ray R ｃ and the compensation surface CP of the water surface.

[0094] In this embodiment, point P ２ is the object T １ Bounding box B that surrounds １ The waterline point is not limited to a point on the waterline of the target object or a point close to the waterline.

[0095] Referring to FIG. 16, point P １ , point P ２ and point P ３ The relationship between the coordinate axis X and ｃ , Y ｃ , Z ｃ defines the camera coordinate system (the coordinate system of the camera itself). １ indicates the origin of the camera coordinate system (the center of the camera). ２ The two-dimensional coordinates of point P are given by pixel coordinates (u, v) in the uv coordinate system of the image plane IP. ２ is converted into three-dimensional coordinates in the camera coordinate system. ３ is a point in the camera coordinate system, and point P １ and point P ２ The image plane IP is the plane of the image captured by the camera, also called the reference frame. The image plane IP is perpendicular to the optical axis of the camera and is located, for example, at the focal length of the camera (z=f). The principal point (C ｘ , C ｙ) is the intersection of the optical axis and the image plane IP, and in FIG. 16 is the origin of the camera coordinate system (xyz coordinate system) in the reference frame. The z-axis is the optical axis of the camera (the center point P １ Starting from the principal point (C ｘ , C ｙ ) is a straight line passing through the line . The image plane IP may be located at a position other than the focal length.

[0096] The distance calculation unit 135 calculates the camera ray R ｃ and the intersection point P with the compensation surface CP of the water surface ３ The method for determining the intersection between the camera ray and the compensation plane for the water surface will be described in detail below.

[0097] The camera ray in 3D space expressed in the camera coordinate system is derived by multiplying the inverse camera matrix (which is ideally corrected for lens distortion) by the homogeneous 2D pixel coordinate vector. This gives the origin of the camera coordinate system (point P in Figure 16). １ ) to the corresponding pixel (point P in FIG. 16 ２ ) is found. This vector is found as a homogeneous 4x1 array, where the first three entries are the x, y, and z coordinates, respectively.

[0098] The camera ray equation (ray equation) is expressed by Equation 3. Here, the vector p ｐ indicates the point through which the camera ray passes, and the vector v ｒ is the direction vector of the camera ray. Note that the subscript r denotes the camera ray vector transformed into the reference frame. The subscript p denotes the origin of the camera coordinate system in the reference frame (the same coordinate system in which the water surface is described).

[0099]

[0100] The intersection of the camera ray and the compensation surface can be found by substituting the equation of the compensation surface (such as the plane equation) into the ray equation, solving for the ray parameter t, and then substituting the resulting parameter t into the ray equation. If the water surface is represented as a plane or sphere, the parameter t can be easily found analytically as follows:

[0101] When the compensation surface is a plane represented by Equation 2, the parameter t is given by Equation 4.

[0102]

[0103] When the compensation surface is a spherical surface as expressed by Equation 1, the parameter t is given by Equation 5, where the vector c ｓ denotes the center of the sphere. r denotes the mean radius of the Earth.

[0104]

[0105] In addition, the vector v ｒ , vector p ｐ , vector c ｓ are as follows:

[0106]

[0107] The reference frame may be in the camera coordinate system of the camera selected for detecting the waterline of the object (the camera from which the ray originates). In this case, x p , y p , z p become zero, and the above equation simplifies significantly.

[0108] Also, if the compensation surface is spherical, there are at most two solutions for t. If there are two solutions, the smaller value of t (i.e., the intersection point closer to the camera) is selected.

[0109] Furthermore, when the compensating surface is expressed by a complex, generally nonlinear function such as an ellipse, a numerical solution for t may be obtained instead of an analytical solution. Note that even when the compensating surface is not complex, such as a flat or spherical surface, t may be obtained by numerical analysis.

[0110] Step S52: The distance calculation unit 135 calculates the distance between the reference point P ４ and intersection point P ３ Find the shortest line (geodesic) between the intersection point P ３ is a point on the compensation surface CP. ４In this embodiment, is a point on the waterline of the ship 1, but may be another point on the water surface. If the compensation surface for the water surface is calculated using a method other than planar approximation in step S4, the line obtained in this step will be a curved line rather than a straight line.

[0111] A reference point on the compensation surface of the water surface (for example, point P in FIG. 14) ４ ) must be selected or calculated. For example, the origin of the coordinates of the platform-mounted reference frame is projected onto the water surface along the normal to the water surface. This projection can be done analytically if the water surface compensation function is planar or spherical, since the normal vector n passing through the origin of the reference frame can be easily obtained.

[0112] When the compensation surface is a plane expressed by Equation 2, the normal vector n is given by Equation 7.

[0113]

[0114] When the compensation surface is a spherical surface as expressed by Equation 1, the normal vector is given by Equation 8. Here, the vector p ｃ is the reference point of the platform.

[0115]

[0116] Note that Equation 8 simplifies if the origin of the platform's reference frame coincides with the reference frame chosen to represent the water surface compensation function. In that case, the vector p ｃ All components of are 0.

[0117] The intersection of the water surface correction function with a ray emanating from the origin of the platform's reference frame in the direction of the surface normal of the water surface correction function is then calculated, e.g., using a procedure similar to that described above for finding the intersection of the camera ray with the water surface. The intersection is the reference point on the water surface.

[0118] Then, the reference point on the water surface (point P in FIG. 14) ４ ) and the intersection point of the camera ray passing through the water line point of the object and the water surface defined by the water surface compensation function (point P in FIG. 16 ). ３) and calculate the shortest path (geodesic) between the platform and the target object. This gives the distance from the platform to the target object. Generally, finding the shortest path requires solving a constraint optimization problem. However, if the compensation surface is flat or spherical, a simpler method can be used:

[0119] If the compensation surface is a plane as described by Equation 2, the shortest path is a straight line, the length of which is given by Equation 9. Here, the first term, vector (pPp), represents the point where the origin of the reference frame is projected onto the water surface (compensation surface). The second term, vector (pPr), represents the intersection of the water surface and the camera ray. Both vectors are expressed in the same arbitrary coordinate system, as derived above with respect to a common reference frame.

[0120]

[0121] When the compensating surface is a sphere expressed by Equation 1, the shortest path is a curve, and its length is given by Equation 10, where r is the mean radius of the Earth. The length calculated by Equation 10 is the length of an arc on a great circle that has the center of the Earth as its origin and passes through two points (the projection point and the intersection point).

[0122]

[0123] The distance calculation unit 135 may correct an error when approximating the water surface with an n-order curved surface. For example, when approximating the water surface with a plane (linear surface), in order to correct an error caused by ignoring the curvature of the Earth, the corrected distance D ｔ where r is the mean radius of the Earth.

[0124]

[0125] α and β in Equation 11 are given by Equations 12 and 13, respectively. D is the distance (approximate distance before correction) calculated assuming the water surface is a flat surface. H is the length (height) of the perpendicular line drawn from the stereo camera point to the water surface (flat surface).

[0126]

[0127]

[0128] The length of the shortest line calculated as above is the distance from the ship 1 to the object (for example, object T １ If multiple objects are detected in step S2, the distance from the ship 1 to each object may be calculated.

[0129] The processing unit 13 (for example, the distance calculation unit 135) may calculate the relative orientation of the target object with respect to the vessel 1. The relative orientation is the angle between the defined platform reference or the forward direction projected onto the water surface and the shortest path from the platform reference frame to the object on the water surface at the origin of the reference frame. This angle is calculated using Equation 14.

[0130]

[0131] Here, the vector pvp is given by Equation 15, and the vector pvr is given by Equation 16. The vector n is given by Equation 7 when the compensating surface is a plane, and by Equation 8 when the compensating surface is a spherical surface.

[0132]

[0133]

[0134] The processing unit 13 (e.g., the distance calculation unit 135) may also calculate the absolute position of the target object (such as a geographical position in a geodetic coordinate system) based on the distance and relative orientation calculated as described above. This calculation may be performed by obtaining the absolute position and orientation of the ship 1 on the Earth's surface using data from an additional sensor, such as a GPS receiver, an inertial navigation system based on a gyroscope or an IMU, or a compass. The absolute position of the object may be calculated by adding the relative position of the object to the position of the ship 1 using the Haversine equation, for example.

[0135] Furthermore, the processing unit 13 (for example, the distance calculation unit 135) may calculate the velocity of an object from the change in the absolute position of the object over time.

[0136] Information such as the distance to the object, the relative orientation, the absolute position and / or the velocity of the object calculated as described above may be transmitted from the distance measuring device 10 to the display device 4. The display device 4 displays the information received from the distance measuring device 10.

[0137] 17 shows an example of a screen displayed on the display device 4. In this example screen, the absolute positions of the ship 1 and the object T are calculated using the absolute positions calculated as described above. １ , T ２ , T ３ is displayed on the map. １ , L ２ is object T １ , T ２ The velocity vector of the object T ３ Since is a buoy and is not moving, no arrow indicating the velocity vector is displayed. Note that the trajectory of the object may be displayed on the screen based on the time change of the position information.

[0138] Additionally, calculated object information (e.g., relative position, absolute position) and other information extracted from sensor data may be fused with data and target attributes obtained from other sensors or sources, such as RADAR, LIDAR, AIS, etc. Well-known algorithms for sensor fusion include variations of Kalman filters and neural network-based methods, which can provide a more complete picture of the platform's environment and improve situational awareness, sensing accuracy, and reliability.

[0139] 18 shows another example of the screen displayed on the display device 4. Safe navigation area A on the screen ＳＮ indicates the area where the ship 1 can safely navigate. To obtain this area, the boundary contour of the mask for extracting the water surface area is identified, and the identified contour is approximated by a polygon. Then, the vertices of the polygon are converted into geospatial coordinates by applying the above-mentioned object positioning method. That is, the safe navigation area A ＳＮ The boundaries of indicate the boundaries of the visible water surface determined in the manner described above. In this way, knowledge of the location of the water surface area around the platform may be used to determine the navigable waters for the vessel.

[0140] <Operation and Effect> As described above, in this embodiment, a pair of images including the water surface are captured by the stereo camera 3 mounted on a platform such as the ship 1, and it is detected whether an object is present on the water surface in the acquired pair of images. If an object is present, a three-dimensional point cloud of the water surface is acquired based on the pair of images, a compensation surface for the water surface is calculated based on the acquired three-dimensional point cloud, and the distance between the platform and the object is calculated based on the image in which the object is detected and the compensation surface.

[0141] As a result, according to this embodiment, accurate distance measurement can be performed over a wide range from short distances to long distances.

[0142] Furthermore, according to this embodiment, since images captured by a camera are used, the resolution is higher than that of methods using radar or the like, and objects can be classified with high accuracy.

[0143] Furthermore, this embodiment has a higher real-time capability than the method using AIS, can target small objects, and there is no risk of information being tampered with.

[0144] In the above description, a pair of images (two images) captured by a stereo camera having two cameras is used, but distance measurement according to the above embodiment may also be performed using three or more images captured by a stereo camera having three or more cameras (multi-view stereo camera).

[0145] The invention according to this embodiment may be applied to part of the recognition, situational awareness, collision avoidance, navigation or control system of an autonomous surface vessel (ASV), also known as a USV (unmanned surface vessel) or a MASS (marine autonomous surface vessel).

[0146] Based on the above description, a person skilled in the art may conceive additional effects and various modifications of the present disclosure, but the aspects of the present disclosure are not limited to the individual embodiments described above. Elements from different embodiments may be combined as appropriate. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present disclosure, which is derived from the content defined in the claims and their equivalents.

[0147] A program that realizes at least some of the functions of the distance measuring system may be distributed via a communication line (including wireless communication) such as the Internet. Furthermore, the program may be encrypted, modulated, or compressed and distributed via a wired or wireless line such as the Internet, or stored on a recording medium.

[0148] REFERENCE SIGNS LIST 1 Ship 2 Mast section 3 Stereo camera 3L, 3R, 3LL, 3RL, 3LR, 3RR, 3LB, 3RB Camera 4 Display device 10 Distance measuring device 12 Storage section 13 Processing section 131 Image acquisition section 132 Object detection section 133 3D point cloud acquisition section 134 Compensation surface calculation section 135 Distance calculation section 1000 Distance measuring system A Ｗ Water surface area A Ｓ Water area A ＳＮ Safe navigation area B １ , B ２ , B ３ , B ４ Bounding box CP Compensation plane DM Disparity map IP Image plane I １ , I ２ , I Ｕ , I Ｌ , I ｗ Image L １ , L ２ Arrow M Mask P Point P １ , P ２ , P ３ , P ４ Point PM 3D point cloud map R ｃ Camera light T １ , T ２ , T ３ , T ４ object

Claims

1. A distance measuring device comprising: an image acquisition unit that acquires at least two images including the water surface taken by a stereo camera mounted on a platform; a 3D point cloud acquisition unit that acquires a three-dimensional point cloud of the water surface based on the at least two images in which an object on the water surface is detected; a compensation surface calculation unit that calculates a compensation surface of the water surface based on the three-dimensional point cloud; and a distance calculation unit that calculates the distance between the platform and the object based on the image in which the object is detected and the compensation surface.

2. The distance measuring device of claim 1, wherein the 3D point cloud acquisition unit generates a disparity map based on the at least two images, performs segmentation processing on one of the at least two images to extract pixels of the water surface area that constitutes the water surface, performs binary processing on the image from which the water surface area has been extracted to generate a mask, applies the mask to the disparity map to exclude pixels not related to the water surface, and performs three-dimensional reprojection to acquire a three-dimensional point cloud of the water surface.

3. The distance measuring device according to claim 1, wherein the compensation surface calculation unit calculates the compensation surface, which is planar or curved, by fitting a compensation function to the three-dimensional point cloud.

4. The distance measuring device according to claim 1, wherein the distance calculation unit determines the intersection of the compensation surface with a camera ray passing through a point of the stereo camera and a water line point of the object in the image, and determines the shortest line between the reference point of the platform and the intersection.

5. The distance measuring device according to claim 1, wherein, when the compensation surface is a plane, the distance calculation unit corrects the distance based on the calculated distance, the mean radius of the Earth, and the length of a water line drawn from the point of the stereo camera to the plane.

6. A distance measurement method comprising: acquiring at least two images including the water surface captured by a stereo camera mounted on a platform; acquiring a three-dimensional point cloud of the water surface based on the at least two images in which an object on the water surface is detected; calculating a compensation surface of the water surface based on the three-dimensional point cloud; and calculating the distance between the platform and the object based on the image in which the object is detected and the compensation surface.

7. A ranging program that causes a computer to perform the following processes: acquire at least two images including the water surface taken by a stereo camera mounted on a platform; acquire a three-dimensional point cloud of the water surface based on the at least two images in which an object on the water surface is detected; calculate a compensation surface of the water surface based on the three-dimensional point cloud; and calculate the distance between the platform and the object based on the image in which the object is detected and the compensation surface.

8. A ranging system comprising: a stereo camera mounted on a platform so as to be able to capture images including the water surface; and a ranging device that calculates the distance from the platform to an object on the water surface based on at least two images captured by the stereo camera, wherein the ranging device has: an image acquisition unit that acquires the at least two images; a 3D point cloud acquisition unit that acquires a three-dimensional point cloud of the water surface based on the at least two images in which the object is detected; a compensation surface calculation unit that calculates a compensation surface of the water surface based on the three-dimensional point cloud; and a distance calculation unit that calculates the distance between the platform and the object based on the images in which the object is detected and the compensation surface.

Images