Method and system for calibrating a camera

Abstract

A method and system for calibrating a camera (24) disposed on an ownship (70) is disclosed. The method includes receiving image data (ID) of an image (40) captured by the camera (24), the image (40) showing at least one imaged solid object (46, 48) and at least one imaged solid object interface (50, 52, 54, 56) between the imaged solid object (46, 48) and the imaged sky (46) or between the imaged solid object (44, 48) and the imaged water surface (42) of a body of water in which the ownship (70) is navigating, extracting the imaged solid object interface (50, 52, 54, 56) from the image (40), determining an attitude of the ownship (70) at the time the image (40) was captured, and receiving reference data (RD), the reference data (RD) including a reference to the imaged solid object interface (50, 52, 54, 56) in the image (40). and representing at least one real solid object interface (64, 82, 84, 86) in the real world (80) corresponding to the imaged solid object (50, 52, 54, 56), wherein the real solid object interface (64, 82, 84, 86) is between the corresponding real solid object (66, 94) and the real sky (96) or between the corresponding real solid object (66, 94) and the corresponding real water surface (92) of the body of water, respectively, and comprising: extracting the real solid object interface (64, 82, 84, 86) from reference data (RD); determining a difference between the extracted imaged solid object interface (50, 52, 54, 56) and the extracted real solid object interface (64, 82, 84, 86) taking into account the determined attitude of the own ship (70); and determining a calibration value (CV) of the camera (24) according to the difference so that the difference is reduced.

Description

[Technical Field]

[0001] The present invention relates to the field of camera calibration, in particular to a method and system for calibrating cameras located on an ownship, which may be used for example for monocular vision based target detection. [Background technology]

[0002] Camera calibration is a well-established process used to estimate a camera's intrinsic and extrinsic parameters, as well as optional lens distortion. The former parameters define the camera's internal properties, such as optical center, focal length, skew, and / or distortion, while the latter parameters define the camera's position and orientation in the real world. For many applications, e.g., navigation purposes, knowing these parameters is essential as they can contribute to defining a mapping between the camera's image coordinate system and the real world that can be used for navigation. For example, if a camera is placed on an ownship sailing over a body of water, this mapping may be used to estimate the distance and bearing of another vessel on the same body of water.

[0003] Manual camera calibration is a time-consuming and labor-intensive process. Furthermore, maintaining calibration across multiple cameras is a difficult task, as even slight changes in camera position require the calibration process to be repeated. While not as accurate as manual calibration, automatic calibration is a desirable alternative because it does not suffer from the same problems.

[0004] The Automatic Identification System (AIS) is currently in use by a significant number of vessels. In addition to the unique identifier, the system transmits other vessel-related information such as geographic location, surface course, and surface speed.

[0005] "Harbor surveillance with cameras calibrated with AIS data," by Palmieri F., Castaldo F., and Marino G., in the 2013 IEEE Aeronautics and Astronautics Conference (pp. 1-8), IEEE, March 2013, describes how to use corresponding geolocation information for camera calibration. However, the current prior art has two problems: it does not take into account the range of other vessels, and it does not take into account that the positions of GPS receivers on other vessels are not provided or are inaccurate. Therefore, a more accurate method for calibrating cameras is needed.

[0006] Calibrated cameras are also needed for other applications, such as determining the position of an object on the water surface of another ship using monocular vision. A known procedure involves calculating a ray or line in the real world corresponding to a given point in an image captured by the camera. To calculate this line, the camera calibration, including intrinsic parameters, extrinsic parameters, and lens distortion, must be known. Intrinsic parameters, such as focal length and principal point, as well as lens distortion characteristics, can be obtained before installing the camera on the ship using a known camera calibration procedure using a checkerboard pattern. Obtaining the camera's extrinsic parameters, which describe the camera's position and orientation on the ship, is more challenging. Orientation is challenging because it requires accuracy on the order of tenths of a degree. Position is also challenging due to the practical difficulty of making precise measurements on ship structures. Because manual calibration is time-consuming, accuracy depends on the experience of the person calibrating the camera, there may be currents, and inaccurate calibration leads to poor performance of the entire system, a corresponding automatic calibration is desirable.

[0007] The current method for obtaining at least a subset of the extrinsic parameters, i.e., pitch and roll for a known camera height, is to automatically detect the open sea horizon, essentially the water-air boundary, on the image captured by the camera to be calibrated. However, this method has several drawbacks. First, it is not possible to calibrate the camera position, but only the roll and pitch angles of the camera. Second, for this method to work, a clear view of the horizon (water-air boundary) is required. If the horizon is, for example, a land-air boundary, the camera cannot be calibrated because the altitude of the land is unknown. Summary of the Invention

[0008] It is an object of the present invention to provide a method for calibrating cameras located on an ownship that is highly accurate and / or that can be performed in a cost-effective manner, in particular automatically.

[0009] Another object of the present invention is to provide a system for calibrating cameras located on an own ship that is highly accurate and / or operates in a cost-effective manner, in particular automatically.

[0010] These objects are achieved by the subject matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

[0011] The object of the present invention is achieved by a method for calibrating a camera disposed on an ownship, the method comprising: receiving image data of an image captured by the camera, the image showing at least one imaged solid object and at least one imaged solid object interface between the imaged solid object and the imaged sky or between the imaged solid object and the imaged water surface of a body of water in which the ownship is navigating; extracting the imaged solid object interface from the image; determining an attitude of the ownship at the time the image was captured; receiving reference data, the reference data representing at least one real solid object interface in the real world corresponding to the imaged solid object interface in the image, where the real solid object interface is between the corresponding real solid object and the real sky or between the corresponding real solid object and the real water surface of the corresponding body of water, respectively; extracting the real solid object interface from the reference data; determining a difference between the extracted imaged solid object interface and the extracted real solid object interface taking into account the determined attitude of the ownship; and determining a calibration value for the camera according to the difference such that the difference is reduced (e.g., minimized).

[0012] Comparing real solid objects, in particular real interfaces between real solid objects and real water surfaces and / or between real solid objects and real sky, to imaged representations of the solid objects, i.e., imaged solid objects, in particular solid object interfaces, i.e., imaged solid object interfaces, for camera calibration purposes allows for automatic, and thereby cost-effective, accurate, calibration of the camera even when the horizon, i.e., the boundary between water and sky, is not visible. "Imaged" herein may mean that the corresponding objects are shown in the image, while "real" herein may mean that the corresponding objects are in the real world.

[0013] The image data may be received by an entity that performs the above method. The entity may be, for example, a general-purpose computer on the ownship or a system for calibrating a camera disposed on the ownship. The body of water may be a lake, ocean, sea, or river. The calibration values ​​may be external calibration values, internal calibration values, and / or distortion parameters, such as camera lens distortion. The calibration values ​​may comprise the position of the camera on the ownship and / or the orientation of the camera with respect to the ownship's ship coordinate system. The camera position may correspond to the position of the camera's center. The camera center may be given by the camera's pinhole. The calibration values ​​may be represented by the vector θ. The camera orientation may include yaw, pitch, and roll. The attitude of the ownship may be determined by a self-positioning system of the ownship. Determining the attitude of the ownship may include receiving attitude data from the self-positioning system representing the attitude of the ownship. The attitude may comprise the ownship's position, yaw, pitch, roll, and / or draft. The attitude may be determined by GPS, compass, and / or IMU data. The imaged solid object interfaces may be extracted by image processing techniques and / or neural networks, as described, for example, in unpublished patent application EP 22180131.9, while the real solid object interfaces may be extracted by image projection techniques and / or by coordinate transformation of the corresponding reference data.

[0014] The difference may be any mathematical concept suitable for describing or defining the deviation between an imaged solid object interface and a real solid object interface. In particular, the difference between the imaged solid object interface seen in the image and the projection of the real solid object interface on the image may be calculated. For example, the difference may be given by one or more distances between one or more points along the imaged solid object interface and corresponding points along the real solid object interface projected onto the image.

[0015] In general, the interface between a solid object and the surface of water, or between a solid object and the sky, may be a physical interface or a non-physical interface: a physical interface exists in the real world, and a non-physical interface exists only in the image and does not exist in the real world.

[0016] The solid object may be land or another ship. If the solid object is land, the solid object interface may be a physical land-water interface between the water surface and the land, such as the coastline of the land, or a physical land-air interface between the sky and the land, i.e., the skyline of the land. Alternatively, if the solid object is land, the solid object interface may be a non-physical land-water interface between the water surface and the land, such as the interface between the water surface and the altitude of the land above the coastline. If the solid object is another ship, the solid object interface may be a physical ship-water interface between the water surface and the other ship, such as the waterline of the other ship, or a physical ship-air interface between the sky and the other ship, i.e., the skyline of the other ship. Alternatively, if the other ship is a solid object, the solid object interface may be a non-physical ship-water interface between the water surface and a portion of the other ship above the waterline of the other ship. This may be valid for imaged solid object interfaces as well as for real solid object interfaces.

[0017] Real solid object interfaces, like imaged solid object interfaces, are also 2D lines. In particular, the interface between an imaged solid object and an imaged water surface, or between an imaged solid object and an imaged sky, is always a 2D line in the camera's 2D image, while the interface between a real solid object and a real water surface is always a 3D line, and the interface between a real solid object and a real sky is always a 3D surface in the real world. However, for the purposes of this invention and in this description, only the portion of the 3D line or 3D surface in the real world that corresponds to the 2D imaged solid object interface is used. Therefore, for the purposes of this invention and in this description, real solid object interfaces, like the corresponding imaged solid object interfaces, are also 2D lines, such as coastlines, waterlines, or skylines.

[0018] The method may be performed several times. For example, the method may be implemented as a loop. The calibration loop may include an optimization algorithm, such as a nonlinear optimization algorithm. The method, i.e., the loop, may be executed until a predetermined condition is met, for example, until the difference is less than a predetermined difference threshold. When the method is executed for the first time, a predetermined calibration value may be received from a memory of a system for calibrating the camera and adjusted to determine the calibration value of the camera. In this case, the predetermined calibration value may be pre-stored in memory, for example, by the system manufacturer, e.g., as a fixed starting value. Alternatively, a "best guess" calibration value or a calibration value determined from the last calibration may be used as a starting point when the method is executed for the first time. When the method is executed for a second or subsequent time during a calibration, the predetermined calibration value may be the adjusted calibration value from the previous loop. Thus, the method may provide a calibration loop for adjusting the calibration value.

[0019] According to one embodiment, extracting the imaged solid object interface from the image comprises extracting a path of the imaged solid object interface in the image, and extracting the real solid object interface from the reference data comprises extracting a path of the real solid object interface in the real world, wherein a difference between the imaged solid object interface and the real solid object interface corresponds to a difference between the extracted path of the imaged solid object interface and the extracted path of the real solid object interface. The extracted path of the imaged solid object interface may include a form and a position of the imaged solid object interface in the image, in particular coordinates of corresponding pixels in the image. The extracted path of the real solid object interface may include a form and a position of the real solid object interface in the real world, in particular coordinates of corresponding pixels in the real world.

[0020] According to one embodiment, determining a difference between a path of the imaged solid object interface and a path of the real solid object interface comprises transforming the path of the imaged solid object interface and / or the path of the real solid object interface into the same coordinate system and determining a misfit function between the imaged solid object interface and the real solid object interface in the coordinate system as the difference, and determining a calibration value of the camera such that the difference is reduced comprises determining a calibration value such that the misfit function is reduced, e.g., minimized. Alternatively, a cost function may be used instead of the misfit function. The "same" coordinate system may be a real-world coordinate system given by the reference data, an image coordinate system given by the camera and the image, or another coordinate system used to determine only the difference.

[0021] According to one embodiment, the predetermined calibration values ​​are varied such that the misfit function is minimized. If a cost function is used, the difference can be reduced by reducing, e.g., minimizing, the cost of the cost function.

[0022] According to one embodiment, when extracting the real solid object interface from the reference data, only the portion of the real solid object interface that is visible to the camera is extracted, and determining the difference between the imaged solid object interface and the real solid object interface comprises determining the difference between the imaged solid object interface and only the determined visible portion of the real solid object interface. The real solid object may comprise an interface of the real solid object with a water surface or sky that is not visible to the camera. Thus, the image may not contain an imaged solid object interface that corresponds to that real solid object interface. Determining these portions of the real solid object interface that are visible to the camera and determining the difference between the imaged solid object interface and the determined visible portion of the real solid object interface may only contribute to solving this problem.

[0023] According to one embodiment, a method includes determining a positional relationship between a camera and a real solid object interface from reference data and determining a portion of the real solid object interface that is visible to the camera according to the determined positional relationship. The positional relationship may be determined by transforming the positions of the camera and the real solid object interface into the same coordinate system, e.g., a 2D coordinate system. For example, the reference data may comprise 2D data, e.g., 2D map data, or 3D data, e.g., 3D map data, which may be transformed into 2D data, e.g., 2D map data, and the camera position may be transformed into a corresponding 2D map. The positional relationship may then be extractable and / or visible from the 2D map comprising a representation of the real solid object interface and the camera position. The positional relationship may be defined by one or more distances between the camera position and one or more points along the representation of the real solid object interface in the corresponding 2D map.

[0024] According to one embodiment, the reference data comprises 2D map data representing a 2D map of the surroundings of the vessel. The 2D map data may comprise 2D coordinates of one or more land polygons corresponding to one or more coastlines of a body of water. The 2D map data may be obtained from a digital electronic map, for example OpenStreetMap.

[0025] According to one embodiment, the portion of the real solid object interface that is visible to the camera is determined from the 2D map data according to the determined positional relationship between the camera and the real solid object interface by ray casting. The ray casting may be 2D uniform ray casting. The ray casting may contribute to determining the portion of the real solid object interface that is easily and / or accurately visible to the camera.

[0026] According to one embodiment, the reference data comprises AIS data. The AIS data may comprise vessel-related information relating to one or more other vessels proximate to the vessel. In particular, the AIS data may relate to at least one other vessel in the body of water, where the other vessel may be used as a real solid object. The vessel-related information may comprise the other vessel's unique identifier, geographic position, ground position, and ground speed. Furthermore, the AIS data may comprise the range of the other vessel relative to the geographic location of the GPS receiver. This allows the range of the other vessel to be taken into account relative to the geolocation of the GPS receiver when using a corresponding real solid object interface, i.e., the vessel-water interface between the other vessel and the water surface, to determine the calibration value.

[0027] According to one embodiment, the AIS data includes the geolocation of the other vessel's GPS receiver and the range of the other vessel relative to the geolocation of the other vessel's GPS receiver. From the range and geolocation of the other vessel's GPS receiver, a real vessel-water interface between the real water surface and the other vessel is estimated. Depending on the estimated real vessel-water interface and the camera position, the portion of the real vessel-water interface visible to the camera is determined. The visible portion of the real vessel-water interface is used as a real solid object interface for comparison with the corresponding imaged solid object interface. This contributes to a highly accurate determination of the calibration value. The real vessel-water interface may be determined depending on the range of the other vessel by constructing a bounding box, e.g., a perfect fit, into which the other vessel fits, and by constructing an ellipse, e.g., a perfect fit, into the bounding box, where the ellipse is transformed into the real-world coordinate system and used as the real vessel-water interface. Similarly, other geometric shapes that more accurately model the vessel's contour may be used besides an ellipse.

[0028] According to one embodiment, the reference data comprises elevation map data of the real land surrounding the body of water, which allows for determining real solid object interfaces, in particular real land-air interfaces. The elevation map data may represent a 3D map and / or topography of the land, and may be referred to as a Digital Elevation Model (DEM), as known in the art.

[0029] According to one embodiment, the portion of the real solid object interface visible to the camera is determined from the elevation map data by generating a textureless rendering image and extracting the portion from the textureless rendering image. The textureless rendering image may be generated by a renderer as a synthetic view of the land surrounding the water body and / or taking into account predetermined calibration values ​​and the attitude of the ownship. The textureless rendering image may comprise sky, land, and water surfaces in different colors. The visible portion may then be extracted by image processing as known in the art.

[0030] According to one embodiment, the calibration values ​​are adjusted until a predetermined condition is met. In other words, the steps of the method may be performed repeatedly until the predetermined condition is met. For each loop of the method, a new image may be captured by the camera and received by the system for calibrating the camera, a new imaged solid object interface may be extracted from the image, and a corresponding new real solid object interface may be extracted from the reference data. The predetermined condition may be that the difference is smaller than a predetermined difference threshold.

[0031] According to one embodiment, the calibration value is determined by adjusting a predetermined calibration value of the camera so that the difference is reduced, e.g., minimized. The predetermined calibration value may be stored in a memory of a system for calibrating a camera. The predetermined calibration value may be known by the system manufacturer or may be derived from the camera specifications of the camera and stored in memory. This may be advantageous when the method is being performed for the first time. Alternatively or additionally, the predetermined calibration value may be a final calibration value determined by the last calibration of the camera. This may be advantageous when the method has already been performed one or more times.

[0032] The object is achieved by a system for calibrating a camera arranged on an own ship. The system comprises a memory configured to store reference data and / or calibration values, e.g. predetermined calibration values, and a processing unit configured to perform the above method. Furthermore, the system may comprise or be part of the camera. Furthermore, the system may comprise means for receiving reference data, e.g. via radio and / or satellite communication.

[0033] It should be understood that the features of the method for calibrating a camera located on an own ship as described above and below may also be features of the system for calibrating a camera located on an own ship as described above and below.

[0034] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. [Brief explanation of the drawings]

[0035] The subject matter of the present invention will be explained in more detail in the following description with reference to exemplary embodiments illustrated in the accompanying drawings. [Figure 1] FIG. 1 shows a block diagram illustrating an exemplary embodiment of a system for calibrating cameras located on an ownship. [Figure 2]FIG. 2 shows a flowchart of an exemplary embodiment of a method for calibrating a camera. [Figure 3] FIG. 3 shows an example of an image showing another vessel in a body of water. [Figure 4] FIG. 4 shows an example picture of the real world. [Figure 5] Figure 5 shows an example of a visualization of the real ship-water interface estimated from AIS data. [Figure 6] FIG. 6 illustrates the principle for determining the portion of the estimated ship-water interface that is visible to the camera.

[0036] The reference signs used in the drawings and their meanings are listed in summary form in the list of reference signs. As a rule, identical parts in the figures are given the same reference signs. DETAILED DESCRIPTION OF THE INVENTION

[0037] 1 shows a block diagram illustrating an exemplary embodiment of a system for calibrating a camera 24 disposed on an ownship 70 (see FIG. 6). The system may comprise a positioning device 22 for determining position data PD of the ownship 70, a camera 24 for generating image data ID of an image 40 (see FIG. 3), an acquisition component 26 for acquiring reference data RD, an extraction component 28 for extracting calibration features CF from the image data ID, and a calibration loop 30 for determining a calibration value CV for calibrating the camera 24 according to the position data PD, the calibration features CF, and the reference data RD. Additionally, the system may comprise a memory (not shown) configured to store the reference data RD and / or the calibration value CV, and a processing unit (not shown) configured to perform a method for calibrating a camera 24 disposed on the ownship 70, for example as described with respect to FIG. 2.

[0038] The camera 24 is configured to capture one or more images 40 of the surroundings of the ownship 70 and generate corresponding image data ID. The positioning device 22 is configured to determine the attitude of the ownship 70 at the time the image 40 was captured and generate corresponding attitude data PD. The extraction component 28 is configured to extract calibration features CF from the image data ID, i.e., at least one imaged solid object interface from the image 40. The acquisition component 26 may be configured to acquire reference data RD from the system's memory or from an external source, for example via satellite, radio, and / or internet. The acquisition component 26 may comprise a separate extraction component for extracting at least one real solid object interface from the reference data RD.

[0039] The reference data RD represents at least one real solid object interface in the real world 80 (see FIG. 4 ) that corresponds to an imaged solid object interface in the image 40, where the real solid object interface is between the corresponding real solid object and the real sky 94, or between the corresponding real solid object and a corresponding real water surface 92 of a body of water. The processing unit is configured to receive the reference data RD and, taking into account the determined attitude of the ownship 70, determine a difference between the imaged solid object interface and the real solid object interface, and determine a calibration CF value for the camera 24 depending on the difference, such that the difference is reduced, e.g., minimized.

[0040] The functionality and / or operation of the system will be described in more detail below with respect to the method performed by the system, and the method steps shown in FIG. 2 will be explained with reference to FIGS.

[0041] 2 shows a flowchart of an exemplary embodiment of a method for calibrating a camera 24 disposed on an ownship 70. The ownship 70 on which the camera 24 is disposed may navigate a body of water, which may be a lake, an ocean, a sea, or a river.

[0042] In step S2, image data ID of an image 40 (see FIG. 3 ) captured by the camera 24 is received. The camera 24 may generate the image data ID representing the image 40. The image data ID may be sent to and / or received by a processing unit that executes a method for calibrating the camera 24. The image data ID may be stored in a memory of a system for calibrating the camera 24, where the memory is coupled to the processing unit. When the method is executed, the processing unit may receive the image data ID directly from the camera 24 or from the memory. The camera 24 may be calibrated with respect to its position and / or orientation relative to the ownship 70 by the above-described systems and methods.

[0043] 3 shows an example image 40 showing an imaged other vessel 48 that is visible from ownship 70 and that may be navigating on a body of water. The body of water may be represented in image 40 by the imaged water surface 42 of the body of water. Image 40 may show the imaged water surface 42, an imaged land area 44, e.g., two imaged land areas, an imaged sky 46, an imaged other vessel 48, and an imaged bounding box 58 surrounding the imaged other vessel 48.

[0044] The image 40 may show at least one imaged solid object and at least one imaged solid object interface between the imaged solid object and the imaged sky 46 or between the imaged solid object and the imaged water surface 42 of the body of water through which the ownship 70 is navigating. For example, the imaged solid object may be imaged land 44 or an imaged other vessel 48. In the case of imaged land 44 as the imaged solid object, the solid object interface may be an imaged land-water interface 50 between the imaged water surface 42 and the imaged land 44, i.e., an imaged coastline of the imaged land 44, or an imaged land-sky interface 52 between the imaged sky 46 and the imaged land 44, i.e., an imaged skyline of the imaged land 44. In the case of the imaged other vessel 48 as an imaged solid object, the imaged solid object interface may be an imaged vessel-water interface 54 between the imaged water surface 42 and the imaged other vessel 48, i.e., the imaged waterline of the imaged other vessel 48, or an imaged vessel-sky interface 56 between the imaged sky 46 and the imaged other vessel 48, i.e., the imaged skyline of the imaged other vessel 48.

[0045] In step S4, the imaged solid object interface, i.e., the imaged land-water interface 50, the imaged land-air interface 52, the imaged vessel-water interface 54, or the imaged vessel-air interface 56, is extracted from the image 40, particularly from the image data ID, for example by the extraction component 28. Extracting the imaged solid object interface from the image 40 may comprise extracting a path of the imaged solid object interface within the image 40. The path of the imaged solid object interface may include a form and a position of the imaged solid object interface within the image 40, where the image data ID may comprise pixel coordinates of corresponding pixels of the image 40. To extract the imaged solid object interface, particularly the corresponding pixel coordinates, an imaged bounding box 58 may be determined, and the path of the imaged solid object interface may be searched only within the bounding box 58.

[0046] Extraction component 28 may comprise a first neural network configured to receive image data ID and extract an imaged solid object interface from the image data ID. For example, if imaged vessel-water interface 54 is to be extracted and used as the imaged solid object interface, the imaged solid object interface may be extracted from image data ID by determining at least one vessel-water value for each pixel of image 40 from the image data ID, where the vessel-water value may represent a probability of the corresponding pixel indicating an imaged solid object—in this case, another imaged vessel 48, the imaged water surface 42, or the imaged sky 46. The imaged vessel-water interface 54 may then be determined from the vessel-water values ​​by one or more post-processing steps, such as edge detection and / or thresholding of the vessel-water values.

[0047] The ship-water value may be determined by semantic segmentation of the images 40. The semantic segmentation may be performed by a first neural network trained to distinguish between solid objects, water surfaces, and sky in an image, for example, by supervised learning using correspondingly labeled images. For example, the first neural network may be pre-trained by supervised learning using a volume of images 40 showing imaged solid objects, e.g., other ships 48 on a body of water, and labeled with respect to the corresponding imaged water surfaces 42, imaged sky 46, and imaged other ships 48. Additionally, images 40 that do not show any solid objects may also be involved in training the first neural network.

[0048] A method for extracting solid object interfaces, in particular ship-water interfaces, from images that can be used in this context is known from and described in detail in unpublished patent application EP 22180131.9. Although this patent application focuses only on the extraction of the ship-water interface, the corresponding method can be easily extended by a person skilled in the art to the extraction of solid object interfaces in general, for example by training a first neural network to be able to determine other solid object interfaces, i.e., the imaged land-water interface 50, the imaged land-air interface 52, or the imaged ship-air interface 56.

[0049] In step S6, the attitude of the ownship 70 at the time the image 40 was captured is determined. Determining the attitude of the ownship 70 may comprise receiving attitude data PD from the positioning device 22, the attitude representing the attitude of the ownship 70. The attitude may comprise the position, yaw, pitch, roll, and / or draft of the ownship 70.

[0050] In step S8, reference data RD is received. The reference data RD represents at least one real solid object interface in the real world 80 that corresponds to an imaged solid object interface in the image 40. The reference data RD may comprise a path of the real solid object interface in the real world 80. The path of the real solid object interface may include the form and position of the real solid object interface in the real world 80. For example, the real world coordinates of the path of the real solid object interface may be encoded in the reference data RD.

[0051] In step S9, the real solid object interface, particularly the path of the real solid object interface, is extracted from the reference data RD. The extracted real solid object interface, particularly the extracted path of the real solid object interface, essentially corresponds to the calibration features CF extracted from the image data, i.e., the corresponding imaged solid object interface, except for differences described below. The method for extracting the real solid object interface may depend on the type of reference data RD that may be used. For example, if the reference data RD includes a 2D map of the surroundings of the ownship 70, the real solid object interface may correspond to a coastline represented in the 2D map, and the real solid object interface may be extracted from the corresponding 2D map by simply extracting corresponding coordinates from the 2D map. Alternatively, if the reference data RD includes AIS data, such as that described with reference to FIG. 5, the real solid object interface may correspond to the waterline of the corresponding other vessel 48, and the waterline is determined from the AIS data, as described below. Alternatively, if the reference data RD comprises elevation map data, for example as described with respect to FIG. 4, the real solid object interface may correspond to a corresponding land skyline, or a physical or non-physical water-land interface, as described below.

[0052] 4 shows an exemplary picture of a real world 80, e.g., as provided by reference data RD, e.g., by an elevation map. The portion of real world 80 shown in the picture of FIG. 4 does not correspond to the portion of real world 80 shown in image 40, but is shown to illustrate at least some of the real solid object interfaces usable in the present invention. Generally, the real solid object interfaces are between the corresponding real solid objects and real sky 96, or between the corresponding real solid objects and the corresponding real water surface 92 of a body of water. In this example, the real solid objects could be real land 94, or other real vessels (not shown in FIG. 4) that correspond, for example, to the imaged other vessels 48 (shown in FIG. 3). In the case of real land 94 as a real solid object, the real solid object interface may be a first real land-water interface 82 (continuous line) between the real water surface 92 and the real land 94, e.g., the real coastline of the real land 94, also referred to as a physical land-water interface, or a second real land-water interface 84 (dashed line) between the real water surface 92 and the real land 94, e.g., the line between the altitude of the real land 94 and the real water surface 92, which is also referred to as a non-physical land-water interface because it is only the land-water interface as seen from the ownship 70, not in the real world. Alternatively, the real solid object interface may be a real land-air interface 86 (dashed-dotted line) between the real sky 96 and the real land 94, i.e., the real skyline of the real land 94. If the real solid object is a real other ship, the interface of the real solid object may be a real ship-water interface (not shown) between the real water surface 92 and the real other ship, e.g., the real waterline of the real other ship, or a real ship-sky interface (not shown) between the real sky 96 and the real other ship, i.e., the real skyline of the real other ship.

[0053] In optional step S10, the portions of real solid object interfaces visible to camera 24 may be determined. A real solid object, such as real land 94, may comprise a real water surface 92 or a real sky 96 interface that is not visible to camera 24. Thus, image 40 may not include and / or show an imaged solid object interface that corresponds to that real solid object interface. For example, a first real land-water interface 82 comprises a portion of the coastline of the real land 94 visible to camera 24, but does not include a portion of the coastline of the real land 94 that is covered by the altitude of the real land 94. Thus, the first real land-water interface 82 is occluded by a second real land-water interface 84. The second real land-water interface 84 does not comprise the coastline of the real land 94, but only comprises a 2D line between the real water surface 92 and the altitude of the real land 94 as seen from ownship 70. Furthermore, only a portion of the real skyline of real land 94 may be visible to camera 24, for example, real land-sky interface 86 as shown in Figure 4. The portion of the real solid object interface visible to ownship 70, and in particular camera 24, may be determined by ray casting, as described below with respect to Figure 6.

[0054] In optional step S12, which may be performed after optional step S10, in particular, a positional relationship between the camera 24 and the real solid object interface may be determined from the reference data RD. In this case, the portion of the real solid object interface visible to the camera 24 may be determined according to the determined positional relationship. The positional relationship may be determined by transforming the position of the camera 24 and the real solid object interface into the same coordinate system, e.g., a 2D coordinate system, by any suitable coordinate transformation method known in the art. For example, the reference data RD may comprise 2D data, e.g., 2D map data, or 3D data, e.g., 3D map data, which may be transformed into 2D data, e.g., 2D map data. The position of the camera 24 may then be transferred to the corresponding 2D map. The positional relationship may then be extractable and / or visible from the 2D map comprising a representation of the real solid object interface and the position of the camera 24. The positional relationship may be defined by one or more distances between the position of the camera 24 and one or more points along the representation of the real solid object interface in the corresponding 2D map. The reference data RD may therefore comprise 2D map data representing a 2D map of the surroundings of the ownship 70, in particular the camera 24. The 2D map data may comprise 2D coordinates of one or more land polygons corresponding to one or more coastlines in the body of water. The 2D map data may be obtained from OpenStreetMap.

[0055] The 2D map data may also be used to extract real solid object interfaces, in particular visible real solid object interfaces, from the corresponding 2D map, which may then be projected onto the image 40.

[0056] Alternatively, the reference data RD may comprise AIS data. In general, the AIS data may comprise vessel-related information regarding one or more other vessels proximate to the own vessel 70. In particular, the AIS data may comprise the geographic location of the GPS receiver 62 of the real other vessel 66 (see FIG. 5 ) that corresponds to the imaged other vessel 48. The vessel-related information may comprise a unique identifier, a ground track, and a ground speed of the real other vessel 66, as well as the geographic location of the GPS receiver 62. Furthermore, the AIS data may comprise the range of the real other vessel 66 relative to the geographic location of its GPS receiver 62.

[0057] FIG. 5 shows an example visualization of a real ship-water interface estimated from AIS data, i.e., estimated real ship-water interface 64. The range of the corresponding real other ship 66 may be given by a width to starboard WS, a width to port WP, a length to bow LB, and / or a length to stern LS, each of which is relative to the position of the GPS receiver 62. The estimated real ship-water interface 64 may be determined by constructing, e.g., a perfect-fit AIS bounding box 60 into which the real other ship 66 fits, depending on the range of the real other ship 66, and by constructing, e.g., a perfect-fit ellipse, into the bounding box, where the ellipse may correspond to the estimated real ship-water interface 64, may be transformed into a real-world coordinate system, and may be used as the real ship-water interface. Alternatively, forms other than an ellipse may be used to estimate the real ship-water interface. Furthermore, the shape extracted from AIS data may be augmented with corresponding information from radar and / or lidar measurements.

[0058] Thus, the real vessel-water interface between the real water surface 92 and the real other vessel 66 may be estimated from the range and geolocation of the real other vessel 66 corresponding to the imaged other vessel 48. The portion of the real vessel-water interface visible to the camera 24 may be determined as a function of the estimated real vessel-water interface 64 and the position of the camera 24. The real vessel-water interface 64, and in particular the visible portion of the real vessel-water interface 64, may be projected onto the image 40 and used to determine the difference between the imaged solid object interface and the real solid object interface.

[0059] Alternatively, the reference data RD may comprise elevation map data of the real land 94 surrounding the body of water. The elevation map data may represent a 3D map and / or topography of the real land 94 and may be referred to as a Digital Elevation Model (DEM), as known in the art. In this case, the portions of the real solid object interfaces visible to the camera 24 may be determined from the elevation map data by generating a textureless rendered image and extracting the portions of the real solid object interfaces visible to the camera 24 from the textureless rendered image. The textureless rendered image may be generated by a renderer and / or as a synthetic view of the real land 94 surrounding the body of water, taking into account, for example, a predetermined calibration value of the camera 24 and the attitude of the ownship 70. The textureless rendered image may comprise representations of the real sky 96, the real land 94, and the real water surface 92, having different characteristics, for example, different colors. The visible portions of the real solid object interfaces visible to the camera 24 may then be extracted by image processing, as known in the art. The extracted real solid object interface, particularly the visible portion of the real solid object interface, may be projected onto the image 40 and used to determine the difference between the imaged solid object interface and the real solid object interface.

[0060] In step S12, a difference between the imaged solid object interface and the real solid object interface is determined, taking into account the determined attitude of the ownship 70. If optional step S10 is performed, only the difference between the imaged solid object interface and the determined visible portion of the real solid object interface may be determined. This difference may be any mathematical concept suitable for describing or defining the deviation of the imaged solid object interface from the corresponding real solid object interface. In particular, the difference between the imaged solid object interface seen in the image 40 and the projection of the real solid object interface on the image 40 may be calculated. For example, the difference may be given by one or more distances between one or more points along the imaged solid object interface, e.g., along the path of the imaged solid object interface, and corresponding points along the real solid object interface projected on the image 40, e.g., along the path of the real solid object interface projected on the image 40. In general, the difference between the imaged solid object interface and the real solid object interface may correspond to the difference between the path of the imaged solid object interface and the path of the real solid object interface transformed into the same coordinate system, e.g., the image coordinate system. Determining the difference between the path of the imaged solid object interface and the path of the real solid object interface may comprise transforming the path of the imaged solid object interface and / or the path of the real solid object interface into the same coordinate system, and determining a misfit function between the imaged solid object interface and the real solid object interface in the coordinate system as the difference.

[0061] For example, the coordinates of one or more of the calibration features CF extracted from image 40, i.e., the solid object interfaces mentioned above, may be given by:

number

[0062] Another set of points expressed in world coordinates can be obtained from a discretization of the visible part of the real solid object interface given by the reference data RD as follows:

number

[0063] Then, the distance from the closest calibration feature CF, i.e., the closest imaged solid object interface, to the corresponding extracted reference data RD, i.e., the corresponding real solid object interface, can be given by:

number

number

[0064] The vector θ is defined by the following equation:

number

[0065] In this context, it must be mentioned that methods for projecting real-world 3D points onto a 2D image 40 are already known in the art, for example using calibration values ​​from the aforementioned reference, i.e., "Multiple View Geometry in Computer Vision", where the projection of real-world points onto the image of an ideal camera is described in Chapter 6, page 154, and how to correct for lens distortion is described in Chapter 7, page 191.

[0066] In step S14, a calibration value CV of the camera 24 may be determined depending on the difference so that the difference is reduced, e.g., minimized. For example, the misfit and / or cost function may be defined as follows:

number

number

[0067] From this optimization, a set of improved calibration values ​​θ* may be determined, which may be improved calibration values ​​CV that may be further improved in subsequent iterations.

[0068] Alternatively, the Frechet distance can be used to determine the difference between the imaged solid object interface and the real solid object interface.

[0069] The calibration value CV may be determined by adjusting a predetermined calibration value of the camera 24 so that the difference is reduced, e.g., minimized. The predetermined calibration value may be stored in a memory of a system for calibrating the camera 24. The predetermined calibration value may be stored in a memory by a manufacturer of the system. Alternatively or additionally, the predetermined calibration value may be a final calibration value CV determined by the last calibration of the camera 24, or may be a "best guess" calibration value.

[0070] The calibration value CV may be adjusted until a predetermined condition is met. In other words, the steps of the method may be performed repeatedly until the predetermined condition is met. For each loop of the method, a new image 40 may be captured by the camera 24 and received by the system for calibrating the camera 24, a new imaged solid object interface may be extracted from the image 40, and a corresponding new real solid object interface may be extracted from the reference data RD. The predetermined condition may be that the difference is less than a predetermined difference threshold.

[0071] FIG. 6 illustrates the principle for determining the portion of the estimated vessel-water interface that is visible to camera 24, i.e., the visible vessel-water interface 74, which is shown by the solid line in FIG. 6 . Camera 24 is located on ownship 70. Camera 24 has a field of view 72 in which the estimated real vessel-water interface 64 is located, where field of view 72 may be larger than depicted in FIG. 6 , and in particular larger than the estimated real vessel-water interface 64. As can be seen from FIG. 6 , the visible vessel-water interface 74 is visible to camera 24, but the remainder of the estimated real vessel-water interface 64 (dashed line) is not visible to camera 24. Therefore, the visible vessel-water interface 74 may be used to determine only the calibration value CV. Numerical methods can be used to distinguish between the visible vessel-water interface 74 and the remainder of the vessel-water interface. For example, a point on the entire vessel-water interface and a line segment connecting that point to the camera center can be defined. Points of the vessel-water interface that have associated line segments that do not intersect with other parts of the vessel-water interface then belong to the visible vessel-water interface 74. The visible vessel-water interface 74 may be determined by ray casting.

[0072] The neural networks and components described above, such as the first neural network, the acquisition component 26, the extraction component 28, and / or the calibration loop 30, may each be implemented by software, hardware, or a combination of software and hardware.

[0073] While the invention has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description are to be considered exemplary or illustrative and not restrictive, and the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processing unit or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Any reference signs in the claims are not to be construed as limiting the scope.

[0074] List of Reference Numbers 20 Calibration System 22 Positioning Device 24 Camera 26 Acquisition Components 28 Extraction Component 30 Calibration Loop 40 images 42 Imaged water surface 44 Land Imaged 46 Photographed sky 48 Other vessels photographed 50 Imaged land-water interfaces 52 Imaged land-sky interface 54 Imaged ship-water interface 56 Imaged ship-air interface 58 Imaged bounding boxes 60 AIS bounding boxes 62 GPS receiver 64 Estimated Realistic Ship-Water Interface 66 Other real ships 70 Own ship 72 Field of view 74 Visible ship-water interface 80 Real World (Picture) 82 The First Real Land-Water Interface 84 The Second Reality Land-Water Interface 86 Real Land-Air Interface 88 Horizontal line 90 Elevation Contours 92 The Surface of Reality 94 Real Land 96 Real Sky S2‐S14 Steps 1 to 14 PD Position Data ID image data RD Reference Data CF Calibration Features CV calibration value WS Width to starboard Width to WP port LB Bow Length LS Length to stern

Claims

1. A method for calibrating a camera (24) located on an own ship (70), comprising the steps of: receiving image data (ID) of an image (40) captured by the camera (24), the image (40) showing at least one imaged solid object (44, 48) and at least one imaged solid object interface (50, 52, 54, 56) between the imaged solid object (44, 48) and the imaged sky (46) or between the imaged solid object (44, 48) and an imaged water surface (42) of a body of water in which the vessel (70) is navigating; extracting the imaged solid object interfaces (50, 52, 54, 56) from the image data (ID); determining the attitude of the ownship (70) at the time the image (40) was captured; receiving reference data (RD), the reference data (RD) representing at least one real solid object interface (64, 82, 84, 86) in the real world (80) corresponding to the imaged solid object interface (50, 52, 54, 56) in the image (40), wherein the real solid object interface (64, 82, 84, 86) is between the corresponding real solid object (66, 94) and the real sky (96) or between the corresponding real solid object (66, 94) and the corresponding real water surface (92) of the body of water; Extracting the real solid object interface (64, 82, 84, 86) from the reference data (RD); determining a difference between the extracted imaged solid object interface (50, 52, 54, 56) and the extracted real solid object interface (64, 82, 84, 86) taking into account the determined attitude of the ownship (70); determining a calibration value (CV) of the camera (24) in response to the difference, such that the difference is reduced.

2. Extracting the imaged solid object interfaces (50, 52, 54, 56) from the image (40) comprises extracting a path of the imaged solid object interfaces (50, 52, 54, 56) within the image (40); extracting the real solid object interface (64, 82, 84, 86) from the reference data (RD) comprises extracting a path of the real solid object interface (64, 82, 84, 86) from the reference data (RD); 2. The method of claim 1, wherein a difference between the imaged solid object interface (50, 52, 54, 56) and the real solid object interface (64, 82, 84, 86) corresponds to a difference between the path of the imaged solid object interface (50, 52, 54, 56) and the path of the real solid object interface (64, 82, 84, 86).

3. determining the difference between the path of the imaged solid object interface (50, 52, 54, 56) and the path of the real solid object interface (64, 82, 84, 86) comprises: transferring the path of the imaged solid object interface (50, 52, 54, 56) and / or the path of the real solid object interface (64, 82, 84, 86) to the same coordinate system; and determining a misfit function between the imaged solid object interface (50, 52, 54, 56) and the real solid object interface (64, 82, 84, 86) in the coordinate system as the difference in the coordinate system; 3. The method of claim 2, wherein determining the calibration value (CV) of the camera (24) such that the difference is reduced comprises determining the calibration value (CV) such that the misfit function is reduced.

4. The method of claim 3 , wherein a predetermined calibration value (CV) is varied such that the misfit function is minimized.

5. when extracting the real solid object interface (64, 82, 84, 86) from the reference data (RD), only the part of the real solid object interface (64, 82, 84, 86) visible to the camera (24) is extracted; 5. The method of claim 1, wherein determining a difference between the imaged solid object interface (50, 52, 54, 56) and the real solid object interface (64, 82, 84, 86) comprises determining a difference between the imaged solid object interface (50, 52, 54, 56) and only the extracted visible portion of the real solid object interface (64, 82, 84, 86).

6. determining a positional relationship between the camera (24) and the real solid object interface (64, 82, 84, 86) from the reference data (RD); and determining a portion of the real solid object interface (64, 82, 84, 86) visible to the camera (24) in response to the determined positional relationship.

7. The method of any one of claims 1 to 6, wherein the reference data (RD) comprises 2D map data representing a 2D map of the surroundings of the own ship (70).

8. 8. The method of claim 7, which is dependent on claim 6, wherein the portion of the real solid object interface (64, 82, 84, 86) visible to the camera (24) is determined from the 2D map data by ray casting according to a determined positional relationship between the camera (24) and the real solid object interface (64, 82, 84, 86).

9. 7. The method of any one of claims 1 to 6, wherein the reference data (RD) comprises AIS data for at least one other vessel (48) in the body of water.

10. the AIS data comprising a geolocation of a GPS receiver (62) of the other vessel (48) and a range (WS, WP, LB, LS) of the other vessel (48) relative to the geolocation of the GPS receiver (62) of the other vessel (48); a real vessel-water interface (64) between the real water surface (92) and the other vessel (48) is estimated from the ranges (WS, WP, LB, LS) and the geolocation of the GPS receiver (62) of the other vessel (48); a portion (74) of the real vessel-water interface (64) visible to the camera (24) is determined as a function of the estimated real vessel-water interface (64) and the position of the camera (24); 10. The method of claim 9, wherein a visible portion (74) of the real vessel-water interface (64) is used as the real solid object interface to be compared with the corresponding imaged solid object interface (54).

11. 7. The method of any one of claims 1 to 6, wherein the reference data (RD) comprises elevation map data of the real land (94) bordering the body of water.

12. 12. The method of claim 11, which relies on claim 6, wherein the portion of the real solid object interface (82, 84, 86) visible to the camera (24) is determined from elevation map data by generating a textureless rendering image and extracting the portion from the textureless rendering image.

13. 13. The method of any one of claims 1 to 12, wherein the calibration value (CV) is adjusted until a predetermined condition is met.

14. 14. The method of any one of claims 1 to 13, wherein the calibration value (CV) is determined by adjusting a predetermined calibration value (CV) of the camera (24) so ​​that the difference is reduced.

15. A system (20) for calibrating a camera (24) arranged on an own ship (70), the system (20) comprising a memory configured to store reference data (RD) and / or calibration values ​​(CV), and a processing unit configured to perform the method according to any one of claims 1 to 14.

Images