Neural network estimation of the distance to ocean objects using a camera

Abstract

A system and method are provided, comprising: obtaining at least one image of an ocean object acquired by an imaging device on an ocean vessel using a processor and memory circuit implementing at least one neural network; feeding at least one image to at least one neural network; and using at least one neural network to estimate the distance between the ocean vessel and the ocean object.

Description

[Technical Field]

[0001] The subject matter of this disclosure relates to the field of the marine environment. [Background technology]

[0002] In a marine environment, marine vessels travel along routes where they may encounter a variety of conditions. Some of these conditions may include hazards, such as obstacles that must be avoided or zones with dangerous weather.

[0003] To enable the control of a marine vessel (by crew and / or autopilot), it is necessary to determine information about marine objects that the vessel encounters.

[0004] It is now necessary to provide new methods and systems to improve the safety and reliability of ocean navigation, to improve the understanding of the marine environment for ocean vessels, and to improve the control of ocean vessels. More generally, it is necessary to develop innovative methods in the marine sector, and more specifically in the field of autonomous vessels.

[0005] overview In particular aspects of the subject matter of the present disclosure, a system is provided comprising a processor and memory circuit (PMC), the PMC being operable to implement at least one neural network, and the PMC being configured to obtain at least one image of an ocean object acquired by an imaging device of an ocean vessel, to feed the image to at least one neural network, and to use at least one neural network to estimate the distance between the ocean vessel and the ocean object.

[0006] In addition to the features described above, a system according to this aspect of the subject matter of the present disclosure may optionally include one or more of the following features (i) to (xvii) in any technically possible combination or rearrangement. i. The system obtains an image of at least one marine object acquired by the imaging device of a marine vessel, and data D that provides information about the height of the imaging device during the period in which the image was acquired. height To obtain the image and data D height It is configured to feed a signal to at least one neural network and to use at least one neural network to estimate the distance between a marine vessel and a marine object. ii. The system is configured to use at least one neural network to determine several different candidate distances between marine vessels and marine objects using images. iii. The system is configured to obtain multiple images of an ocean object acquired by an imaging device on an ocean vessel at different moments in a given period; feed each of the multiple images into at least one neural network; use at least one neural network to estimate one or more candidate distances between the ocean vessel and the ocean object for each of the multiple images, thereby obtaining a set of candidate distances for a given period based on the multiple images; and use the set of candidate distances to estimate the distance between the ocean vessel and the ocean object within a given period. iv. The system provides data D that gives information about the frequency distribution of candidate distances within the set of candidate distances. distribution It is configured to determine and to use the frequency distribution to estimate the distance between a marine vessel and a marine object. v. The system provides data D, which gives information about the expected movement of ocean objects. physical To obtain and estimate the distance between ocean vessels and ocean objects, data D physical It is configured to use and perform vi. The system obtains multiple images of an ocean object acquired by the ocean vessel's imaging device at different moments in a given period, feeds each of the multiple images into at least one neural network, and uses at least one neural network to estimate one or more candidate distances between the ocean vessel and the ocean object for each of the multiple images, thereby obtaining a set of candidate distances for a given period based on the multiple images, and determines whether at least one of the candidate distances in the set of candidate distances is acceptable as an estimate of the distance between the ocean vessel and the ocean object, using data D physical It is configured to use and perform vii. The system is configured to obtain at least one image of an ocean object acquired by an imaging device on an ocean vessel, to feed the image to at least one neural network, and to use at least one neural network to determine a distribution including multiple candidate distances, each of which is associated with a probability, and to generate data indicating that at least one neural network must be retrained if it detects that the distribution does not meet the criteria. viii. The system is configured to determine that distance estimation performed by at least one neural network does not meet quality criteria for one or more given images of one or more oceanic objects acquired by imaging devices corresponding to the same given scenario. ix. The system is configured to retrain at least one neural network with a training set of images that match a given scenario. x. The system is configured to identify a given scenario using data associated with a given image, the data including at least one of (a), (b), (c), (d), or (e): (a) the type of one or more marine objects, (b) data providing information about the environment in which the given image was acquired, (c) a range of distances in which one or more marine objects are located, (d) data providing information about the orientation of one or more marine objects as they appear in the given image, and (e) a range of heights of the imaging device in which the given image was acquired. xi. The system obtains an image of at least one marine object acquired by an imaging device on a marine vessel, and data D that provides information about the environment in which the image was acquired. environmental To obtain the image and data D environmental It is configured to feed a signal to at least one neural network and to use at least one neural network to estimate the distance between a marine vessel and a marine object. xii. The system is configured to provide distance to the controller of the ocean vessel, and the controller is able to operate using the distance to control the trajectory of the ocean vessel. xiii. The system is configured to obtain one or more images of an ocean object, to feed one or more images to at least one neural network, and to use at least one neural network to estimate the distance between the ocean vessel and the ocean object, wherein all one or more images of the ocean object used by the neural network to estimate the distance between the ocean vessel and the ocean object are acquired by the same single imaging device on the ocean vessel. xiv. The system is configured to estimate the distance between a marine vessel and a marine object using images of the marine object acquired by the marine vessel's imaging device, without using data that provides information about the marine object acquired by a sensor different from the imaging device, by using at least one neural network. xv. At least one neural network is trained using a set of images and labeled data, where each given image in the set of images includes a given marine object acquired by a given imaging device of a given marine vessel, and the labeled data includes, for each given image, an estimated value of the distance between the given marine vessel and the given marine object, and an estimated value of the height of the imaging device during the period when the given image was acquired. xvi. The system obtains at least one image of a marine object acquired by an imaging device of a marine vessel, and obtains data D that provides information regarding the orientation of the imaging device during the period when the image was acquired. orientation obtains the image and data D orientation and supplies the image and data D to at least one neural network, and is configured to use at least one neural network to estimate the distance between the marine vessel and the marine object. xvii. The neural network is trained using a set of images and labeled data, where each given image in the set of images includes a given marine object acquired by a given imaging device of a given marine vessel, and the labeled data includes, for each given image, an estimation of the distance between the given marine vessel and the given marine object, and the labeled data further includes, for at least one given image, data that provides information regarding the orientation of the given marine object as displayed in the given image.

[0007] In particular aspects of the subject matter of the present disclosure, a system is provided comprising a processor and memory circuit (PMC), the PMC being operable to implement at least one neural network, the PMC being configured to obtain a plurality of images, each given image of the plurality of images comprising a given ocean object acquired by a given imaging device of a given ocean vessel, each given image being associated with a given label indicating the distance between the ocean vessel and the given ocean object, and the PMC being configured to obtain a plurality of images and to feed each given image of the plurality of images to the at least one neural network with a given label in order to train the at least one neural network, the at least one neural network being available after training to estimate the distance between the ocean vessel and the ocean object using images of the ocean object acquired by the imaging device of the ocean vessel.

[0008] In addition to the features described above, a system conforming to this aspect of the subject matter of this disclosure may optionally include one or more of the following features (xviii) to (xxiv) in any technically possible combination or rearrangement. xviii. The system provides data D for each given image in a set of multiple images, which gives information about the height of a given imaging device during the period in which the given image was acquired. height With the given image and data D obtained, height It is configured to supply this to at least one neural network for its training. The xix system is configured to obtain a given label indicating the distance between a given ocean vessel and a given ocean object, using data provided by at least one of (a) a sensor of a given ocean vessel different from the imaging device, or (b) a sensor of a given ocean object, for a given image of multiple images. xx. The system is configured to obtain first position data that gives information about the position of a first ocean object, the first position data being obtained based on a given image acquired by a given imaging device of a given ocean vessel; obtain second position data that gives information about the position of a second ocean object, the second position data being obtained based on data acquired by at least one sensor of a given ocean vessel, the at least one sensor being different from a given imaging device, the at least one sensor being the same as at least some of the first ocean objects being the same as at least some of the second ocean objects; determine that one ocean object of the first ocean object and one ocean object of the second ocean object correspond to the same given ocean object acquired by the given imaging device and at least one sensor, respectively; and determine a given label indicating a given distance between a given ocean vessel and a given ocean object using data provided by at least one sensor. xxi. The system provides (a) data D that gives information about the environment in which the given image was acquired, for a given image from a set of images. environmental With the given image and data D obtained, environmental (b) providing a neural network with data D that gives information about the orientation of a given ocean object as shown in a given image. orientation Obtaining data, the given image and data D orientation It is configured to supply this to at least one neural network for training. xxii. The system is configured to determine if the distance estimation by the neural network does not meet quality standards for a given scenario in which a given image of a given ocean object is acquired by an imaging device, and then retrain the neural network with a training set of images that match the given scenario. xxiii. After training, the neural network can be used to estimate the distance between a marine vessel and a marine object using images of marine objects acquired by the marine vessel's imaging device, without using data that provides information about marine objects acquired by sensors different from the imaging device. xxiv. After training, the neural network is available to estimate the distance between a marine vessel and a marine object using one or more images of marine objects acquired by the marine vessel's imaging device, and all one or more images of marine objects used by the neural network to estimate the distance between the marine vessel and the marine object are acquired by the same single imaging device on the marine vessel.

[0009] According to a particular aspect of the subject matter of this disclosure, a system comprising a processor and memory circuit (PMC) wherein the PMC obtains first position data that gives information relating to the position of a first ocean object, the first position data being obtained based on a given image acquired by a given imaging device of a given ocean vessel, and obtains second position data that gives information relating to the position of a second ocean object, the second position data being obtained based on data acquired by at least one sensor of the given ocean vessel, the at least one sensor being different from the given imaging device, and at least one of the first ocean object A system is provided which is operable to obtain, determine that one ocean object of the first ocean object and one ocean object of the second ocean object correspond to the same given ocean object acquired by a given imaging device and at least one sensor, respectively, determine a given distance between a given ocean vessel and a given ocean object using data provided by at least one sensor, and generate a labeled image, the labeled image comprising a given image and a label including the given distance.

[0010] In addition to the features described above, a system conforming to this aspect of the subject matter of this disclosure may optionally include one or more of the following features (xxv) to (xxvii) in any technically possible combination or rearrangement. xxv. The system provides data D that gives information about at least one of the height and orientation of the imaging device during the period in which a given image was acquired by the imaging device. camera It is configured to determine, and data D camera According to the criteria, D camera The system optimizes the agreement between at least some of the positions of the first oceanic objects determined using the first position data and at least some of the second position data of the second oceanic objects. xxvi. The system is configured to determine the type of a given marine vessel using data provided by at least one sensor and to associate the type with a labeled image. xxvii. At least one sensor is not an imaging device, and / or at least one sensor includes at least one of radar and an automatic identification system (AIS), and / or at least one sensor includes a first sensor and a second sensor, the second sensor being of a different type from the first sensor, and the first sensor and the second sensor are not imaging devices.

[0011] A method is provided which, according to a particular aspect of the subject matter of the present disclosure, includes: obtaining an image of at least one ocean object acquired by an imaging device of an ocean vessel using a processor and memory circuit (PMC) implementing at least one neural network; feeding the image to at least one neural network; and using at least one neural network to estimate the distance between the ocean vessel and the ocean object.

[0012] In addition to the features described above, a method following this aspect of the subject matter of this disclosure may optionally implement one or more of the features (i) to (xvii) described above (as described with reference to the corresponding system) in any technically possible combination or rearrangement.

[0013] In particular aspects of the subject matter of this disclosure, a non-temporary storage device is provided that is machine-readable and tangibly embodies a program of machine-executable instructions for performing an operation, the operation of which is described with reference to the above method.

[0014] In particular aspects of the subject matter of the present disclosure, a method is provided comprising obtaining a plurality of images by a processor and memory circuit (PMC) implementing at least one neural network, wherein each given image of the plurality of images includes a given ocean object acquired by a given imaging device of a given ocean vessel, and each given image is associated with a given label indicating the distance between the given ocean vessel and the given ocean object; and feeding each given image of the plurality of images and a given label to a neural network for training at least one neural network, wherein, after training, the at least one neural network is available to estimate the distance between the ocean vessel and the ocean object using images of the ocean object acquired by the ocean vessel's imaging device.

[0015] In addition to the features described above, a method following this aspect of the subject matter of this disclosure may optionally implement one or more of the features described above (xviii) to (xxiv) (as described with reference to the corresponding system) in any technically possible combination or rearrangement.

[0016] In particular aspects of the subject matter of this disclosure, a non-temporary storage device is provided that is machine-readable and tangibly embodies a program of machine-executable instructions for performing an operation, the operation of which is described with reference to the above method.

[0017] According to a particular aspect of the subject matter of this disclosure, a method comprising obtaining, by a processor and memory circuit (PMC), first position data that provides information relating to the position of a first ocean object, wherein the first position data is obtained based on a given image acquired by a given imaging device of a given ocean vessel, and second position data that provides information relating to the position of a second ocean object, wherein the second position data is obtained based on data acquired by at least one sensor of the given ocean vessel, wherein the at least one sensor, unlike the given imaging device, of the first ocean object A method is provided which includes obtaining second position data, at least some of which are the same as at least some of the second ocean objects; determining that one ocean object of the first ocean object and one ocean object of the second ocean object correspond to the same given ocean object acquired by a given imaging device and at least one sensor, respectively; determining a given distance between a given ocean vessel and a given ocean object using data provided by at least one sensor; and generating a labeled image, wherein the labeled image includes a given image and a label including the given distance.

[0018] In addition to the features described above, a method following this aspect of the subject matter of this disclosure may optionally implement one or more of the features (xxv) to (xxvii) described above (as described with reference to the corresponding system) in any technically possible combination or rearrangement.

[0019] In particular aspects of the subject matter of this disclosure, a non-temporary storage device is provided that is machine-readable and tangibly embodies a program of machine-executable instructions for performing an operation, the operation of which is described with reference to the above method.

[0020] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object in an accurate and efficient manner.

[0021] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object using a single camera.

[0022] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object in real time or near real time.

[0023] According to some embodiments, the proposed solution allows for the estimation of the distance between a marine vessel and a marine object using multiple cameras.

[0024] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object using only images acquired by a camera, without requiring input from other sensors.

[0025] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object using images acquired by a camera, even when the representation of the marine object in the image has a limited size.

[0026] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object using images acquired by a camera, while being substantially undetectable to the movement of the marine vessel due to the movement of the camera and / or the presence of waves.

[0027] According to some embodiments, the proposed solution enables the estimation of distances between marine vessels and marine objects, even over long distances.

[0028] According to some embodiments, the proposed solution enables the estimation of the distance between a marine vessel and a marine object while taking environmental conditions into account.

[0029] According to some embodiments, the proposed solution can be easily deployed on offshore vessels.

[0030] According to some embodiments, the proposed solution automatically generates a dataset of labeled images of oceanic objects that can be used to train a neural network to determine the distance to the oceanic objects using the labeled images of the oceanic objects.

[0031] According to some embodiments, the proposed solution allows for evaluation of the contexts in which the neural network is inefficient in predicting the distance to an ocean object, in order to retrain the neural network with a training set focused on this context.

[0032] According to some embodiments, the proposed solution enables accurate and efficient distance estimation in the maritime domain, which is a challenging technical area because it can include adverse environmental conditions and changes in the height of marine vessels over time. Conversely, standard methods for distance estimation (such as triangulation and stereoscopic vision) cannot be used and are inaccurate and impractical in the maritime domain. [Brief explanation of the drawing]

[0033] To understand the present invention and how it can be put into practice, embodiments are described with reference to the accompanying drawings as non-limiting examples. [Figure 1] The following are examples of a system that can be used to perform one or more of the methods described below. [Figure 2A] This document illustrates an embodiment of a method for determining the distance to an ocean object using images acquired by an imaging device on an ocean vessel. [Figure 2B] This document illustrates an example of images of marine objects acquired by imaging devices on marine vessels. [Figure 2C] A schematic diagram illustrating the distance between a marine vessel and an object at sea will be used as an example. [Figure 2D] This diagram illustrates the effect of the height of an imaging device on estimating the distance between a marine vessel and an ocean object. [Figure 2E] An example of the distribution of candidate distances estimated by a neural network is illustrated. [Figure 3A] This document illustrates an embodiment of a method for determining the distance to an ocean object using multiple images acquired by an imaging device on an ocean vessel. [Figure 3B] This paper illustrates examples of multiple distributions of candidate distances estimated over time by a neural network. [Figure 3C] An embodiment of a system that uses multiple distributions of candidate distances estimated over time by a neural network to determine candidate distances is illustrated. [Figure 3D] To estimate candidate distances, we illustrate an embodiment of a method for determining the frequency distribution within a set of candidate distances. [Figure 4] This document illustrates an embodiment of a method for training a neural network to estimate the distance to an ocean object using images acquired by an imaging device on an ocean vessel. [Figure 5] This example illustrates the influence of the orientation of marine objects on their representation in images. [Figure 6A] This document illustrates an example of a method for evaluating the performance of a trained neural network. [Figure 6B] This example illustrates a distribution of candidate distances output by a trained neural network that demonstrates the performance of the trained neural network does not meet quality standards. [Figure 7]An embodiment of a method for determining data that provides information regarding the height and / or orientation of an imaging device on an offshore vessel is illustrated. [Figure 7A] A modified version of the method shown in Figure 7 will be illustrated as an example. [Figure 7B] This document illustrates an example of tracking a marine object across multiple images acquired by an imaging device on a marine vessel. [Figure 7C] This document illustrates a method for determining data that provides information about the height and / or orientation of imaging devices on marine vessels over time. [Figure 7D] An example of an action that can be performed according to the method shown in Figure 7 will be illustrated. [Figure 7E] This paper schematically illustrates the projection of the positions of marine objects acquired by an imaging device and those acquired by another sensor onto a common reference. [Figure 7F] This document illustrates an embodiment of a method for projecting the location of marine objects from an image (independently of the image) onto a global / absolute reference. [Figure 7G] The parameters that can be used in the method shown in Figure 7F are illustrated below. [Figure 7H] The parameters that can be used in the method shown in Figure 7F are illustrated below. [Figure 7I] The parameters that can be used in the method shown in Figure 7F are illustrated below. [Figure 7J] The parameters that can be used in the method shown in Figure 7F are illustrated below. [Figure 7K] An example of the output of the method shown in Figure 7D will be illustrated. [Figure 7L] An example of an action that can be performed according to the method shown in Figure 7A will be illustrated. [Figure 7M] To improve the resolution of the optimization problem in the method shown in Figure 7L, we illustrate an embodiment that uses ocean object tracking data. [Figure 7N] To improve the resolution of the optimization problem in the method shown in Figure 7L, we illustrate an embodiment that uses ocean object tracking data. [Figure 8A]To improve the resolution of the optimization problem in the methods of Figures 7D and 7L, an embodiment using the type of ocean object is illustrated. [Figure 8B] To improve the resolution of the optimization problem in the methods of Figures 7D and 7L, an embodiment using the type of ocean object is illustrated. [Figure 9A] An example of the method shown in Figure 7 will be illustrated. [Figure 9B] Another embodiment of the method shown in Figure 7 will be illustrated. [Figure 10] This document illustrates an embodiment of a method for automatically generating labeled images of ocean objects, associated with their distance. [Modes for carrying out the invention]

[0034] The following detailed description includes numerous specific details to provide a complete understanding of the invention. However, it will be understood by those skilled in the art that the subject matter of this disclosure can be implemented without these specific details. In other cases, well-known methods are not described in detail so as not to obscure the subject matter of this disclosure.

[0035] Unless otherwise specifically stated, as will be apparent from the following considerations, throughout this specification, any consideration using terms such as “obtain,” “use,” “supply,” “determine,” “estimate,” and “generate” refers to computer operations and / or processes that manipulate and / or transform data into other data, where such data is expressed as a physical quantity such as an electronic quantity, and / or where such data represents a physical object.

[0036] The terms “computer” or “computerized system” should be interpreted broadly to include any kind of hardware-based electronic device having data processing circuits (e.g., digital signal processors (DSPs), GPUs, TPUs, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), microcontrollers, microprocessors, etc.). A processing circuit may comprise one or more processors operably connected to computer memory, loaded with executable instructions for performing operations, as further described below. A processing circuit may comprise a single processor or multiple processors, which may be located in the same geographical zone, or at least partially in different zones, and may communicate together.

[0037] Figure 1 illustrates an embodiment of a computerized system 100 that can be used to perform one or more of the methods described below. As shown, the system 100 comprises a processor and a memory circuit (PMC) 110.

[0038] System 100 can be embedded in an offshore platform. Specifically, the offshore platform can be a mobile offshore platform. The mobile offshore platform can be, for example, an offshore vessel 125. Examples of offshore vessels include ships, boats, hovercraft, etc.

[0039] In some embodiments, the system 100 can be embedded on an offshore platform that can be stationary, or at least temporarily stationary.

[0040] The embodiments are described with reference to the ocean vessel 125, but it should be understood that these embodiments are similarly applicable to stationary ocean platforms.

[0041] As shown in Figure 1, system 100 can obtain data from one or more sensors 130. At least some of the sensors 130 can be located on the offshore vessel 125 on which system 100 is located (or on at least one other offshore vessel / object that communicates with the offshore vessel 125 on which system 100 is located).

[0042] The sensor 130 collects data during the voyage of the ocean vessel 125. The voyage includes a portion of the voyage in motion, but may also include a portion of the voyage in which the ocean vessel is substantially stationary (for example, when the ocean vessel 125 is moored or docked in a port).

[0043] The sensor 130 includes an imaging device 120 (e.g., a camera) mounted on the marine vessel 125.

[0044] In some embodiments, the camera may be an infrared camera, a night camera, or a day camera.

[0045] In some embodiments, the sensor 130 includes a plurality of imaging devices 120 (which may be separate). In some embodiments, the imaging devices 120 may have different fields of view (which do not overlap at all) or may have fields of view that at least partially overlap.

[0046] According to some embodiments, the sensor 130 may include one or more additional sensors 115 (not necessarily imaging devices) such as radar (any type of radar), LiDAR, Automatic Identification System (AIS) (located on the ocean vessel 125 and / or on an ocean object separate from the ocean vessel 125), a transponder communicating with GPS located on another ocean object, a system including a laser located on the ocean vessel 125, and an optical reflector located on another ocean object located by the ocean vessel 125 (the reflection of the laser by the reflector enables the location of the other ocean object) (this list is not limiting).

[0047] Specifically, the sensor 115 provides information that can be used to locate marine objects surrounding the marine vessel 125.

[0048] The marine vessel 125 itself may include a geolocation system (e.g., GPS), an IMU, speed and acceleration sensors, a gyrocompass, and other sensors.

[0049] As described below in this specification, the system 100 can process data collected by one or more of the sensors 130.

[0050] In some embodiments, data output by system 100 can be transmitted via a telecommunications network 140 to a central station 150, which may include, for example, at least one processor and memory circuit (PMC).

[0051] In some embodiments, the central station 150 can perform at least some of the tasks of the PMC 110 located on the offshore vessel 125.

[0052] The remote communication link can support, for example, broadband cellular networks (e.g., 4G, 5G, LTE, etc.), satellite communication networks, and wireless communication networks (e.g., wireless VHF - very high frequency).

[0053] The data can be transmitted using a communication system located on a marine vessel 125 suitable for transmitting data over a telecommunications network. Examples of such communication systems include antennas, emitters, and transponders.

[0054] As shown in Figure 1, the PMC110 processor can be configured to implement a neural network (NN) 160. In some embodiments, the neural network 160 can be a deep neural network.

[0055] Specifically, the processor can execute several computer-readable instructions implemented on the computer-readable memory provided in the PMC, and the execution of these computer-readable instructions enables data processing by a neural network. As described below, the NN160 enables data processing of one or more images of one or more oceanic objects in order to output distance information of one or more oceanic objects.

[0056] In non-limiting embodiments, the layers of NN160 can be organized according to a convolutional neural network (CNN) architecture, a recurrent neural network architecture, a generative adversarial network (GAN) architecture, or other methods. In some embodiments, at least some of the layers can be organized into multiple DNN subnetworks. Each layer of a DNN may contain multiple basic computational elements (CEs), typically referred to in the art as dimensions, neurons, or nodes.

[0057] Generally, the computational elements of a given layer can be connected to the CEs of preceding and / or succeeding layers. Each connection between the CE of a preceding layer and the CE of a succeeding layer is associated with a weighting value. A given CE can receive inputs from the CE of the previous layer through each connection, and each given connection is associated with a weighting value that can be applied to the input of the given connection. The weighting value can determine the relative strength of the connections, and therefore the relative influence of each input on the output of a given CE. A given CE can be configured to calculate an activation value (e.g., a weighted sum of inputs) and further derive an output by applying an activation function to the calculated activation. The activation function can be, for example, an identity function, a decision-making function (e.g., linear, sigmoid, threshold, etc.), a probability function, or other suitable function. The output from a given CE can be transmitted to the CE of a succeeding layer through each connection. Similarly, as described above, each connection at the output of a CE can be associated with a weighting value that can be applied to the output of the CE before it is received as an input to the CE of a succeeding layer. In addition to weighted values, thresholds (including limiting functions) associated with connections and CEs may exist.

[0058] System 100 can be used to perform one or more of the methods described below.

[0059] Now, let's focus on Figure 2A.

[0060] The method includes obtaining an image of at least one of the marine objects acquired by the imaging device 120 of the marine vessel 125 (operation 200).

[0061] In some embodiments, the image may include multiple different marine objects.

[0062] Examples of marine objects include other marine vessels, icebergs, and buoys. Marine objects generally include at least a portion of them that are located above sea level.

[0063] A non-limiting embodiment of Figure 280 is illustrated in Figure 2B, where an imaging device (not shown in Figure 2B) mounted on an offshore vessel 125 acquires an image of an offshore object 270.

[0064] The method further includes supplying the image to the neural network 160 (operation 210).

[0065] According to some embodiments, the images are preprocessed before being fed to the neural network 160. Specifically, the PMC (PMC 110, etc.) can execute an algorithm configured to detect marine objects present in the images acquired by the imaging device 120.

[0066] This detection can rely, for example, on image processing algorithms. In some embodiments, a machine learning module 159 (for example, implementing a neural network such as a deep neural network) is trained to detect marine objects present in images acquired by an imaging device on a marine vessel. In some embodiments, the machine learning network implemented in the machine learning module 159 can be separate from the neural network 160, but this is not required.

[0067] This training may include supervised learning in which multiple annotated images containing ocean objects are fed to a machine learning module. Each annotated image may include, for example, a bounding box provided by an operator indicating the location of the ocean object in the image.

[0068] This is not limited to automated training and / or unsupervised learning.

[0069] In some embodiments, the machine learning module 159 can provide information about the type of object (e.g., ocean vessels, types of ocean vessels, types of ocean objects such as icebergs, etc.). This can be obtained by performing supervised learning on the machine learning module, where labeled images containing ocean objects (along with their types corresponding to the labels) are supplied to the machine learning module 159 for its training.

[0070] The output for identifying marine objects in an image may include a geometric representation (e.g., a bounding box) indicating the estimated location of each marine object in the image. A non-limiting embodiment is provided in Figure 2B, where a bounding box 275 surrounds a marine vessel 270 in image 280.

[0071] In some embodiments, the neural network 160 can be trained on its own to identify marine objects in images received from the imaging device 120 (or another imaging device).

[0072] As seen in Figure 2A, the method further includes using a neural network 160 (operation 220) to estimate the distance between the ocean vessel 125 and the ocean object. An example of such a distance is drawn in Figure 2C, which corresponds to the distance 276 between the ocean vessel 125 and the ocean object 270 (the distance can be expressed along the horizontal direction at a constant altitude, for example, as drawn in Figure 2C, but this is not limiting). If the sea level is considered to be the XY plane and the Z axis corresponds to the height from the sea level, the distance can be expressed, for example, between a point on the sea level located at the same X / Y coordinates as the imaging device 120 of the ocean vessel 125 and the ocean object 270. This is not limiting, and other rules can be used.

[0073] In some embodiments, the distance is estimated from the center of the geometric representation (e.g., bounding box) surrounding the ocean vessel 125 and the ocean object 270 in the image.

[0074] As described below, the neural network 160 is pre-trained to output an estimated distance between the ocean vessel (where the imaging device acquiring the image is located) and the ocean object, based on the image of the ocean object.

[0075] According to some embodiments, the neural network 160 provides an estimate of the distance between the marine vessel 125 and the marine object in real time or near real time after obtaining an image from the imaging device 120.

[0076] The distance determined by the neural network 160 can be output to the user (e.g., using a display device such as a screen) and / or can be provided to another system, such as an autopilot system for the ocean vessel 125, which uses the estimated distance to control the trajectory of the ocean vessel 125 over time. The autopilot system can generate commands to actuators (e.g., motors, rudders) to control the trajectory of the ocean vessel 125. For example, if the distance to an ocean object is below a threshold (indicating a potential collision), the autopilot system can send commands to the actuators of the ocean vessel 125 to change its trajectory, thereby avoiding a collision. This embodiment is not limiting.

[0077] According to some embodiments, the neural network 160 provides data D that gives information about the orientation and / or height of the imaging device 120 during the period in which the image was acquired (in addition to the image of the ocean object acquired by the imaging device 120 of the ocean vessel 125). camera It is supplied (operation 230).

[0078] The neural network 160 provides an estimate of the distance to the ocean object using data D camera This can be taken into consideration.

[0079] In some embodiments, this additional input can improve the accuracy of estimating the distance between the ocean vessel 125 and the ocean object.

[0080] Figure 2B illustrates an example in which the height of the imaging device 120 affects the display of marine objects in the image.

[0081] In the two images of Figure 2B, the distance 276 to the ocean object 270 is the same. However, the height of the imaging device 120 is not the same. In the lower part of Figure 2D, the imaging device 120 is mounted on a ship 126 that is larger in size than the ship 125 illustrated in the upper part of Figure 2D. Therefore, the height H2 of the imaging device 120 on ship 126 is higher than the height H1 of the imaging device 120 on ship 125.

[0082] This is because, since distance estimation depends on the image from the imaging device 120, the same ocean object 270 (located at the same distance) can appear differently in the images acquired by the imaging device 120, which can affect distance estimation.

[0083] In some embodiments, the neural network 160 can supply the height (and / or orientation) of the imaging device 120 during the period in which the image is acquired in order to prevent the height of the imaging device 120 from altering the distance estimate.

[0084] As described below, in some embodiments, the estimated height (and / or orientation) can be supplied to the neural network 160 during training. Thus, the neural network 160 can learn to predict the distance to the ocean object, while the height of the imaging device (and subsequently the representation of the ocean object in the image) can change.

[0085] In some embodiments, the estimated height (and / or orientation) can be fed to a neural network 160 during the prediction phase (the neural network 160 estimates the distance to the ocean object based on one or more images).

[0086] It should be noted that the height of the imaging device 120 may also change over time for a given vessel due to changes in wave height at sea, changes in cargo on the vessel, changes in the number of passengers on the vessel, etc. Estimation of the height of the imaging device 120 can be performed and used (as input to the neural network 160) to prevent changes in height from altering the distance estimation.

[0087] In some embodiments, for each image acquired at time T, the corresponding estimate of the height (and / or orientation) of the imaging device 120 at time T (or close to T at time T1, if there is a processing delay) is fed to the neural network 160.

[0088] D camera Embodiments for determining (including the height and / or orientation of the imaging device) are provided below.

[0089] According to some embodiments, the neural network 160 receives data D that provides information about the environment in which the image was acquired (in addition to the image of the marine object acquired by the imaging device 120 of the marine vessel 125). environmental It is supplied (see operation 240).

[0090] The neural network 160 provides an estimate of the distance to the ocean object using data D environmental This can be taken into consideration.

[0091] In some embodiments, this additional input can improve the accuracy of estimating the distance between the ocean vessel 125 and the ocean object.

[0092] D environmentalThis could include data that provides information about environmental conditions, such as whether it is daytime or nighttime, and weather conditions (such as sunny, foggy, cloudy, or rainy).

[0093] D environmental This may include data characterizing the environment in which the image was acquired (for example, whether the image was taken near a port, canal, coast, or distant coast).

[0094] D environmental This can be obtained, for example, using sensors on the marine vessel 125 and / or by data provided by a third party. For example, meteorological conditions can be measured using temperature sensors, pressure sensors, or provided by a third party. Data providing information about the type of environment encountered by the marine vessel 125 can be determined, for example, using the location of the marine vessel 125 (e.g., using GPS / AIS) and maps providing information about the type of environment at sea (such as mapping the location of a port). These embodiments are not limiting.

[0095] It should be noted that the environment in which the image was acquired (such as weather conditions) affects the distance estimation process. In fact, the visibility of marine objects in an image is not the same depending on the environmental conditions. Determining the distance to a marine object is more difficult under low visibility conditions (e.g., at night and / or in the presence of fog) than under high visibility conditions (e.g., during the day in the open sea). Data D (in prediction and / or training) environmental The use of this helps enable the neural network 160 to estimate the distance to ocean objects even under challenging environmental conditions.

[0096] According to some embodiments, the neural network 160 can determine multiple candidate distances between a marine vessel 125 and a marine object for a given image. For each candidate distance, the neural network 160 can provide the probability that the candidate distance is a correct estimate of the distance between the marine vessel and the marine object.

[0097] A non-restrictive example is shown in Figure 2E, where for each candidate distance (the X-axis of the graph corresponds to the estimated distance value D between the ocean vessel 125 and the ocean object), a probability (the Y-axis of the graph corresponds to the probability P associated with each distance value) is provided. A higher value of P indicates a higher probability that the corresponding distance value is correct according to the neural network 160.

[0098] In some embodiments, the distance output by the neural network 160 can be selected as the candidate distance with the highest probability. This is not limiting. In some embodiments, statistical analysis can be performed on the probability distribution of the candidate distances. The statistical analysis includes, for example, 1-sigma, 2-sigma, etc. (or other statistical methods).

[0099] It should be noted that the method in Figure 2A can be performed in parallel for multiple different ocean objects. For example, suppose there are multiple N different ocean objects in the image. The pixel location of each ocean object can be identified in the image (as described above). The image (along with the location of each ocean object in the image) can be fed into the neural network 160, which outputs an estimate of the distance from the ocean vessel 125 to each given ocean object.

[0100] According to some embodiments, additional data can be supplied to the neural network 160 to enable distance estimation, such as the type of ocean object. The type of ocean object can be inferred from the image using an image processing algorithm or information provided by another sensor.

[0101] As can be seen from the method in Figure 2A, according to some embodiments, it is possible to determine the distance to an ocean object using data acquired by an imaging device without using data that provides information about the ocean object acquired by a sensor other than the imaging device (e.g., AIS, radar, etc.). In some embodiments, the distance to the ocean object can be determined using only data acquired by the imaging device.

[0102] According to some embodiments, the neural network 160 is supplied with one or more images of a given ocean object. All of the one or more images supplied to the neural network 160 are acquired by the same single imaging device on the ocean vessel (it should be noted that this does not preclude the ocean vessel from having other imaging devices embedded), and the neural network 160 uses these images to determine the distance to the ocean object. In other words, this makes it possible to determine the distance to the ocean object using a single camera on the ocean vessel.

[0103] Now, let's focus on Figure 3A.

[0104] The method shown in Figure 3A includes obtaining multiple images (a series of images) of an ocean object acquired by an imaging device at different time periods (operation 300).

[0105] The method further includes supplying each image of a group of images to the neural network 160 (operation 310). The images can be supplied to the neural network 160 sequentially, for example, one by one. Alternatively, they can be supplied to the neural network 160 simultaneously.

[0106] According to some embodiments, the time interval between the acquisition of the first image and the acquisition of the last image in a series of images is less than a threshold. For example, the first image is acquired at time T1, the second image is acquired at time T2 > T1, and the last image is

number

[0107] In some embodiments, the threshold can be chosen such that the displacement of the ocean object relative to the ocean vessel 125 between different images is negligible. In this case, the threshold depends on the relative position of the ocean object. If the ocean object is located near the horizon, the threshold can be up to 1 minute, and if the ocean object is located near the bottom of the image, the threshold can be up to 1 second. Note that these values ​​are not limiting.

[0108] As described above, marine objects can be identified in each image, for example, using the machine learning module 159. A tracking method can be used to track marine objects across different images. The tracking method can be implemented by a PMC such as PMC110. The tracking method can use, for example, a Kalman filter or other adapted tracking method. The tracking method makes it possible to understand the movement of each marine object across multiple images.

[0109] The method further includes using a neural network 160 to estimate one or more candidate distances between the ocean vessel 125 and the ocean object for each image in a group of images (operation 320). This operation is repeated for each image (taken at different time periods) so that a set of candidate distances over time is obtained.

[0110] For example, as illustrated in Figure 3B, a distribution of candidate distances 375 is obtained for the image of the ocean object 370 acquired at T1 (each candidate distance is associated with a probability), and a distribution of candidate distances 376 is obtained for the image of the ocean object 370 acquired at T2 (each candidate distance is associated with a probability), T N For the images of the oceanic objects 370 acquired, a distribution of candidate distances 377 is obtained.

[0111] Therefore, the set of candidate distances 380 is obtained (this is given over a period ([T1;T N (This may include all or at least some of the candidate distances determined in ]). The set of candidate distances 380 is a given period ([T1;T N It can be used to estimate the distance between marine vessels and marine objects within the ])

[0112] In some embodiments, the set of candidate distances 380 may include only the candidate distances for distributions 375, 376, and 377 that have a probability higher than a threshold, for example. However, this is not limiting, and the set of candidate distances 380 may include all candidate distances for various distributions 375, 376, and 377.

[0113] In some embodiments, a state estimator 385 implemented by the PMC110 can be supplied with a set of candidate distances 380, such as a Kalman filter, a deep neural network, or a high-pass or low-pass filter. The state estimator 385 then calculates the distances over a given period ([T1;T N A set of candidate distances 380 can be used to output an estimated distance 390 between the ocean vessel 125 and the ocean object within ]). The state estimator 385 can smooth the distance estimate over time, for example, by filtering out sudden fluctuations in the distance estimate.

[0114] In other embodiments, the neural network 160 can be configured to process time-series data. For example, a recurrent neural network, an LSTM (Long Short-Term Memory (LSTM)), or other adapted neural network can be used to determine the distance to an ocean object over time using a series of images acquired at different time periods.

[0115] According to some embodiments, the method provides data D that gives information about the frequency distribution of candidate distances within a set of candidate distances 380. distribution This includes determining (action 391 in Figure 3D). In fact, each candidate distance D iM is within the set of 380 candidate distances. i times(M i (where is 1 or greater) can exist. For example, neural network 160 estimates that the distance between a marine vessel and a given marine object is D1 at time T1, and the distance between the marine vessel and the given marine object is D1 at time T J (If J is different from 1) then assume again that it is D1. As a result, distance D1 appears twice in the set of candidate distances 380 (M1 is equal to 2). Assume that the set of candidate distances 380 contains K different candidate distances. Each candidate distance D i The frequency distribution (also called the frequency of occurrence) is, for example,

number

[0116] The frequency distribution of candidate distances can be used to estimate the distance between the ocean vessel 125 and the ocean object (operation 392).

[0117] In some embodiments, the candidate distance with the highest frequency distribution can be selected as the estimate of the distance between a marine vessel and a marine object. This is not limited and can output candidate distances obtained at different frequencies (not necessarily only the highest frequency).

[0118] The method is different for different periods [T N+1 ;T M This can be repeated.

[0119] According to some embodiments, data D provides information about the expected movement of an ocean object in order to determine the distance to an ocean vessel. physical It is possible to use it.

[0120] Assume that a neural network 160 is used to estimate one or more candidate distances between a marine vessel and a marine object for each image in a set of multiple images, thereby obtaining a set of candidate distances for a given period based on the multiple images.

[0121] Data D physical This can be used to determine that at least one of the candidate distances in the set of candidate distances is incompatible with a (realistic / correct) estimate of the distance between a marine vessel and a marine object.

[0122] In fact, if it is known that an oceanic object has a velocity that cannot exceed a threshold, the difference between the candidate distance determined for the image at the first time and the candidate distance determined at the second time cannot exceed a given threshold, and therefore this can be used to filter out false / unrealistic candidate distances.

[0123] Now, let's look at Figure 4, which describes how to train at least one neural network 160.

[0124] The method includes obtaining a set of images (training set) and labeled data (operation 400). Each given image in the set of images includes a given ocean object (or multiple image ocean object) acquired by a given imaging device on a given ocean vessel. Note that in the training set, the ocean object may differ from one image to another. In some embodiments, the images may be acquired by imaging devices mounted on different ocean vessels. In some embodiments, the images may be acquired by imaging devices of different types, and / or that shake at different heights and / or have different orientations.

[0125] The labeled data includes a given label for each given image in the training set, which provides information about the distance between a given ocean vessel and a given ocean object. Embodiments for generating the labeled data are provided below.

[0126] According to some embodiments, the labeled data includes, for each given image, a geometric representation (e.g., a bounding box) indicating the location of a given ocean object in the given image. This location in the image can be determined, for example, by machine learning module 159 or by using a different method.

[0127] The method further includes supplying multiple images and labeled data to at least one neural network 160 for training (operation 410).

[0128] The neural network 160 attempts to predict the distance to an ocean object, and this prediction, compared to the distance provided by the labeled data, can be used to update the weights associated with the neurons in the layers of the neural network 160. Methods such as backpropagation can be used to train the neural network 160.

[0129] The weights and / or thresholds of the neural network 160 can be initially selected before training and can be further iteratively adjusted or modified during training to achieve an optimal set of weights and / or thresholds in the trained neural network. After each iteration, the difference (also called the loss function) can be determined between the actual output produced by the neural network 160 and the target output associated with each training set of data (e.g., the distance provided to the label). This difference may be referred to as the error value. Training can be determined to be complete when the cost or loss function indicating the error value is less than a predetermined value, or when a limited change in performance between iterations is achieved. If the neural network 160 includes multiple NN subnetworks, they can be trained separately before training the entire NN.

[0130] According to some embodiments, the labeled data supplied to the neural network 160 for training includes, for each given image in the training set, data D that provides information about the orientation and / or height of the imaging device 120 during the period in which the given image was acquired. camera This includes (operation 430).

[0131] As explained above with reference to Figure 2D, providing the height of the imaging device 120 to the neural network 160 during training can help the neural network 160 adapt to any height of the imaging device. Thus, the robustness of distance prediction is improved.

[0132] D camera Embodiments for determining this are provided below.

[0133] According to some embodiments, the labeled data D gives information about the environment in which the image was acquired for each given image in the training set (or for at least some of them). environmental Includes. D environmental Examples of the data concerning this, as well as non-limiting embodiments for obtaining this data, are provided above, and it should be noted that this description applies below.

[0134] Therefore, D environmental It can also be supplied to the neural network 160 as input for training.

[0135] According to some embodiments, additional data, such as the type of ocean object, can be supplied to the neural network 160 for training. The type of ocean object can be provided along with a labeled image. Embodiments for automatically determining the type of ocean object are provided below.

[0136] According to some embodiments, the neural network 160 is trained using a set of images and labeled data, where each given image in the set of images includes a given ocean object acquired by a given imaging device of a given ocean vessel, and the labeled data includes data 440 (see Figure 4) that gives information about the orientation of a given ocean object as it appears in the given image.

[0137] Non-limiting embodiments are provided in Figure 5.

[0138] In the left portion of Figure 5, the ocean vessel 520 is moving toward the ocean vessel equipped with an imaging device that acquires image 530.

[0139] In the right portion of Figure 5, the same marine vessel 520 is moving along a trajectory perpendicular to the optical axis of the imaging device that acquires the image 540.

[0140] As can be seen from the embodiment in Figure 5, the size and dimensions of marine vessels are not the same depending on their orientation in the image. This can affect the distance estimation performed by the neural network 160.

[0141] Therefore, in some embodiments, data 440 providing information about the orientation of ocean objects in the image is supplied to the neural network 160. This can help improve the accuracy of distance estimation when training the neural network 160.

[0142] In some embodiments, the data 440 providing information about the orientation of the ocean object can be determined using the AIS (which provides the position and course of the ocean object) of the ocean object and the position of the ocean vessel on which the imaging device is installed (known using the ocean vessel's sensors). Based on this data, it is possible to estimate the trajectory of the ocean object relative to the ocean vessel and then estimate the orientation of the ocean object in the images acquired by the ocean vessel's imaging device.

[0143] In some embodiments, the data 440 can also be used during prediction; that is, the data 440 can be fed to the trained neural network 160 (in addition to images of ocean objects acquired by the imaging vessel) to improve the estimation of the distance to ocean objects by the trained neural network 160. However, this is not mandatory.

[0144] In some embodiments, the neural network 160 is trained using images acquired by one or more imaging devices of one or more ocean vessels, which are not necessarily the same ocean vessels used to acquire images during the prediction phase.

[0145] Now, let's focus on Figure 6A.

[0146] Assume that the neural network 160 is used to determine the distance from the ocean vessel 125 to a given ocean object. In some embodiments, the performance of the neural network in this distance estimation can be evaluated (operation 610). Specifically, it can be determined that the distance estimation by the neural network 160 does not meet quality criteria.

[0147] Operation 610 may include, for example, comparing the distance output by the neural network with the true distance (which can be provided using other sensors, such as AIS). If the difference exceeds a threshold, this indicates that the estimate does not meet the quality criteria. In a non-limiting embodiment, the threshold is approximately 200m (this value is not limiting). Note that the threshold may vary depending on the distance to the ocean object.

[0148] Operation 610 may include checking the confidence level of the neural network 160 in its estimation. In fact, in some embodiments, the neural network 160 may provide a confidence level (or probability) that its output is correct. If the confidence level is below a threshold, this indicates that the estimation does not meet the quality criteria.

[0149] Operation 610 may include analyzing the distribution of candidate distances. This distribution should tend to be a Gaussian curve (this is not restrictive). However, a distribution in which candidate distances have similar probabilities indicates that the neural network 160 was unable to estimate the correct distance (see an unrestrictive example of such a distribution 600, as illustrated in Figure 6B). This indicates that the estimation does not meet the quality criteria. In other words, the distribution of candidate distances does not meet the criteria (e.g., does not conform to a Gaussian curve), and therefore the neural network 160 should be retrained, as described below.

[0150] If the distance estimation by the neural network 160 does not meet the quality criteria, the method in Figure 6A may further include determining the scenario in which the images were acquired (operation 620). Note that the scenario can also be determined based on multiple images of one or more oceanic objects in which the neural network tends to perform poorly in distance estimation.

[0151] In some embodiments, it can be determined that an image was acquired under given environmental conditions. This is, for example, data D environmental This can be used to make a determination. For example, it can be determined that distance estimation using neural network 160 tends to perform poorly on rainy days.

[0152] In some embodiments, it is possible to determine that the acquired image is an image of a given type of marine object. For example, it may be possible to determine that distance estimation by the neural network 160 tends to perform poorly for fishing vessels.

[0153] In some embodiments, it is possible to determine that the acquired image corresponds to a specific location at sea (e.g., on the coast) or a specific background in the image (e.g., the coast and not the open sea).

[0154] In some embodiments, it can be determined that the acquired image corresponds to a specific height or range of heights of the imaging device.

[0155] In some embodiments, it can be determined that an image was acquired at a given distance or within a range of distances. For example, it can be determined that distance estimation by the neural network 160 tends to perform poorly at distances greater than 5 miles.

[0156] In some embodiments, it can be determined that an image was acquired for a specific orientation of one or more marine objects as shown in the given image. For example, it can be determined that distance estimation by the neural network 160 tends to perform poorly for marine objects moving toward a marine vessel.

[0157] In other words, this indicates that the deep neural network 160 tends to underperform in a given scenario, and therefore, it should be retrained specifically for this type of given scenario.

[0158] The method in Figure 6A may further include retraining the neural network with a training set that matches the scenario according to the matching criteria (operation 630).

[0159] For example, if an image is determined to have been taken on a rainy day, a training set containing images taken on rainy days can be used.

[0160] Alternatively, if an image is determined to have been acquired at a long distance from an ocean object (e.g., more than 5 miles), the training set will include images of ocean objects located at distances greater than 5 miles.

[0161] In other words, retraining a neural network is performed by focusing on input data scenarios / types in which the neural network tends to perform poorly.

[0162] For example, it has been detected that neural network 160 tends to underperform in distance estimation for fishing boats located on the coast at a distance of more than 5 miles (this embodiment is not limiting). A training set of images corresponding to this scenario can be obtained and fed into neural network 160 for retraining, thereby enabling neural network 160 to improve its performance in scenarios where it tends to underperform.

[0163] It should be noted that in some embodiments, it is possible to generate detailed benchmarks for analyzing the performance of the neural network 160. For each of several given scenarios, the distance estimation of the neural network 160 is evaluated to verify whether it meets quality criteria. If quality criteria are established for a given scenario, the neural network 160 can be retrained, specifically using images acquired under conditions corresponding to this given scenario.

[0164] Here, data D provides information about the height and / or orientation of the imaging device. camera Figure 7 describes a method for determining this. As described below, the method can also be used to generate automatic labeling of images of ocean objects (labels including the distance to the ocean object).

[0165] As shown in Figure 7, the method includes obtaining first data that gives information about a first ocean object (operation 700).

[0166] The first data is obtained based on images acquired by imaging devices 120 of the offshore vessel 125. In some embodiments, the first data is obtained based on images acquired by multiple imaging devices 120 of the offshore vessel 125. The first data includes first position data that provides information about the position of a first offshore object.

[0167] Generally, the first position data is represented in reference to the imaging device 120. The PMC is configured to detect marine objects present in a given image acquired by the imaging device 120. As mentioned above, this detection can rely on an image processing algorithm.

[0168] When a given ocean object is detected in an image acquired by the imaging device 120, its position in the image (for example, a pixel position including its position along the X-axis and its position along the Y-axis of the image) can be obtained. Therefore, for each ocean object acquired by the imaging device 120, its position in the image can be obtained. In some embodiments, the first position data may include the position of each given object of the first ocean object in the image in which the given object was detected.

[0169] According to some embodiments, the imaging device 120 acquires multiple images over multiple periods. As a result, it is possible to obtain the position of the first marine object over time.

[0170] Specifically, the method may include obtaining a first set of positional data (see operation 7001 in Figure 2A). The first set of positional data includes the position of each object of the first ocean object over multiple time periods.

[0171] In fact, since the ocean vessel 125 moves over time, and / or at least some of the first ocean objects move over time, the position of the first ocean objects in the image acquired by the imaging device 120 can change over time.

[0172] According to some embodiments, it is possible to track a first marine object in multiple images acquired by the imaging device 120 in order to generate a first set of position data.

[0173] Non-limiting embodiments are provided with reference to Figure 7B.

[0174] Assume that at time t1, an image is acquired by the imaging device 120. In this image, three ocean objects are detected. The first ocean object is located at position 730, the second ocean object is located at position 731, and the third ocean object is located at position 732.

[0175] At time t2 (different from t1), another image is acquired by the imaging device 120. In this image, three ocean objects are detected. The first ocean object is located at position 733, the second ocean object is located at position 734, and the third ocean object is located at position 735.

[0176] At time t3 (different from t2), another image is acquired by the imaging device 120. In this image, three ocean objects are detected. The first ocean object is located at position 736, the second ocean object is located at position 737, and the third ocean object is located at position 738.

[0177] The tracking method can be used to track marine objects across different images. The tracking method can be implemented by a PMC (Private Management Company). The tracking method can implement, for example, a Kalman filter or other adapted tracking method.

[0178] In some embodiments, the ocean object may appear in some images and disappear in subsequent images. This can be due to the relative motion between the ocean object and the ocean vessel 125.

[0179] In the embodiment shown in Figure 7B, the tracking method reveals that the ocean object located at position 730 at time t1, the ocean object located at position 733 at time t2, and the ocean object located at position 736 at time t3 correspond to the same object at different time periods. Therefore, the same tracking ID (in this embodiment, "(1)") can be assigned to indicate that the same ocean object exists at different positions in different images.

[0180] Similarly, the tracking method reveals that the ocean object located at position 731 at time t1, the ocean object located at position 734 at time t2, and the ocean object located at position 737 at time t3 correspond to the same object at different time periods. Therefore, the same tracking ID (in this embodiment, "(2)") can be assigned to it.

[0181] Similarly, the tracking method reveals that the ocean object located at position 732 at time t1, the ocean object located at position 735 at time t2, and the ocean object located at position 738 at time t3 correspond to the same object. Therefore, the same tracking ID (in this embodiment, "(3)") can be assigned to them.

[0182] The method includes obtaining second data that gives information about a second ocean object (710).

[0183] Figure 7 illustrates the sequence in which the first position data is obtained first, followed by the second position data. However, this is not mandatory; it can be done in reverse order or simultaneously. This also applies to the sequence illustrated in Figure 7A.

[0184] The second data is obtained based on data acquired by at least one sensor 115 of the offshore vessel 125. Sensor 115 is different from the imaging device 120. In some embodiments, sensor 115 is not an imaging device (e.g., not a camera). Various embodiments are provided above for sensor 115 (e.g., radar, AIS, etc.).

[0185] The second data includes second position data that provides information about the position of a second ocean object encountered by the ocean vessel 125 during its voyage. According to some embodiments, as described below, the method projects the position of a target detected by various sensors (which can be represented by different references) into a common reference.

[0186] In some embodiments, multiple sensors 115 are available, including different types of sensors 115 (for example, the first sensor is radar, the second sensor is AIS, and the third sensor is GPS). In this case, for each sensor, position data of the ocean object detected by that sensor is obtained. As described below, each sensor may detect different ocean objects, but at least several ocean objects surrounding the ocean vessel 125 are detected by different sensors 115.

[0187] At least some of the first and second ocean objects correspond to the same physical ocean object. For example, the first ocean object includes a given first ocean vessel, a given second ocean vessel, and a buoy. The second ocean object includes a given first ocean vessel, a given second ocean vessel, and an iceberg.

[0188] This difference between the first and second ocean objects can be attributed to the fact that the imaging device 120 and the other sensors 115 have different fields of view and / or different lines of sight and / or different capabilities to detect objects (for example, depending on the type of sensor, its ability to detect objects may depend, for example, on weather conditions, the size of the object, the type of object, etc.). In addition, the imaging device 120 and the other sensors 115 may present other differences (for example, the imaging device 120 can be used to classify ocean objects, which is not possible for all sensors 115 such as radar).

[0189] The distance between the marine vessel 125 and the marine object can also affect the detection of the marine object by the marine vessel's sensors. For example, radar operates to detect marine objects at medium to long ranges but has a dead zone at short ranges, while imaging devices 120 perform better at detecting marine objects at short ranges than at long ranges. Therefore, not all marine objects are detected by all sensors on the marine vessel 125.

[0190] In some embodiments, the first ocean object and the second ocean object are the same. This means that all sensors 130 can detect the same ocean object.

[0191] The second position data is represented by a reference that can depend on sensor 115.

[0192] For example, if sensor 115 is an AIS, the absolute position of the marine object (latitude and longitude in world coordinates) can be obtained.

[0193] If sensor 115 is a radar, the position of the ocean object relative to the ocean vessel 125 is obtained (for example, expressed as a range and angular position relative to the radar and / or ocean vessel 125).

[0194] According to some embodiments, the second data may include additional data (in addition to the second position data of the second ocean object).

[0195] According to some embodiments, the second data includes identification data for a second marine object. For example, AIS provides unique identification data for each object, which enables its identification.

[0196] According to some embodiments, the second data includes data that provides information about the type of marine object (e.g., the type of marine vessel) (this can be provided by sensor 115 and / or derived from data provided by sensor 115).

[0197] For example, AIS can provide the type of marine object.

[0198] According to some embodiments, sensor 115 acquires data over a plurality of periods (e.g., while marine vessel 125 is moving). As a result, it is possible to obtain the position of the second marine object over time. As described below, the tracking data of the marine object can be used to improve the matching / association between the first marine object and the second marine object. However, this is not essential.

[0199] Specifically, the method can include obtaining a set of second position data (operation 7101 in FIG. 7A). The set of second position data includes the positions of a given object over a plurality of periods for each given object of the second marine object.

[0200] Several methods can be used to track the second marine object over different acquisitions.

[0201] In some embodiments, when sensor 115 provides identification data unique to each object, it is possible to track the object over a plurality of periods, thereby enabling the generation of a set of second position data. For example, if sensor 115 is AIS, each object is associated with specific identification data provided by AIS, so it is possible to track the position of the object over time.

[0202] When sensor 115 is a radar, the object can be tracked over various radar acquisitions (i.e., over a plurality of periods) using normal radar tracking.

[0203] According to some embodiments, first position data (each set of first position data) that provides information about the position of a first ocean object corresponds to the position in a first period (each of a plurality of first periods), and second data (each set of first position data) that provides information about the position of a second ocean object corresponds to the position in a second period (each of a plurality of second periods).

[0204] The first period (each referred to as the first period) and the second period (each referred to as the second period) satisfy the synchronization criterion. The synchronization criterion ensures that the time difference between each first period and each second period is less than a threshold. For example, the synchronization criterion can ensure that the time difference between each first period and each second period is less than 1 second. However, this value is not limiting. Specifically, if each first period and each second period do not satisfy the synchronization criterion, it is possible to perform upsampling of the data provided by the sensor (for example, using a Kalman filter, and this is not limiting). Similarly, downsampling can be performed if necessary.

[0205] As a result, the first period and the second period are substantially identical.

[0206] The method further includes using first position data and second position data to estimate data that provides information about at least one of the height and orientation of the imaging device 120 (operation 720). In some embodiments, both or only a portion of the data that provides information about the height and orientation of the imaging device is estimated (for example, because at least some of this data is known using other sensors and / or external inputs). Generally, at least one of the height and orientation of at least one imaging device is changeable over time because the orientation and / or position of the offshore vessel evolves over time.

[0207] The data providing information about the orientation of the imaging device 120 includes at least one of the following: the roll of the imaging device 120, the pitch of the imaging device 120, and the yaw of the imaging device 120. This orientation can be expressed in a similar manner to the roll / pitch / yaw of a ship (for example, the roll axis is an imaginary line running horizontally along the length of the ship, passing through its center of gravity and parallel to the waterline; the pitch axis is an imaginary line running horizontally across the ship and passing through its center of gravity; and the yaw axis is an imaginary line running vertically across the ship and passing through its center of gravity).

[0208] Data providing information about the height (also called altitude or elevation) of the imaging device 120 can also be estimated. The height of the imaging device 120 can be expressed, for example, relative to the sea level (also called mean sea level - MSL, or static water level - SWL).

[0209] Once one or more parameters of the imaging device 120 are determined, additional positional data (e.g., the absolute position of the imaging device) can be determined.

[0210] As mentioned above, in some embodiments, positional data of oceanic objects over multiple time periods is obtained.

[0211] According to some embodiments, the method may include using a first set of position data (which includes the position of a first ocean object over multiple periods derived from images acquired by the imaging device 120) and a second set of position data (which includes the position of a second ocean object over multiple periods derived from data acquired by the sensor 115) to estimate data that provides information about at least one of the position and orientation of the imaging device 120 over time (operation 7201 in Figure 2A).

[0212] As described below, estimating data that provides information about the height and / or orientation of the imaging device 120 may include attempting to match the position of a first ocean object with the position of a second ocean object (to reflect the fact that they correspond to the same ocean object acquired by different sensors) by correcting the height and / or orientation values ​​of the imaging device 120 (to be estimated). In other words, estimating the height and / or orientation of the imaging device 120 also makes it possible to match images with various targets acquired by one or more additional sensors.

[0213] According to some embodiments, a filter (e.g., a stochastic filter) can be used to predict the expected changes in the orientation and / or height of the imaging device 120 (for example, depending on weather conditions). This is useful for filtering out estimates of the height and / or orientation of the imaging device 120 that are unrealistic and noisy.

[0214] Figure 7C illustrates a method by which data providing information about at least one of the height and orientation of an imaging device 120 on an ocean vessel 125 is estimated over time. As shown in Figure 7C, the method is iterative over time.

[0215] The method is as follows: 1,i First marine object FIRSTMOBJ 1,i ~FIRSTMOBJ N,i The objective is to obtain first position data that provides information about the position of, the first position data being obtained based on images acquired by the imaging device 120 of the marine vessel 125 (operation 700) i ) includes. Operation 700 i This is the same as operation 200.

[0216] The method is the second period T 2,i Second oceanic object SNDMOBJ 1,i ~SNDMOBJ M,i The objective is to obtain second position data that provides information about the position of the first period T1,i and the second period T 2,i This means that the synchronization criteria are met (see possible definitions of this criterion above), and that the operation 710 i ) including. The second position data is obtained based on data acquired by at least one sensor 115 of the ocean vessel 125. Operation 210 i This is the same as operation 210.

[0217] First ocean object FIRSTMOBJ 1,i ~FIRSTMOBJ N,i At least some of them are the second ocean object SNDMOBJ 1,i ~SNDMOBJ M,i It is the same as at least some of them.

[0218] The method uses first position data and second position data to estimate data that provides information about at least one of the height and orientation of the imaging device 120 of the ocean vessel 125 (operation 720). i ) includes. Operation 220 i This is similar to operation 220. As a result, data giving information about at least one of the height and orientation of the imaging device 120 is obtained over a given period T'. i This is estimated to be the first period T 1,i and the second period T 2,i This substantially matches (as mentioned, the first period T) 1,i and the second period T 2,i (This is because it satisfies the synchronization criteria and therefore is essentially a match). In other words, T' i ≒T 1,i ≒T 2,i That is the case.

[0219] As shown in Figure 2C, operation 700 i This is a different first period T 1,i+1 This is repeated (this is shown in the flowchart in Figure 2C, T 1i (This occurs after -i is increased by 1). Therefore, time T 1、i+1 So, the first ocean object, FIRSTMOBJ 1,i+1~FIRSTMOBJ N,i+1 First position data providing information on the position of the first marine object FIRSTMOBJ is obtained at time T i+1 of the first marine object FIRSTMOBJ 1,i+1 ~FIRSTMOBJ N,i+1 is the first marine object FIRSTMOBJ at time T i of the first marine object FIRSTMOBJ 1,i ~FIRSTMOBJ N,i Note that it may be different from... However, this is not essential and depends on the scenario (in some cases, there may be partial overlap).

[0220] Operation 710 i is repeated over a different second period T 2,i+1 (occurring after T 2,i ). Thus, at time T 2、i+1 second position data providing information on the position of the second marine object SNDMOBJ is obtained. At time T 1,i+1 ~SNDMOBJ M,i+1 of the second marine object SNDMOBJ i+1 is the second marine object SNDMOBJ at time T 1,i+1 ~SNDMOBJ M,i+1 is the second marine object SNDMOBJ at time T i of the second marine object SNDMOBJ 1,i ~SNDMOBJ M,i Note that it may be different from... However, this is not essential and depends on the scenario (in some cases, there may be partial overlap).

[0221] T 1,i+1 and T 1,i+1 meet the synchronization criteria.

[0222] At least some of the first marine objects FIRSTMOBJ 1,i+1 ~FIRSTMOBJ N,i+1 are the same as at least some of the second marine objects SNDMOBJ 1,i+1 ~SNDMOBJ M,i+1 .

[0223] Operation 720 iThis is repeated to estimate data that provides information about at least one of the height and orientation of at least one imaging device among the ocean vessels. As a result, data that provides information about at least one of the height and orientation of imaging device 120 is obtained over a given period T'. i+1 This is estimated to be the first period T 1,i+1 and the second period T 2,i+1 This substantially matches (as mentioned, the first period T) 1,i+1 and the second period T 2,i+1 (This is because it satisfies the synchronization criteria and therefore is essentially a match). In other words, T' i+1 ≒T 1,i+1 ≒T 2,i+1 That is the case.

[0224] Therefore, the method makes it possible to estimate at least one of the height and orientation of the imaging device 120 over time.

[0225] According to some embodiments, the height and / or orientation of the imaging device 120 is estimated in real time or near real time (a small delay may exist due to the time it takes for the ocean vessel's sensor to acquire data and the time it takes to process this data).

[0226] Assume that data providing information about at least one of the height and orientation of the imaging device 120 is estimated during a given period (corresponding to a given iteration i in the method of Figure 7C). In some embodiments, this data can be used to improve the estimate of at least one of the height and orientation of the imaging device 120 in a subsequent period (corresponding to a given iteration i+1 in the method of Figure 7C). For example, the estimate obtained in iteration i can be used as a starting point for an algorithm attempting to estimate the height and / or orientation in iteration i+1. In other words, the results of a previous iteration can be used to improve the estimate in a subsequent iteration.

[0227] In some embodiments, and as described below, the estimation of the height and / or orientation of the imaging device 120 includes determining an association or coincidence between a first ocean object and a second ocean object in a given iteration. The association determined in a given iteration "i" of the method can be reused as input to the method in subsequent iterations "i+1" (or more generally, in iteration "j" if j>i) to improve the determination of association in subsequent iterations. For example, if two given ocean objects were identified as coincidence in a previous iteration of the method in Figure 7C, as described below, a term (reward) can be introduced into a loss function (calculated in a later iteration of the method; see below for embodiments that depend on the loss function) that takes this information into account.

[0228] Figure 7D shows operation 720 or operation 7201 or operation 720 i This illustrates an embodiment of a method that can be used to carry out the task.

[0229] As shown in Figure 7D, the method includes projecting first positional data, which gives information about the position of a first ocean object, and second data, which gives information about the position of a second ocean object, onto a common reference (721).

[0230] Figure 7D illustrates a sequence in which the first position data is projected first, followed by the second position data, but this is not limited to this; it can also be done in reverse or simultaneously.

[0231] Common references can correspond to global / absolute references, such as world coordinates (latitude, longitude). This is not limited, and other references can be used. For example, a given set of coordinates sharing the same plane can be used (e.g., a set of coordinates is represented relative to the position of a seaplane chosen as the origin of the set of coordinates).

[0232] As described above, the first position data is generally represented in the image reference (reference of the imaging device 120).

[0233] To convert the first position data into a common reference, assumptions can be made regarding data that provides information about the height and / or orientation of the imaging device 120.

[0234] Based on this assumption and the known position of the first ocean object in the image reference acquired by the imaging device 120, it is possible to project the first position data from the image reference to the common reference.

[0235] The method for projecting the first position data onto the common reference is described below with reference to Figure 7F.

[0236] In some embodiments, at least some of the positional data is already represented in the common reference.

[0237] For example, AIS can provide location data in world coordinates.

[0238] If positional data is provided by radar, it is possible to transform the positional data in world coordinates by using the position of the ocean vessel 125. In fact, since the radar provides relative position (range / azimuth) and the position of the ocean vessel 125 is known (for example, using the ocean vessel 125's GPS / AIS or other positioning system), it is possible to project the positional data into world coordinates (or another common reference).

[0239] Figure 7E illustrates the projection of the first and second location data onto a common reference 752 (global / absolute reference such as the Earth reference).

[0240] The projection of the first position data onto the common reference 752 depends, in particular, on the height and orientation of the imaging device 120.

[0241] At this stage, the height and / or orientation of the imaging device 120 are unknown (or known by error), so the first position data is projected randomly, and therefore the position of the first ocean object (drawn as a triangle) does not coincide with the position of the second ocean object (drawn as a circle).

[0242] Figure 7F illustrates a method for projecting the first position data from an image reference to a common reference (a global / absolute reference independent of the image reference).

[0243] Please note that this method is provided only as an example and not as an extension.

[0244] Assume that a given ocean object 799 (as depicted in Figure 7J) is identified in the image acquired by the imaging device 120.

[0245] For example, a bounding box containing a given ocean object 799 (see Figure 7G, "target_bounding_box") is obtained. The following notation is assumed. i.

number

number

number

[0246] The method is coordinate (target x ,target y This includes using (operation 790) to convert the coordinates of the two endpoints of the bounding box to a single point.

number

[0247] The method is as detailed below, using the coordinates of a given ocean object (target x ,target y This includes converting ) to a bearing (denoted as global_bearing) of a given ocean object represented in absolute reference (e.g., Earth reference) (791).

number

[0248] The method further includes determining the equation of an artificial horizon in the image (792) (the artificial horizon corresponds to a reference in which the imaging device 120 has zero roll and zero pitch). Figure 7H illustrates some non-limiting embodiments of the parameters used to determine the artificial horizon and its equation.

[0249] Normalization function old_to_new_value(old value old range new range ) is defined, and here, old value These are values ​​that require normalization, old range is the current range (old min old max ) corresponds to new range The expected range of values ​​(new min new max ) corresponds to new value This corresponds to the output of the function. The function old_to_new_value can be defined as follows:

number

[0250] Operation 792 may further include calculating the position of an artificial horizon in the image in pixels.

[0251] This is the following pixel pitch This may include determining the following:

number

[0252] This is the following pixel pitch It can further include updating. pixelpitch =abs(cos(cam roll ))pixel pitch This can further include defining (x1, y1) as follows:

number

[0253] This can further include defining (x2, y2) as follows:

number

number

number

[0254] The equation for the artificial horizon can be calculated using (x1, y1) and (x2, y2).

[0255] The method further includes determining the angle (denoted as angle_to_artificial_horizon) of a given ocean object with respect to an artificial horizon (see Figure 7I) (793). Operation 793 may include determining the distance in pixels in the image between the bounding box of a given ocean object (see Figure 7I, target_bounding_box) and the artificial horizon (see Figure 7I, pixel_dist_to_artificial_horizon) (this can be done by a simple trigonometric calculation). Operation 793 may then include determining angle_to_artificial_horizon using the following calculation:

number

[0256] The method is as detailed below, and involves the distance (Euclidean distance) between the ocean vessel 125 and a given ocean object 799. dist Operation 794 further includes determining (794) the orthogonal distance (see Figure 7J, hereafter referred to as ortho_dist) between the ocean vessel 125 and a given ocean object 799. Operation 794 may include a preliminary step of determining the orthogonal distance (see Figure 7J, hereafter referred to as ortho_dist) between the ocean vessel 125 and a given ocean object 799. ortho dist =tan(90-angle_to_artificial_horizon)cam height

[0257] In some embodiments, 、 ortho dist This can be corrected to take into account the curvature of the Earth (see, for example, https: / / earthcurvature.com).

[0258] Operation 794 may then include performing a calculation.

number

[0259] The method further includes determining the absolute coordinates (latitude, longitude) of a given ocean object 799 (795). Operation 795 may include the following calculations:

number

[0260] In this equation, d is euclidean dist Equals to the radius of the Earth, where R is the radius of the Earth and b is global bearing It is equal to.

[0261] The method in Figure 7D further includes solving an optimization problem (operation 722). Specifically, data D provides information about the height and / or orientation of the imaging device 120. camera D camera And it is estimated to allow a match between the positions of at least some of the first ocean objects determined using the first position data and the second position data of at least some of the second ocean objects. camera If the parameter cam is changed, the projection of the first ocean object from the image reference (first position data) to the common reference 252 will change (as seen in the equation provided above, referring to Figures 7F-7J). pitch cam roll cam yaw and cam height (See reference).

[0262] This agreement can follow a criterion (as described below, the criterion may, for example, define a minimum value for the number of iterations of the method and / or the loss function).

[0263] As mentioned above, imaging device 120 ("D camera Correction of the height and / or orientation estimation of the first ocean object leads to a correction of the projection of the position of the first ocean object from the image reference (first position data) to the common reference 752. camera The correction does not affect the projection of the position of the second ocean object, acquired by other sensors, onto Common Reference 752.

[0264] Solving optimization problems involves common references, (D camera To optimize the agreement between the position of the first ocean object (which is recalculated using the first position data) and the position of the second ocean object, the height and / or orientation D of the imaging device 120 is estimated. camera This may include optimizing the position of a first ocean object and the position of a second ocean object that correspond to the same ocean object.

[0265] In some embodiments, the field of view of the imaging device 120

number

[0266] In this case, solving the optimization problem also involves determining the position of the first ocean object (D) in the common reference. camera , D field_of_view This may include optimizing the field of view values ​​to optimize the agreement between the position of the second ocean object and the position of the first ocean object (and the position recalculated using the first position data).

[0267] As shown in Figure 7D, the method is generally iterative (see reference 723). In other words, solving the optimization problem involves D in each iteration until the convergence criterion is met. camera (and / or D field_of_view The process can include various iterations (e.g., N iterations for N>1) that are refined to optimize the loss function. In some embodiments, the convergence criterion can define the number of iterations, the value to be reached for the loss function, and so on.

[0268] In some embodiments, the convergence criterion depends on the number of associations / matches (e.g., absolute number or ratio) performed between the first and second ocean objects. In fact, a larger number of associations / matches between the first and second ocean objects increases the likelihood of finding an optimal solution to the optimization problem (as well as better estimation of the orientation and / or height of the imaging device).

[0269] In each iteration, an attempt is made to improve the estimation of the height and / or orientation of the imaging device 120 so as to improve the agreement between the positions of the first ocean object and the second ocean object in the common reference.

[0270] The iteration of the method may include repeating operations 721 and 722.

[0271] Figure 7K illustrates the projection of the first and second position data onto the common reference 752 after N iterations of the method.

[0272] As shown, the first ocean object, having an initial position 750 in the first iteration of the method, has an optimized position 755 (after N iterations of the method) that coincides with the position 751 of the second ocean object.

[0273] Similarly, each of the multiple first ocean objects has an optimized position that coincides with the position of each of the multiple second ocean objects.

[0274] However, there may be one or more first ocean objects that do not coincide with any of the second ocean objects. In Figures 7E and 7K, the first ocean object having position 755 in the first iteration of the method has an optimized position 760 after N iterations of the method that does not coincide with any position of the second ocean object. This can be attributed to the fact that this ocean object was acquired only by the imaging device 120 (and not by other sensors 115), or to various factors such as noise.

[0275] Let's look at Figure 7L.

[0276] As described above (see Figure 7A or Figure 7C), in some embodiments, the positions of the first and second ocean objects are acquired over multiple periods (a set of first position data and a set of second position data). This can be used to improve the estimation of the height and / or orientation of the imaging device 120.

[0277] The first position data and the second position are obtained at time t i Assume that it will be obtained.

[0278] The method shown in Figure 2L is time t i First data giving information about the position of the first ocean object at time t, and time t i This includes projecting second data, which gives information about the position of a second ocean object, onto a common reference (operation 724). Operation 724 is similar to operation 721.

[0279] The method involves providing data D that gives information about the height and / or orientation of the imaging device 120. camera (and / or Data D field_of_view ) but (D camera (and time t) so that it is recalculated using the first position data i The position and time t of the first ocean object i This further includes solving an estimated optimization problem (operation 725) to enable matching with the second position data of the second ocean object.

[0280] Operation 725 is the same as operation 722.

[0281] Time t i For the positional data, the method in Figure 7L can be iteratively repeated until the convergence criteria are met (see reference 726). Therefore, the height and / or orientation of the imaging device 120 can be estimated over time t i This information is obtained.

[0282] The method is time t i+1 (time t) iIt can be run again (different from) (see reference 727). Time t i+1 Therefore, the positions of the first and / or second oceanic objects may be advanced in common reference.

[0283] Time t i The height and / or orientation of the imaging device 120 is estimated because the height and / or orientation of the imaging device 120 can change during the voyage of the ocean vessel 125 (due to the various factors mentioned above), so time t i+1 This is not necessarily effective.

[0284] Therefore, the method is time t i+1 To estimate the height and / or orientation of the imaging device 120, operations 724 and 725 (these operations can be performed iteratively as drawn in reference 726) may be performed.

[0285] In some embodiments, it is possible to use tracking of ocean objects over time to improve the match between a first ocean object and a second ocean object. Specifically, if a match is determined to exist between two given objects (a given object of the first ocean object and a given object of the second ocean object) over different periods, it is likely that the two given objects correspond to the same ocean object. Therefore, in subsequent periods (action 725) in which attempts are made to match the positions of the first and second ocean objects, the match between the two given objects should be assigned a high weight in the optimization problem. This can be done by introducing a term (reward) in the loss function that takes this information into account.

[0286] Non-limiting embodiments are provided with reference to Figure 7M.

[0287] At time t1 (for example, after multiple iterations of the method as drawn in reference 726), assume that in common reference 752, the position 770 of the first ocean object having tracking ID (1,1) coincides with the position 771 of the second ocean object having tracking ID (2,1), and the position 772 of the first ocean object having tracking ID (1,2) coincides with the position 773 of the second ocean object having tracking ID (2,2).

[0288] Assume that at time t2 (for example, after multiple iterations of the method), in common reference 752, the location 774 of the first ocean object having tracking ID (1,1) coincides with the location 771 of the second ocean object having tracking ID (2,1), and the location 776 of the first ocean object having tracking ID (1,2) coincides with the location 777 of the second ocean object having tracking ID (2,2).

[0289] At time t3, in the first iteration of the method, a first ocean object has position 778 and tracking ID (1,1), another first ocean object has position 780 and tracking ID (1,2), a second ocean object has position 779 and tracking ID (2,1), and another second ocean object has position 781 and tracking ID (2,2).

[0290] When attempting to match the positions of the first and second ocean objects (operation 725), it is possible to take tracking IDs into consideration (as illustrated in operations 785 and 786 of Figure 7N). In fact, at times t1 and t2, it is established that the first ocean object with tracking ID (1,1) corresponds to the second ocean object with tracking ID (2,1). At times t1 and t2, it is established that the first ocean object with tracking ID (1,2) corresponds to the second ocean object with tracking ID (2,2).

[0291] Therefore, at time t3, the data providing information about the height and / or orientation of the imaging device 120 can be estimated to attempt to match the position of the first ocean object with tracking ID (1,1) with the position of the second ocean object with tracking ID (2,1), and the position of the first ocean object with tracking ID (1,2) with the position of the second ocean object with tracking ID (2,2) (since each of these positions is likely to correspond to the same respective ocean object).

[0292] Therefore, the agreement between the first and second ocean objects may depend not only on their location but also on the time-series tracking data of the first and second ocean objects (and / or other parameters as described below).

[0293] Let's look at Figures 3A and 3B.

[0294] As mentioned above, in some embodiments, the first data providing information about a first ocean object and the second data providing information about a second ocean object include data providing information about the type of ocean object.

[0295] This can be used to improve the agreement between the first ocean object and the second ocean object.

[0296] In fact, if it is known that two ocean objects correspond to the same type of object, then (in operations 722 or 725) a higher weight should be assigned in the optimization problem to match these two ocean objects. This can be done by introducing a term (reward) in the loss function that takes this information into account.

[0297] Conversely, if two ocean objects correspond to different types of objects, then (in operations 722 or 725) a lower weight should be assigned in the optimization problem to match these two ocean objects. This can be done by introducing a term (penalty) in the loss function that takes this information into account.

[0298] Non-limiting embodiments are shown in Figure 8B.

[0299] Assume that the first ocean object has position 805 in common reference 820. The first data includes the type of object and indicates that the first ocean object is an ocean vessel.

[0300] Assume that the second ocean object has position 815 in common reference 820. The second data indicates that this second ocean object is an ocean vessel.

[0301] Assume that another second ocean object has position 810 in common reference 820. The second data indicates that this second ocean object is a buoy.

[0302] The position 805 of the first ocean object is closer to the position 810 of the second ocean object than to the position 315 of the other second ocean object. However, since both of these ocean objects correspond to ocean vessels (the second ocean object with position 810 corresponds to a buoy, which is a different ocean object), data giving information about the height and / or orientation of the imaging device 120 should be estimated in order to improve the agreement between the position 805 of the first ocean object and the position 815 of the second ocean object.

[0303] This is illustrated in operations 850 and 860 of Figure 3A, which illustrate the use of data that provides information about the type of the first and second oceanic objects in order to improve the agreement between the positions of the first and second oceanic objects.

[0304] More generally, the method may use various parameters or additional inputs that can help improve the match (or association) between the first ocean object and the second ocean object.

[0305] Now, let us focus on Figure 9A, which depicts a specific embodiment of the method shown in Figure 7.

[0306] The method includes obtaining first data (operation 900) which includes first position data that gives information about the position of a first ocean object derived from an image acquired by the imaging device 120. Operation 900 is the same as operation 700.

[0307] The method includes obtaining second data (operation 910) which includes second position data that gives information about the position of a second ocean object provided by another sensor 115. Operation 910 is similar to operation 710. This other sensor is, unlike the imaging device 120, generally a sensor that is not a camera.

[0308] As mentioned above, at least some of the first and second ocean objects correspond to the same ocean object.

[0309] The method further includes determining the current state of data that provides information about the height and / or orientation of the imaging device 120 (operation 920).

[0310] In the first iteration of the method, the exact height and / or orientation of the imaging device 120 is unknown. Therefore, operation 920 may include generating random values ​​for the height and / or orientation of the imaging device 120.

[0311] In some embodiments, a first estimate of the height and / or orientation of the imaging device 120 may be available. This first estimate may be provided, for example, by an operator and / or manufacturer who have first knowledge regarding the height and / or orientation of the imaging device 120 (for example, due to the fact that they installed the imaging device 120 on an ocean vessel 125). However, due to the various factors mentioned above, this first estimate is no longer accurate during the voyage of the ocean vessel 125, and therefore the parameters of the imaging device 120 need to be estimated.

[0312] In some embodiments, a first estimate of the height and / or orientation of the imaging device 120 can be provided by an operator located on the offshore vessel 125 who measures the first values ​​of the height and / or orientation of the imaging device 120.

[0313] Once the current state is available for the height and / or orientation of the imaging device 120, the first position data of the first ocean object can be projected onto a common reference (e.g., Earth reference, but not limited to this) (operation 930). An example of this projection is provided in Figure 7F.

[0314] Similarly, the second positional data can be projected into a common reference, as already explained above.

[0315] The method further includes determining data that provides information about at least one of the height and orientation of the imaging device in order to optimize the agreement between at least some positions of the first ocean object and at least some positions of the second ocean object (operation 940). Examples of optimization algorithms that can be used include, for example, MSE (mean squared error), gradient descent, MAE (mean average error), minimum L2 distance (Euclidean distance), etc. These examples are not limiting.

[0316] As mentioned above, operation 940 may include using various additional data, such as the type of ocean object and tracking data of the ocean object, in order to improve the matching between the first ocean object and the second ocean object.

[0317] The loss function can be calculated to reflect the optimization problem. If the loss function does not meet the convergence criteria (for example, because its value is above a threshold), the method can be repeated by repeating operation 940, in which an attempt is made to improve the estimation of the height and / or orientation of the imaging device 120 in order to improve the agreement between the loss function and the convergence criteria.

[0318] When the loss function of the optimization algorithm meets the convergence criteria (e.g., its value falls below a threshold and / or a sufficient number of iterations have been performed), the current state (current estimate) of the height and / or orientation of the imaging device 120 can be output (operation 950). Similarly, a match between a first ocean object and a second ocean object can also be output for further use, such as automatic labeling of images from the training set, as described below (e.g., two ocean objects acquired by different sensors can be considered a match if their positions in a common reference determined using the estimated height / orientation of the imaging device are substantially similar, or if their distance is below a threshold).

[0319] According to some embodiments, the method of Figure 9A includes obtaining first data, which includes first position data giving information about the location of a first ocean object derived from an image acquired by the imaging device 120; second data, which includes second position data giving information about the location of a second ocean object provided by a first sensor (see reference 115); and third data, which includes third position data giving information about the location of a third ocean object provided by a second sensor (see reference 115). The first and second sensors are not imaging devices, and the first sensor may be of a different type than the second sensor. At least some of the first, second, and third ocean objects correspond to the same ocean object acquired by different sensors. In this embodiment, the method of Figure 9A can be similarly performed by projecting the locations of all ocean objects in a common reference (similar to operation 930), and attempting to determine the height and / or orientation of the imaging device 120 to optimize the coincidence between the respective locations of the first, second, and third ocean objects in the common reference. For example, the height and / or orientation of the imaging device is modified so that the projected position of each of the first ocean objects matches, as far as possible, the position of at least one of the second and third ocean objects.

[0320] For a given ocean object, it may be detected only by, for example, the imaging device and the first sensor, and for another ocean object, it may be detected only by the first sensor and the second sensor, or only by the imaging device and the second sensor.

[0321] As described below, once the height and / or orientation of the imaging device 120 is estimated, this data can be used for different marine applications.

[0322] According to some embodiments, we assume that an agreement has been found between a given ocean object of the first ocean object and a given ocean object of the second ocean object, as shown in Figure 9A (as mentioned above, in some embodiments this agreement can be obtained after several iterations of the optimization method).

[0323] In other words, this indicates that the given ocean object was identified as the same ocean object acquired by both the imaging device 120 and the other sensor 115. Therefore, if the other sensor 115 provides second position data that gives information about the given ocean object (e.g., expressed in a global / absolute reference such as a world reference, e.g., sensor 115 is AIS), the second position data can be used to determine the position of the given ocean object acquired by the imaging device 120, and then the distance between the ocean vessel 125 and the given ocean object (operation 960).

[0324] Now, let's look at Figure 9B, which shows a modified version of the method in Figure 9A.

[0325] The method includes obtaining first data (operation 900) which includes first position data that gives information about the position of a first ocean object derived from an image acquired by the imaging device 120. Operation 400 is the same as operation 200.

[0326] The method includes obtaining second data (operation 910) which includes second position data that gives information about the position of a second ocean object provided by a first sensor (see reference 115). Operation 910 is similar to operation 710. This first sensor is, in general, a sensor that is not a camera, unlike the imaging device 120.

[0327] The method includes obtaining third data (operation 915) which includes third position data that gives information about the position of a third ocean object provided by a second sensor (see reference 115).

[0328] The second sensor is different from the first sensor and the imaging device 120. According to some embodiments, the second sensor is not a camera.

[0329] According to some embodiments, the second sensor is of a different type from the first sensor (for example, the first sensor is an AIS and the second sensor is a radar or LIDAR, but this is not limited to these).

[0330] The method may include an intermediate operation 916 in which at least some of the second ocean objects and at least some of the third ocean objects are merged to obtain an aggregated (unified) set of ocean objects. However, this operation is not limited.

[0331] Each ocean object in an aggregated set of ocean objects is assigned location data, which may correspond to, for example, a second location data and / or a third location data.

[0332] Operation 916 can be performed by merging ocean objects whose distance between their locations (in a common reference) is less than and / or the minimum threshold.

[0333] Operation 916 may involve solving an optimization problem in which the distance between each pair of ocean objects is minimized, and each pair includes the ocean object of the second ocean object and the ocean object of the third ocean object. The optimization algorithms mentioned above can be used.

[0334] For example, suppose the first sensor is an AIS and the second sensor is a radar. The AIS provides the latitude / longitude of the second ocean object, and the relative range / azimuth measurements of the radar and the position of the ocean vessel 125 can be used to determine the latitude / longitude of the third ocean object. Thus, it is possible to merge the second and third ocean objects into an aggregated set of ocean objects.

[0335] The method in Figure 9A is depicted using two sensors (in addition to the imaging device 120), but this is not limiting, and any additional, adapted sensors (providing information about the positions of marine objects surrounding the marine vessel) can be used.

[0336] The method further includes determining the current state of data that provides information about the height and / or orientation of the imaging device 120 (operation 920), similar to the method in Figure 9A.

[0337] In the current state of data that provides information about the height and / or orientation of the imaging device, the method includes projecting the first position data onto a common reference (an absolute reference such as a globe reference) (operation 930).

[0338] With respect to the position data of an aggregated set of ocean objects, in some embodiments, this position data is already represented in a common reference. In fact, if at least one given sensor (among the first and second sensors) provides position data in a common reference, then after merging the second and third ocean objects into an aggregated set of ocean objects (see operation 916), it is possible to assign the position data in the common reference to each object in the aggregated set of ocean objects, as provided by the given sensor.

[0339] The method is D camera To optimize the match between the positions of at least some of the first ocean objects determined using and the first position data and positions of at least some of the ocean objects in the aggregated set of ocean objects, data D provides information about at least one of the height and / or orientation of the imaging device 120. camera The operation further includes determining (operation 939). Operation 939 is similar to operation 940, but differs in that operation 939 includes a match between the first ocean object and an aggregated set of ocean objects (obtained using at least two sensors). In some embodiments, operation 939 is D field_of_view This may include determining the following:

[0340] The method can be iterated until the convergence criterion is met and there is an agreement between the loss function and the convergence criterion (for example, operation 939 is D camera (This can be repeated to fine-tune the estimate.)

[0341] Once the convergence criteria are met, the height and / or orientation estimates of the imaging device 120 can be output (see operation 950).

[0342] As described below, once the height and / or orientation of the imaging device 120 is estimated, this data can be used for different marine applications.

[0343] Now, let's look at Figure 10.

[0344] The method includes obtaining first position data that provides information relating to the position of a first ocean object, wherein the first position data is obtained based on a given image acquired by a given imaging device 120 of a given ocean vessel 125 (operation 1000). Operation 1000 is the same as operations 700 and 900 already described above and will not be described again.

[0345] The method includes obtaining second position data that gives information relating to the position of a second ocean object, the second position data being obtained based on data acquired by at least one sensor of a given ocean vessel 125, the at least one sensor being different from a given imaging device 120, and at least some of the first ocean objects being the same as at least some of the second ocean objects (operation 1010). Operation 1010 is similar to operations 710 and 910 already described above and will not be described again.

[0346] As described in the various embodiments above, matching can be performed between the position of a first ocean object (acquired by the imaging device 120) and the positions of a second (or even third, or more) ocean object (acquired by other sensors 115).

[0347] Various methods for achieving this matching are described above (as mentioned above, these methods also provide data D that gives information about the height and / or orientation of the imaging device 120 that acquired the image). camera (This can include the determination of whether or not it is true.)

[0348] Therefore, the method may include the operation 1015) determining that one ocean object of the first ocean object and one ocean object of the second ocean object correspond to the same given ocean object acquired by a given imaging device and at least one sensor, respectively. As mentioned above, this matching between the first ocean object and the second ocean object can be performed for multiple ocean objects.

[0349] Once this matching is performed, it is known that one ocean object of the first ocean object and one ocean object of the second ocean object correspond to the same given ocean object acquired by both the imaging device 120 and the sensor 115 (because their positions are substantially identical or differ by less than a threshold). In other words, an association is made between the first ocean object and the second ocean object.

[0350] To enhance our knowledge of a given ocean object, it is possible to use this association and the various parameters provided by different sensors.

[0351] Specifically, (since it is known that the acquisitions by the imaging device and at least one sensor 115 correspond to the same physical object) it is possible to determine the parameters of a given ocean object using data provided by other sensors 115 that have detected this given ocean object.

[0352] According to some embodiments, at least one sensor 115 (e.g., AIS) can provide information regarding the position of an ocean object. Thus, it is possible to determine the distance to a given ocean object in the image (operation 1020).

[0353] As a result, it is possible to automatically generate labeled images (operation 1030) that include an image of an ocean object and a label indicating the distance to the ocean object.

[0354] More generally, it is possible to automatically generate a set of labeled images that include images of ocean objects and labels indicating the distance to those objects.

[0355] In some embodiments, automatic labeling may include associating marine objects in an image with other parameters such as the type of marine object (or other / additional parameters such as the ocean condition or the distance of the marine object). Thus, automatic labeling of marine objects (sensor labeling) is achieved. The labeled images can then be used, for example, for supervised training of a neural network 160 configured to determine the distance of marine objects in the image. Thus, training at a higher granularity is achieved.

[0356] For example, AIS provides the type of a given ocean object (e.g., "cargo"). However, this information is provided by the ocean object itself and can therefore be corrupted. Imaging devices can be used to determine the true type of a given ocean object (e.g., using a neural network that detects the type of object based on the image).

[0357] The set of extended data / parameters can be determined for each ocean object.

[0358] As a result, a database of images of marine objects (with corresponding augmented data / parameters) can be created, which can be used to train the neural network 160. Specifically, for each image, labeled data including the distance to the marine object, the type of marine object, etc., can be provided for training the neural network 160.

[0359] Embodiments of the subject matter of this disclosure are not described with reference to any particular programming language. It should be understood that various programming languages ​​may be used to implement the teachings of the subject matter disclosed herein, as described herein.

[0360] The present invention envisions a computer program that is readable by a computer to perform one or more methods of the present invention. The present invention further envisions a machine-readable memory that embodies a program of machine-executable instructions to perform one or more methods of the present invention.

[0361] It should be noted that the various features described in the various embodiments can be combined according to all possible technical combinations.

[0362] It should be understood that the present invention is not limited in its application to the details described herein or illustrated in the drawings. Other embodiments of the present invention are possible and can be carried out and implemented in a variety of ways. Accordingly, it should be understood that the expressions and terms used herein are for the purposes described and should not be considered limiting. Accordingly, those skilled in the art will understand that the concepts on which this disclosure is based can be readily used as a basis for designing other structures, methods, and systems to accomplish some of the purposes of the subject matter of this disclosure.

[0363] Those skilled in the art will readily understand that various modifications and changes can be applied to the embodiments of the invention described herein, without departing from the scope defined in the appended claims and without departing from the scope defined by the appended claims.

Claims

1. A system comprising a processor and a memory circuit (PMC), wherein the PMC performs the following operations: Obtaining first position data that provides information relating to the position of a first ocean object, wherein the first position data is obtained based on at least one image acquired by a given imaging device of a given ocean vessel, To obtain second position data that provides information about the position of a second ocean object, the second position data being obtained based on data acquired by at least one sensor that is not an imaging device and is different from the given imaging device, and at least some of the first ocean objects are the same as at least some of the second ocean objects, The PMC determines that, for each of the multiple ocean objects of the first ocean object, each ocean object and each ocean object of the second ocean object correspond to the same ocean object acquired by the given imaging device and the at least one sensor, respectively, and the PMC To obtain first data that provides information about the type of the first marine object, and which is based on at least one image acquired by an imaging device, To obtain second data that provides information regarding the type of the second marine object, and which is based on data acquired by the at least one sensor, It is further possible to match the first and second ocean objects as corresponding using acquired data that provides information regarding the type of the first ocean object and the type of the second ocean object. To make a judgment, Data D provides information about at least the orientation of the given imaging device, including at least the roll of the imaging device, during the period in which the at least one image is acquired by the given imaging device. ｃａｍｅｒａ Determine the common reference, (a) the recalculated position of at least some of the first oceanic objects, D ｃａｍｅｒａ (b) optimizing the agreement between the recalculated position, which is determined using the first position data, and the second position data for at least some of the second ocean objects, At least D ｃａｍｅｒａ Based on this, the distance between the given ocean vessel and a given ocean object among the first ocean objects, acquired by the given imaging device, is determined using the data provided by the at least one sensor. Based on the aforementioned at least one image, a labeled image is generated which includes the given ocean object and a label including the given distance. A system capable of operating.

2. The system according to claim 1, configured to determine the type of a given marine vessel using data provided by at least one of the sensors and to associate the type with the labeled image.

3. The system according to claim 1, wherein the second position data does not provide information regarding the position of the given ocean object.

4. The system according to claim 1, wherein the PMC is capable of iteratively correcting the estimated orientation values ​​of the imaging device, including at least the estimated roll of the imaging device, to match the positions of at least some of the first ocean objects with the positions of at least some of the second ocean objects, and further determining the distance between a given ocean vessel and a given ocean object based on the result of this match.

5. The aforementioned at least one image includes a plurality of images, and the PMC is Obtaining the first position data which provides information about the positions of different marine objects at separate points in time, and which is based on the plurality of images acquired by the given imaging device, To obtain the second position data that provides information about the position of a different second ocean object at different points in time, Processing the first position data to generate first tracking information for at least one first ocean object that provides information about the position of each of the first ocean objects at separate points in time, Processing the second position data to generate second tracking information for at least one second ocean object, which provides information about the position of each of the second ocean objects at separate points in time; Furthermore, based on the first tracking information and the second tracking information, the first ocean object and the second ocean object are matched. The system according to claim 4, which is capable of performing the operation.

6. The system according to claim 1, wherein the PMC is capable of iteratively correcting the estimated field of view values ​​of the given imaging device to match the positions of at least some of the first ocean objects with the positions of at least some of the second ocean objects, and further determining the distance between the given ocean vessel and the given ocean objects based on the result of this match.

7. The PMC provides data D that gives information about both the height and orientation of the given imaging device during the period in which at least one image was acquired by the given imaging device. ｃａｍｅｒａ The system according to claim 1, which is operable to determine the following.

8. The PMC, during the voyage of the given ocean vessel, receives the data D ｃａｍｅｒａ The system according to claim 1, wherein the system is operable to determine the height and orientation of the given imaging device, the height and orientation of the given imaging device change during the voyage, and the first position data is obtained based on the at least one image acquired by the given imaging device during the voyage at a point in time when the height and orientation of the given imaging device are unknown.

9. A method for determining the distance between a given ocean vessel and a given ocean object, the method being performed by a processor and a memory circuit (PMC), Obtaining first position data that provides information relating to the position of a first ocean object, wherein the first position data is obtained based on at least one image acquired by a given imaging device of a given ocean vessel, To obtain second position data that provides information about the position of a second ocean object, the second position data being obtained based on data acquired by at least one sensor that is not an imaging device and is different from the given imaging device, and at least some of the first ocean objects are the same as at least some of the second ocean objects, Determining that, for each of the multiple ocean objects of the first ocean object, each ocean object and the ocean object of the second ocean object correspond to the same ocean object acquired by the given imaging device and the at least one sensor, respectively, and matching the first ocean object and the second ocean object as corresponding, To obtain first data that provides information about the type of the first marine object, and which is based on at least one image acquired by an imaging device, To obtain second data that provides information relating to the type of the second marine object, the second data being based on data acquired by at least one sensor, the at least one sensor including at least one of radar and an automatic identification system (AIS), This includes matching the first and second ocean objects using acquired data that provides information regarding the type of the first ocean object and the type of the second ocean object. To make a judgment, Data D provides information about at least the orientation of the given imaging device, including at least the roll of the imaging device, during the period in which the at least one image is acquired by the given imaging device. ｃａｍｅｒａ (a) The recalculated position of at least some of the first oceanic objects, D ｃａｍｅｒａ (b) optimizing the agreement between the recalculated position, which is determined using the first position data, and the second position data for at least some of the second ocean objects, At least D ｃａｍｅｒａ Based on this, the distance between the given ocean vessel and a given ocean object among the first ocean objects, acquired by the given imaging device, is determined using the data provided by the at least one sensor. Methods that include...

10. the data D providing information regarding both the height and orientation of the given imaging device during the period in which the at least one image was acquired by the given imaging device ｃａｍｅｒａ determining ｃａｍｅｒａ , and causing positions of at least some of the first marine objects to coincide with positions of at least some of the second marine objects by repeatedly modifying a value of an estimated orientation of the imaging device including at least an estimated roll of the imaging device, the determining of the distance further includes determining the distance between the given marine vessel and the given marine object based on a result of this coincidence, the method of claim 9

11. This includes aligning the position of at least some of the first ocean objects with the position of at least some of the second ocean objects by iteratively correcting the estimated orientation value of the imaging device, including at least the estimated roll of the imaging device. Determining the aforementioned distance further includes determining the distance between the given ocean vessel and the given ocean object based on the result of this match, The at least one image comprises a plurality of images, the first position data provides information about the position of different marine objects at different time points in time, the first position data is based on the plurality of images acquired by the given imaging device, and the second position data provides information about the position of a different second marine object at different time points in time, and the method is Processing the first position data to generate first tracking information for at least one first ocean object that provides information about the position of each of the first ocean objects at separate points in time, This includes processing the second position data to generate second tracking information for at least one second ocean object, which provides information about the position of each of the second ocean objects at separate points in time, The method according to claim 9, wherein the matching further includes matching the first ocean object and the second ocean object based on the first tracking information and the second tracking information.

12. The steps of obtaining the first position data, obtaining the second position data, determining the correspondence between the first ocean object and the second ocean object, and the data D ｃａｍｅｒａ The steps of determining the distance between the given ocean vessel and the given ocean object are performed during a first time when the given ocean vessel is located at a first height and in a first orientation, and the method is In a second time when the given ocean vessel is located at a second height different from the first height and in a second orientation different from the first orientation, the steps are: to obtain the first position data, to obtain the second position data, to determine the correspondence between the first ocean object and the second ocean object, and the data D ｃａｍｅｒａ The method further includes determining a second distance between the given ocean vessel and the given ocean object by repeating the steps of determining the given distance between the given ocean vessel and the given ocean object, The method according to claim 9, wherein the distance between the first height and the second height and the distance between the first orientation and the second orientation affect the distance estimation, such that the given ocean object appears differently in the at least one image from which the first position data for the first time was obtained, compared to the at least one image from which the first position data for the second time was obtained.

13. A non-temporary storage device that is machine-readable and tangibly embodies a program of machine-executable instructions for performing an action, wherein the action is Obtaining first position data that provides information relating to the position of a first ocean object, wherein the first position data is obtained based on at least one image acquired by a given imaging device of a given ocean vessel, To obtain second position data that provides information about the position of a second ocean object, the second position data being obtained based on data acquired by at least one sensor that is not an imaging device and is different from the given imaging device, and at least some of the first ocean objects are the same as at least some of the second ocean objects, Determining that, for each of the multiple ocean objects of the first ocean object, each ocean object and the ocean object of the second ocean object correspond to the same ocean object acquired by the given imaging device and the at least one sensor, respectively, and matching the first ocean object and the second ocean object as corresponding, To obtain first data that provides information about the type of the first marine object, and which is based on at least one image acquired by an imaging device, To obtain second data that provides information relating to the type of the second marine object, the second data being based on data acquired by the at least one sensor, the at least one sensor including at least one of radar and an automatic identification system (AIS), This includes matching the first and second ocean objects using acquired data that provides information regarding the type of the first ocean object and the type of the second ocean object. To make a judgment, Data D provides information about at least the orientation of the given imaging device, including at least the roll of the imaging device, during the period in which the at least one image is acquired by the given imaging device. ｃａｍｅｒａ (a) The recalculated position of at least some of the first oceanic objects, D ｃａｍｅｒａ (b) optimizing the agreement between the recalculated position, which is determined using the first position data, and the second position data for at least some of the second ocean objects, At least D ｃａｍｅｒａ Based on this, the distance between the given ocean vessel and a given ocean object among the first ocean objects, acquired by the given imaging device, is determined using the data provided by the at least one sensor. Non-temporary memory devices, including [specific type of device].

Images