Systems and methods for maritime object detection

Abstract

Disclosed is a method for performing maritime object detection. A computer processing device receives imaging data indicating at least one image generated by an imaging system configured to capture a maritime environment relative to a structure. The computer processing device also receives detection data indicating one or more detected objects of the at least one image, wherein the one or more detected objects are generated by a detection system operating on the at least one image. The computer processing device also receives, from one or more maritime sensors, sensing data indicating one or more proposed objects of the at least one image. The computer processing device also projects, using localization data associated with the structure and / or the imaging system, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects, and generates label data to label the at least one image by associating the one or more projected proposed objects with the one or more detected objects.

Description

"Systems and methods for maritime object detection" Cross Reference

[0001] The present application claims priority from Australian Provisional Patent Application No 2024903368 filed on 18 October 2024, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] The present disclosure relates to the detection and optional tracking of maritime objects, and more particularly to systems and methods for enhancing the output of a computer vision system operating in a maritime environment, such as, for example, to provide improved navigational guidance to a maritime vessel or mobile structure. Background

[0003] Recently image and / or video based computer vision systems have been used in various maritime related applications, including the monitoring of coastal regions, and the navigation of maritime vessels. These vision systems utilize one or more cameras, sensors, and / or other imaging devices, to capture real-time visual data, which is processed to identify and / or track one or more objects and their corresponding location in the maritime environment. Object detection and / or tracking is often performed relative to a structure, which may be a mobile structure (e.g., a maritime vessel operating in the maritime environment) or a fixed structure (e.g., an offshore building). Information in relation to the detection of maritime objects may be used for the purpose of maritime navigation, either with respect to the structure itself, or for other vessels or other entities in the vicinity of the structure.

[0004] For example, achieving autonomous navigation of a maritime vessel or mobile structure is dependent on the ability of the vision system to detect objects such as othervessels, buoys, and obstacles that may affect the ability of the mobile structure, such as a boat or submarine, to travel through the environment, such as an ocean or waterway, unimpeded.

[0005] Machine learning (ML) models have been implemented to enhance the accuracy of the object detection and tracking capability of vision systems. For maritime navigation related applications, ML models may be trained using datasets containing one or more maritime objects and corresponding labels, allowing the model to learn the patterns and features associated with the presence of different objects in the maritime environment. The trained ML model is then applied to an image, or a series of images, captured from the maritime environment in real-time by the imaging devices, to produce a set of detections indicating objects within the environment. By processing the detections output by the vision system, the location of objects within the maritime environment may be determined relative to that of the mobile structure, for example to permit the structure to be guided within the environment to avoid obstacles.

[0006] Any discussion of documents, acts, materials, devices, articles or the like which has been included in the present specification is solely for the purpose of providing a context for the present invention. It is not to be taken as an admission that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present invention as it existed before the priority date of each claim of this application.

[0007] Throughout this specification the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element, integer or step, or group of elements, integers or steps, but not the exclusion of any other element, integer or step, or group of elements, integers or steps. Summary

[0008] There is provided a method for performing maritime object detection, the method comprising: receiving imaging data indicating at least one image generated byan imaging system configured to capture a maritime environment relative to a structure; receiving detection data indicating one or more detected objects of the at least one image, wherein the one or more detected objects are generated by a detection system operating on the at least one image; receiving, from one or more maritime sensors, sensing data indicating one or more proposed objects of the at least one image; projecting, using localization data associated with the structure and / or the imaging system, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and generating label data to label the at least one image by associating the one or more projected proposed objects with the one or more detected objects.

[0009] In some embodiments, the one or more proposed objects are located within a field-of-view of the imaging system and / or a vicinity of the structure.

[0010] In some embodiments, the method further comprises processing the sensing data to filter the one or more proposed objects based on the size and / or dimensions of the respective proposed object in an image plane.

[0011] In some embodiments, generating the one or more projected proposed objects comprises, for each proposed object in each image of the at least one image: determining a geographical location of the proposed object in a global reference frame; and projecting the geographical location of the proposed object to an image location within the image.

[0012] In some embodiments, determining the geographical location of the proposed object comprises using the localization data to convert geographical coordinates of the proposed object to the global reference frame.

[0013] In some embodiments, projecting the geographical location of the proposed object to the image location comprises: (i) transforming the geographical coordinates of the proposed object from the global reference frame to generate corresponding local coordinates in a local reference frame of structure; and (ii) projecting the correspondinglocal coordinates into a plane of the image to obtain a set of pixel coordinates representing a position of the projected proposed object in the image.

[0014] In some embodiments, projecting the corresponding local coordinates into a plane of the image comprises: transforming each point of the projected proposed object in the local reference frame of the structure to a point in a reference frame of the imaging system, based on a position and / or orientation of the imaging system relative to the structure; and mapping each point in the reference frame of the imaging system to the corresponding set of pixel coordinates in the image.

[0015] In some embodiments, the method further comprises applying a distortion model to each point of the projected proposed object in the reference frame of the imaging system.

[0016] In some embodiments, the structure is a mobile structure, and the localization data comprises a geographical location and an orientation of the mobile structure.

[0017] In some embodiments, the geographical location and the orientation of the mobile structure is determined in real-time by a localization system of the mobile structure.

[0018] In some embodiments, associating the one or more projected proposed objects with the one or more detected objects comprises: for each of the one or more projected proposed objects in each image of the at least one image, calculating a distance value in the image between the respective projected proposed object and each of the one or more detected objects.

[0019] In some embodiments, the distance value is calculated as a distance between: a position of an object detection marker of the respective detected object; and the location of the respective projected proposed object, in the image.

[0020] In some embodiments, the label data indicates at least one of: respective ones of the projected proposed objects do not match to any of the one or more detected objects; and respective ones of the detected objects do not match to any of the one or more projected proposed objects, wherein a match is determined to occur by comparing the respective distance value against at least one threshold value.

[0021] In some embodiments, the at least one threshold value is determined based on a degree of accuracy of the localization data.

[0022] In some embodiments, the at least one image is a series of images captured from the maritime environment over a time window, and wherein the method comprises validating a temporal sequence of detections associated with each of the one or more detected objects against the one or more projected proposed objects by determining a number of the matches occurring over the series of images.

[0023] In some embodiments, each track is a model of a position of a target object represented by the corresponding detected object in the image space of the image.

[0024] In some embodiments, the label data represents one or more bounding boxes encompassing the corresponding one or more detected objects of each track in the image.

[0025] In some embodiments, the label data indicates that one or more tracks of the series of images are not valid.

[0026] In some embodiments, the one or more detected objects are obtained by applying the at least one image as input to a maritime object detector to generate the detection data as output of the detector.

[0027] In some embodiments, the maritime object detector comprises a machine learning model having one or more model parameters determined by a model training process.

[0028] In some embodiments, the method further comprises adapting the machine learning model by causing an execution of the model training process using at least: the imaging data; and the label data.

[0029] There is also provided a system for performing maritime object detection, the system comprising: a controller having: a communications interface to receive data; at least one computer processor to execute program instructions; and a memory, coupled to the at least one computer processor, to store program instructions for execution by the at least one computer processor, wherein the controller is configured to communicate with one or more maritime sensors, an imaging system, and a localization system configured to provide localization data associated with the imaging system and / or a structure, and wherein the controller is further configured to: receive imaging data indicating at least one image generated by the imaging system configured to capture a maritime environment relative to the structure; receive detection data indicating one or more detected objects of the at least one image, wherein the one or more detected objects are generated by a detection system operating on the at least one image; receive sensing data from the one or more maritime sensors, the sensing data indicating one or more proposed objects of the at least one image; project, using the localization data, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and generate label data to label the at least one image by associating the one or more projected proposed objects with the one or more detected objects.

[0030] In some embodiments, the controller of the system is further configured to perform any of the methods for performing maritime object detection that are described herein.

[0031] In some embodiments, the one or more maritime sensors comprise respective devices configured to receive data from one or more of: an Automatic Identification System (AIS); an Electronic Navigational Chart (ENC) system; and an Automatic Radar Plotting Aid (ARPA) system.

[0032] In some embodiments, the imaging system comprises an optical sensing system having at least one camera.

[0033] In some embodiments, the at least one camera is configured to generate the at least one image as a portion of a video feed of the maritime environment.

[0034] In some embodiments, the structure is a mobile structure, and wherein the localization system is a navigation system comprising a satellite-based positioning unit configured to provide a geographical location of the structure as part of the localization data.

[0035] In some embodiments, the navigation system further comprises an inertial navigation system unit configured to provide an orientation of the mobile structure as part of the localization data.

[0036] There is also provided an apparatus comprising: an imaging system configured to generate imaging data indicating at least one image capturing a maritime environment relative to a structure; a detection system configured to generate detection data indicating one or more detected objects of the at least one image; one or more maritime sensors configured to generate sensing data indicating one or more proposed objects of the at least one image; a localization system configured to generate localization data associated with the imaging system and / or the structure; and a controller having: a communications interface to receive data; at least one computer processor to execute program instructions; and a memory, coupled to the at least one computer processor, to store program instructions for execution by the at least one computer processor, wherein the controller is configured to communicate with the one or more maritime sensors, the imaging system, and the localization system at least to receive the imaging data, the detection data, the sensing data, and the localization data, and wherein the controller is further configured to: project, using the localization data, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and generate label data to label the atleast one image by associating the one or more projected proposed objects with the one or more detected objects.

[0037] In some embodiments, the controller of the apparatus is further configured to perform any of the methods for performing maritime object detection that are described herein. Brief Description of Drawings

[0038] Some embodiments of the invention will now be described with reference to the accompanying drawings, in which:

[0039] Fig.1 is a diagram of a sensor fusion-based object detection apparatus in accordance with some embodiments of the proposed technology;

[0040] Fig.2a is a schematic diagram of an example configuration of the apparatus of Fig.1 for performing maritime object detection and optional tracking;

[0041] Fig.2b is a schematic diagram of a top-down view of the operation of the apparatus to perform object detection and labelling according to the configuration of Fig.2a;

[0042] Fig.2c is an illustration of a set of images of the maritime environment and corresponding labels generated by the apparatus of the view depicted in Fig.2b;

[0043] Fig.3 is a flow diagram of a method for performing maritime object detection and optional tracking according to some embodiments of the proposed technology;

[0044] Fig.4a is a flow diagram of a method for receiving and processing sensing data of the method of Fig.3;

[0045] Fig.4b is a schematic diagram of a representation of the height of an object in one or more captured images associated with the apparatus and in the real-world of the method of Fig.3;

[0046] Fig.5 is a flow diagram of a method for generating the one or more projected proposed objects for an image of the captured image(s) of the method of Fig.3;

[0047] Fig.6 is a flow diagram of a method for performing object projection in association with the method of Fig.3;

[0048] Fig.7 is an illustration of a spheroid depicting the process of obtaining cartesian coordinates in a local reference frame, as performed during the object projection of the method of Fig.6;

[0049] Fig.8 is a flow diagram of a method for projecting an object from the local reference frame to an image plane, as performed during the object projection of the method of Fig.6;

[0050] Fig.9 is an illustration of a transformation of a point of a projected proposed object between representations, as performed during the object projection of the method of Fig.6;

[0051] Fig.10 is a flow diagram of a method for associating one or more projected proposed objects and one or more detected objects of the method of Fig.3; and

[0052] Fig.11 is an illustration of a distance-based association between a detected object and projected proposed objects in a pixel coordinate space of a captured image, as performed during the method of Fig.10. Description of Embodiments

[0053] Achieving effective autonomous navigation of a vessel or watercraft is dependent on the ability to accurately detect objects in a corresponding maritimeenvironment, such as by operating a vision-based system to image the environment. These vision-based imaging and detection systems typically utilize ML models that are constructed with a set of training data representative of the objects that are to be identified and tracked in the environment of the mobile structure or vessel. The training process involves providing the detection model with training data including a set of one or more input images and corresponding labels, allowing the resulting detection model to have parameters that represent the features associated with different objects captured within the image(s).

[0054] Conventional detection systems utilizing the trained ML models have imperfect accuracy when applied to imaging data that differs from the training data, such as for example images obtained from a maritime environment that is previously unseen and / or gathered in real-time by the vision-based imaging system. With respect to a single image generated by the cameras or other optical devices of the system, the corresponding detection system may produce two types of errors: (i) false positives, where the set of objects detected in the image by the detection system includes one or more objects that each do not truly exist in the environment as captured by the image; and (ii) false negatives, where the set of objects detected in the image by the detection system omits one or more objects that each truly exist in the environment as captured by the image.

[0055] The accuracy of maritime detection models may be improved by using supervised learning, in which one or more images of the training data set are explicitly labelled with data indicating the objects of interest prior to the construction of the detection model.

[0056] While existing supervised learning-based object detectors can achieve reasonable performance when trained on sufficiently large labelled datasets, their operation in the specific domain of maritime applications is hampered by the lack of ground truth data from real-world environments. That is, the ability to produce an accurate detection model is dependent on both an accurate labelling of the training data, and the existence of a sufficiently diverse set of images within the a priori training datato properly represent objects that are present within corresponding evaluation data imaged from the maritime environment during the application use case (e.g., navigating the structure).

[0057] One approach to addressing this issue is to implement continuous monitoring and enhancement of the deployed object detectors, for example to utilize the imaging data gathered from the vision system in real-time to enhance the trained models. However, this is challenging as it requires independent identification of any errors in the detections made by the system in order to use this output as feedback to refine the trained models, and thereby improve the maritime object detection capability.

[0058] Conventionally, the identification of errors in the detections of the vision system is performed manually, which is prohibitive both in terms of cost and time associated with reviewing and correcting the images and detection labels. A user must often review the imaging data, including video and / or image feed from the camera system, in conjunction with external data (e.g., a nautical chart or map, etc.) to determine an error in the detection output. Typically, the systems for displaying the imaging data and the external data are separate which the requires the user to perform additional processing to correlate the imaging data and the external data to determine whether one or more objects in the environment have been falsely detected or omitted from the detection output (e.g., to combine or merge data into a common interface or visual display means).

[0059] Further, detecting false negatives remains a significant challenge for a user reviewing the image data and the detection output in this manner, particularly in the context of using the detections to navigate a vessel in the environment. This is at least because the indications of the detected objects, and those of the external data, are expressed with different relative geolocation information, and since this different information must be resolved against corresponding navigational data of the mobile structure which changes in real-time (i.e., as the vessel moves in the environment during the imaging). Thus, the presence of false negative detections is particularlydetrimental to improving maritime object detection models according to the conventional approach.

[0060] It is therefore desired to develop an improved approach to vision-based maritime object detection that utilizes supervised learning models, and particularly to improve the ability to utilize the image-based detection outputs of the system, for example in the context of providing navigational guidance to a vessel, watercraft or other mobile structure in a maritime environment. Overview

[0061] Disclosed herein are methods, systems, and apparatus for detecting, and optionally tracking, objects in a maritime environment by integrating computer vision- based sensing and maritime specific sensing. Performing maritime object detection according to the proposed technology involves using an imaging system to capture the maritime environment relative to a structure. One or more images generated by the imaging system are processed to fuse object data within the images with corresponding data received from maritime specific sensors (e.g., data from one or more of an Automatic Identification System (AIS), an Electronic Navigational Chart (ENC), and an Automatic Radar Plotting Aid (ARPA) system). The one or more images are labelled based on associations between one or more objects resulting from the fusion, and a corresponding set of objects that are detected within the one or more images, as generated by a detection system (e.g., an image-based object recognition system).

[0062] In some embodiments, performing maritime object detection involves the fusion of data from one or more portable and / or fixed sensors associated with the structure, which may be a mobile structure (e.g., a watercraft, aircraft, motor vehicle, submersible vehicle, and / or other vehicle with a movement capability relative to the maritime environment). In other embodiments, the structure is non-mobile and has a fixed geographic location relative to the maritime environment (e.g., an offshore building).

[0063] In some embodiments, the one or more sensors include imaging devices (e.g., cameras), localization and / or navigation devices (e.g., sonar systems, radar systems, other ranging sensor systems, GNSS systems, position sensors, orientation sensors, inertial measurement devices, such as gyroscopes, accelerometers, and / or speed sensors providing measurements of an orientation, a position, an acceleration, and / or a speed), and maritime sensors providing information about entities associated with the maritime environment.

[0064] The sensor(s) may be coupled to or otherwise associated with the structure. For example, the sensors may be mounted to or within the structure, integrated with other sensor assemblies, or integrated within a portable device (e.g., smartphones, tablets, portable computers, portable sensor suites, cameras, and other devices). The sensor(s) provide data in the form of measurements of attributes and / or properties of the structure and the associated maritime environment, such as for example of objects imaged within the maritime environment and real-world entities present in the environment.

[0065] A computer processing device (i.e., a “controller”) is configured to receive imaging data, including the one or more images of the maritime environment, detection data representing the detected objects, and sensing data from the maritime sensors. The sensing data provides information about one or more objects located within the maritime environment imaged by the imaging system (referred to as “proposed” objects). For example, the sensing data may provide a ground truth for the existence of ships, markers and / or hazards that may be visible in a field-of-view of optical devices of the imaging system.

[0066] The controller processes the imaging data and the maritime sensing data by projecting the proposed objects indicated by the maritime sensors into the one or more images of the imaging data. The projection is performed using localization data associated with the structure and / or the imaging system to convert between coordinates of locations of the proposed objects and the location of the structure (e.g., as navigation data comprising a position and / or orientation measurement of a mobile structure that isdetermined in real-time). The controller generates labels representing one or more errors in the detections output by the detection system for the respective image(s). For example, missed detections (i.e., false negatives) and erroneous detections (i.e., false positives) of the objects initially detected in the image may be labelled by associating objects proposed by the maritime sensing data with the image-based detections.

[0067] In some embodiments, labelling is performed by determining associations between detected and proposed objects for each image generated by the imaging system, and by determining errors in the detection system output for each image individually.

[0068] In other embodiments, the labelling is performed by tracking the presence (or absence) of an object (referred to as a “target”) in an image space of a series of images generated by the imaging system. A track is established for each object in the image space by applying a distance-based association strategy to associate object proposals with existing detections (or absences) of objects over all images of the series. Labels are generated with respect to the one or more tracks enabling the identification of errors in the output of the detection system even in the case of imperfect maritime sensor data.

[0069] Advantageously, the labels may be used to update, modify and / or retrain the detection system, either in real-time or in an offline mode, to improve the underlying detection model. This improves the capability of the system to identify and track maritime objects in complex real-world scenarios.

[0070] The images and associated labels may be further used to enhance the overall reliability and effectiveness of a navigation capability of a maritime vessel, including the structure from which the maritime environment is imaged (i.e., in the case of a mobile structure) and / or any other vessel configured to receive the images and associated labels, in dynamically changing operational settings.

[0071] For example, the labelling of erroneous or absent detections allows a mobile structure to utilize imaging data and the corresponding labels in real-time controlapplications, for example to locate and avoid a previously undetected object, or to navigate through an erroneously detected object, in the maritime environment. The proposed techniques thereby assist the navigation of a mobile structure in the maritime environment by augmenting the output of the image-based object detections. Sensor fusion-based object detection apparatus

[0072] Fig.1 illustrates a sensor fusion-based object detection apparatus 100 in accordance with some embodiments of the proposed technology. Apparatus 100 comprises: an imaging system 102; a detection system 104; a localization system 106; a maritime sensing system 118; and a controller 110, collectively configured to image a maritime environment relative to a structure 101, and to automatically generate labels to associate the image-based object detections of the detection system 104 with object data provided by the sensing system 118. Imaging system

[0073] Imaging system 102 is configured to capture one or more images of a maritime environment surrounding the structure 101 (e.g., as a scene). The imaging system 102 comprises one or more imaging devices 103, which may include a single one or multiple types of sensors, scanners, emitters or vision capture devices (also referred to as “imaging sensors” herein). For example, the imaging sensors 103 may include devices configured to perform optical imaging of a scene of the environment, such as but not limited to a digital camera and / or other integrated image sensors, including a charge-coupled device (CCD) sensor, a complementary metal-oxide semiconductor (CMOS) sensor, an electron multiplying CCD (EMCCD), a scientific CMOS (sCMOS) sensor, and / or other image sensors configured to generate image signals of visible light received from the scene.

[0074] In some embodiments, the imaging sensors 103 may include sensors configured to capture electromagnetic radiation in other wavelengths in addition to visible light. For example, imaging sensors 103 may include an infrared (IR) imaging sensor, and / or other elements responsive to IR radiation.

[0075] The one or more imaging sensors 103 are configured with a field of view that extends substantially over a portion of the maritime environment representing the scene to be captured during operation of the imaging sensors 103 (also referred to as the field- of-view of the imaging system 102). In some embodiments, the imaging system 102 includes the one or more imaging sensors 103 and further includes an imaging control unit 105. In other embodiments, at least one of the sensor(s) 103 and the imaging control unit 105 is external to the imaging system 102 and are configured to exchange data with the imaging system 102 via a physical or wireless communication channel.

[0076] Imaging control unit 105 is in electronic communication with the imaging sensor(s) 103 to perform one or more control operations associated with imaging the maritime environment, including: activating one or more of the sensor(s) 103 to capture at least one image of the maritime environment within the field of view of the sensor(s) 103, in response to receiving a control signal; receiving, from the sensor(s) 103 image capture data (e.g., two dimensional images and / or three dimensional images) representing the environment in the field of view; and processing the image capture data received from the sensor(s) 103 to generate imaging data of the maritime environment.

[0077] In some embodiments, the imaging control unit 105 is configured as an embedded system with a processor and memory implemented as an integrated microcontroller with a RISC architecture, and the imaging sensor(s) 103 are configured as peripheral devices providing data to, and receiving control data from, the microcontroller. In other embodiments, the imaging control unit 105 may be implemented as one or more full-scale computer systems, such as an Intel Architecture computer system. In other embodiments, the imaging sensor(s) 103 may be integrated with the imaging control unit 105, thereby enabling an exchange of data between operational modules of the imaging control unit 105 and the imaging sensor(s) 103 via an internal controller bus or similar structure.

[0078] The imaging system 102 operates the one or more imaging sensors 103, for example via the imaging control unit 105, to produce imaging data comprising a seriesof one or more images of a maritime environment as viewed by the imaging system 102 (e.g., as frames of a video feed).

[0079] Imaging system 102 is configured to generate imaging data in a reference frame 102’ of the imaging sensor(s) (referred to as the “imaging reference frame”). For example, the imaging reference frame may be defined by one or more of: a 3D cartesian coordinate vector; a roll offset; and a pitch offset, that describes the configuration of the imaging system relative to the structure 101. In some embodiments, the imaging reference frame is defined using other non-Cartesian coordinate systems (e.g., polar coordinates). Detection system

[0080] The one or more images of the imaging data are input to the detection system 104, which is configured to operate on the one or more images to output detection data representing one or more objects detected within each image (e.g., as a set of “detected objects” for the respective image).

[0081] In some embodiments, the detection system 104 is configured with one or more detection models (not shown) indicating the expected characteristics of maritime objects that may be captured in the one or more images generated by the imaging system 102. One or more of the detection models may be configured as a machine learning model having one or more model parameters determined by a model training process to recognize maritime objects (e.g., ships, hazards, etc.) via supervised learning on a training image data set.

[0082] For example, detection system 104 may include a detection network, such as a multilayer perceptron (MLP), a recurrent neural network (RNN), or a convolutional neural network (CNN) trained on a set of training data specific to one or more features to be detected from the one or more images input into the detection system 104. Training of the detection network may be performed with, for example, a set of one or more training images and associated labels indicating visual features of maritime objects within the set of training images. In some embodiments, other types of machinelearning and / or pattern recognition models may be used as the one or more detection models of the detection system 104.

[0083] The detection system 104 is configured to receive the image(s) generated by the imaging system 102 as input to a maritime object detector module of the detection system 104 to generate the detection data as output of the detector. For example, the maritime object detector may comprise one or more detection models implemented using an object detection algorithm, such as Faster-RCNN (see [1]) and You Only Look Once (YOLO) (see [2]) to recognize objects based on the trained detection model(s).

[0084] In various embodiments, the detection system 104 is configured to track the presence of one or more objects within a series of images captured over a temporal window. For example, the detection system 104 may implement an image-based object tracking module implementing one or more object tracking methods (e.g., see [4], [5], and [6]) that operate on a set of images of the maritime environment (e.g., as generated by the imaging system 102 from taking a video of the environment). The detection system 104 generates detection data further indicating object tracks for one or more of the detected objects, where each track is a model of a position of a target object represented by the corresponding detected object in the image space of the image. For example, the motion model may be used to predict the target (detected) object's position based on its past trajectory as determined from a series of images. Localization system

[0085] Apparatus 100 is configured to generate measurements related to the location and pose (if appropriate) of the structure 101. Localization system 106 comprises one or more sensors and / or devices configured to measure orientation, position, acceleration and / or speed of the structure 101 as localization data, which may comprise real-time (or non-real-time) navigation data for a mobile structure 101. The form and / or type of the navigation data may depend on the nature of the mobile structure 101. For example, the localization system 106 may be configured to provide navigation data for a particular type of mobile structure 101, such as a drone, a watercraft, an aircraft, arobot, a vehicle, and / or other types of mobile structures. Alternatively, for a non-mobile structure 101 the localization system 106 may be configured to generate and / or maintain position and / or orientation measurements of the structure that are substantially constant over time.

[0086] In some embodiments, the localization system 106 comprises a positioning unit 107 and an inertial navigation system (INS) unit 108. Positioning unit 107 comprises one or more modules configured to provide a position of the structure 101, for example for the purpose of locating the structure 101 relative to the environment. For example, positioning unit 107 may be implemented according to any global navigation satellite system (GNSS), including a GPS, GLONASS, and / or Galileo based receiver and / or other device capable of determining absolute and / or relative position of structure 101 based on wireless signals received from an external source.

[0087] The positioning unit 107 generates position data indicating a geolocation of the structure 101 represented as a set of coordinates, such as a latitude and longitude pair, and output data related to the same part of the localization data. In some embodiments, positioning unit 107 may be configured to further determine a velocity or speed of the structure 101 (e.g., using a time series of position measurements). In various embodiments, one or more logic devices of apparatus 100 (not shown in Fig.1) may be adapted to determine a calculated speed of the structure 101 from position and / or location measurements generated by the positioning unit 107.

[0088] INS unit 108 is configured to generate measurements of the orientation of the mobile structure 101. For example, INS unit 108 may comprise one or more sensors configured to measure the roll, pitch, and / or yaw of the mobile structure 101 and output data related to the same as part of the localization data.

[0089] In some embodiments, the INS unit 108 comprises one or more inertial sensors and / or devices such as gyroscopes, accelerometers, and / or other devices capable of measuring angular velocities / accelerations and / or linear accelerations (e.g., to measure a direction and magnitude) of the mobile structure 101. The gyroscope(s) and / oraccelerometer(s) of the INS unit 108 may be configured to generate measurements relative to a part of the INS unit 108, positioning unit 107, localization system 106 or another system of the mobile structure 101.

[0090] The localization system 106 generates location and pose measurements for the structure 101 in the global reference frame 100’. The structure 101 has a local structure reference frame 101’ (“local reference frame”) which, in some embodiments, has a coordinate system (e.g., 3D cartesian) with an origin at a predetermined part of the localization system 106 or structure 101 (e.g., an antenna). Controller 110, or another logic device, is configured to maintain an indication of a relative positional and / or pose deviation or offset between the local reference frame 101’ and the imaging reference frame 102’. The location and pose measurements provided by the localization system 106 are thereby processed (e.g., by the controller 110 or other logic device) to convert object coordinates in the global reference frame 100’ to local coordinates in the local reference frame 101’, and subsequently to coordinates in the imaging reference frame 102’, in accordance with the techniques described herein.

[0091] In some embodiments, the localization system 106 generates measurements of orientation, position, acceleration and / or speed of the structure 101 relative to a known measurement point of the structure 101. For example, the sensors and / or devices of the localization system 106, such as the GNSS / GPS receivers of the positioning unit 107, and / or the gyroscope(s) and / or accelerometer(s) of the INS unit 108, may be implemented in a common housing and / or module that is attached to, or located on, the structure 101 at the measurement point. In other embodiments, the sensors and / or devices of the localization system 106 are movable relative to the structure 101, such as when the respective sensors and / or devices are carried by a user of the structure 101. In such embodiments, the controller 110 or other logic devices of the apparatus 100 maintain an indication of the global reference frame 100’ frame and a relative spatial arrangement between reference frames of the imaging system 102 and the structure 101 that are configured to generate the imaging and localization data respectively.

[0092] The form and representation of the spatial arrangement varies according to the configuration of the imaging system 102 relative to the structure 101. For a fixed relative relationship, the spatial arrangement may be a function indicating a position and orientation offset of the one or more imaging sensors 103 to a part of the localization system 106 that defines the local reference frame 101’ (e.g., an antenna). For a moveable relative relationship, the spatial arrangement is a function of time and accounts for changes in the position and / or orientation of the one or more imaging sensors 103 with respect to the localization system 106. For example, the controller 110 may be configured to dynamically track the position and orientation of the one or more sensors 103 relative to a point on the structure 101 to which the sensor(s) 103, and / or the imaging system 102, are coupled to. Sensing system

[0093] Sensing system 118 comprises one or more maritime sensors 120 configured to receive maritime data indicating the presence of, for example, ships, markers, hazards and other entities within the maritime environment. Maritime sensor(s) 120 may implemented as one or more devices configured to receive data from corresponding maritime information sources, such as but not limited to: an Automatic Identification System (AIS); an Electronic Navigational Chart (ENC) system; and an Automatic Radar Plotting Aid (ARPA) system, and to provide corresponding sensing data to one or more other components or devices of the apparatus 100, such as the controller 110.

[0094] In some embodiments, the maritime sensor(s) 120 receive a real-time data stream representing, at least, information associated with entities within the maritime environment. The sensing system 118 processes data received by the sensor(s) 120 and generates the sensing data to provide an indication of corresponding objects that may be expected to appear within the one or more images of the environment generated by the imaging system 102 (referred to herein as “proposed” objects).

[0095] In various embodiments, maritime sensor(s) 120 generate streams of sensing data corresponding to the respective maritime information sources. The maritimesensor(s) 120 may be configured to generate the sensing data based on input data provided by the apparatus 100. For example, the ARPA data stream of maritime sensor(s) 120 may be generated using measurements of a one or more other systems 170 of the structure 101 configured to detect and track other vessels or structures in a vicinity of the structure 101.

[0096] Alternatively, or in addition, the maritime sensor(s) 120 may be configured to generate one or more streams of the sensing data based on input data received from an auxiliary system 180. In some embodiments, the auxiliary system 180 is a computing system, comprising a processing device 182 and a database 184, residing externally to the apparatus 100 and, in some implementations, located physically apart from the structure 101. The auxiliary system 180 is configured to provide input data to the maritime sensor(s) 120 of the sensing system 118, or to other sensors or devices of the apparatus 100, via a communications network 150, which may include wireless and / or wired transmission media and one or more local or wide area networks. Controller

[0097] Controller 110 is configured to exchange data with components of the apparatus 100, including the imaging system 102, detection system 104, localization system 106 and sensing system 118, to execute, store and receive instructions for controlling operations of the apparatus 100.

[0098] Controller 110 is configured to receive: imaging data comprising at least one image generated by the imaging system 102 capturing the maritime environment relative to the structure 101; detection data comprising one or more detected objects of the at least one image, where the one or more detected objects are generated by the detection system 104 operating on the at least one image; and sensing data from the maritime sensor(s) 120 indicating one or more proposed objects of the at least one image. The controller 110 is configured to process the imaging data, the detection data, and the sensing data to automatically label the one or more images to provide improved detection and tracking of objects in the maritime environment.

[0099] In some embodiments, the imaging data and the detection data are received by the controller 110 as a live data stream provided by the respective imaging system 102 and detection system 104 components in real-time. Alternatively, the imaging and detection data may be provided to the controller 110 at a time after the generation of the respective data by the imaging system 102 and the detection system 104 (i.e., such that the controller 110 performs offline processing of the data).

[0100] In some embodiments, controller 110 is implemented as a standalone computing device, and comprises a central system bus (not shown), a memory system 116, one or more processors 115, and a communications module 130. The processor(s) 115 may be any microprocessor which performs the execution of sequences of machine instructions, and may have architectures consisting of a single or multiple processing cores such as, for example, a system having a 32- or 64-bit Advanced RISC Machine (ARM) architecture (e.g., ARMvx). The processor(s) 115 issues control signals to other device components via the system bus, and has direct access to at least some form of the memory system 116.

[0101] The memory system 116 provides internal media for the electrical storage of the machine instructions required to execute the user application. The memory system 116 may include random access memory (RAM), non-volatile memory (such as ROM or EPROM), cache memory and registers for fast access by the processor(s) 115, and high volume storage subsystems such as hard disk drives (HDDs), or solid state drives (SSDs).

[0102] In some embodiments, the processes executed by the controller 110 are implemented as programming instructions of one or more software modules stored on non-volatile storage of the memory system 116. The modules include: a filtering unit 111 configured to analyze and process the sensing data received from the sensing system 118; a projection unit 112 configured to project, using localization data associated with the structure 101 and / or the imaging system 102, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and a labelling unit 113 configured togenerate label data to label the at least one image by associating the one or more projected proposed objects generated by the projection unit 112 with the one or more detected objects of the detection data generated by the detection system 104. In some other embodiments, the modules may be implemented as one or more dedicated hardware components of the computing device, and / or another logic or processing device such as field programmable gate arrays (FPGAs) and / or application-specific integrated circuits (ASICs).

[0103] Memory system 116 may also include one or more general application programs providing methods, data structures or other software services that define data or perform functions as required by the controller 110 (e.g., an operating system). The data and instructions may reside in multiple parts of the memory system 116, including registers, cache, main memory, and high volume storage.

[0104] In the some embodiments, one or more of the imaging system 102, the detection system 104, localization system 106, and / or sensing system 118 are connected to the controller 110 via a specialized I / O connector (not shown) enabling the exchange of data between the controller 110 and other systems or components in real-time, or substantially real-time. In some embodiments, the controller 110 is configured to store the received or otherwise obtained images, detected objects, proposed objects, and / or associated labels as a function of time in order to enable post- processing of the data. In other embodiments, the data received by the controller 110 is only processed dynamically in real-time, for example by the invocation of the software modules 111, 112, 113 with the received data.

[0105] Communications module 130 is configured to enable the establishment of a logical connection between the controller 110 and other components of the apparatus 100 through a wireless or wired transmission media. For example, communications module 130 may be configured as one or more devices that receive and transmit sensor signals, control signals, and other signals from and to elements of apparatus 100 using a variety of wired and / or wireless communication techniques, including voltage signaling, Ethernet, WiFi, Bluetooth, Zigbee, Xbee, Micronet, or other medium and / orshort range wired and / or wireless networking protocols and / or implementations. In such embodiments, each system of apparatus 100 may include one or more modules supporting communication with the communications module 130 of the controller 110, such as for example a network interface implementing the IEEE 802.xx family of networking protocols.

[0106] The controller 110 implements one or more service modules including a data storage and retrieval module (not shown) enabling data to be stored in, and retrieved from, a data store 117. In some embodiments, the data store 117 includes, for example, a SQL database and / or a file management system. In some embodiments, the data store 117 is formed within the memory system 116 and includes data tables, or other structures, configured to store any one or more of the: imaging data, detection data, localization data and / or navigation data, sensing data, and label data.

[0107] Controller 110 further comprises a user interface 114 configured to accept user input and / or provide feedback to a user of the apparatus 100. For example, user interface 114 may comprise one or more displays, touch screens, and / or input peripheral devices, such as but not limited to a keyboard, mouse, steering wheel, or any other device adapted to provide user input as a type of signal and / or information to the controller 110. The user input may include directives for the apparatus 100 to execute instructions, such as software instructions, implementing any of the various processes and / or methods described herein.

[0108] In some embodiments, the user interface 114 is configured to receive user input that: determines a particular communication protocol and / or parameters for communication between the apparatus 100 and a user, or a user device; selects one or more view or perspectives for one or more of the image(s) of the imaging data, the detected object(s) of the detection data, the proposed object(s) of the sensing data, the projected proposed object(s) generated by projecting the proposed object(s) into each of image(s), the labels of the generated label data, and / or the position, orientation, and / or any other measurements of the localization data and / or navigation data; adjusts a position and / or orientation of imaging system 102 and / or the structure 101; and / orotherwise facilitates operation of apparatus 100 and the associated systems and / or devices. Motor control system

[0109] In some embodiments in which structure 101 is a mobile structure, controller 110 is further configured to provide signals and data to a motor control system 109. The motor control system 109 comprises one or more devices or components configured controlling the movement of the mobile structure 101 in response to the signals and data received from the controller 110. In some embodiments, the motor control system 109 comprises a propulsion system implemented as a propeller, turbine, or other device configured to generate thrust-based propulsion, a mechanical wheel or other device configured to generate tracked propulsion, a sail, fan, or other device configured to generate propulsion based on air around the mobile structure 101, and / or other devices configured to generate a motive force to move the mobile structure 101. The controller 110 is configured to instruct the propulsion system to set or vary the amount of motive force provided to the mobile structure 101 such as to control a speed of the movement of the mobile structure 101.

[0110] In various embodiments, the motor control system 109 further comprises one or more devices or components configured to steer or control a direction of the movement of the mobile structure 101. For example, the motor control system 109 may include a steering sensor / actuator to set and / or adjust a direction of the generated motive force and / or thrust, such as to steer the mobile structure 101 in a desired heading expressed relative to the world coordinate frame of mobile structure 101. Controller 110 determines the desired heading by processing any one or more of the navigation data, imaging data, detection data, sensing data and label data, as generated by the processes and methods described herein.

[0111] For example, the controller 110 may use a current position and an orientation of the mobile structure 101, as indicated by the navigation data, in combination with at least one image captured by the imaging system 102 of a real-time maritime environment of the mobile structure 101, and one or more corresponding labels ofobjects in the at least one image as generated by the controller 110 according to the methods described herein, to set or adjust a heading and a speed of the mobile structure 101 such as to avoid the objects while moving through the environment. Other systems

[0112] Apparatus 100 may include one or more other sensors, devices, or systems 170 such as for example a sonar system, a steering actuator, a speed sensor, and / or other modules configured to measure the dynamic characteristics of the structure 101. For example, the other systems 170 may include a radar system, or other ranging sensors, such as one or more light detection and ranging (LIDAR) devices, laser scanning devices, stereo cameras, and / or radar devices, and / or other environmental sensors, such as a humidity sensor, a wind and / or water temperature sensor, a barometer, and / or a salinity sensor such as a sea surface salinity sensor.

[0113] The skilled person in the art will appreciate that many other embodiments may exist including variations in the configuration of apparatus 100, and the distribution of program data and instructions to execute the processes and methods described herein. For example, the imaging system 102 and the detection system 104 may be implemented as part of a computer-vision system residing externally to the apparatus 100. In some examples, localization system 106, motor control system 109, sensing system 118 and motor control system 109 are also external to the apparatus 100. Systems external to the apparatus 100 may be configured to communicate with the controller 110 via the communications network 150. For example, each external system may include one or more communication devices, such as a modem or transceiver, configured to exchange data with communications module 130 via communication network 150. Example configuration

[0114] Fig.2a illustrates an example configuration of the apparatus 100 for performing maritime object detection, and optional tracking, through the fusing of vision-based object detections and maritime sensing data to automatically label imagesof a maritime environment relative to a mobile structure 101 (also referred to as a “vessel”, or “ves” as shorthand herein).

[0115] In other embodiments, the apparatus 100 may be configured to perform maritime object detection, and optional tracking, of an environment relative to a non- mobile structure through techniques that are analogous to those described herein for a mobile structure 101 (e.g., with various parameters determining the local reference frame 101’ being constant, and where the localization system 106 is configured to maintain an indication of the fixed parameters, without the use of an INS unit 108).

[0116] In the configuration depicted by Fig.2a, imaging system 102 comprises a set of imaging sensors in the form of one or more optical cameras 103. The optical cameras 103 are calibrated with one or more intrinsic parameters including, for example, an optical center, focal length, and one or more distortion parameters. In some embodiments, one or more of the optical cameras 103 may include a maritime camera, such as for example a FLIR M300C Marine High Definition Camera.

[0117] The optical cameras 103 are configured to generate images of the maritime environment in the imaging reference frame 102’ of the imaging system 102. The apparatus 100 maintains a set of extrinsic imaging system calibration parameters describing a relationship between the imaging reference frame 102’ and the global reference frame 100’, which may for example describe a coordinate format for geographical location and orientation values used by external sources of maritime data. For example, the extrinsic imaging system calibration parameters may include a set oftranslation parameters (^^cam, ^^cam, ^^cam) describing a translation offset between thereference frames in 3D cartesian coordinates, and a set of orientation parameters(ϕcam, θcam, ψcam) describing an orientation offset between the reference frames as aset of roll, pitch and yaw values.

[0118] In the configuration depicted by Fig.2a, the detection system 104 is a machine learning (ML) model that is configured to perform image-based object detection and tracking using a maritime object detector, such as for example, an image classifierbased on Faster-RCNN or YOLO techniques. The ML model 104 receives the set of images of the maritime environment generated by the optical cameras 103 as input and identifies target objects within one or more of the images using a pre-trained object network of the ML model 104. The ML model 104 outputs a set of image object detections (i.e., one or more detected objects, also referred to as “image-based detections” ^^Detect,img) as detection data. In some implementations, the detection data indicates markers (e.g., bounding boxes) that correspond to positions within the respective images.

[0119] Localization system 106 comprises a positioning unit 107 in the form of a GPS / GNSS receiver configured to provide a geolocation of the mobile structure 101 in the global reference frame 100’, as given by the geographic coordinates (latitude,longitude), denoted as {φves,world, λves,world} . The localization system 106 comprises anINS / Compass 108 having at least one gyroscope configured to provide measurements of the orientation of the mobile structure as a set of roll, pitch, and yaw values in theglobal reference frame 100’ as denoted respectively by {ϕves,world, θves,world, ψves,world}.

[0120] The apparatus 100 receives data from multiple auxiliary maritime information sources via the sensing system 118. In the configuration depicted by Fig.2a, the sensing system 118 comprises individual sensors for: an Automatic Identification System (AIS); an Electronic Navigational Chart (ENC) system; and an Automatic Radar Plotting Aid (ARPA) system. The individual sensors are configured to provide at least geographical locations for maritime objects relevant to the maritime environment captured by the imaging system 102.

[0121] Controller 110 is configured to receive: imaging data and detection data from the detection system 104; localization data, which may be in the form of navigation data including the measurements of position and orientation of the mobile structure 101 from the localization system 106; and sensing data comprising respective data streams of the AIS, ENC and ARPA sensors from the sensing system 118. The one or more images representing the captured maritime environment of the imaging data are denoted ^^IN.

[0122] Projection unit 112 of the controller 110 projects the proposed objects contained within the data streams, each denoted as ^^AIS, ^^ENCand ^^ARPAinto the image pixel space of the image(s) ^^INof the imaging data using the pose (i.e., position and orientation) measurements of the navigation data. The projected proposed objects as denoted respectively as ^^AIS,img, ^^ENC,imgand ^^ARPA,img.

[0123] The controller 110 fuses the projected proposed objects ^^AIS,img, ^^ENC,imgand ^^ARPA,imgwith the image-based detections ^^Detect,imgand provides the resulting data to the labelling unit 113. In the configuration of Fig.2a, the labelling unit 113 comprises an object detector 113a and a pseudo-label generator 113b.

[0124] Object detector 113a may be set to determine erroneous detections on a frame- by-frame basis, with each image being processed independently. Alternatively, the object detector 113a may be set to determine the erroneous detections by conducting the projection of proposed objects over a series of images, such as to identify a track identity of the detected and projected proposed objects.

[0125] Pseudo-label generator 113b labels the image(s) of the imaging data by associating the projected proposed objects ^^AIS,img, ^^ENC,imgand ^^ARPA,imgwith the image-based detections ^^Detect,img. In some configurations, the labelling unit 113 generates labelling data that indicates errors associated with detections ^^Detect,imgrelative to the proposed objects ^^AIS,img, ^^ENC,imgand ^^ARPA,imgin the image space. For example, one or more false negative detections are determined for an image based on the association, which may include, for example, evaluating a distance metric between locations of the detected objects (i.e., from ^^Detect,img) and a given (base) projected proposed object (i.e., from ^^AIS,img, ^^ENC,imgand ^^ARPA,img) in the image space coordinates. One or more false positive detections are determined for an image by evaluating a distance metric between locations of the projected objects (i.e., from ^^AIS,img, ^^ENC,imgand ^^ARPA,img) and a given (base) detected object (i.e., from ^^Detect,img) in the image space coordinates.

[0126] The pseudo-label generator 113b performs the association of the projected proposed objects (^^AIS,img, ^^ENC,imgand ^^ARPA,img) with the one or more detected objects (^^Detect,img) using distance based criteria calculated over the image space.

[0127] In some configurations, the labelling unit 113 associates the one or more projected proposed objects (^^AIS,img, ^^ENC,imgand ^^ARPA,img) with the one or more detected objects (^^Detect,img) using sensor-specific object confidence metrics (i.e., for each of the AIS, ENC, and ARPA data streams).

[0128] For example, the identification of false negatives may be performed by determining a number of associations of a projected object to each detected object, where the number of associations varies depending on the auxiliary source of the projected object (e.g., a lower number for AIS due to its low false negative rate). The number of associations needed to register a false negative or false positive error may be determined by a confidence score calculated using the number of associations and based on one or more of the auxiliary sources of the projected objects.

[0129] In some configurations, pseudo-label generator 113b labels an erroneous or absent detection to indicate the proposed object or the detected object (e.g., by providing a bounding box of the object within the image). The pseudo-label generator113b provides label data representing object tracks (i.e., labelsB labels ) for one or moreoutput images ^^OUTof the input image data ^^^^^^. In some configurations, the one or more images output by the pseudo-label generator 113b include a subset of the imagesI flagged that have been identified or flagged for use to improve the ML model 104 (i.e.,the challenging frames for which the detections initially determined by the ML model 104 are inaccurate).

[0130] For example, the controller 110 may initiate retraining, adaptation, or other modification of the ML model 104 using the subset of flagged images and corresponding object labels to improve the future detection of objects by the ML model 104 in response to receiving further input images. This reduces the number of futurefalse negative and / or false positive errors made by the ML model 104 during maritime objection detection.

[0131] Fig.2b schematically illustrates a top-down view 200’ of the operation of the apparatus 100 to perform object detection and labelling according to the configuration of Fig.2a. In the embodiment depicted by view 200’, apparatus 100 is located on a mobile structure 101 and is configured to image a portion of the environment visible in a field-of-view 202’ of the imaging system 102 relative to the mobile structure 101. Objects 210, 212, 214 and 216 are located within the environment. Maritime sensors provide sensing data indicating ARPA, AIS and ENC proposed objects corresponding to the objects 210, 212 and 214, and 216 respectively.

[0132] Fig.2c illustrates a set of images 200 of the maritime environment and corresponding labels generated by the apparatus 100 of the view 200’ depicted in Fig. 2b. Apparatus 100 processes object proposals obtained from each sensor of the sensing system 118 in the global reference frame 100’, with detections generated by the ML model 104 (depicted as bounding boxes 220, 222 for detected objects 210, 212), and fuses the objects in the image plane of the imaging system 102. The apparatus 100 subsequently performs automatic labelling of the objects to generate a set of flaggedframes I flagged 250 (e.g., those images with incorrect or absent detections), and acorresponding set of pseudo-labels B labels 240 representing corrected object tracksdetermined from the fused object data. Method for maritime object detection

[0133] Fig.3 illustrates an exemplary method 300 executed by the controller 110 of the apparatus 100 of Fig.1 for performing maritime object detection, according to the proposed techniques.

[0134] At step 302, the controller 110 receives imaging data indicating at least one image of the maritime environment (“captured image(s)” herein). The captured image(s) are generated by the imaging system 102 as configured relative to thestructure 101. The controller 110 also receives receiving detection data indicating one or more detected objects of the captured image(s). The detected object(s) are denoted by the set of image-based detections ^^Detect,imgin the examples below.

[0135] Detection data is generated by an object detector, such as the ML model 104 of Fig.2a, operating on the image(s) as input. In some embodiments, the captured images(s) of the maritime environment may be generated by an optical imaging system 102 (also referred to as an “optical sensing” system) with at least one camera 103. The captured image(s) may be generated as a single image, a set of independent images, or as a series of images captured with temporal continuity over a time window, such as for example as a portion of a video feed of the maritime environment.

[0136] At step 304, the controller 110 receives, from the one or more maritime sensor(s) 120 of the sensing system 118, sensing data indicating one or more proposed objects of the captured image(s). In some embodiments, the sensing data is a set of one or more data streams of the maritime sensor(s) 120 which are devices configured to receive data from one or more of the following maritime data sources: an Automatic Identification System (AIS); an Electronic Navigational Chart (ENC) system; and an Automatic Radar Plotting Aid (ARPA) system. In the examples described herein, the apparatus 100 utilizes data from each of the AIS, ENC, and ARPA sources, however it will be appreciated that any combination of one or more data sources may be used in other examples. Proposed objects from the AIS, ENC, and ARPA data sources are denoted as ^^AIS, ^^ENCand ^^ARPArespectively.

[0137] At step 306, the controller 110 projects the one or more proposed objects (i.e., ^^AIS, ^^ENCand ^^ARPA) into each of the captured image(s) to generate one or more corresponding projected proposed objects (i.e., ^^AIS,img, ^^ENC,imgand ^^ARPA,img). To perform the projection, the controller 110 uses localization data generated by the localization system 106. In various embodiments, the localization data comprises position and pose measurements associated with the structure 101 (e.g., to provide a geographical location and orientation of the structure 101, which may be determined and / or updated in real-time by the localization system 106). The controller 110 isconfigured to determine corresponding position and pose measurements associated with the imaging system by processing the localization data (e.g., based on a known spatial relationship between the local reference frame 101’ and the imaging system 102). In other embodiments, the localization data directly specifies position and / or pose measurements of the imaging system 102.

[0138] At step 308, the controller 110 generates label data to label the captured image(s) by associating the one or more projected proposed objects (i.e., ^^AIS,img, ^^ENC,imgand ^^ARPA,img) with the image-based detections ^^Detect,img. The labels generated by the controller 110 may comprise image frame independent labels, such as to identify erroneous or absent detections on a per-image basis of the captured image(s). Alternatively, or in addition, the controller 110 may generate pseudo-labels over a plurality of images to validate object tracks with the temporal information provided by the maritime sensor data.

[0139] The imaging data and label data generated by the execution of the method 300 advantageously provides improved maritime object detection and tracking for the environment. This enables apparatus 100 to provide navigational guidance to a vessel operating within the maritime environment, including (mobile) structure 101 or another mobile structure.

[0140] Further details for implementing the method 300 are provided in the following sections with respect to an exemplary embodiment of apparatus of 100, as depicted in Figs.1 and 2a-c, comprising a mobile structure 101. However, it will be appreciated that other embodiments exist in which the apparatus 100 comprises a non-mobile structure 101. Maritime sensing

[0141] Fig.4a illustrates an exemplary method 400 performed by the controller 110 for receiving and processing the sensing data. At step 402, the controller 110 determines a location and pose of the mobile structure 101 and / or the imaging system102. The location and pose is provided by, or derived from, the localization data of the localization system 106.

[0142] At step 404, the controller 110 receives sensing data from the sensing system 118 from AIS, ENC and ARPA sensor(s) 120. In some embodiments, the sensing data is retrieved as a set of real-time data streams, with the controller 110 configured to select data relating to one or more proposed objects ^^AIS, ^^ENCand ^^ARPAbased on the location and pose provided by the localization data, and relative to the mobile structure 101 and / or imaging system 102. For example, the one or more proposed objects may be selected as those objects specified by any one of the maritime sources that are located within a field-of-view of the imaging system 102, and / or a vicinity of the mobile structure 101.

[0143] Optionally, at step 406 the controller 110 processes the sensing data to filter the one or more proposed objects ^^AIS, ^^ENCand ^^ARPAbased on the size and / or dimensions of the respective proposed object in an image plane.

[0144] For example, the AIS and ENC sources provide information about the exact object distance and size. The controller 110 is configured to estimate the pixel height of objects when projected from the global reference to the image plane. Based on this calculation, the controller 110 sets a threshold to filter out objects that have a height of less than β pixels. In some implementations, an ENC source may indicate that there are channel markers of a certain size in front of the mobile structure 101. The controller 110 may be configured to remove channel marker object candidates that have a height of less than β pixels in the image on the basis that there is too little pixel information to be of any benefit in improving image detections.

[0145] Fig.4b illustrates a representation 450 of an object's height in the image and in the real-world. With the assumption that the object's height in pixel space and real world are parallel, the controller 110 is configured to calculate the object's pixel height ℎpixas shown in Eqns. (1.i), (1.ii) and (1.iii) where the image object height (pixels) is denoted as ℎ, the real-world object height (mm) as ^^, the focal length ^^ (mm), distanceto object ^^ (mm), image sensor height (mm) as ^^^^and image pixel resolution height (pixels) as ^^ℎ: (1.i)(1.ii)(1.iii)

[0146] The controller 110 is configured to filter out candidates for the proposed objects if the object height ℎ is less than threshold ^^. Object coordinate conversion and projection

[0147] Fig.5 illustrates an exemplary method 500 executed by the controller 110 for generating the one or more projected proposed objects (i.e., ^^AIS,img, ^^ENC,imgand ^^ARPA,img) for an image of the captured image(s).

[0148] At step 502, the controller 110 determines a geographical location of a proposed object in the global reference frame 100’. In some embodiments, the controller 110 determines the geographical location of the proposed object by using the localization data of the mobile structure 101 to convert geographical coordinates of the proposed object to the global reference frame 100’. Conversion of range-bearing to geographic coordinates

[0149] Conversion of objects from a maritime sensor source to a geographic (latitude, longitude) coordinate involves one or more transformation operations performed by the controller 110 on object data from the sensing data. For example, the controller 110may be configured to convert the range / bearing from ARPA data into a geographic coordinate.

[0150] While AIS and ENC data sources provide geolocation in the global reference frame (“world frame”) for proposed objects, ARPA often represents objects in a range- bearing format relative to the heading ^^vesof the mobile structure 101.The controller 110 is therefore configured to determine the geographical location of the ARPA object by using the localization data to convert its geographical coordinates to the global reference frame 100’.

[0151] Detected objects in ARPA contain a range ^^ and relative bearing ^^ (in radians). The controller 110 converts the range and relative bearing representation to a relative cartesian location offset using Eqns. (2.i) and (2.ii), where ^^vesis the mobile structure 101 heading, and Δ^^ and Δ^^ are the eastward and northward location offset from the mobile structure’s geolocation respectively in metres. ^x = d ^sin( ^ ves + ^ ) (2.i)^y = d ^cos( ^ ves + ^ ) (2.ii)

[0152] From this, the controller 110 converts the object's Δ^^ and Δ^^ to a relativegeolocation (latitude ^^obj and longitude ^^obj) with Eqns. (3.i) and (3.ii), where ^^ =6378137 and ^^ = 6356752.3142 are the radii of the semi-major and semi-minor axesof the WGS84 spheroid respectively.

[0153] After converting to degrees, the controller 110 adds the relative geolocation of the object to the latitude ^^ves, and longitude ^^ves, as determined by the positioning unit 107, to determine the absolute geolocation of the ARPA object.

[0154] In some embodiments, the controller 110 uses a geoid model that is different to the WGS84 spheroid to determine the relative geolocation, such as for example the GRS80 spheroid. Alternatively, or in addition, in various embodiments the controller 110 implements one or more gravitation models such as an EGM2008 model to determine the relative geolocation. Unlike spheroids, gravitation models often advantageously have data values that can be easily 2D linear interpolated to represent the ocean surface as a function of latitude and longitude. It will therefore be appreciated that the controller 110 can be configured to determine the relative geolocation of an object using any appropriate geoid and / or gravitation model without any substantial adjustment to the methods and techniques described for use with the WGS84 geoid model.. Projection of an object from the global reference to the image plane

[0155] At step 504, the controller 110 projects the geographical location of the proposed object to an image location within the image. The controller 110 performs the projection of the world geographic coordinates of the object to corresponding image plane pixel coordinates using localization data, including position and pose measurements, such as for example provided by the positioning unit 107 and INS unit 108.

[0156] In various embodiments, the controller 110 executes one or more processes to perform the object projection step 504. The controller 110 obtains the object's pixel coordinates by converting the object's geolocation to cartesian coordinates and projecting the same into the image plane. Projection into the image plane involves first transforming the object's geolocation from the world frame, ^^world, to its cartesian coordinates in the imaging (“camera”) frame, ^^cam, and subsequently projecting intothe image plane with the intrinsic camera parameters to obtain the pixel coordinates ^^pix.

[0157] Fig.6 illustrates a method 600 performed by the controller 110 for performing the object projection step 504 according to some embodiments, which comprises two steps: (i) (Step 604) transforming the geographical coordinates of the proposed object from the global reference frame to generate corresponding local coordinates in a local reference frame 101’ of the structure 101; and (ii) (Step 606) projecting the corresponding local coordinates into a plane of the image to obtain a set of pixel coordinates representing a position of the projected proposed object in the image. Details of exemplary processes for performing these operations are provided below. Transforming the object from the global (world) frame to the local frame

[0158] At step 604, the controller 110 converts the object's geographic coordinates inglobal reference frame 100’ Oworld = {φobj,world,  λobj,world} to cartesian coordinates inthe local reference frame 101’ ^^local = {^^local, ^^local, ^^local}. In some embodiments, thelocal frame is centered on the GNSS / INS unit 108 where the forward +^^ axis faces a predetermined direction relative to the mobile structure 101 (e.g., the bow of the vessel).

[0159] The GNSS / INS unit 108 provides the geolocation of the mobile structure 101orientation {ϕves,world, θves,world, ψves,world} in the worldframe; the local reference frame 101’ is thereby referred to as an “INS frame” in below. In this example, both the world and INS frame are in the North-East-Down (NED) convention of reference.

[0160] Fig.7 illustrates an exemplary WGS84 spheroid 700 depicting the process of obtaining cartesian coordinates in the INS frame, as performed by the controller 110 during step 604. The cartesian coordinates of the object 210 are the north and east arc length relative to the mobile structure 101 (e.g., ego-vessel), with the assumption that itlies directly on the WGS84 (see [3]) spheroid at ^^ = 0.

[0161] For accurate arc length calculations, the controller 110 obtains accurate radii of the earth at the latitude of the mobile structure 101. Firstly, the controller 110 computes the geocentric radius ^^gcat φvesin Eqn. (4) where φvesis in radians. This provides the radius for computing the north distance ^^localwhere ^^ and ^^ are the semi- major and semi-minor radii of the WGS84 spheroid (see [3]).

[0162] To compute the east distance ^^local, the controller 110 determines a cross- section at φvesto provide a smaller circle κ for which the controller 110 calculates the radius ^^^^as

[0163] The controller 110 converts the object's geographic coordinates ^^world={φobj,world, λobj,world} to cartesian coordinates ^^local-world = {^^1, ^^1, ^^1} by computing thecircle arc length of the relative geographic coordinates, ^^obj − ^^ves, of the object withrespect to the INS frame as shown in Eqns. (6.i), (6.ii) and (6.iii). At this stage, ^^local-worldis centered on the INS frame but still aligned in orientation with the world frame.z1 = 0 (6.iii)

[0164] To complete the transformation of the coordinates to the INS frame, thecontroller 110 rotates the ^^local-world by the 3 × 3 rotation matrixto obtain ^^localin Eqn. (7).Projecting the object from the local frame to the image plane

[0165] With reference to method 600 of Fig.6, at step 606 the controller 110 transforms object coordinates in the local reference frame 101’ to the imaging system reference frame and subsequently projects the object point into the image. In this example, the objects in the local reference frame 101’ are denoted as ^^ins, since local reference frame 101’ is defined relative to the positioning unit 107 and / or INS unit 108, and objects in the imaging system (camera) frame are denoted ^^cam.

[0166] Fig.8 illustrates an exemplary method 800 performed by the controller 110 for the step 606 of method 600 for projecting the object from the local frame to the image plane. First, the controller 110 transforms each point of the projected proposed object in a local reference frame 101’ of the structure (i.e., ^^AIS,ins, ^^ENC,insand / or ^^ARPA,ins) to a point in a reference frame of the imaging system 102, based on a position and / or orientation of the imaging system 102 relative to the structure 101.

[0167] Fig.9 illustrates a depiction 900 of the transformation of the point of the projected proposed object between representations in the INS frame of apparatus 100. Object coordinates in the INS frame are oriented in the NED convention of reference. Therefore, at step 802 the controller 110 transforms each point of the projected proposed object from an INS (NED) local representation 902 to an INS Front-Left-Up (FLU) representation 904 with ^^FNLEUD.

[0168] It will be appreciated that step 802 of the exemplary method 800 utilizes INS (NED) and INS (FLU) representations for the local reference and image reference respectively. However, in some other embodiments the controller 110 performs analternative step 802’ involving a transformation from an alternative local representation to another corresponding representation (e.g., where the controller 110 transforms each point of the projected proposed object from an INS East North Up (ENU) local representation 902’ to an INS Forward, Right Down (FRD) representation 904’).

[0169] At step 804, the controller 110 applies the homogenous 4 × 4 transformmatrix ^^cianms(905) to transform the coordinates from the FLU representation 904 to the camera frame 906. This is process is shown in Eqns. (8.i) and (8.ii). Oins, flu = R FNLEUDO ins (8.i)(8.ii)

[0170] In the described embodiments, controller 110 converts ^^ins, fluto homogenous coordinates before multiplying with ^^cianms. In other embodiments, this process may be performed in a single joint transformation matrix that combines ^^FNLEUDand ^^cianms.

[0171] At step 806, the controller 110 maps object point ^^cam(910) into the image with the intrinsic camera parameter matrix ^^ to obtain pixel coordinates ^^uv(908). In some embodiments, the projection of the object point into the image plane involves the controller 110 performing a perspective projection by normalizing x,y coordinates by their depth to obtain a normalized point ^^^^. The controller 110 subsequently applies thematrix ^^ which contains the focal lengths ^^^^, ^^^^, and principle point coordinates ^^^^, ^^^^in pixels. The projection to image coordinates is shown in Eqns. (9.i) and (9.ii) where^^, ^^ are ^^, ^^ coordinates in the pixel space.

[0172] In some embodiments, the controller 110 applies a distortion model to each point of the projected proposed object in the reference frame of the imaging system. The controller 110 determines if an image has not been undistorted, and if so apply an appropriate distortion model to distort the object points resulting from the projection process.

[0173] For example with a rectilinear lens, the controller 110 applies the plumb bob distortion model to distort the object points. This involves the controller 110 first applying the distortion to ^^^^in Eqn. (9.i) before the projection to the image plane. Thedistortion model is described by Eqn. (10) where (^^1, ^^2, ^^3) and (^^1, ^^2) are theradial and tangential distortion coefficients respectively.

[0174] For fisheye lenses, the radial distortion can be corrected with Eqn. (11) whereθ = ^^^^^^−1(^^).

[0175] After distorting the object points ^^^^, the controller 110 projects the points into the image plane as shown in Eqn. (12) to obtain the distorted image points ^^^^and ^^^^.

[0176] With reference to Fig.5, at steps 506 and 508, the controller 110 repeats steps 502 and 504 for each proposed object in the image, and repeats the execution of method 500 for each captured image. Generating labels for the captured images

[0177] The controller 110 is configured to perform label generation for projected proposed objects in the image space (i.e., ^^AIS,img, ^^ENC,imgand ^^ARPA,img) by processing of the captured image(s) individually (i.e., to correct errors in per-image detections) and / or by validating tracks of the objects detected over time through a plurality of the captured images. Per-image labelling of false negatives and / or false positives

[0178] In some embodiments, the controller 110 is configured to generate label data (i.e., at step 306 of method 300) by associating each of the projected AIS, ARPA, ENC objects in the pixel space (^^AIS,img, ^^ENC,imgand ^^ARPA,img) with the one or more detected objects identified by the detection data (^^Detect,img).

[0179] Fig.10 illustrates an exemplary method 1000 performed by the controller 110 for image distance-based association between the one or more projected proposed objects and the one or more detected objects. At step 1002, the controller 110 retrieves the coordinates of a projected proposed object (i.e., an object from any of ^^AIS,img,^^ENC,img and ^^ARPA,img) in the image space, denoted as ^^pix = {^^u, ^^v}.

[0180] Then, at step 1004 the controller 110 calculates a distance value in the image between the respective projected proposed object ^^pixand each of the one or more detected objects identified by the detection data (^^Detect,img). For example, the controller 110 may calculate the distance value as a distance between: a position of an object detection marker (e.g., a bounding box) of the respective detected object ^^Detect,img; and the location of the respective projected proposed object, in theimage ^^pix = {^^u, ^^v}.

[0181] For each image frame the controller 110 determines a set of detection markers ^^pixand a set of objects ^^pixand determines a corresponding distance between eachobject and detection ^^(^^pix, ^^pix), as shown in Eqn. (13) using a Euclidean distance.

[0182] In other embodiments, alternative metrics may be used to determine the distance between each object and detection. For example, the controller 110 may be configured to determine a Manhattan distance or a Weighted Manhattan distance. Different distance metrics may be calculated in accordance with one or more different parameterizations of the coordinates of the projected proposed objects and the detected objects (e.g., as x / y degrees from the center of the camera 103).

[0183] Fig.11 is a depiction 1100 of a distance-based association between a detected object and projected proposed objects 1102 in a pixel coordinate space 908 of a captured image. Detected object ^^Detect,img(1106) has a bounding box 1104determined by the detection model of the detection system 104. Distance ^^(^^pix, ^^pix1)(1114) is determined between a center point of the bounding box (1110) of a first projected proposed object ^^pix1(1108) and the box 1104 of detected object ^^Detect,img(1106). The distance calculation is repeated between detected object ^^Detect,img(1106) and a second projected proposed object ^^pix2(1116) based on the bounding box of the same (1118).

[0184] With reference to method 1000 of Fig.10, at step 1005 the controller 110 utilizes at least one threshold to determine whether or not the projected proposed object matches to any of the one or more detected objects. For example, the controller 110 may use a single threshold value α defining the maximum pixel distance between a detection marker and a corresponding object. In some embodiments, the value of α is set or adjusted according to a degree of accuracy of the measurements of the localization data (e.g., the position / location and / or orientation values) generated by thelocalization system 106. The controller 110 is configured to maintain a counter for the number of matches (and optionally a number of non-matches) of each projected proposed object matched as ^^matched(and unmatched ^^unmatched) as according to Eqns. (14.i) and (14.ii).

[0185] At step 1006 the controller 110 repeats the distance-based association process to determine distances and matches for each object ^^pixin the set of objects ^^pix. In some embodiments, the controller 110 performs matching between each projected proposed object (or detected object) and a corresponding detected object (projected proposed object) that is closest within the image space.

[0186] The controller 110 processes at least the ^^matchedvalue of each projectedproposed object ^^pix to determine if there is at least one match (i.e., if ^^matched ≥ 1) forthat object to a corresponding detected object (represented by ^^pix) in the image. Ifthere is no match (i.e., ^^matched = 0) then the proposed object has not been detected(i.e., a false negative detection has occurred with respect to the one or more detected objects of the detection data). The controller 110 may be configured to determine the corresponding ^^matchedvalue for each detected object, where if there are no matches of proposed objects to the detected object then a false positive detection has occurred.

[0187] At step 1008, the controller 110 generates labels based on the number of matches of the projected proposed objects against the detected objects. In various embodiments, the controller 110 generates label data for each captured image to represent each projected proposed object as a set of projected object markers ^^′pix, with subsets denoting: respective ones of the projected proposed objects that do not match to any of the one or more detected objects (i.e., indicating one or more false negatives inthe image ^^′pix, FalseNeg); and respective ones of the detected objects that do not match to any of the one or more projected proposed objects (i.e., indicating one or more false positives in the image ^^′pix, FalsePos). In some embodiments, the label data includes data associated with the set of projected proposed objects ^^pixin addition to the set of projected object markers ^^′pix, ^^′pix, FalseNeg, ^^′pix, FalsePos. Validating object tracking

[0188] In various embodiments, the controller 110 is configured to generate label data for the captured images to validate a tracking of one or more detected objects over time, as determined by the detection system 104. In such embodiments, the captured images comprise a series of images of the maritime environment over a time window (e.g., frames from a continuous portion of a video feed).

[0189] The controller 110 is configured to validate a temporal sequence of detections associated with each of the one or more detected objects (^^Detect,img) against the one or more projected proposed objects (^^AIS,img, ^^ENC,imgand ^^ARPA,img). In some embodiments, object track validation is performed by determining a number of the matches (referred to as “hits” in this context) occurring over the series of images, for a respective detected object against a projected proposed object. Details of exemplary techniques, performed as one or more processes by the controller 110, for object track validation are provided below.

[0190] To fuse information provided by object detections (i.e., as represented by bounding boxes of detected objects) and projected proposed objects within the image space at time ^^, the controller 110 computes the distances between detected and proposed objects, as described above.

[0191] The detection data additionally comprises data from one or more object trackers (see [5] and [6]) indicating a track identity based on a confidence threshold σ and a “number of hits" threshold ^^hitsfor the bounding boxes. In some configurations,new tracks undergo a probationary period requiring ^^hitsto exceed σ confidence to validate their existence and minimize tracking of false positives. Track identities that do not pass their probationary period are terminated. In some configurations, track initialization occurs only when the presence of an object is confidently established.

[0192] In some embodiments, the controller 110 utilizes maritime sensor-specific object confidence metrics to determine false positive tracks over the series of captured images. For example, an AIS source has a typically low false negative rate, and thereby permits a high-confidence track initiation upon any detection. Conversely, an ARPA source may have a higher false positive rate thereby benefiting from matching the detections to the projected proposed objects (from the ARPA source) over a series of images to increase the confidence that an identified track is correct.

[0193] The controller 110 applies the aforementioned distance-based association technique to iteratively associate matches between each detected object and the one or more projected proposed objects to validate tracks identified in each image of the series. In some embodiments, the controller 110 updates or adjusts the identified track of a detected object (i.e., as representing the target object's motion model) according to the temporal matching.

[0194] In some embodiments, the sensing data indicates a heading and / or speed of the proposed object (e.g., AIS), and the controller 110 is optionally configured to use the heading and / or speed data of the sensing data to project the position of a target (detected) object forward in time (i.e., if the object is not detected in one or more of the images in the series of captured images, but has an associated identified object track indicated by the detection data). In some embodiments, the controller 110 is configured to remove markers of the target object from other images of the series of captured images if the target object remains undetected for a number ^^lostframes, where ^^lostmay be predetermined or set dynamically by the controller 110 (e.g., based on parameters of the imaging system 102, detection system 104, localization system 106, or other system of the apparatus 100).

[0195] In some embodiments, the controller 110 is configured to adjust the identified tracks of one or more detected objects, such as for example to smooth or filter the track parameters backward and / or forward in time.

[0196] The controller 110 generates label data indicating, for example, a validity or reliability of the one or more identified detected object tracks. In some embodiments, the label data represents one or more bounding boxes encompassing the corresponding one or more detected objects of each track in the image (e.g., to insert a bounding box into an image of the series of captured images corresponding to a detected object). Alternatively, or in addition, the controller 110 generates labels to indicate that one or more existing identified tracks (and / or the associated detected objects) are not valid or reliable, thereby enabling the filtering of the track(s) in a post-processing operation.

[0197] In some embodiments, the controller 110 determines the labels based on one or more track attributes, including length of track and number of corresponding detections within the track as generated by the detection system 104. In some embodiments, the controller 110 generates labels for select tracks as identified, for example, based on the number of matches / hits and / or the track properties. Labelled tracks (or, alternatively, unlabelled tracks) may be flagged for review during post- processing.

[0198] In various embodiments, the controller 110 generates coordinates for bounding box based labels of objects within a selected track by processing the sensing data over a time window associated with the series of captured images. The bounding boxes are pseudo-labels incorporated into the corresponding label data. The label data and imaging data including the series of captured images may be advantageously used in one or more post-processing operations to improve the object detector and / or tracker of detection system 104. For example, at least the imaging data and the label data may be used to execute a model training process to adapt a machine learning model of the detection system 104.

[0199] Alternatively, or in addition, the controller 110 may flag one or more images of the captured images, as based on the generated label data, imaging data, and / or other data, for post-processing and / or use to improve the detection system 104. For example, images with a low count of detections of a particular object within a validated track may be flagged and used in a detection model retraining process.

[0200] It will be appreciated by persons skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments, without departing from the broad general scope of the present disclosure. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive. References [1] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, vol.28, 2015. [2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp.779–788. [3] E. P. S. Group, “Epsg:4326 - wgs84, world geodetic system 1984,” https: / / epsg.io / 4326, retrieved 24 July 2024. [4] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in 2017 IEEE international conference on image processing (ICIP). IEEE, 2017, pp.3645–3649. [5] J. Cao, J. Pang, X. Weng, R. Khirodkar, and K. Kitani, “Observation-centric sort: Rethinking sort for robust multi-object tracking,” in Proceedings of theIEEE / CVF conference on computer vision and pattern recognition, 2023, pp. 9686–9696. [6] Y. Zhang, P. Sun, Y. Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “Bytetrack: Multi-object tracking by associating every detection box,” in European conference on computer vision. Springer, 2022, pp.1–21.

Claims

CLAIMS:

1. A method for performing maritime object detection, the method comprising: receiving imaging data indicating at least one image generated by an imaging system configured to capture a maritime environment relative to a structure; receiving detection data indicating one or more detected objects of the at least one image, wherein the one or more detected objects are generated by a detection system operating on the at least one image; receiving, from one or more maritime sensors, sensing data indicating one or more proposed objects of the at least one image; projecting, using localization data associated with the structure and / or the imaging system, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and generating label data to label the at least one image by associating the one or more projected proposed objects with the one or more detected objects.

2. The method of claim 1, wherein the one or more proposed objects are located within a field-of-view of the imaging system and / or a vicinity of the structure.

3. The method of any of claims 1 to 2, further comprising processing the sensing data to filter the one or more proposed objects based on a size and / or a dimension of the respective proposed object in an image plane.

4. The method of any of claims 1 to 3, wherein generating the one or more projected proposed objects using the localization data comprises, for each proposed object in each image of the at least one image: determining a geographical location of the proposed object in a global reference frame; and projecting the geographical location of the proposed object to an image location within the image.

5. The method of claim 4, wherein determining the geographical location of the proposed object in the global reference frame comprises using the localization data to convert geographical coordinates of the proposed object to the global reference frame.

6. The method of any of claims 4 to 5, wherein projecting the geographical location of the proposed object to the image location within the image comprises: (i) transforming the geographical coordinates of the proposed object from the global reference frame to generate corresponding local coordinates in a local reference frame of structure; and (ii) projecting the corresponding local coordinates into a plane of the image to obtain a set of pixel coordinates representing a position of the projected proposed object in the image.

7. The method of claim 6, wherein projecting the corresponding local coordinates into a plane of the image comprises: transforming each point of the projected proposed object in the local reference frame of the structure to a point in a reference frame of the imaging system, based on a position and / or orientation of the imaging system relative to the structure; and mapping each point in the reference frame of the imaging system to the corresponding set of pixel coordinates in the image.

8. The method of claim 7, further comprising applying a distortion model to each point of the projected proposed object in the reference frame of the imaging system.

9. The method of any of claims 1 to 8, wherein the structure is a mobile structure, and the localization data comprises a geographical location and an orientation of the mobile structure.

10. The method of claim 9, wherein the geographical location and the orientation of the mobile structure is determined in real-time by a localization system of the mobile structure.

11. The method of any of claims 1 to 10, wherein associating the one or more projected proposed objects with the one or more detected objects comprises: for each of the one or more projected proposed objects in each image of the at least one image, calculating a distance value in the image between the respective projected proposed object and each of the one or more detected objects.

12. The method of claim 11, wherein the distance value is calculated as a distance between: a position of an object detection marker of the respective detected object; and a location of the respective projected proposed object, in the image.

13. The method of any of claims 11 to 12, wherein the label data indicates at least one of: respective ones of the projected proposed objects do not match to any of the one or more detected objects; and respective ones of the detected objects do not match to any of the one or more projected proposed objects, wherein a match is determined to occur by comparing the respective distance value against at least one threshold value.

14. The method of claim 13, wherein the at least one threshold value is determined based on a degree of accuracy of the localization data.

15. The method of any of claims 13 to 14, wherein the at least one image is a series of images captured from the maritime environment over a time window, and wherein the method comprises validating a temporal sequence of detections associated with each of the one or more detected objects against the one or more projected proposed objects by determining a number of the matches occurring over the series of images.

16. The method of claim 15, wherein each track is a model of a position of a target object represented by the corresponding detected object in an image space of the image.

17. The method of claim 16, wherein the label data represents one or more bounding boxes encompassing the corresponding one or more detected objects of each track in the image.

18. The method of any of claims 16 to 17, wherein the label data indicates that one or more tracks of the series of images are not valid.

19. The method of any of claims 1 to 18, wherein the one or more detected objects are obtained by applying the at least one image as input to a maritime object detector to generate the detection data as output of the detector.

20. The method of claim 19, wherein the maritime object detector comprises a machine learning model having one or more model parameters determined by a model training process.

21. The method of claim 20, further comprising adapting the machine learning model by causing an execution of the model training process using at least: the imaging data; and the label data.

22. A system for performing maritime object detection, the system comprising: a controller having: a communications interface to receive data; at least one computer processor to execute program instructions; and a memory, coupled to the at least one computer processor, to store program instructions for execution by the at least one computer processor, wherein the controller is configured to communicate with one or more maritime sensors, an imaging system, and a localization system configured to provide localization data associated with the imaging system and / or a structure, and wherein the controller is further configured to:receive imaging data indicating at least one image generated by the imaging system configured to capture a maritime environment relative to the structure; receive detection data indicating one or more detected objects of the at least one image, wherein the one or more detected objects are generated by a detection system operating on the at least one image; receive, from the one or more maritime sensors, sensing data indicating one or more proposed objects of the at least one image; project, using the localization data, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and generate label data to label the at least one image by associating the one or more projected proposed objects with the one or more detected objects.

23. The system of claim 22, wherein the controller is further configured to perform the method of any of claims 2 to 21.

24. The system of claim 23, wherein the one or more maritime sensors comprise respective devices configured to receive data from one or more of: an Automatic Identification System (AIS); an Electronic Navigational Chart (ENC) system; and an Automatic Radar Plotting Aid (ARPA) system.

25. The system of any of claims 23 to 24, wherein the imaging system comprises an optical sensing system having at least one camera.

26. The system of claim 25, wherein the at least one camera is configured to generate the at least one image as a portion of a video feed of the maritime environment.

27. The system of any of claims 23 to 26, wherein the structure is a mobile structure, and wherein the localization system is a navigation system comprising a satellite-based positioning unit configured to provide a geographical location of the structure as part of the localization data.

28. The system of claim 27, wherein the navigation system further comprises an inertial navigation system unit configured to provide an orientation of the mobile structure as part of the localization data.

29. An apparatus comprising: an imaging system configured to generate imaging data indicating at least one image capturing a maritime environment relative to a structure; a detection system configured to generate detection data indicating one or more detected objects of the at least one image; one or more maritime sensors configured to generate sensing data indicating one or more proposed objects of the at least one image; a localization system configured to generate localization data associated with the imaging system and / or the structure; and a controller having: a communications interface to receive data; at least one computer processor to execute program instructions; and a memory, coupled to the at least one computer processor, to store program instructions for execution by the at least one computer processor, wherein the controller is configured to communicate with the one or more maritime sensors, the imaging system, and the localization system at least to receive the imaging data, the detection data, the sensing data, and the localization data, and wherein the controller is further configured to: project, using the localization data, the one or more proposed objects into each of the at least one image to generate one or more corresponding projected proposed objects; and generate label data to label the at least one image by associating the one or more projected proposed objects with the one or more detected objects.

30. The apparatus of claim 29, wherein the controller is further configured to perform the method of any of claims 2 to 21.

Images