Navigation system and method for continuously updating ML

Abstract

A vessel management system and method are described that includes receiving input data including at least radar input data indicative of a first field of view and image input data indicative of a second field of view at least partially overlapping with the first field of view; processing the radar input data to determine data indicative of one or more reflective objects within the overlapping portion of the first field of view; determining one or more respective locations within the second field of view at which the one or more reflective objects are identified and obtaining radar metadata for the one or more reflective objects; processing the input image data at the respective locations within the overlapping portion of the second field of view; determining one or more pieces of image data corresponding to one or more sections of the image data associated with the one or more reflective objects; using the radar metadata to generate label data and generating output data including the one or more image data sections and the label data, thereby facilitating a connection between the radar metadata and image data of one or more objects to enable machine learning training for object detection based on the image data.

Description

[Technical field]

[0001] The present invention relates to techniques for automatic navigation, traffic avoidance, and collision prevention using Artificial Intelligence (AI) control technologies, and in particular to autonomous and automatically updated AI by using automatic and "on-the-fly" machine learning training in maritime traffic and marine environments. [Background technology]

[0002] The growing interest in autonomous vehicles is also directed towards autonomous ships. Although marine environments and routes may present promising candidates for autonomous navigation and processing, existing International Maritime Organization (IMO) regulations and rules dictate the presence of a 24 / 7 visual human watch on the bridge of the ship to make a thorough assessment of the situation and collision risk. Such regulations dictate the use of optical imaging and optical image processing as inputs for the ship's automatic or autonomous surveillance functions and navigation control. Summary of the Invention

[0003] Autonomous or automatic control of a vessel generally requires the operation of the vessel in several aspects, including navigation, obstacle avoidance, and resolution of route and traffic collisions with other vessels. Maritime regulations require the use of visual inputs through the use of visual human lookouts to make a thorough assessment of the situation and collision risks. Control of the automatic and autonomous navigation of a vessel requires processing of visual optical inputs. To this end, a suitable control system for such automatic or autonomous navigation of a vessel is trained to detect and analyze various objects in different visual conditions to enable object detection, object classification, and determination of characteristics of object behavior, as well as route replanning or temporary rerouting to avoid collisions and resolve various navigational or possible potential collisions in a timely manner.

[0004] To enable automatic and / or autonomous marine navigation in a variety of conditions while complying with regulations, the present techniques provide ongoing learning and training of AI techniques through automatic and "on-the-fly" machine learning control to enable detection, classification, and generation of behavior profiles of various objects in various visual conditions.

[0005] To this end, the present invention utilizes sensing equipment on the vessel, including one or more optical imaging devices (such as one or more camera units) and one or more radar systems, and a navigational control system (control system) comprising one or more computer processors and memory units. The control system may also comprise a local or remote display, one or more ship control units providing control over the ship autopilot and engine throttles, and a ship interface unit. The control system generally comprises input and output communication ports operatively connected to the sensing equipment for receiving input image data and input radar data. The control system is generally connected to the vessel engines and steering and navigation module for controlling the vessel's steering and thrust, thereby controlling, piloting and navigating the vessel.

[0006] The control system may further include one or more processing modules, including, for example, a safety module configured to receive and process input data, at least image data, and determine a risk level associated with objects around or in the path of the vessel, and an auto captain module configured to receive and process data regarding the vessel and its surrounding environment, apply one or more decision-making algorithms, and generate operational commands to the engine and steering modules. The control system may also include additional modules, such as an efficiency module for monitoring and optimizing the vessel's fuel usage, and a cyber security module configured to monitor and verify data communications to limit security breaches. In general, the control system may include an object analysis module configured to receive input image data from one or more imaging devices, identify one or more objects in the image data, determine one or more classification data for the objects so identified, and provide a corresponding output to at least one of the safety module and the auto captain module. Typically, in a marine environment, the object analysis module may be trained to detect, recognize, and classify various types of vessels, including, for example, ships, boats, rafts, obstacles, various platforms (e.g., oil rigs), etc., as well as to analyze behavioral profiles of such vessels over stationary objects. Additionally, the analysis module may be trained to determine object characteristics such as object lights, smoke, flags, shapes, sails, ship vectors and steering, trailing water wakes, forward (nose) water waves, water bubbles, etc. Furthermore, the object analysis module may be trained to recognize land areas such as islands and shores.

[0007] Generally, the object analysis module, as well as one or more of the safety module and the auto captain module, may utilize a corresponding artificial intelligence (AI) processing module. To this end, the AI ​​module is typically trained in various scenarios and in accordance with maritime regulations. The present technique generally utilizes ongoing training of the object analysis module to provide the vessel with retraining and continuous learning capabilities to detect and classify various types of objects that the vessel may encounter. To this end, the present technique utilizes a training data generator module connectable to the sensing device to receive input data including image input data and radar input data, and to process the input data to generate one or more labeled image sections associated with objects found around the vessel. Each image data section is a cut section of one or more input image frames and includes metadata labels that depict an object and provide one or more classification data regarding the object.

[0008] More specifically, the training data generator is configured and operable to receive input data from a sensing device. The input data includes image data acquired by one or more imagers / camera units and radar input data acquired by one or more radar units. The radar input data is indicative of a first field of view and the image input data is indicative of a second field of view that at least partially overlaps the first field of view. The training data generator is configured to process the radar input data to determine data indicative of one or more reflecting objects and to determine data related to the reflecting objects including respective locations of the objects and radar metadata. The training data generator is further configured to process the input image data to detect one or more objects within the image data. In general, the training data generator may process image data within respective locations of objects detected in the radar input data to determine one or more pieces of image data corresponding to one or more sections of the image data associated with the one or more reflecting objects. The training data generator may be operable to determine correlations between object characteristics acquired from the radar input data and respective objects detected within the image data. The correlation level may be used to determine the usability of data regarding the detected object, and the score threshold may be selected according to weather and visibility conditions, as well as day and night conditions.

[0009] Using the radar metadata, the training data generator creates labeling data and thus generates output data including the one or more image data sections and corresponding label data. The labeling data may also include object characteristics determined from processing of the image data to provide image metadata and data associated with correlations between the image data sections and the reflected radar data. The output data may typically be in the form of labeled image data pieces indicative of objects found around the vessel and corresponding radar metadata. The training data generator thus facilitates a connection between the metadata obtained from the radar input data and the image data of one or more objects, thus enabling machine learning training for object detection based on the image data. The training data generator may augment the corresponding label data with ship systems data and an almanac database.

[0010] The training data generator may thus continuously generate labeled image data fragments for various objects found in the ship's surroundings and utilize the labeled image data fragments to enable continuous, on-the-fly training of an AI module associated with at least one of the object analysis module, the safety module, and the autocaptain module, thus enabling continuous training of the ship AI module using automatic machine learning while the ship is operating.

[0011] Thus, according to a broad aspect, the invention provides a method, implemented, for example, by one or more processors and memory circuits, comprising: receiving input data including at least radar input data indicative of a first field of view and image input data indicative of a second field of view at least partially overlapping with the first field of view; processing the radar input data to determine data indicative of one or more reflective objects within the overlapping portion of the first field of view, determining one or more respective locations within the second field of view at which the one or more reflective objects are identified, and obtaining radar metadata for the one or more reflective objects; processing the input image data at the respective locations within the overlapping portion of the second field of view to determine one or more pieces of image data corresponding to one or more sections of the image data associated with the one or more reflective objects, using the radar metadata to generate label data; and generating output data including the one or more image data sections and the label data, thereby facilitating a connection between the radar metadata and image data of one or more objects to enable machine learning training for object detection based on the image data.

[0012] According to some embodiments, radar metadata includes one or more pieces of data indicative of radar signatures of one or more objects reflecting the radar signal.

[0013] According to some embodiments, the radar metadata includes one or more pieces of data selected from the group of object size, object distance, object closing rate, object aspect, object location, angle, azimuth, vector, Doppler, cross section, and signature.

[0014] According to some embodiments, the method further includes providing the output data comprising one or more pieces of image data and labeling data for training processes of one or more artificial intelligence (AI) modules, thereby enabling continuous training for object recognition and / or classification in the image data.

[0015] According to some embodiments, the AI ​​module is adapted for object detection in a marine environment and enables collision avoidance in vessels according to said image data.

[0016] According to some embodiments, the AI ​​module is further configured to receive location data from one or more location detection units (GPS) and to determine a navigation route for the vessel.

[0017] According to some embodiments, an AI module may be connected to the steering controls of the vessel, thereby enabling at least partially autonomous operation of the vessel. For example, an autocaptain module may be connected to the drive controls and drive systems of the vessel.

[0018] According to some embodiments, the method may further include processing the radar input data to determine data indicative of one or more reflective objects within the first field of view, and upon determining that one or more respective locations at which a reflective object is identified are outside the second field of view, generating an operational command to acquire image data from the one or more respective locations.

[0019] According to some embodiments, the method may further include providing data relating to the positions of the one or more reflective objects, utilizing an automatic identification system (AIS) and the positions of the one or more reflective objects to obtain data relating to the identity of the one or more reflective objects, and using the data relating to identity to generate additional label data, and generating output data comprising the one or more image data sections and the label data and the additional label data.

[0020] According to another broad aspect of the present invention, there is provided a vessel management system comprising at least one processor and memory circuitry, one or more camera units, and one or more radar units, the at least one processor comprising an autocaptain module, an object detection training module, and a training data generator.

[0021] The autocaptain module comprises an artificial intelligence (AI) module that is continuously trainable based on the labeled image data and is configured to receive image data from the one or more camera units and process the image data to determine data regarding one or more objects within a selected field of view around the vessel.

[0022] The training data generator is configured and operable to receive input data from the one or more camera units and one or more radar units, the input data including at least radar input data indicative of a first field of view and image input data indicative of a second field of view that at least partially overlaps with the first field of view; process the radar input data to determine data indicative of one or more reflective objects within the overlapping portion of the first field of view, determine one or more respective locations within the second field of view at which the one or more reflective objects are identified, and obtain radar metadata for the one or more reflective objects; process the respective locations within the overlapping portion of the second field of view of the input image data to determine one or more pieces of image data corresponding to one or more sections of the image data associated with the one or more reflective objects, use the radar metadata to generate label data; and generate output data including the one or more image data sections and the label data, thereby facilitating an association of the radar metadata with image data of one or more objects to enable machine learning training for object detection based on the image data.

[0023] An object detection training module is configured to receive the labeled output data and update training of the AI ​​module of the AutoCAPTAIN module to detect objects based on the labeled output data, thereby enabling training of the AI ​​module to be continuously updated.

[0024] According to some embodiments, the AI ​​module is adapted to process input image data received from the one or more camera units and determine data regarding one or more objects identified in the image data, thereby providing object recognition of one or more objects from the image data.

[0025] According to some embodiments, radar metadata includes one or more pieces of data indicative of radar signatures of one or more objects reflecting the radar signal.

[0026] According to some embodiments, the radar metadata includes one or more pieces of data selected from the group of object size, object distance, object closing rate, object aspect, object location, angle, azimuth, vector, Doppler, cross section, and signature.

[0027] According to some embodiments, the AI ​​module is further configured to receive location data from one or more location detection units (GPS) and to determine a navigation route for the vessel.

[0028] According to some embodiments, an autocaptain module is connectable to a steering control of the vessel and configured to vary at least one of the speed and heading of the vessel, thereby enabling at least partially autonomous operation of the vessel.

[0029] According to some embodiments, the system may further include processing the radar input data to determine data indicative of one or more reflective objects within the first field of view, and upon determining that one or more respective locations at which a reflective object is identified are outside the second field of view, generating an operational command to acquire image data from the one or more respective locations.

[0030] According to some embodiments, the system may further comprise an Automatic Identification System (AIS) module, wherein the training data generator is further configured to provide data regarding the positions of the one or more reflective objects, obtain the data regarding the identity of a vessel located at the position from the AIS module, use the data regarding the identity to generate additional label data, and generate output data including the one or more image data sections and the label data and the additional label data.

[0031] According to yet another broad aspect, the present invention provides a system for generating labeled training data, the system comprising a processing utility comprising one or more processors, a memory unit, and a communication module connectable to one or more camera units and one or more radar units. The processing utility is configured to receive input data including at least radar input data indicative of a first field of view and image input data indicative of a second field of view that at least partially overlaps with the first field of view; process the radar input data to determine data indicative of one or more reflective objects within the overlapping portion of the first field of view, determine one or more respective locations within the second field of view at which the one or more reflective objects are identified, and obtain radar metadata for the one or more reflective objects; process the input image data at the respective locations within the overlapping portion of the second field of view to determine one or more pieces of image data corresponding to one or more sections of the image data associated with the one or more reflective objects, use the radar metadata to generate label data; and generate output data including the one or more image data sections and the label data, thereby facilitating a connection between the radar metadata and image data of one or more objects to enable machine learning training for object detection based on the image data.

[0032] According to some embodiments, the processing utility comprises a radar reflection detector, a radar signature processing module, and an FOV analyzer. The radar reflection detector is configured to receive input data from one or more radar units and determine data of one or more radar signal reflections indicative of one or more objects in a field of view of one or more radar units and a location of the one or more objects. The radar signature processing module is configured to receive and process data regarding the radar signal reflections and determine data regarding radar signatures of the one or more objects. The FOV analyzer is configured to receive input image data from the one or more camera units and the data regarding the location of the one or more objects and process the input image data to determine one or more pieces of image data associated with the location of the one or more objects. The processing utility generates output data including the pieces of image data and labeling data associated with the data regarding the radar signatures of the one or more objects, thereby generating labeled training data for training one or more artificial intelligence (AI) modules.

[0033] According to some embodiments, radar metadata includes one or more pieces of data indicative of radar signatures of one or more objects reflecting the radar signal.

[0034] According to some embodiments, the radar metadata includes one or more pieces of data selected from the group of object size, object distance, object closing rate, object aspect, object location, angle, azimuth, vector, Doppler, cross section, and signature.

[0035] According to some embodiments, the processing utility is further configured to provide the output data including one or more pieces of image data and labeling data for training processing of one or more AI modules, thereby enabling continuous training for object detection within the image data.

[0036] According to some embodiments, the one or more AI modules are adapted for object detection in a marine environment and enable collision avoidance in vessels according to said image data.

[0037] According to some embodiments, the one or more AI modules are further configured to receive location data from one or more location detection units (GPS) and to determine a navigation route for the vessel.

[0038] According to some embodiments, one or more AI modules are connectable to steering controls of the vessel, thereby enabling at least partially autonomous operation of the vessel.

[0039] According to some embodiments, the processing unit is configured to identify data indicative of one or more reflective objects within the first field of view, and in response to determining that one or more respective locations at which a reflective object is identified are outside the second field of view, generate an operational command to acquire image data from one or more additional respective locations.

[0040] According to some embodiments, the system may further comprise an Automatic Identification System (AIS) module, wherein the training data generator is further configured to provide data regarding the positions of the one or more reflective objects, obtain data regarding the identity of a vessel located at the positions from the AIS module, use the data regarding the identity to generate additional label data, and generate output data comprising the one or more image data sections and the label data and the additional label data.

[0041] It will be appreciated that the system according to the invention may be a suitably programmed computer. Likewise, the invention contemplates a computer program readable by the computer for carrying out the method of the invention. The invention further contemplates a machine-readable memory tangibly embodying program instructions executable by a computer to carry out the method of the invention.

[0042] Thus, according to yet another broad aspect, the invention provides a machine-readable program storage device tangibly embodying program instructions executable by the machine to perform a method implemented by one or more processors and memory circuits, the method including receiving input data including at least radar input data indicative of a first field of view and image input data indicative of a second field of view at least partially overlapping with the first field of view; processing the radar input data to determine data indicative of one or more reflective objects within the overlapping portion of the first field of view, determining one or more respective locations within the second field of view at which the one or more reflective objects are identified, and obtaining radar metadata for the one or more reflective objects; processing the input image data at the respective locations within the overlapping portion of the second field of view to determine one or more pieces of image data corresponding to one or more sections of the image data associated with the one or more reflective objects, using the radar metadata to generate label data; and generating output data including the one or more image data sections and the label data, thereby facilitating a connection between the radar metadata and image data of one or more objects to enable machine learning training for object detection based on the image data. [Brief description of the drawings]

[0043] In order to better understand the subject matter disclosed herein, and to illustrate how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Figure 1] 1 illustrates generally a vessel (e.g., a watercraft) according to some embodiments of the present invention; [Diagram 2] 1 illustrates major modules and operation of a vessel according to some embodiments of the present invention. [Diagram 3] 1 illustrates decision-making operations in an automatic and / or autonomous vessel, according to some embodiments of the present invention. [Figure 4]1 illustrates a technique for continuously generating labeled training data according to some embodiments of the present invention. [Diagram 5] 1 illustrates generally a system for continuously generating labeled training data according to some embodiments of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] As indicated above, the present technique provides a vessel that operates automatically and / or autonomously, typically a vessel that can continuously learn and update its operation according to collected data about the vessel's environment and arena. In this regard, reference is made to FIG. 1, which illustrates a system 50 of a vessel, such as a vessel, ship, boat, etc. The system 50 may include a particular drive system 180 (including, for example, engines and drive and navigation means such as rudders and propellers), sensing devices 200, and a control system 100. The control system is generally configured as a computerized control system including one or more processors, memory units 190, input / output communication ports, and may include one or more user interface modules such as a display screen and a keyboard. The control system 100 is connected to the sensing devices 200 to provide operation instructions / commands and receive input sensory data, and to the drive system 180 to provide operation commands related to the operation of the drive system and sometimes receive input data related to the state of the drive system 180. To this end, the control system 100 includes processor and memory circuitry (PMC) that has one or more processors and memory, and operates one or more artificial intelligence (AI) modules. The one or more processors of the control system 100 may be configured to execute several functional modules according to computer readable instructions embodied on the non-transitory computer readable memory 190. Such functional modules are hereinafter referred to as being included in the PMC.

[0045] The sensing device 200 includes one or more imagers or camera units 210 configured to collect image data in the visible and / or infrared spectrum of the vessel's surroundings, and a radar unit configured to scan the vessel's 50 surroundings using RF radiation. In this context, the term camera unit or imager should be broadly understood as relating to a unit configured to collect input data about its surroundings using optical wavelength ranges, possibly including IR and / or UV wavelengths.

[0046] The control system 100 may include an autocaptain module 130 configured to manage the operation of the vessel 50. The autocaptain module 130 is generally configured to obtain input data including, for example, data regarding the vessel's selected route, sensory data regarding the environment around the vessel, and data regarding the operation and effectiveness of the drive system 180. The autocaptain module 130 is configured to process the input data and determine a selected operation of the drive system 180 to steer the vessel 50, vary the route as necessary, and resolve route conflicts. The autocaptain module 130 may also be operative to issue alerts and provide reroute recommendations as necessary, and communicate data and alerts regarding vessel status (if any) to vessel personnel and / or a remote control center and / or other vessels. Generally, the autocaptain module 130 may utilize a pre-trained AI module to process the input data utilizing pre-stored data and data regarding the vessel's status. Such data may include almanac data indicating sun and moon paths, star charts, weather conditions, lighting conditions, etc.

[0047] It should be noted that the vessel 50 may typically include an autopilot module operable to navigate the vessel along a predetermined route to a selected destination. The autopilot module may be associated with the drive system 180 and / or the autocaptain module 130.

[0048] The autocaptain module 130 may be connected to the safety module 120 to receive input data and operational recommendations for various safety issues around the vessel 50. Such safety issues may be associated with various objects around the vessel 50, changing environmental conditions, etc. The safety module may utilize an AI module trained for various situations with appropriate safety related issues and solution options. Additionally, the autocaptain module may also be connected to an efficiency module 140 configured to receive and process data regarding the operation of the drive system 180 and provide operational recommendations regarding the operation to optimize efficiency, such as, for example, fuel efficiency, electrical efficiency, time savings, etc. The autocaptain module 130 may also be connected to a cyber module 150 configured to monitor incoming and outgoing communications and detect and intercept various cyber security threats. To this end, the cyber module may be configured to monitor and process incoming communications and verify the signature of operational commands according to authorized command input paths.

[0049] In general, the autocaptain module 130 may be operable based on image input collected by one or more imaging devices or camera units 210 in order to comply with IMO regulations requiring surveillance of vessels. To process the collected image data and detect and analyze objects therefrom, the control system 100 may include an object analysis module 110 configured to receive and process input image data from the one or more camera units 210, identify the presence of objects in the image data, determine object location, classify the type of object, and provide the object data to the autocaptain module 130 and / or the safety module 120.

[0050] Typically, the autocaptain module 130 and / or safety module 120 may also utilize data from an Automatic Identification Systems (AIS) module (not shown) to determine the location and / or identity of vessels detected in the vicinity of the vessel 50. AIS systems are typically required modules on vessels and provide globally accessible data regarding the location and identity of ships, boats, platforms, and other vessels.

[0051] The object analysis module 110 may generally be configured to utilize AI processing trained by machine learning, e.g., continuously trained by the ML training module 170. The object analysis module 110 is trained for object recognition and classification of various objects that may be detected in an environment in which the vessel may be used. For example, the vessel 50 may be a marine vessel, such as a ship, or any other type of vessel for use in waterborne transportation. Thus, the object analysis module 110 may be trained for recognition and classification of various types of vessels, rafts, marine platforms (e.g., oil rigs), and icebergs, land masses, islands, or shore areas. The object analysis module 110 is preferably trained for object recognition and classification in various weather conditions that typically occur during marine travel, such as rain, fog, etc. The object analysis module may also operate to determine correlations between input sensory data (image data and / or radar data), location data provided by the vessel positioning system (e.g., GPS), and AIS data regarding the identities of various vessels located in the vicinity of the vessel 50. Additionally, object analysis module 110 may be trained to determine object and / or environmental features such as object lights, smoke, flags, shapes, sails, ship vectors and maneuvers, trailing water wakes, forward (nose) water waves, water bubbles, etc. Furthermore, object analysis module 110 may be trained to recognize land areas such as islands and shores.

[0052] The technique further utilizes one or more radar units 220 associated with the sensing device, a training data generator 160, and an ML training module 170 to enable continuous learning of the vessel 50. The radar unit 220 utilizes RF signals to obtain data regarding the environment surrounding the vessel 50. More specifically, the radar unit 220 operates to transmit RF signals toward a first field of view around the vessel 50 and collect data regarding reflected RF signals reflected from various objects within the first field of view around the vessel 50. Thus, the sensing device 200 is configured to provide input sensed data including radar data indicative of RF reflection data from the first field of view and imager data collected by the one or more imager / camera units 210 from a second field of view. Typically, the first and second fields of view are at least partially overlapping. Furthermore, in some preferred embodiments, the first and second fields of view may include a 360° horizontal field of view around the vessel and a selected vertical field of view, for example, about a 30° vertical field of view.

[0053] The training data generator 160 is configured to receive input data from the sensing device 200, the input data including radar input data and image input data. The training data generator 160 is configured to process the radar input data to determine data indicative of one or more reflecting objects. Typically, in a marine environment, the radar data may include a wave reflection background, and thus the processing may be directed to identifying reflections against background noise associated with reflections of RF signals from waves around the vessel. Generally, such background noise may be filtered out by one or more processing units associated with the radar unit 220, or in some configurations, by a suitable filtering module of the system 100. The training data generator 160 may include a radar signal analyzer (not specifically shown) configured to analyze collected radar signals that exhibit reflections above a noise threshold and determine radar metadata regarding the objects causing such collected reflections. The radar metadata may include various pieces of data regarding the objects reflecting RF signals that form the radar signature of the object. Radar metadata generally includes data regarding object range, object location, azimuth and elevation angles, and radar signature data, such as data regarding object size, object aspect, effective reflective properties, return size, Doppler shift components, closing rate, and any other object characteristics that can be determined using radar technology.

[0054] The training data generator 160 is further configured to process the input image data, specifically sections of the image data associated with one or more locations where radar returns are identified. In this regard, the training data generator 160 may utilize one or more image processing and object recognition techniques to determine data regarding one or more objects captured in the image data. The training data generator 160 operates to determine one or more pieces of image data, typically one or more sections of the image data, associated with locations within the field of view where reflecting objects are identified. These image sections may generally include image data of objects identified in the radar data.

[0055] The training data generator 160 may include a correlator module (not specifically shown) configured to determine correlations between radar metadata associated with detected objects and image data depicting corresponding regions, sometimes using data determined for objects from the image data. In general, a high correlation may indicate that the imager data section is associated with the same object as that detected in the radar input data. The thresholds for determining a sufficient correlation level may be related to day or night, weather, and general visibility conditions.

[0056] Using the radar metadata, the training data generator 160 is further operable to generate label data and attach the label data to each of the one or more pieces of image data to form output data. The output data is generally in the form of one or more pieces of image data that are sections of image data collected by the one or more camera units 210 that have been labeled with the radar metadata acquired from the respective locations. Thus, the training data generator facilitates a connection between the radar metadata acquired from the one or more reflective objects and the image data of the respective one or more objects. Thus, the output may be used to enable machine learning training for object detection based on image data.

[0057] The retraining data generator 160 may also utilize AIS and location (e.g., GPS) data to further include vessel identity in the image label data, further enabling training based on ship identity and corresponding visual features in addition to radar metadata.

[0058] In general, because the radar input data and image data may be associated with slightly different fields of view, the training data generator 160 may be connectable to one or more imager / camera units 210 to provide operational instructions for acquiring image data from one or more selected locations. Thus, if the field of view of the input image data (second field of view) does not include each location where a radar return signal is identified, the training data generator 160 may instruct the one or more camera units 210 to acquire image data from each location where an object is identified in the radar input data.

[0059] The training data generator 160 may store the labeled image data pieces in the memory unit 190 for later use. Additionally, the training data generator 160 may provide the labeled image data pieces to the ML training module 170 for additional ML training of the AI ​​modules, including one or more of the object analysis module 110, the safety module 120, and the autocaptain module 130. The ML training module 170 may operate for additional training periodically or in response to collection of a selected volume of labeled image data pieces. This allows the vessel's control system 100 to continuously train and learn based on various objects and different environmental conditions experienced during operation. Thus, limitations associated with controlled ML training based on pre-provided training data are removed. The updated learning may provide the automatic or autonomous vessel with an updated ability to identify and classify nearby objects in various weather conditions, such as rain, fog, etc., as well as in different lighting conditions and different weather or general visibility conditions between day and night.

[0060] In general, the ML training module may utilize one or more machine learning techniques and a set of labeled image data pieces collected by the training data generator 160 over a selected collection period to train one or more AI modules of the object analysis module 110, the safety module 120, and the auto-captain module 130. The ML training module may utilize automated ML techniques and / or other machine learning training techniques to train the AI ​​modules. The training techniques may include operational limitations associated with IMO regulations. Additionally, the training is typically constructive and utilizes previous training.

[0061] Typically, the autocaptain module 130 may utilize a variety of data to operate the vessel. For example, the autocaptain module 130 may utilize pre-stored data tables such as sun tables including sunrise and sunset data and sun paths, moon tables including moon paths and moonrise and moonset times, location data (e.g., using a location determination system such as a global positioning system (GPS)), acceleration data (e.g., provided by one or more inertial accelerometers), ocean current maps, geographical maps, and any other data that may be used to control the operation of the vessel. The autocaptain module 130 may also be configured to communicate with one or more remote stations to receive weather updates and forecast data, as well as data regarding wave height, visibility, paths of other nearby vessels, and the like. The autocaptain module may utilize such data to operate the vessel 50 according to a target objective. More specifically, the autocaptain module 130 may operate to determine the vessel's route, operate the autopilot module to maintain the route, or alter route and / or speed in response to changing weather conditions or detected possible route conflicts.

[0062] The control system 100 may typically be configured, using the autocaptain module 130 and / or the safety module 120, to utilize data regarding changing conditions and / or data regarding various objects located in the vicinity of the vessel 50 to resolve route conflicts and thus avoid the risk of collision. To this end, the autocaptain module 130 may be trained on marine route protocols and route conflict rules.

[0063] For example, in response to input data from object analysis module 110 regarding the detection of one or more vessels in the vicinity of ship 50, autocaptain module 130 may operate one or more signal units (not specifically shown), such as a horn, optical signal, RF beacon, etc., and recalculate the navigation of ship 50 according to its training.

[0064] Please refer to FIG. 2, which illustrates a schematic structural diagram of a vessel 50 according to some embodiments. As shown, the autocaptain module 130 is configured to acquire and / or receive input data related to vessel operation and utilize the input data according to pre-stored data and ML training to operate the vessel drive system 180. Furthermore, the autocaptain module 130 may communicate with one or more remote stations 195 when necessary. The input data may include one or more pieces of data associated with optical imaging data 210 acquired by one or more camera units, radar input data 220 acquired by a radar system, data related to nearby ships acquired from the AIS 240, location data 230, operating condition data 300, speed log 310, and vessel heading or gyro data 320. As indicated above, external input data such as the input image data 210 and radar input data 220 may be processed for object recognition by the object analysis module (OAM) 110 to determine and classify objects identified within the collected data. Additionally, the object analysis module 110 may process sensory data, including image data and radar input data, to determine data regarding waves, horizon, weather, and general visibility conditions.

[0065] If an object or other condition that may endanger the vessel is detected within view or within a selected range, the respective data is transmitted to the safety module 120 to determine the risk level (e.g., risk of collision). The autocaptain module 130 utilizes such data to determine vessel behavior. The autocaptain module 130 is configured to process the different types of input data described above and determine conditions associated with one or more actions to assist the vessel in establishing one or more operational goals. For example, the autocaptain module 130 may operate to determine a level of potential conflict and select a suitable conflict resolution action. The condition resolution action may include, for example, generating an instruction to operate the vessel signal unit and alert the vessel 50 of the detected vessel. If the safety module 120 determines that the route of the detected ship / object has a collision probability that exceeds a selected or predefined threshold, the autocaptain module 130 may generate an action instruction to operate the vessel drive system 180 and change the vessel's route to avoid such collision according to a selected route change option.

[0066] In general, the autocaptain module 130 may be associated with an electronic chart display and information system (ECDIS) 135 that includes pre-stored and / or periodically updated data related to traffic and route planning, alert interpretation data, and COLREG (International Regulations for Preventing Collisions at Sea). The autocaptain module 130 may utilize the pre-stored data to determine suitable condition resolution actions in response to different situations.

[0067] As described above, different data sources are used that provide input data to the autocaptain module 130. The autocaptain module may determine weights for the data of the different sources to determine the state of the vessel and its surroundings and appropriate condition resolution actions. Thus, the input data may include various types of input data including, for example, environmental data in addition to data associated with different objects detected around the vessel. The environmental data may be determined by processing of image data collected by one or more imaging devices. For example, the environmental data may include light level data, relative horizon, sea surface clutter levels (both the foam of the waves and the waves themselves), solar or lunar glint, weather, and visibility conditions, etc. The autocaptain module 130 may also utilize pre-stored data from a memory unit (not shown) to retrieve pre-stored data or communicate with additional sensors, which may be one or more remote stations and / or the location of the vessel. The pre-stored data or data retrieved by remote communication may include, for example, solar and lunar calendar data, world ship database, retrieved by the AIS system. Furthermore, the additional data may include weather conditions such as temperature, cloud data, humidity, sea spray levels, fog conditions, precipitation levels (eg, in millimeters), wave size / height, visibility, etc.

[0068] As mentioned above, the autocaptain module 130 may be formed by or may utilize one or more artificial intelligence modules trained with one or more decision trees according to predefined rules for resolving various conflict types. In this regard, reference is made to FIG. 3 illustrating decision-making schemes according to several configurations associated with the detection of one or more objects in the vicinity of the ship 50. Sensory data is collected by the sensing devices and transmitted to the control system 100. The control system is operable by ship navigation commands that operate one or more processors to process input data from the sensing devices (3010). Such data may include image data from one or more imaging devices, radar data, and / or other sensors. The detected sensor data may generally be transmitted for object recognition / classification to determine one or more characteristics of the object (3020). The recognition and / or classification may utilize the training of the object analysis module 110 and may also include data obtained from a positioning system combined with AIS data regarding the ship identity. The object data may be further analyzed and / or correlated along a particular time period to determine an estimated trajectory / path of the object (3030). The analysis may include determining a probability of route conflict and / or collision. This may be related to the determined route of the detected object and / or the planned route of the vessel 50. Decision making based on the detected object is typically related to ECDIS data, including guidelines on collision resolution, navigation and traffic regulations, signaling protocols, etc. (3040). Based on the analysis and the ECDIS guidelines, the process determines a level of route boundary conflict with the detected object (3050). If no route boundary conflict is determined, for example if the detected object is outside the vessel route envelope, the system may operate to issue an internal alarm (3060). If the detected object is within the envelope but does not pose any risk of conflict, appropriate external signaling (e.g., acoustic, RF, or visual signals using a horn) may be used (3070). Such signaling may typically resolve various possible route conflicts.However, if the detected object results in a path boundary conflict that may result in a collision above a certain probability threshold, the decision may also be associated with suitable operation of the vessel drive system, for example, to verify that the risk of trajectory conflict is minimized (3090). If the detected object and its analysis indicate a possible trajectory conflict path boundary conflict with a risk of conflict above a predetermined threshold, an appropriate set of collision prevention actions is taken (3080). Such a set of collision prevention actions may include one or more operational actions to operate at least the vessel drive and signaling systems, and is typically determined according to the ECDIS and corresponding guidelines. Thus, the autocaptain may operate to modify the vessel trajectory and vary the operation of the drive system to eliminate the risk of collision (3090).

[0069] As mentioned above, the technique further provides a continuous learning capability for the machine learning system of the control system 100. Such continuous learning capability may be associated with object recognition of various objects detected in the vicinity of the vessel 50. See FIG. 4, which illustrates a technique for continuously generating training data in the form of labeled image data pieces according to some embodiments of the technique. As mentioned above, the technique may be implemented using one or more processors and memory circuitry, such as the control system 100 described above. As shown, the technique includes providing input radar data acquired by a radar unit (4010) and providing input image data acquired by one or more camera units (4020). In general, the radar input data represents a first particular field of view and the image data represents a second field of view, the first and second fields of view at least partially overlapping. In some preferred examples, the radar data is collected from a field of view extending 360° horizontally and at least 30° in elevation. The image data may preferably be collected from a similar field of view, although there may be a certain delay between the collection of the radar data and the collection of the image data of the same section of the field of view.

[0070] The input data formed by the radar input data and the image data is processed to generate labeled image data. The input radar data may be filtered (4015) to remove noise associated with reflections from waves in a marine environment, for example. It is noted that such filtering may relate to changes in wave reflections due to different sea conditions, utilizing input data regarding weather and wave conditions. Filtering of the radar data may generally be provided by a radar unit. The radar data is processed (4030) for detection of one or more reflected signals associated with various objects in the vicinity of the vessel. Once an object is detected, the location and radar metadata of the object is determined (4035, 4050). The location of the object may be determined using suitable processing and vessel positioning systems as a relative location with respect to the vessel 50 and / or as a global location of the object. The radar metadata is determined based on the input radar reflection signature, the Doppler shift, and respective variations between detection instances when the object is detected in two or more radar sweeps. The metadata may include one or more pieces of data such as object size, aspect, cross section, distance, azimuth, elevation, object location, rate of closure, Doppler shift, and object radar signature. In some configurations, typically for vessels, the techniques may utilize AIS data regarding the presence of one or more vessels at the location of the detected object (4060). AIS data typically includes vessel identities and their global positions, thereby allowing for obtaining direct information of nearly complete data regarding the detected ships, including flag data, size, direction, signature color, etc.

[0071] Utilizing the object relative location, a corresponding section of the image data is selected, typically cropped, to generate an image data piece depicting the object (4040). If the image data does not include image data for the respective location, the technique may operate to generate operational instructions to one or more of the camera units to collect image data for the respective location.

[0072] In some configurations, the image data sections and radar metadata may undergo further processing to determine correlations between objects detected in the images and the radar data in corresponding location coordinates (4045). To this end, the techniques may include processing of the input image data for object detection as well as for additional data such as wave height and direction, horizontal curve angle, cloud data, etc.

[0073] If the correlation between the imager data section and the radar signal of a detected object is determined to be above a selected threshold, the associated data is used as labeled output data. Some faintly detected object returns may be omitted in response to a correlation level with the image data below the threshold. The correlation threshold is generally determined based on weather, lighting, and other visibility conditions.

[0074] The collected data, including the image data pieces, radar metadata, and AIS data when used, are used to generate labeled output data (4070). The labeled output data facilitates and provides a connection between the radar input data and the imager data associated with objects detected in the vicinity of the vessel. The technique may typically operate to continuously collect input data and generate labeled output data indicative of various detected objects ranging from other vessels, marine platforms, boats, oil rigs, icebergs, islands, and coastlines, as well as any other type of object detected around the vessel. The collected labeled output data may generally be used to continuously or periodically train the ML module of the control system to recognize various objects around the vessel (4080). This allows for the provision of an automatic and / or autonomous vessel capable of continuous learning beyond that provided by initial training and periodic updates.

[0075] FIG. 5 illustrates the configuration of a system for generating labeled training data according to some embodiments of the present technique. As mentioned above, the system may be configured in the form of a control system 100 or a training data generator (TDG) module 160. In general, the system 100 includes one or more processors, a memory unit 190, and a communication module connectable to one or more camera units and one or more radar units, one or more processor and memory circuitry (PMC) utilizing an I / O port 195. The TDG 160 is configured to receive input data associated with at least radar input data and image input data, process the input data to generate one or more labeled image data pieces, and provide the labeled image data to the ML training module 170 to enable continuous training. The labeled image data pieces are generally labeled based on connections between received signal returns of the radar data and corresponding sections of image data depicting objects around the vessel. In this example, the TDG 160 may include software or hardware modules, including a radar reflection detector 410, a radar signal processing module 420, a field of view analyzer 430, an object data correlator 435, and a labeled output data generator 440.

[0076] The radar reflection detector 410 is configured to receive data regarding collected RF signals collected by the radar system using one or more collection beams and process the collected signals to determine signal portions indicative of objects within the field of view. To this end, the radar reflection detector 410 may utilize noise filtering, summation of RF collection channels with suitable phase changes, and / or peak detection techniques to detect collected signal portions indicative of objects relative to background noise. Upon detecting a reflected signal caused by one or more objects, the radar reflection detector 410 transmits data regarding the detected reflections to the radar signal processing module 420 for processing the reflected signal components and determining one or more characteristics of the objects based on the reflected RF signals. The radar signal processing module 420 may utilize various processing algorithms to determine data regarding one or more objects reflecting the collected RF signals. Such processing may include determining Fourier components of the collected signals, determining a Doppler shift, processing the collected channels, determining one or more additional collection beams to distinguish the reflected signals from one or more objects located within the group, etc. The radar signal processing module 420 is configured to determine radar metadata regarding the objects, including pieces of data selected from the number of objects, object location, azimuth, elevation, distance from the vessel, object size, length, width, height, object aspect, cross section, rate of closure, fractional Doppler shift (e.g., associated with internal movement of elements within the object), and the like. The radar signal processing module 420 may further utilize two or more detection instances of a common object to determine the lateral velocity of the objects. The radar metadata so determined is packaged into labeling data for labeling of corresponding sections of the image data. The radar signal processing module 420 may also utilize positioning data (e.g., GPS data) to determine the global position of the detected objects based on the vessel position and the relative location of the objects with respect to the vessel 50. To this end, the radar signal processing module 420 provides data regarding object location, including at least azimuth, elevation, and distance, to the field of view (FOV) analyzer 430.

[0077] The FOV analyzer 430 operates to receive data regarding the location of one or more detected objects and image data collected by one or more camera units, and to process the image data to determine one or more pieces of image data associated with a section of image data collected from each location where an object was detected. The FOV analyzer 430 may utilize data regarding object distance and object size / cross-section to determine a size of the image section, typically based on a solid angle covered within the image section. In general, if the image data does not include sufficient data for a respective location, the FOV analyzer 430 may generate a request to the sensing device to operate one or more camera units and collect additional image data for the respective location.

[0078] The FOV analyzer 430 may include or be associated with an image processing module configured to process one or more relevant sections of image data to determine data regarding objects appearing in the respective sections. To this end, the FOV analyzer 430 may utilize image data acquired at different time instances, or at one or more different times, allowing detection of moving objects on a substantially stationary background. Additionally or alternatively, the FOV analyzer 430 may utilize gradient detection or other techniques (including AI image processing) to determine data regarding objects appearing in the image data.

[0079] The object correlator 435 may be configured to process different pieces of data indicative of detected objects. More specifically, the object correlator 435 is configured to receive radar metadata about detected objects and image data sections for each one or more locations, and process the radar and image data to determine correlations between object detections in different technologies. Correlation levels may be included in the labeling data. Additionally, correlation levels may be processed according to weather, visibility, and / or lighting conditions. If there is a correlation between the sensor representation of the detected object and below a selected threshold, the technique may operate to obtain additional sensing data to improve the correlation and / or omit the respective sensing data.

[0080] The cropped image data pieces and radar metadata are then sent to the labeled output data generator 440 for packaging as labeled output data. As mentioned above, the labeled output data generator may utilize the AIS system 240 and / or remote communications to obtain data regarding one or more vessels that may be found at the location of the detected object. Such AIS data, if present, may provide additional labeling data fields that enhance the value of the labeled output data. For example, the AIS data may indicate ship identity and clarify that the detected object is a particular type of ship. Additionally, the AIS data may indicate ship flag data, color scheme, heading, etc. The labeled output data generator 440 thus operates to generate output data formed from the labeled image data pieces formed by the images of the one or more objects and the radar metadata collected about the objects, and may also include AIS data about the objects. The labeled output data is typically stored in the memory unit 190 for later use and is typically used to enable continuous training of the ML object analysis module using the ML training module 170.

[0081] Thus, the present technique utilizes combined RF and optical signal collection to continuously generate labeled image data and enable continuous learning of an autonomous vessel. The present technique is particularly relevant for use on marine vessels, utilizing a common analytical field of view that enables the technique to uniquely and efficiently correlate radar input data and image data with respect to a common object.

Claims

A method implemented by one or more processors and a memory circuit for operating a ship, the method comprising: Receiving input data including at least radar input data indicating a first field of view and image input data indicating a second field of view that at least partially overlaps the first field of view; Processing the radar input data to determine data indicating one or more reflective objects within the overlapping portion of the first field of view; Determining one or more respective locations within the second field of view where the one or more reflective objects are identified; Obtaining radar metadata of the one or more reflective objects; Processing the input image data at the respective locations within the overlapping portion of the second field of view; Determining one or more corresponding pieces of image data corresponding to one or more sections of the image data associated with the one or more reflective objects; Using the radar metadata to generate label data and generating output data including the one or more image data sections and the label data, thereby facilitating the connection between the radar metadata and the image data of one or more objects and enabling machine learning training for object detection based on the image data; Including; The method further includes providing the output data including one or more pieces of image data and labeling data for training processing of one or more artificial intelligence (AI) modules, thereby enabling continuous training for object recognition and / or classification in the image data; Method.

2. The method according to claim 1, wherein the radar metadata includes one or more pieces of data selected from the group consisting of object size, object distance, object approach speed, object aspect, object location, angle, azimuth, vector, Doppler, cross-section, and signature.

3. The method according to claim 1, wherein the AI module is adapted for object detection in a marine environment and enables collision prevention in a ship according to the image data.

4. The method according to claim 3, wherein the AI module is further configured to receive location data from one or more location detection units (GPS) and to determine the navigation route of the ship.

5. The method according to claim 4, wherein the AI module is connectable to the steering control of the ship, thereby enabling at least a partially autonomous operation of the ship.

6. The method according to claim 1, further comprising: processing the radar input data to determine data indicating one or more reflective objects within the first field of view; and when it is determined that each of the one or more locations where a reflective object is identified is outside the second field of view, generating an operation command to acquire image data from each of the one or more locations.

7. The method according to claim 1, further comprising: providing data regarding the positions of the one or more reflective objects; using an automatic identification system (AIS) and the positions of the one or more reflective objects to obtain data regarding the identities of the one or more reflective objects; using the data regarding the identities to generate additional label data; and generating output data including the one or more image data sections, the label data, and the additional label data.

8. A ship management system comprising at least one processor and a memory circuit, and one or more camera units and one or more radar units, wherein the at least one processor comprises an auto-captain module, an object detection training module, and a training data generator. The auto-captain module includes an artificial intelligence (AI) module that can be continuously trained based on labeled image data, receives image data from the one or more camera units, and is configured to process the image data to determine data regarding one or more objects within a selected field of view around the ship. The training data generator is configured to receive input data from the one or more camera units and the one or more radar units, the input data including at least radar input data indicating a first field of view and image input data indicating a second field of view that at least partially overlaps with the first field of view; process the radar input data to determine data indicating one or more reflective objects within the overlapping portion of the first field of view, determine one or more respective locations within the second field of view where the one or more reflective objects are identified, and obtain radar metadata of the one or more reflective objects; process the respective locations within the overlapping portion of the second field of view of the input image data to determine one or more image data pieces corresponding to one or more sections of the image data associated with the one or more reflective objects, use the radar metadata to generate label data, and generate output data including the one or more image data sections and the label data, thereby facilitating the connection between the radar metadata and the image data of one or more objects and enabling machine learning training for object detection based on the image data. The object detection training module is configured to receive the labeled output data and, based on the labeled output data, update the training of the AI module of the auto captain module for detecting an object, thereby enabling continuous updating of the training of the AI module. The AI module is trained to comply with maritime regulations in a scenario. A ship management system. Claim 9 The system according to claim 8, wherein the AI module is adapted to process input image data received from the one or more camera units and determine data regarding one or more objects identified in the image data, thereby providing object recognition of the one or more objects from the image data. Claim 10 The system according to claim 8, wherein the radar metadata includes one or more data pieces indicating a radar signature of one or more objects that reflect radar signals. Claim 11 The system of claim 8, wherein the radar metadata includes one or more data pieces selected from the group consisting of object size, object distance, object approach speed, object aspect, object location, angle, azimuth, vector, Doppler, cross section, and signature.

12. The system of claim 8, wherein the AI module is further configured to receive location data from one or more location detection units (GPS) and to determine a navigation route of the ship.

13. The system of claim 8, wherein the auto-captain module is connectable to the steering control of the ship and is configured to change at least one of the speed and the bow azimuth of the ship, thereby enabling at least a partially autonomous operation of the ship.

14. The system of claim 8, further comprising generating an operation command to acquire image data from each of the one or more locations when the radar input data is processed to determine data indicating one or more reflecting objects within the first field of view and it is determined that each of the one or more locations where a reflecting object is identified is outside the second field of view.

15. The system of claim 8, further comprising an Automatic Identification System (AIS) module, wherein the training data generator provides data regarding the positions of the one or more reflecting objects, obtains data regarding the identity of the ship located at the positions from the AIS module, uses the data regarding the identity to generate additional label data, and is further configured to generate output data including the one or more image data sections and the label data and the additional label data.

Images