Optimizing vision perception in autonomous maritime surface vehicles

Abstract

An autonomous maritime surface vehicle (AMSV) operating in a maritime environment includes an on-board vision perception optimization system, which operates on raw image data generated by an on-board passive remove sensing system. The vision perception optimization system may optimize the raw image data based on respective entropies of sub-areas within the raw image data, e.g., by detecting sub-areas of the raw image data having lower entropy, and filtering or cropping the raw image data to include only the detected sub-areas. An on-board vision perceptor may apply one or more computer vision techniques to the optimized image data to detect presences of objects within the field-of-view of the AMSV, measure respective distances of the objects from the AMSV, detect and / or track the movements of the objects, etc. The AMSV may responsively control one or more operations of the AMSV based on the information generated from the applied computer vision techniques.

Description

Attorney Docket No. 33894-S003 PC(PATENT)OPTIMIZING VISION PERCEPTION IN AUTONOMOUS MARITIME SURFACE VEHICLESCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 698,453, entitled “Autonomous Maritime Surface Vehicles” and filed September 24, 2024, and this application also claims priority to U.S. Provisional Patent Application No. 63 / 701 ,166, entitled “Autonomous Maritime Surface Vehicles” and filed September 30, 2024, the entire disclosures of which are hereby expressly incorporated by reference herein.FIELD OF THE DISCLOSURE

[0002] The present disclosure generally relates to autonomous maritime surface vehicles (AMSVs) and associated systems, components, methods, and techniques related to on-board optimizing of the vision perception of autonomous maritime surface vehicles.BACKGROUND OF THE DISCLOSURE

[0003] Maritime vehicles, or vehicles designed for use on or in the water, are commonly used for transportation, recreation, defense, scientific research, and other purposes. Examples of maritime vehicles include boats, watercraft, submarines, and amphibious vehicles. As such, a maritime vehicle can be a surface vehicle which operates while generally floating on or in bodies of water, e.g., so that during its operations a majority of the surface vehicle is generally disposed above the waterline, and a maritime vehicle can be a partially or entirely submersible vehicle which operates while a majority or all of the vehicle is disposed beneath the waterline. Maritime vehicles can be manned (i.e., operated by an onboard human) or unmanned. Unmanned maritime vehicles can be remotely controlled.

[0004] Computer vision systems typically include computers which attempt to understand, interpret, and extract meaningful information from visual data, such as visual data included in digital images, digital videos, and / or digital streams. Generally speaking, computer vision involves the analysis and processing of digital images or videos to replicate or simulate aspects of human visual perception. For example, computer vision techniques aim to extract and analyze features, depth, motion, and other visual cues to improve visual perception and enable computers to comprehend and interact with visual data in a manner similar to humans. As such, computer vision systems may operate on stereo images respectively obtained by a pair of associated cameras or image sensors, and / or may operate on images which are obtained sequentially over time. Typically, computer vision systems apply one or more computer vision processing techniques to the entirety of a captured image. Computer vision systems may be utilized in various domains, such as in autonomous car and truck operations, robotics, medical imaging, and the like.SUMMARY

[0005] In an embodiment, a method of controlling operations of maritime vehicles in a maritime environment includes detecting a first sub-area of a first image captured by a first image sensor included in aAttorney Docket No. 33894-S003 PC(PATENT) stereo vision system mounted on a maritime vehicle, where the first sub-area depicts a respective at least a portion of an object located remotely from the maritime vehicle, and the first sub-area has a respective entropy less than a first threshold. The method also includes detecting a second sub-area of a second image captured by a second image sensor included in the stereo vision system, where the second sub-area depicts a respective at least a portion of the object, and the second sub-area has a respective entropy less than a second threshold. Additionally, the method includes cropping the first image to include only the first sub-area, cropping the second image to include only the second sub-area, and determining a distance between the maritime vehicle and the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image. For example, the method may include determining disparity between the two images resulting from the baseline distance between the two image sensors, and determining the distance between the maritime vehicle and the object based on the disparity. Further, the method includes controlling an operation of the maritime vehicle based on the distance between the maritime vehicle and the object.

[0006] In an embodiment, a method of controlling operations of maritime vehicles in a maritime environment includes detecting, at a first time, a first sub-area of a first image captured by a vision system mounted on a maritime vehicle, where the first sub-area depicts a respective at least a portion of an object located remotely from the maritime vehicle, and the first sub-area has a respective entropy less than a first threshold. The method also includes detecting, at a second time, a second sub-area of a second image captured by the vision system, where the second sub-area depicts a respective at least a portion of the object, and the second sub-area has a respective entropy less than a second threshold. Additionally, the method includes cropping the first image to include only the first sub-area, cropping the second image to include only the second sub-area, and detecting a movement of the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image. Further, the method includes controlling an operation of the maritime vehicle based on the detected movement of the object.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The features of this invention which are believed to be novel are set forth with particularity in the appended claims. The invention may be best understood by reference to the following description taken in conjunction with the accompanying drawings, in which like reference numerals identify like elements in the several figures, in which:

[0008] FIG. 1 A is a top perspective view of an example autonomous maritime surface vehicle (AMSV) in which embodiments of the vision perception optimization systems, components, methods, and / or techniques of this disclosure may be implemented;

[0009] FIG. 1 B is a front view of the maritime vehicle of FIG. 1 A;

[0010] FIG. 1 C is a rear view of the maritime vehicle of FIG. 1A;Attorney Docket No. 33894-S003 PC(PATENT)

[0011] FIG. 1 D is a bottom perspective view of the maritime vehicle of FIG. 1 A;

[0012] FIG. 1 E is a side view of the maritime vehicle of FIG. 1 A;

[0013] FIG. 2 is a block diagram of an example AMSV in which embodiments of the vision perception optimization systems, components, methods, and / or techniques of this disclosure may be implemented;

[0014] FIGS. 3A-3D depict various example perception hardware configurations and corresponding fields of view (FOVs), in accordance with various embodiments described herein;

[0015] FIGS. 4A-4K depict various example perception software analyses using the hardware configurations of FIGS. 3A-3D, in accordance with various embodiments described herein;

[0016] FIGS. 5A-5E depict various flow diagrams representing example computer-implemented methods, in accordance with various embodiments described herein;

[0017] FIG. 6A is a block diagram of an example vision perception optimization system which may be included in an autonomous maritime surface vehicle;

[0018] FIGS. 6B-6E are reproductions of images obtained while using at least some of the techniques disclosed herein;

[0019] FIG. 7A depicts a flow diagram of an example computer-implemented method performed by a vision perception optimization system on-board an AMSV; and

[0020] FIG. 7B depicts a flow diagram of an example computer-implemented method performed by a vision perception optimization system on-board an AMSV.DETAILED DESCRIPTION

[0021] The present disclosure is generally directed to systems, components, and methods of optimizing vision perception systems of autonomous maritime surface vehicles (AMSVs) that can be used for military purposes (e.g., for naval defense, patrolling waters and enforcing laws, reconnaissance, naval exploration, monitoring, etc.), and can be used for other non-military purposes (e.g., non-military or civilian transport of goods and / or people, patrolling and monitoring recreational water areas, scientific exploration, etc.) if desired. An autonomous maritime surface vehicle is small(er), durable, and configured to quickly, efficiently, and stealthily traverse a body of water once dispatched (e.g., from other maritime vehicles, beachheads, or an airdrop). During operations, the autonomous maritime surface vehicle generally floats on the body of water; that is, typically an AMSV does not operate when fully submersed within the body of water. The autonomous maritime surface vehicle may typically be unmanned and can operate autonomously without requiring the use of any real-time human instructions and without being remotely controlled. The autonomous maritime surface vehicle is modular, with components that can be flexibly altered, removed, or added as desired in accordance with the mission of the AMSV. The autonomous maritime surface vehicle can operate singly or in collaboration with other similar maritime vehicles and / or assets when necessary, and the autonomousAttorney Docket No. 33894-S003 PC (PATENT) maritime surface vehicle (and indeed, groups of AMSVs) may preferably be unmanned operate autonomously during the missions.

[0022] When operating in an active mode, the AMSV can operate to perform a mission with which it (either singly or in cooperation with other AMSVs) has been charged to perform, such as a task, an operation, a coordinated maneuver, etc. Some missions may be military-related, for example, performing reconnaissance along a stretch of coastline or within a certain area of the ocean, patrolling enemy lines or presences, and / or travelling to within striking distance of a target (e.g., an enemy vessel or asset) and discharging its pay load (e.g., bomb, explosive device, etc.) to engage or hit the target. Some missions may not be related to any military at all, for example, delivering commercial shipping containers to a series of ports via maritime environments, operating water taxis or other types of public or private maritime transportation, patrolling a recreational boating, fishing, and / or swimming area, and the like. Missions may involve only the AMSV or may involve a group of AMSVs (e.g., a “swarm” of AMSVs) which cooperatively operate to perform the mission. For example, a swarm of AMSVs may be charged with a mission to find a particular target, surround the target, and engage the target with respective pay loads in a pre-defined order over an interval of time. Further, during any mission, the AMSV (whether operating singly or cooperatively with other AMSVs) may autonomously operate to respond to unexpected conditions, such as the loss of another AMSV, a failure of the AMSV to deploy payload, a fault or degradation in performance of one of the AMSVs components, etc.

[0023] Example Autonomous Maritime Surface Vehicle (AMSV) - Perspective Views

[0024] FIGS. 1 A-1 E illustrate an example of an autonomous maritime surface vehicle (AMSV) 100 in embodiments of the vision perception optimization systems, components, methods, and / or techniques of this disclosure may be implemented. Generally speaking, the AMSV 100 is an unmanned vehicle configured to autonomously traverse a body of water. The maritime vehicle 100 generally includes a hull 104 and a cap 108 that is coupled to the hull 104 to secure various components within the maritime vehicle 100. The hull 104 is at least partially disposed in the body of water in which the maritime vehicle 100 is traversing. The hull 104 in this example is a mono-hull that has a front (or bow) 112, a rear (or stern) 116, two sides 120, and a keel 124 coupled to another. The front 112, the rear 116, the sides 120, and the keel 124 can be welded together or can be coupled to one another in a different manner. The hull 104 is configured such that the hull provides a continuous planning surface that allows the maritime vehicle 100 to be highly maneuverable and to ride along the top of a body of water at high speeds, even in extreme weather conditions and difficult to navigate bodies of water. Meanwhile, the cap 108 is coupled to the hull 104 to cover and / or conceal the components of the maritime vehicle 100 disposed in and carried by the hull 104 as the maritime vehicle 100 traverses the body of water.

[0025] In this example, the hull 104 and the cap 108 each have a length that is equal to approximately 6 feet. In other examples, however, the length can vary. For example, the length can be equal to approximately 14 feet. The hull 104 is preferably entirely made of aluminum but can be partially or entirely be made of fiberglass and / or one or more other materials. In other examples, the maritime vehicle 100 can include two orAttorney Docket No. 33894-S003 PC (PATENT) more hulls (e.g., two parallel hulls) instead of the mono-hull as depicted. In this example, the cap 108 entirely covers the hull 104 (and the components therein). In other examples, however, the maritime vehicle 100 need not include the cap 108 or the cap 108 may only partially cover the hull 104 (and the components disposed therein).

[0026] In some examples, the cap 108 can be removably coupled to the hull 104 via a locking system. For example, as illustrated in FIGS. 1 A-1 E, the locking system can take the form of a plurality of latch mechanisms 128 disposed around at least a portion (if not the entirety) of a perimeter of the maritime vehicle 100. Thus, the cap 108 can be removed to allow access to the interior of the hull 104. In other examples, however, the cap 108 can be permanently coupled to the hull 104 to permanently conceal the components within the maritime vehicle 100. The autonomous maritime surface vehicle 100 can include a plurality of bulkheads arranged within the hull 104 (not depicted). The bulkheads divide the maritime vehicle 100 into a plurality of different compartments for receiving and retaining different components in the maritime vehicle 100.

[0027] The autonomous maritime surface vehicle 100 also includes a sensor system that is generally configured to collect data about various components of the maritime vehicle 100 as well as data about the environment surrounding the maritime vehicle 100 (including data about objects in that environment). To this end, the sensor system generally includes a plurality of sensors disposed on an exterior or an interior of the maritime vehicle 100. The sensors can include, for example, one or more pressure sensors (e.g., positioned to detect the pressure of the ambient air external to the maritime vehicle 100, the pressure of the water in which the maritime vehicle 100 is disposed, the pressure within the maritime vehicle 100), one or more temperature sensors (e.g., positioned to measure a temperature of a component of the maritime vehicle 100, a temperature of ambient air external to the maritime vehicle 100, a temperature of water in which the maritime vehicle 100 is disposed), one or more acoustic sensors (e.g., sonar sensors), one or more LIDAR sensors, one or more location sensors (e.g., GPS sensors, compass sensors), one or more motion sensors (e.g., accelerometers, gyroscopes), one or more infrared sensors, one or more water sensors (e.g., a float switch, a capacitive sensor, an ultrasonic sensor, an electrical water sensor, etc.) to determine when water is present and / or present to a given extent (e.g., at a certain volume or level), one or more humidity sensors, one or more power sensors (e.g., configured to detect charging or fueling levels), one or more lighting sensors (e.g., daylight sensors), one or more imaging sensors (e.g., CCD sensors, CMOS sensors), one or more magnetic sensors, or combinations thereof.

[0028] Additionally, the autonomous maritime surface vehicle 100 may include a vision system 130 which is generally configured to capture, process, and analyze images obtained by the one or more image sensors and other data (e.g., data obtained by other sensors in the sensor system). In the example AMSV 100 of FIGS. 1 A-1 E, the vision system 130 is depicted as a stereoscopic vision unit that includes two independent stereoscopic cameras or image processors which may be infrared (IR) image processors or electro-optical (ER) image processors. The vision system 130 can identify or classify the environment surrounding the AMSVAttorney Docket No. 33894-S003 PC(PATENT)100 (including objects in that environment). In some embodiments, the vision system 130 is included in the sensor system. Notably, the vision system 130 can be an vision perception optimization system which can optimize image data for vision perception purposes, such as in manner discussed in more detail elsewhere herein.

[0029] The autonomous maritime surface vehicle 100 also includes a power system that is generally configured to power the maritime vehicle 100 (and the components of the maritime vehicle 100). The power system generally includes a thrust system and one or more power sources configured to power the thrust system (and the other components within the maritime vehicle 100). The thrust system is generally configured to propel the maritime vehicle 100 in / on / along the water. For example, the thrust system can be a propellerbased thrust system or can be a jet pump-based thrust system. The one or more power sources can include, for example, one or more batteries, fuel (e.g., gasoline, diesel) stored in tanks carried by the maritime vehicle 100, hydrogen stored in hydrogen tanks carried by the maritime vehicle 100, solar panels (e.g., mounted to an exterior of the vehicle 100), or other sources. The AMSV 100 illustrated in FIGS. 1A-1 E includes four battery assemblies each including a rechargeable battery, and includes a retention assembly for the four battery assemblies. The maritime vehicle 100 generally also includes a cooling system configured to cool the thrust system and / or the one or more power sources, thereby preventing these components from overheating and leading to failure of the maritime vehicle 100. For example, the maritime vehicle 100 can include a micro-keel cooler or other types of coolers.

[0030] In operation, the autonomous maritime surface vehicle 100 may be used to deploy and / or retrieve payloads such as, for example, persons, weapons (e.g., drones, missiles, mines, bombs), cargo (e.g., food), scientific instruments, or other equipment. Payloads can be deployed aerially (into the air), underwater, or on the surface of the water. Payloads can also be retrieved from the air, from underwater, or the surface of the water. Payloads to be deployed can be disposed in the hull 104, attached to the exterior surface of the hull 104, or attached to the exterior surface of the cap 108 prior to deployment. Likewise, retrieved payloads can be stored in the hull 104, attached to and stored on the exterior surface of the hull 104, or attached to and stored on the exterior surface of the cap 108.

[0031] The autonomous maritime surface vehicle 100 can also include other systems to help with the operation of the AMSV 100, for example a ballast system, the navigation system, and a pay load control system. The ballast system is generally configured to stabilize the maritime vehicle 100 in the water, regardless of whether the maritime vehicle 100 is stationary or on the move. To this end, the maritime vehicle 100 may include one or more ballast tanks or chambers selectively filled with water or air to vary the buoyancy of the maritime vehicle 100. Alternatively or additionally, the ballast system may include and utilize one or more inflatable devices to vary the buoyancy of the maritime vehicle 100. The ballast system may also provide for the selective submerging and re-surfacing of the maritime vehicle 100 in a similar manner. The navigation system, which may for example be an inertial navigation system, utilizes the sensors of the sensor system to track the position and orientation of the maritime vehicle 100 and to guide the maritime vehicle 100Attorney Docket No. 33894-S003 PC (PATENT) to its desired location in the body of water (or in a different body of water). Finally, the payload control system is configured to deploy or retrieve payloads.

[0032] The autonomous maritime surface vehicle 100 further includes a communications system that is generally configured to facilitate communication (i) between the maritime vehicle 100 and one or more central (remote) controllers, (ii) between the maritime vehicle 100 and and / or one or more other maritime vehicles 100 and / or other assets (e.g., planes, ships), and (iii) between different components of the maritime vehicle 100. The communications system generally includes one or more local controllers and one or more communication modules (e.g., one or more antennae, one or more receivers, one or more transmitters, one or more radios, one or more ethernet switches) to effectuate wired or wireless communication between the maritime vehicle 100 and the central controller(s) or other maritime vehicles 100. For example, the maritime vehicle 100 can include a plurality of antennae disposed on an exterior of the cap 108 as well as a plurality of antennae disposed in the hull 104.

[0033] As mentioned above, the autonomous maritime surface vehicle 100 may include one or more local (e.g., on-board) controllers which are generally configured to communicate data and / or information (e.g., data and / or information from the sensor system and other on-board systems, data and / or information received from other AMSVs 100 and / or other types of maritime vehicles or assets) and to perform automated operations of the AMSV 100 based on that data and / or information (e.g., by generating and sending operational instructions, control signals, etc.). In some examples, the AMSV 100 includes a plurality of different local or on-board controllers. For example, the AMSV 100 can include one or more sensor controllers (for controlling the sensors in the sensor system), one or more vision system controllers (for controlling the vision system), one or more payload controllers (for deploying or retrieving payloads), one or more navigation controllers (for controlling the operations of the navigation system), one or more thrust controllers (for controlling the operations of the thrust system), one or more communications controllers (for communicating data and / or information to off-board entities such as central (remote) controllers and / or other maritime assets), and one or more ballast controllers (for controlling the ballast system). It will be appreciated that each of the one or more controllers may be implemented as hardware (e.g., processor, die, integrated device), software (e.g., non-transitory processor readable medium), and / or combinations thereof, in one or more devices (e.g., processor, chip, computer, tablet, mobile device).

[0034] While not explicitly described or illustrated herein, it will be appreciated that the autonomous maritime surface vehicle 100 includes several additional components. For example, the AMSV 100 includes various sealing elements configured to provide seals between different components of the vehicle 60 (or between the vehicle 100 and the environment surrounding the vehicle 100). As another example, the AMSV 100 also includes various fasteners that help to couple the components of the AMSV 100 together. As yet another example, the AMSV 100 includes cabling and / or wiring that helps to communicatively couple components of the AMSV 100 together. As yet another example, the AMSV 100 includes various electricalAttorney Docket No. 33894-S003 PC(PATENT) components that help to operate the AMSV 100, e.g., one or more relay boards, one or more DC-DC converters, etc.

[0035] Generally speaking, the perception techniques / systems of the present disclosure allow an AMSV to accurately, reliably, and passively sense, perceive / detect, and identify targets within a marine, littoral, riparian, or other environment. More specifically, the techniques / systems of the present disclosure sense radiation from an external environment of an AMSV using a passive sensing system (e.g., a stereovision infrared (IR) camera) to detect objects within data representative of the radiation. The techniques / systems of the present disclosure thereby improve over conventional perception techniques / systems at least by accurately and reliably detecting targets in two and three dimensions within an AMSV external environment (e.g., marine environment) without emitting radiation detectable by such targets.

[0036] Further, due to the optimization of the perception techniques / systems of the present disclosure, and as discussed in more detail elsewhere herein, an AMSV can accurately, reliably, and passively sense, perceive / detect, and identify targets within a marine, littoral, riparian, or other environment and operate responsively more quickly and by using less (limited) on-board resources than is able to be achieved using conventional computer vision techniques.

[0037] Of course, it should be appreciated that the advantages and technical improvements described above and elsewhere herein are not the only advantages and / or technical improvements that may be realized as a result of the techniques described herein. Other advantages and / or technical improvements to the functioning of a computer itself or other technologies or technical fields may be apparent to one of ordinary skill in the art. For example, while described herein primarily in the maritime context, the vision perception optimization techniques described herein may be readily applied in any suitable field for any suitable purpose.

[0038] Example Autonomous Maritime Surface Vehicle (AMSV) - Block Diagram

[0039] FIG. 2 depicts a block diagram of an example autonomous maritime surface vehicle (AMSV) 200 in which embodiments of the on-board vision perception optimization systems, components, methods, and / or techniques of this disclosure may be implemented. For example, the AMSV 200 may be the AMSV 100 of FIGS. 1 A-1 E, or may be another AMSV. As shown in FIG. 2, the AMSV 200 includes a passive remote sensing system 212, a control system 215, a locomotion system 218, and a group of n communication interfaces 220a-220n via which the AMSV 200 may communicatively connect to off-board devices and systems (e.g., “off-board” or “external” communication interfaces 220a-220n), where typically, but not necessarily, n is an integer greater than one. The AMSV 200 may be, for example, the AMSV 100 of FIGS. 1A-1 E, in an embodiment, or another autonomous maritime surface vehicle. For example, the passive remote sensing system 212 may be implemented by at least a portion of the sensor system and / or the vision system of the AMSV 100, the control system 215 may be included in the one or more local or on-board controllers of the AMSV 100 or vice versa, the locomotion system 218 may include the navigation system and / or the thrustAttorney Docket No. 33894-S003 PC(PATENT) system of the AMSV 100, and the communication interfaces 220 may be included in the communications system of the AMSV 100, for example.

[0040] As shown in FIG. 2, the control system 215 of the AMSV 200 (also interchangeably referred to herein as “the AMSV control system 215” or the “vehicle control system 215”) includes one or more processors 222 and one or more memories 225 storing an AMSV control module 230 (also interchangeably referred to herein as a “vehicle control module” 230), a local situational awareness module (LSA) 232, a swarm situational awareness (SSA) module 235, and optionally one or more other modules (not shown). The one or more memories 225 may also store locally generated detection data generated by the AMSV, remotely generated detection data received from others of the AMSVs in the swarm, mission definition data, tracks generated by the LSA module based on the locally generated detection data and / or the remotely generated detection data, fused tracks received from the active SSA module, and other data as required for navigation, operation, and mission execution. The AMSV control system 215 also includes one or more communication interfaces 238 (e.g., one or more “on-board” or “internal” communication interfaces 238) via which the AMSV control system 215 may communicate with one or more other systems, components, and / or modules onboard the AMSV 200, such as the passive remote sensing system 212, the locomotion system 218, the off- board communication interfaces 220a-220n, an active remote sensing system 248 (if included in the AMSV 200), and other on-board systems, components, and / or modules.

[0041] Typically, each of the AMSV control module 230, the LSA module 232, and the SSA module 235 includes a respective set of computer-executable instructions that are executable by the one or more processors 222 to cause the AMSV 200 to perform one or more control methods and / or techniques described elsewhere herein; however, in some implementations, at least one of the modules 230, 232, 235 may be implemented at least partially using firmware and / or at least partially using hardware. Further, although FIG. 2 depicts the AMSV control module 230, the LSA module 232, and the SSA module 235 as being separate and distinct modules, this only for the purposes of clarity of discussion and is not limiting. For example, at least a portion of the AMSV control module 230 and at least a portion of the SSA module 235 may be implemented as an integral module, at least a portion of the LSA module 232 and the SSA module 235 may be implemented as an integral module, all three modules 230-235 may be implemented as a single integral module, at least a portion of at least one of the modules 230-235 may be implemented integrally with some other module of the control system 215 (not shown), etc.

[0042] Generally speaking, and as will be described in more detail below, the passive remote sensing system 212 operates to passively detect or sense the presence of objects (and, in some cases, the absence of the presence of one or more objects or of any object) within its field-of-view (FOV). Objects may include, for example, other AMSVs, other friendly maritime vehicles, enemy maritime vehicles, other types of enemy maritime assets (e.g., floating mines, enemy communication towers, etc.), enemy land-based assets (e.g., disposed on the coastline or shore), and the like.Attorney Docket No. 33894-S003 PC(PATENT)

[0043] Data indicative of the results of the sensing performed by the passive remote sensing system 212 may be provided to an image data optimizer 260 which may operate to optimize data (e g., optimize image data) provided by the passive remote sensing system 212, such as in manners described in more detail elsewhere herein. Although the image data optimizer 260 is illustrated in FIG. 2 as being included in the passive remote sensing system 212, this is only one of many possible embodiments. For example, at least a part (or an entirety) of the image data optimizer 260 can be included in the passive remote sensing system 212, and / or at least a part (or an entirety) of the image data optimizer 260 can be included in the AMSV control system 215. In some embodiments, at least a part of the image data optimizer 260 can be implemented in a stand-alone module, device, or system on-board the AMSV 200 which is separate and distinct from either the passive remote sensing system 212 or the AMSV control system 215. In an embodiment, the image data optimizer 260 can include a set of computer-executable instructions that are stored on one or more memories and executable by one or more processors to perform one or more of the methods and / or techniques disclosed herein. For example, the image data optimizer 260 can be stored on the memories 225 of the AMSV control system 215 and be executable by the processors 222 of the AMSV control system 215. Indeed, in some implementations, at least part of the image data optimizer 260 and at least part of one or more of the modules 230, 232, 235 of the AMSV control system 215 may operate as an integral module. Additionally or alternatively, the image data optimizer 260 can be stored on one or more memories included in the passive remote sensing system 212 (not shown) and be executable by one or more processors of the passive remote sensing system 212 (also not shown), and / or the image data optimizer 260 can be stored on one or more memories and be executable by one or more processors elsewhere on the AMSV 200. In some implementations, the image data optimizer 260 may be implemented at least partially using firmware and / or at least partially using hardware.

[0044] At any rate, the image data optimizer 260 may provide optimizations of the (raw) data generated by the passive remote sensing system 212 to the AMSV control system 215. The control system 215 may operate on the optimized, passively-sensed data (and optionally operate further based on data provided by at least one of the off-board communication interfaces 220a-220n and / or by other components of the AMSV 200) to generate a control signal, which the control system 215 may provide to the locomotion system 218. Responsive to the control signal, the locomotion system 218 may operate to change (or in some situations, maintain) an orientation of the AMSV 200, a movement of the AMSV 200, or both an orientation and a movement of the AMSV 100 within the body of water. In some situations, the AMSV control system 215 may generate control signals based on the sensing data provided by the passive remote sensing system 212 and information provided by other on-board and / or off-board systems, as will be described in more detail elsewhere herein.

[0045] As discussed above, the passive remote sensing system 212 is configured to passively detect or sense the presence and / or absence of an object (and / or of any objects, for that matter) within the FOV of the passive remote sensing system 212. As such, in an embodiment, the passive remote sensing system 212 may include one or more stereovision cameras 240. Each stereovision camera 240 may include a respectiveAttorney Docket No. 33894-S003 PC (PATENT) group of image sensors 242 which are configured to capture similar electromagnetic radiation across a similar FOV, and which are separated (e.g., fixedly separated) by a baseline distance 245. Typically, the group of image sensors 242 includes a pair of (i.e., two) image sensors 242a, 242b; however, in some implementations, the group of image sensors 242 may include more than two image sensors 242. However, for ease of reading herein and not for limitation purposes, the present disclosure refers to the group of image sensors 242 as including a pair of image sensors 242a, 242b.

[0046] The pair of image sensors 242 may utilize the same passive sensing technology. For example, the pair of images sensors 242b, 242b may be a pair of electro-optical (EO) sensors (e.g., a pair of red-blue- green or “RGB” sensors), a pair of infrared radiation (IR) sensors, etc. In FIG. 2, only a single stereovision camera 240 is depicted. In other embodiments, though (not shown), the AMSV 200 may include multiple stereovision cameras 240. For example, the AMSV 200 may include multiple EO stereovision cameras, multiple IR stereovision cameras, both an EO stereovision camera and an IR stereovision camera, etc. Each stereovision camera of the one or more stereovision cameras 240 may be fixedly disposed at a respective location on the AMSV 200 (e.g., with respect to the body of the AMSV 200), where the respective location of the stereovision camera 240 does not change over time. In some implementations, though, the respective location of a stereovision camera 240 with respect to the body of the AMSV 200 may be dynamically controlled, over time, to change, e.g., may be automatically controlled without any human intervention. For example, a stereovision camera 240 may be automatically raised or lowered with respect to altitude or distance from the deck of the AMSV 200 to accommodate for large waves, to avoid detection, etc. Generally speaking, the passive remote sensing system 212 may operate to obtain sets of data (e.g., images or digital images) indicative of captured electromagnetic radiation within its FOV at discrete time intervals (e.g., periodically at every x seconds and / or when desired), and / or the passive remote sensing system 212 may operate over time to continuously obtain sets of data (e.g., images or digital images) indicative of captured electromagnetic radiation within its FOV, e.g., as quickly as the system 212 can technically do so. For example, the passive remote sensing system 212 may livestream sensed data, where the livestream sensed data may include RGB and / or IR imaging livestreams. A more detailed description of the passive remote sensing system 212 and the time-series or continuous snapshots of sensing data generated by the passive remote sensing system 212 is provided elsewhere within this disclosure.

[0047] In some embodiments, in addition to the passive remote sensing system 212, the AMSV 200 may also include an active remote sensing system 248, such as a RADAR (Radio Detection and Ranging), LIDAR (Light Detection and Ranging), or some other type of active remote sensing system which generally requires the active remote sensing system 248 to expressly or actively initiate transmissions of signals (e.g., radio signals, light signals, sound signals, etc.) to perform the sensing of remote objects. Data indicative of the results of the active sensing performed by the active remote sensing system 248 may be provided to the control system 215, which may operate on the active sensing data provided by the active remote sensing system 248 in conjunction with the optimized, passive sensing data provided by the passive remote sensing system 212 (and optionally the data provided via at least one of the off-board communication interfaces 220a-Attorney Docket No. 33894-S003 PC (PATENT)220n and / or one or more other on-board systems and / or components) to generate control signals that are to be provided to the locomotion system 218. However, an active remote sensing system 248 is not a necessary or required component of the AMSV 200. Indeed, in some embodiments, the AMSV 200 does not include (that is, the AMSV 200 excludes) any type of active remote sensing system 248 at all. In some embodiments, the AMSV 200 includes an active remote sensing system 248 but powers down, deactivates, disables, or turns off the active remote sensing system 248 altogether (e.g., so that that active remote sensing system 248 does not emit any signals and transmissions at all, including not transmitting any heartbeat, scanning, or other administrative types of signals) so that the AMSV 200 is totally “radio-silent” and relies only on optimized, passive sensing data provided by the passive remote sensing system 212 and the image data optimizer 260 to generate control signals for the locomotion system 218.

[0048] Turning back to the passive remote sensing system 212, the passive remote sensing system 212 of the AMSV 200 may be communicatively connected to the image data optimizer 260, and the image data optimizer 260 may be communicatively connected to the AMSV control system 215 on-board the AMSV 200, for example, in a wired manner via communication interfaces 238. The passive sensing data generated by the passive remote sensing system 212 (e.g., FOV sensed data, image data, images, etc.) may be optimized by the image data optimizer 260, and the optimized, passive sensing data may be received by the local situational awareness (LSA) module 232 of the control system 215 and / or may be stored locally on-board the AMSV 200. For example, the passive remote sensing system 212 and / or the image data optimizer 260 may transmit at least some of the optimized, passive sensing data via the communicative connection between the passive remote sensing system 212 and the LSA module 232. Additionally or alternatively, the passive remote sensing system 212 and / or the image data optimizer 260 may store optimized, passive sensing data in local data storage 250 on-board the AMSV 200 (also interchangeably referred to herein as “on-board” data storage 250) and the LSA module 232 of the AMSV control system 215 may access the optimized, passive sensing data stored in local data storage 250. In some situations, the (optimized) passive sensing data may be indicative of a detection of a presence of a remotely located object within the FOV of the passive remote sensing system 212. Upon or after the initial detection of the presence of the object, the LSA module 232 may utilize one or more image and / or vision processing techniques (e.g., image segmentation, object detection, etc.) to detect, classify, and / or identify the object detected within the FOV sensed data (e.g., recreational maritime vehicle, type of enemy vessel, specific enemy vessel, etc.). Additionally, the LSA module 232 may utilize subsequent, optimized, passive sensing data provided by the passive remove sensing system 212 (e.g., snapshots of sensing data over time) to generate and update a local track of the path of the detected object over time, where the local track may be indicative of the direction(s) in which the detected object has moved between various detections over time. The elapsed time interval between attempted detections of objects (e.g., between snapshots of sets of passively sensed data generated by the passive remote sensing system 212) may be a standard or periodic time interval (e.g., every x seconds), and / or the elapsed time intervals between various attempted detections may vary over time.Attorney Docket No. 33894-S003 PC(PATENT)

[0049] For example, the LSA 232 may obtain a plurality of detections of the object over time (e.g., timeseries snapshots of sets of optimized, passively sensed data) and determine a respective relative position of the object with respect to the AMSV 200 for each detection, e.g., based on global positioning system (GPS) or other types of geospatial positioning data indicative of the AMSV’s current physical geospatial location, which may be obtained via off-board communication interface 220a, and / or based on the data provided by the stereovision system 240, for example. The LSA 232 may generate and / or update a local track of the object based on the respective relative positions of the detected object over time with respect to the AMSV 200. As such, the local track of an object may be indicative of a path of travel, over time, of the object as perceived by and with respect to the location(s) of the AMSV 200 over time (e.g., a track that is locally determined at the AMSV 200), and as such the local track may include both a geographical location component as well as a temporal component. The LSA module 232 may generate and update a plurality of local tracks of a plurality of remotely-located objects whose respective presences have been detected by the passive remote sensing system 212 of the AMSV 200. Additionally, indications of local tracks of one or more detected objects and updates thereto may be stored in the local data storage 250. As such, the local data storage 250 may store indications of one or more current local tracks respectively corresponding to one or more detected objects. In some situations, the local data storage 250 may store respective local tracks of all objects which have been detected by the passive remote sensing system 212.

[0050] It is noted that although FIG. 2 depicts the LSA module 232 as being included in the control system 215, in some embodiments (not shown in FIG. 2), at least a portion of the LSA module 232 (or, in some implementations, an entirety of the LSA module 232) may be included in the passive remote sensing system 212. For example, in addition to the passive remote sensing system 212 generating time-series data of attempts to detect objects (e.g., time-series snapshots indicative of any objects sensed within the FOV of the passive remote sensing system 212), the passive remote sensing system 212 may utilize its integral LSA module 232 generate local tracks of detected objects based on the generated time-series data.

[0051] Further, it is noted that although FIG. 2 depicts the local data storage 250 as being separate and distinct from the passive remote sensing system 212, the AMSV control system 215, and the AMSV locomotion system 218, this is only one of numerous possible embodiments. For example, a respective at least a portion of the local data storage 250 may be included in at least one of the passive remote sensing system 212, the AMSV control system 215, and / or the AMSV locomotion system 218, and the local data storage 250 may be accessed by the passive remote sensing system 212, the AMSV control system 215, and / or the AMSV locomotion system 218. Generally speaking, local data storage 250 may include one or more tangible, non-transitory, computer-readable media or memories such as magnetic disks, laser disks, optical discs, semiconductor memories, biological memories, random access memories (RAMs), flash memories, other memory devices, or other storage media.

[0052] T urning now to the AMSV control module 230 of the AMSV control system 215, the AMSV control module 230 may generate a control signal for the locomotion system 218 based on one or more inputs. TheAttorney Docket No. 33894-S003 PC(PATENT) one or more inputs may include for example, one or more local tracks (or indications thereof) generated by the local situational awareness (LSA) module 232, one or more swarm-level tracks (or indications thereof) generated by on-board swarm situational awareness (SSA) module 235 or by another off-board SSA servicing the group or swarm in which the AMSV 200 is included, an indication of a geospatial location of a detected object (e.g., as detected by the passive remote sensing system 212), an indication of a geospatial location of the AMSV 200, (e.g., as indicated via GPS communication interface 220a or similar geospatial coordinate interface), and / or data provided by one or more sensor systems that are generally configured to collect data about various components of the AMSV 200 and data indicative of environmental conditions surrounding the AMSV 200 (e.g., pressure, temperature, water level and / or other water conditions, wind, humidity, speed, direction or orientation, etc.). In some situations, the AMSV control module 230 generates control signals further based on information obtained by the AMSV 200 via the off-board communication interfaces 220a-220n, e.g., information received at the AMSV 200 from GPS systems, other AMSVs, a remotely-located (e.g., off-board) SSA servicing the group or swarm of AMSVs in which the AMSV 200 is included, other maritime vehicles, land-based systems or devices, cloud-based systems, central (remote) controllers, etc. As such, the off-board communication interfaces 220a-220n may include interfaces supporting multiple different wireless technologies, such as a GPS communication interface 220a, one or more satellite communication interfaces 220c, one or more cellular communication interfaces 220d, one or more other types of line-of-sight (LOS) communication interfaces 220b (e.g., optical, Bluetooth, Zigbee, Digimess, Wi-Fi, NearLink, near-field communications (NFC), LPWAN, UWB, IEEE 802.15.4-compatible, etc.), and / or other types of wireless communication interfaces. During operations, the AMSV 200 may power down, deactivate, disable, or turn off any one or more of the off-board communication interfaces 220a-220n as desired. Indeed, during some operations, the AMSV 200 may power down, deactivate, disable, or turn off all of the off-board communication interfaces 220a-220n so that that the AMSV 200 does not emit any wireless signals and transmissions at all, including not transmitting any heartbeat, scanning, or other administrative types of signals) so the AMSV 200 operates in a radio-silent mode. In these situations, the AMSV 200 may rely solely on optimized, passive sensing data provided by the on-board passive remote sensing system 212 to generate control signals for the locomotion system 218 and navigate in its environment.

[0053] As discussed above, in some situations, at least one of the inputs based on which the AMSV control module 230 generates a control signal includes respective local tracks of one or more detected objects, where the one or more local tracks are generated by the local situational awareness (LSA) module 232. In some situations, the AMSV control module 230 generates a control signal additionally or alternatively based on information and / or instructions received from a swarm situational awareness (SSA) module, where the SSA module may be the local SSA module 235 disposed on-board the AMSV, a remote SSA module disposed on-board another AMSV, a remote SSA module disposed in a mobile control system (MGS) (which may be disposed on another maritime vehicle or on land), or some other remotely-located SSA module corresponding to a group or swarm of AMSVs in which the AMSV 200 is included. Generally speaking, anAttorney Docket No. 33894-S003 PC(PATENT)SSA module servicing a group or swarm of AMSVs in which the AMSV 200 is included may operate to receive a plurality of local tracks of a detected object respectively from a plurality of AMSVs included in the group or swarm of AMSVs, and to generate a fused (e.g., a swarm-level) track of the detected object therefrom. The SSA module of the group or swarm may or may not receive and utilize a local track generated by the AMSV 200 to generate the fused or swarm-level track that is utilized by the AMSV 200 to navigate and move. A more detailed discussion of the SSA module and its use in swarms or groups of AMSVs and coordinating control across the swarm or group of AMSVs is discussed elsewhere herein.

[0054] Generally speaking, the SSA modules are operable to receive detection data from each AMSV in a swarm or fleet. Detection data transmitted to the active SSA module are generally transmitted using a guaranteed transmission method that verifies receipt of uncorrupted data. In embodiments, the detections from each AMSV are used to create AMSV-specific tracks of each detected object, and tracks that overlap by a predetermined threshold (e.g., 80% co-detections) are fused into a fused track. Of course, other embodiments of track fusion that could be employed.

[0055] Turning now the locomotion system 218, the locomotion system 218 is generally responsive to control signals generated by the AMSV control module 230, and is configured to orient the AMSV 200 in and move the AMSV 200 through the water, e.g., based on the control signals. Accordingly, the AMSV 200 may include a navigation system configured to orient the AMSV 200 (e.g., orient a direction of the AMSV 200), a thrust system configured to propel the AMSV 200 in / on / along the water, and one or more power sources to power the navigation system, the thrust system, and other components of the locomotion system 218 and the AMSV 200 itself. For example, the locomotion system 218 may be included in the AMSV 100 and may be implemented, for example, by at least portions of the navigation and thrust systems of the AMSV 100. As the locomotion system 218 is controlled via control signals generated by the AMSV control system 215, the AMSV 200 is able to operate autonomously, e.g., without receiving any real time control signals generated by humans. In some situations, the navigation system and the thrust systems may be independently controlled, and in some situations, the navigation system and the thrust systems may be controlled in a coordinated manner.

[0056] In some situations, the AMSV 200 may operate independently of any other AMSV to perform a mission (e.g., a military mission, a civilian mission, a commercial mission, etc.). That is, only one AMSV 200 may autonomously and independently operate to perform one or more tasks (or all tasks) of a mission with which the AMSV 200 has been charged, e.g., without communicating with any other AMSV 200 with respect to the mission tasks and / or at all.

[0057] Example Perception Hardware Configurations

[0058] FIG. 3A depicts a first example perception hardware configuration 300, in accordance with various embodiments described herein. In an embodiment, the passive remote sensing system 212 of FIG. 2 may include the first example perception hardware configuration 300. Generally, the first example perceptionAttorney Docket No. 33894-S003 PC(PATENT) hardware configuration 300 includes two stereovision cameras 302 configured to capture radiation from an external environment of the AMSV (e.g., AMSV 200), upon which, the first example perception hardware configuration 300 is mounted, integrated, and / or otherwise associated. In particular, the two stereovision cameras 302 are each configured to capture radiation using two image sensors 302a / b, 302c / d separated by a baseline distance 304 that mimics human binocular vision and thereby enables depth perception based on the feature disparities within the captured images. It should be appreciated that the image sensors 302c and 302d are separated by a shorter baseline distance than the image sensors 302a, and 302b. For example, the baseline distance 304 represents the distance between the image sensors 302c, 302d, and the image sensors 302a, 302b are separated by the baseline distance 304 in combination with some additional distance (e.g., including the dimensions of the image sensors 302c, 302d). For example, the stereovision cameras 302a / b, 302 c / d may include the sensors 242a / b of FIG. 2, and the distance 304 may be the distance 245 of FIG. 2.

[0059] The two stereovision cameras 302 include an IR stereovision camera comprised of a first IR image sensor 302a and a second IR image sensor 302b and an EO stereovision camera comprised of a first EO image sensor 302c and a second EO image sensor 302d. At least the IR stereovision camera passively captures (e.g., does not include / use an emission source) radiation, but it should be appreciated that any of the perception systems described herein may utilize passive sensing and / or active sensing. Moreover, while the discussion herein focuses primarily on the IR stereovision camera, the descriptions of the IR stereovision camera and corresponding IR image sensors may apply to the EO stereovision cameras, EO image sensors, and / or other sensing systems described herein.

[0060] As illustrated in FIG. 3A, the first IR image sensor 302a and the second IR image sensor 302b have FOVs 306a, 306b, represented by the lines extending diagonally outwards from the first and second IR image sensors 302a, 302b. Both IR image sensor FOVs 306a, 306b have an optical axis 306a1 , 306b1 that correspond to the principal point of the FOVs 306a, 306b at any distance from the image sensors 302a, 302b. Thus, any object located in the AMSV external environment in-line with either optical axis 306a1 , 306b1 will appear at the principal point of the resulting image created by the respective image sensor(s) 302a, 302b.

[0061] These two FOVs 306a, 306b intersect / overlap at a particular distance away from the image sensors 302a, 302b, creating a composite FOV 306c and a blind spot 306d. The composite FOV 306c represents a physical region of the AMSV external environment, from which, both image sensors 302a, 302b capture radiation, and consequently capture representations of the same objects / features within the AMSV external environment. However, because the composite FOV 306c includes different portions of the individual image sensor 302a, 302b FOVs 306a, 306b, the same object / feature representations in the images are included at different positions within the images. For example, in simultaneous image captures of the first IR image sensor 302a and the second IR image sensor 302b, a target vessel located within the composite FOV 306c will generally appear more towards the right edge of the first FOV 306a than the target vessel will appear relative to the right edge of the second FOV 306b because the optical axes 306a1 , 306b1 are parallel.Attorney Docket No. 33894-S003 PC(PATENT)

[0062] The blind spot 306d is a region of the AMSV external environment that is imperceptible by the IR stereovision camera because the IR image sensors 302a, 302b are not oriented and / or the focusing optics are otherwise not configured to capture radiation from this region. It will be appreciated that the FOVs 306a-c and the blind spot 306d in FIG. 3A are not drawn to scale, such that the blind spot 306d may only comprise a relatively small portion of the AMSV external environment, as compared to the portions included / covered by the FOVs 306a-c. Nevertheless, the blind spot 306d may preclude or complicate the AMSV sensing / perception systems described herein from accurately detecting, identifying, and / or otherwise locating objects disposed within this relatively small region proximate to the AMSV. This can lead to issues when the AMSV needs to maneuver precisely relative to objects located within the blind spot 306d, such as when an AMSV path plan involves the AMSV contacting or otherwise maneuvering into very close proximity to a tracked object (e.g., a target vessel).

[0063] To overcome these potential issues, FIG. 3B depicts a second example perception hardware configuration 310, in accordance with various embodiments described herein. The second example perception hardware configuration 310 includes an IR stereovision camera 312 that includes a first IR image sensor 312a and a second IR image sensor 312b separated by a baseline distance 314. For example, the sensors 312a / b may be the sensors 242a / b of FIG. 2, and the distance 315 may be the distance 245 of FIG. 2. The first IR image sensor 312a has a first FOV 312a and the second IR image sensor 312b has a second FOV 312b and the image sensors 312a, 312b are oriented slightly towards one another. As a result, and unlike the optical axes 306a1 , 306b1 of FIG. 3A, the first optical axis 312a1 of the first FOV 312a is not parallel with the second optical axis 312b1 of the second FOV 312b.

[0064] More specifically, the first IR image sensor 312a and the second IR image sensor 312b are oriented towards one another such that a left edge 312a2 of the first FOV 312a is substantially parallel (e.g., within 3° of exactly parallel) to a right edge 312b2 of the second FOV 312b. This configuration of the first IR image sensor 312a and the second IR image sensor 312b yields a central FOV 312c that includes more of the external environment that was previously included as part of the blind spot 306d of FIG. 3A. Thus, the blind spot 312d is significantly smaller than the blind spot 306d and thereby enables the AMSV sensing / perception systems described herein to detect, identify, and / or otherwise locate objects disposed proximate to the AMSV (e.g., near a front or a front portion of the AMSV) more accurately than in the first example perception hardware configuration 300. In some embodiments, the first IR image sensor 312a and the second IR image sensor 312b may be oriented towards one another, but the left edge 312a2 and the right edge 312b2 may not be substantially parallel.

[0065] Further, the first IR image sensor 312a and the second IR image sensor 312b may be physically oriented towards one another and / or may include optical components that yield the FOVs 312a, 312b illustrated in FIG. 3B. For example, the first IR image sensor 312a and the second IR image sensor 312b may include various optical components (e.g., lenses, mirrors, prisms, gratings, etc.) configured to focus, reflect, diffract, and / or otherwise manipulate the incoming radiation that may consequently impact the FOVs 312a,Attorney Docket No. 33894-S003 PC (PATENT)312b. In this configuration 310, any objects within the central FOV 312c will move to the opposite side of the image sensor 312a, 312b from what is intuitively expected. Namely, objects positioned in the central FOV 312c (e.g., at distances greater than a few meters from the IR stereovision camera 312) will be on the left side of the optical axis 312a1 and on the right side of the optical axis 312b1 .

[0066] It should be appreciated that the angular size of the overlap illustrated in FIG. 3B decreases significantly with distance, but stereovision accuracy also becomes significantly less accurate with distance. Thus, the angular alignment of the two image sensors 312a, 312b should be chosen to optimize the total angle of both FOVs 312a, 312b (e.g., the union of FOVs 312a, 312b) and the distance at which the overlap angle becomes too small. Orienting the image sensors 312a, 312b inward past where the edges 312a2, 312b2 are substantially parallel will create an FOV overlap of finite size.

[0067] In some embodiments, the second example perception hardware configuration 310 may facilitate interception of target objects detected / identified by the AMSV. Target objects located within the FOVs 312a, 312b may be detected and identified as target objects, and the AMSV and / or any host device (e.g., MCS 18) may determine an AMSV path plan configured to cause the AMSV to intercept the target object. The AMSV may maneuver in accordance with the AMSV path plan to execute the plan and intercept the target object. For example, the target object may be a friendly vessel, and the AMSV path plan may cause the AMSV to intercept the friendly vessel by maneuvering proximate to the friendly vessel (e.g., within 1-3 meters) to enable the crew of the friendly vessel to board the AMSV and / or otherwise retrieve a deliverable stored in the AMSV. As another example, the target object may be an unfriendly vessel, and the AMSV path plan may cause the AMSV to intercept the unfriendly vessel by maneuvering proximate to the unfriendly vessel (e.g., physically impact or otherwise contact the vessel) and delivering an explosive payload into the unfriendly vessel. Accordingly, the minimal blind spot 312d (also referenced herein as a “reduced” blind spot) enables the AMSV to accurately execute such AMSV path plans at least by reducing the time spent without viewing the target object / location indicated in the AMSV path plan. In any event, the AMSV can maintain a stable course to the target object even without receiving updates (e.g., via radio) from other AMSVs or host devices, in part, because the AMSV can readily view the target object up to the point of contact using only passive sensing as a result of the minimal blind spot 312d.

[0068] In certain embodiments, the imagers 312a, 312b may be faced in opposite directions (e.g., outward), which will decrease the FOV overlap (e.g., size of central FOV 312c) and increase the size of the union of the FOVs 312a, 312b. However, turning the imagers 312a, 312b outward will necessarily create a larger blind spot than the blind spot 312d illustrated in FIG. 3B, such that the systems described herein may lack data of objects proximate to the AMSV.

[0069] In certain instances, the AMSV may benefit from expanding or narrowing the perception system FOVs. For example, a wider FOV enables more robust object tracking within the FOV at least by reducing the likelihood of the object slipping outside of the FOV edges and therefore exceeding the AMSV’s perceptive range. A narrower FOV can increase the accuracy of object detection / identification / tracking by increasing theAttorney Docket No. 33894-S003 PC(PATENT) effective image resolution as a direct result of increasing the pixel density in the observed angular region. FIG. 3C depicts a third example perception hardware configuration 320 that leverages wider / narrower FOVs, in accordance with various embodiments described herein.

[0070] The third example perception hardware configuration 320 includes an IR stereovision camera 322 with a first IR image sensor 322a and a second IR image sensor 322b separated by a baseline distance 324. For example, the sensors 322a / b may be the sensors 242a / b of FIG. 2, and the distance 324 may be the distance 245 of FIG. 2. The first IR image sensor 322a has a relatively wide FOV 326a, as indicated by the first angle 328a. The second IR image sensor 322b has a relatively narrow FOV 326b, as indicated by the second angle 328b. In particular, the first angle 328a is greater than the second angle 328b, and results in a wider FOV 326a than the FOV 326b, as well as the FOVs 306a, 306b, 312a, and 312b illustrated in FIGS. 3A and 3B. By contrast, the second angle 328b results in a narrower FOV 326b than the FOV 326a, as well as the FOVs 306a, 306b, 312a, and 312b illustrated in FIGS. 3A and 3B.

[0071] Using this third example perception hardware configuration 320, the perception systems described herein may detect / identity / track objects located within the composite FOV 326c more accurately based on the narrow FOV 326b and / or may achieve more robust tracking capabilities due to the larger overall FOV from the wide FOV 326a. Namely, the narrow FOV 326b achieves a higher angular pixel density for objects detected within the composite FOV 326c, and the wide FOV 326a may achieve a larger overall FOV (e.g., FOV 326a combined with FOV 326b) to ensure tracked objects do not fall outside of the FOV edges.

[0072] Of course, the example configuration 320 represented in FIG. 3C is for the purposes of discussion only, and it should be appreciated that any combination of image sensors with narrower / wider FOVs and / or image sensors or optics (e.g., lenses, etc.) orientations may be utilized to achieve the desired advantages. For example, a first combination may include an image sensor (e.g., 322b) with the narrow FOV 326b and an image sensor with any of the other FOVs (306a, 306b, 312a, 312b) illustrated and described herein. A second example combination may include an image sensor (e.g., 322a) with the wide FOV 326a and an image sensor with any of the other FOVs illustrated and described herein. Any of these image sensor configurations may yield one or more of the advantages described herein, such as greater pixel density for improved detection / identification / tracking accuracy, larger overall FOV to reduce the likelihood of objects slipping outside of the FOV edges, and / or any other advantages described herein.

[0073] In any event, the combined FOVs (e.g., 306c, 312c, 326c) described herein enable the depth measurements of the stereovision perception systems of the AMSV. As such, the AMSV’s described herein generally maintain at least objects of interest (e.g., targets) within the combined FOV to determine the three- dimensional (3D) position of such objects. FIG. 3D depicts a fourth example perception hardware configuration 330 that highlights the combined FOV and objects disposed within therein, in accordance with various embodiments described herein.Attorney Docket No. 33894-S003 PC (PATENT)

[0074] The fourth example perception hardware configuration 330 includes a stereovision system 332 that includes, for example, a stereovision IR camera and a stereovision EO camera. The stereovision IR camera includes two IR image sensors that each have a FOV, resulting in a combined FOV 336. For example, the stereovision system 332 may be similar to the first example perception hardware configuration 300 of FIG. 3A, and the combined FOV 336 may be an extension of the combined FOV 306c.

[0075] Multiple objects 334a-d are disposed within the combined FOV 336. Thus, both the IR image sensors of the IR stereovision camera will capture radiation reflected or emitted from each of the objects 334a-d, but each of the objects 334a-d will be in a slightly different position within the images captured by the different IR image sensors. For example, the first object 334a will appear more towards the right edge of the left IR image sensor FOV than the first object 334a will appear relative to the right edge of the right IR image sensor FOV. This difference in perceived location represents the disparity between the two image sensors resulting from the baseline distance separating the two image sensors, and enables depth measurements based on these sets of images in accordance with the below equation:

[0077] where D is the depth, f is the focal length of the image sensors, B is the baseline distance between the two image sensors, and 6 is the disparity between the coordinate locations of an object in the two images.

[0078] To illustrate, the IR image sensors may each capture images featuring the object 334b, as represented by the lines of sight 338a, 339a of the respective imagers. The position of the object 334b within the respective images captured by the different IR image sensors is represented by the different angles 338b, 339b of the lines of sight 338a, 339a from the respective optical axes. The object 334b thus appears at different coordinate positions within the images captured by the different IR image sensors, such that the processing components described herein can determine the disparity between the two coordinate locations and the depth of the object 334b based on equation (1). Thus, each of the example perception hardware configurations illustrated herein enable depth measurements based on the principles represented by equation (1) because each hardware configuration includes stereovision cameras separated by a baseline distance.

[0079] It should be appreciated that some / all of the imagers / sensors described herein may be stacked and / or otherwise organized in a manner that maximizes the baseline between each pair of stereo imagers to further improve the vision systems described herein. For example, each IR image sensor of an IR stereovision camera may be stacked below / on top of EO image sensors of an EO stereovision camera at opposite corners of a housing to increase the effective baseline of both stereovision cameras. Further, it should be appreciated that the angular overlap of the stereovision FOVs described herein will decrease with distance, but this does not represent a genuine disadvantage because stereovision techniques generally lack resolving power over these distances. Accordingly, any of the angles described herein can be selected to optimize maximum overlap for a given camera resolution and baseline.Attorney Docket No. 33894-S003 PC(PATENT)

[0080] In any event, the perception techniques described herein use these hardware configurations in combination with various perception algorithms to improve conventional techniques, particularly those for perception in an external environment of an AMSV (e.g., a marine environment). These perception techniques are described further herein in reference to FIGS. 4A-4K.

[0081] Example Perception Software Analyses

[0082] Generally speaking, any of the example perception software analysis scenarios illustrated and described herein may utilize any of the hardware components described herein in reference to FIG. 2 and / or FIGS. 3A-3D. For example, any of the example perception software analysis scenarios may use or include a stereovision system, including a stereovision IR camera with two IR image sensors and / or a stereovision EO camera with two EO image sensors. It should also be appreciated that the perception algorithm described herein may utilize (e.g., simultaneously or otherwise in combination) any one or more of the algorithms, evaluations, analyses, calculations, equations, and / or any other concepts described herein in reference to FIGS. 4A-4K to improve, adjust, and / or otherwise influence the perception algorithm’s depth / distance estimates / measurements. Moreover, any of the perception techniques described herein may be utilized by a single AMSV, multiple AMSVs in combination, and / or at a fleet-level among an entire fleet of AMSVs to create an aggregate / collective perception (e.g., object detection / identification / location) of the external environment for a single AMSV, multiple AMSVs, and / or a fleet of AMSVs.

[0083] FIG. 4A depicts a first example perception software analysis scenario 400 using any of the hardware configurations 300-330 of FIGS. 3A-3D, in accordance with various embodiments described herein. The first example perception software analysis scenario 400 includes a stereovision system 402a with a composite FOV 406. There are multiple objects 404a-d positioned within the composite FOV 406, such that the stereovision system 402a can capture radiation representing each of the objects 404a-d.

[0084] The first example perception software analysis scenario 400 is similar to the fourth example perception hardware configuration 330 of FIG. 3D, but further includes a perception algorithm 402b communicatively coupled with the stereovision system 402a. The perception algorithm 402b is configured to process the image data generated by the stereovision system 402a and detect objects within the image data. In particular, the perception algorithm 402b causes the AMSV processors to perform one or more machine vision techniques (e.g., image segmentation, scale invariant feature transforms (SIFT), histogram of oriented gradients (HOG), implementing a convolutional neural network (CNN), etc.) on the image data generated by the stereovision system 402a. These machine vision techniques may separate / segment the image data into various classes or classifications that correspond to one or more objects.

[0085] In certain embodiments, the perception algorithm 402b may also identify the objects within the image data and / or may further generate and / or output data contributing to track determinations, as described herein. In these embodiments, each AMSV may generate an individual track which can be combined for a collective (e.g., fleet-level) track, as further described herein.Attorney Docket No. 33894-S003 PC(PATENT)

[0086] As a simple example, the stereovision system 402a may capture images of the objects 404a-d located within the composite FOV 406, and the perception algorithm 402b may cause the AMSV processors to analyze these images. The perception algorithm 402b may cause the AMSV processors to execute one or more machine vision techniques that detect each of the four objects 404a, 404b, 404c, and 404d within the image data. Further, based on this machine vision analysis instructed by the perception algorithm 402b, the AMSV processors may determine that the second object 404b is an object of interest (e.g., a target vessel) that the AMSV should track and / or otherwise accurately locate. The perception algorithm 402b may cause the AMSV processors to indicate this identification as an object of interest based on a mask 408 associated with a class / classification of one or more objects of interest.

[0087] The mask 408 may be a segmentation mask, and it should be appreciated that such a mask 408 may appear within an image captured by the stereovision system 402a. Thus, the representation of the mask 408 over the second object 404b within the composite FOV 406 is for the purposes of illustration / discussion only. Similar masks are discussed herein in reference to FIG. 4B, which depicts a second example perception software analysis scenario 410 using any of the hardware configurations 300-330 of FIGS. 3A-3D, in accordance with various embodiments described herein.

[0088] The second example perception software analysis scenario 410 generally is an example image the perception algorithm (e.g., algorithm 402b) analyzes and / or indicates analysis performed to detect objects and / or identify the objects. The example image includes a marine (water) portion 411 and an air portion 412. The marine portion 411 has multiple objects 404a-d floating and / or otherwise disposed therein, including a first object 414a (e.g., a rock), a second object 414b (e.g., a target vessel), third object 414c (e.g., a rock), and a fourth object 414d (e.g., a rock). It should be understood that the bottom of the marine portion 41 1 represents a first distance 413a that is shorter than a second distance 413b represented by the top of the marine portion 411 .

[0089] To detect each of the objects 414a-d in the example image, the perception algorithm may cause the AMSV processors to perform any suitable machine vision techniques or combinations thereof. More specifically, the perception algorithm may cause the AMSV processors to analyze an image by examining the image pixel data to identify patterns, shapes, and / or contrasts that correspond to known characteristics of objects and / or that otherwise differ from the known / consistent characteristics of the background environment (e.g., marine environment). Through techniques such as edge detection, image segmentation, and pattern recognition, the perception algorithm can cause the AMSV processors to differentiate objects from the background environment and determine which pixels likely correspond to a complete “object” within the image.

[0090] For example, the perception algorithm detects each of the objects 414a-d within the example image by determining that each of the pixels comprising those objects 414a-d are sufficiently similar to one another and / or sufficiently different from the pixels representing the surrounding environment that the pixels should be grouped together to represent an object. At this point, the perception algorithm may or may not identify theAttorney Docket No. 33894-S003 PC (PATENT) object (e.g., identification agnostic detection), but may only recognize the presence of a distinct object within the image. Once the perception algorithm detects the objects 414a-d within the example image, the algorithm may proceed to identify each object 414a-d based on many / all of the same pixel characteristics used to detect the objects 414a-d. In certain embodiments, the perception algorithm may simultaneously or nearly simultaneously identify the objects 414a-d as part of the object detection.

[0091] With continued reference to FIG. 4B, each of the multiple objects 414a-d has an associated mask 414a1 , 414b1 , 414c1 , 414d1 corresponding to the machine vision processes performed by the perception algorithm to detect and / or identify each of the objects 414a-d. In embodiments where the perception algorithm identifies each object 414a-d, each mask 414a1-d1 may represent and / or otherwise include an associated class or classification, which the perception algorithm determines is applicable to the respective object 414a- d. For example, the first mask 414a1 , the third mask 414c1 , and the fourth mask 414d1 may each represent and / or include a class / classification indicating that the objects 414a, 414c, 414d referenced by the masks 414a1 , 414c1 , 414d1 are each an environmental object (e.g., rocks). As another example, the second mask 414b1 may represent and / or include a class / classification indicating that the second object 414b referenced by the second mask 414b1 is a non-environmental object (e.g., man-made object) or another vessel (e.g., target vessel).

[0092] In certain embodiments, the masks 414a1 -d 1 may be segmentation masks corresponding to the objects in the image as a result of image segmentation and / or other suitable machine vision techniques performed by the perception algorithm. In some embodiments, the masks 414a1-d1 may appear as part of an image output for display to a user, and may also visually indicate (e.g., via color, patterning, etc.) the classes / classifications associated with each object 414a-d.

[0093] Additionally, the second object 414b also includes a representation of a lowest pixel 414b2. This lowest pixel 414b2 indicates a pixel that the perception algorithm determined corresponds with the second object 414b and has the lowest or smallest vertical position value of any pixel associated with the second object 414b. Broadly speaking, the perception algorithm may analyze the example image such that each pixel has associated coordinate values (e.g., Cartesian coordinates) in addition to the other pixel values corresponding to the image characteristics. For ease of discussion, each pixel in the example image may have a corresponding x-value associated with the pixel’s lateral (e.g., left / right) position within the image and a corresponding y-value associated with the pixel’s vertical (e.g., up / down) position within the image. The lateral position may correspond to the physical, lateral location of the corresponding object in real space, and the vertical position may correspond to a physical height of the corresponding object in real space.

[0094] Thus, the lowest pixel 414b2 is a pixel within the example image that represents the lowest visible point (e.g., height) of the second object 414b in real space. In certain embodiments, the perception algorithm can use this lowest pixel 414b2 to further improve the distance / depth measurements made using the stereovision cameras described herein. For example, FIG. 4C depicts a third example perception softwareAttorney Docket No. 33894-S003 PC (PATENT) analysis scenario 415 using any of the hardware configurations 300-330 of FIGS. 3A-3D, in accordance with various embodiments described herein.

[0095] Specifically, the third example perception software analysis scenario 415 includes an AMSV 416 capturing radiation with a stereovision system 417 having a FOV 417a to generate an image of a target 418. As illustrated in FIG. 3C, the lowest point 418a of the target 418 is within the stereovision system 417 FOV 417a, and therefore appears within the generated image of the target 418. Using the vertical position value of the pixel corresponding to the lowest point 418a and the known height 420a of the stereovision system 417 from the water 419 surface, the perception algorithm can estimate the distance 420b to the target 418.

[0096] For example, the perception algorithm may determine the distance 420b to the target 418 in a two- step process. The perception algorithm may first calculate the angle of depression 417b from the stereovision system 417 to the lowest point 418a of the target 418 based on, e.g., inference using the lowest pixel’s vertical position. The perception algorithm may then calculate the distance 420b to the target 418 using the tangent function in combination with the known height 420a of the stereovision system 417 from the water 419 surface.

[0097] In certain embodiments, the perception algorithm may utilize this distance estimate as a comparison with the depth / distance measurement resulting from the stereovision system 417 image captures, as generally defined by equation (1). In this manner, the perception algorithm may reduce the error associated with the depth measurements resulting from the stereovision system 417 image captures at least by checking that the distance measurements resulting from equation (1) do not differ significantly from the distance measurements resulting from the lowest pixel analysis described in reference to FIG. 4C. Moreover, the perception algorithm may utilize multiple other depth / distance measurement techniques to reduce the error associated with the measurements utilizing the stereovision system 417 images and equation (1).

[0098] For example, FIGS. 4D and 4E depict a fourth example perception software analysis scenario 421 and a fifth example perception software analysis scenario 427, respectively, using any of the hardware configurations 300-530 of FIGS. 3A-3D, and in accordance with various embodiments described herein. Generally speaking, the fourth and fifth example perception software analysis scenarios 421 , 427 depict the same AMSV 422 with a stereovision system 423 and a target 424 with a static point 424a at two distinct times. The fourth example perception software analysis scenario 421 may be at a first time (also referenced herein as a first / second / etc. “time instance”) when the AMSV 422 and the target 424 are significantly, vertically aligned, and the fifth example perception software analysis scenario 427 may be at a second time when the AMSV 422 and the target 424 are significantly, vertically misaligned due to the undulations of the water surface 426.

[0099] Thus, in the fourth example perception software analysis scenario 421 , the static point 424a is at a relatively minimal vertical angle 425a relative to the stereovision system 423 FOV central vertical axis 425b. At this point, the stereovision system 423 may capture images of the target 424 that include the static pointAttorney Docket No. 33894-S003 PC(PATENT)424a. At a high level, the static point 424a may include distinctive visual characteristics (e.g., bright colors, high contrast with surrounding portions of the target 424, etc.) and / or otherwise be readily identifiable by the perception algorithm across subsequent image captures of the target 424. This visual and / or otherwise distinctiveness of the static point 424a is crucial because the perception algorithm may utilize the pixel(s) representing this static point 424a in combination with known and / or measurable height differences between subsequent image captures to create a vertical synthetic baseline between the stereovision system 423 image captures at the first time and the image captures at the second time.

[0100] Namely, at the second time (e.g., in scenario 427), the AMSV 422 may have significantly vertically shifted relative to the target 424 due to the undulations of the water surface 426. As illustrated in FIG. 4E, the AMSV 422 may have lowered (e.g., in a trough) relative to the first time while the target 424 may have elevated (e.g., at a wave crest) relative to the first time. Practically speaking, the elevation differences experienced by the AMSV 422 may be equally experienced by the target 424, such that the vertical baseline measurements described in reference to FIGS. 4D and 4E may be negatively impacted by movement of the target within the stereovision system 423 that is not attributable to the vertical movement of the AMSV 422. However, at least at substantial distances from the target 424, the target’s 424 vertical movement may have a negligible impact on the target’s vertical position in image captures relative to the vertical movement of the AMSV 422, so any vertical displacement of the target between subsequent image captures is approximately attributable solely to the vertical movement of the AMSV 422.

[0101] In any event, due to the vertical movement of the AMSV 422 and the target 424, the static point 424a is at a large vertical angle 428 relative to the stereovision system 423 FOV central vertical axis 425b, and the stereovision system 423 may capture images of the target 424 that include the static point 424a at the large vertical angle 428. The perception algorithm may identify the static point 424a in these subsequent image captures (e.g., based on the distinctive visual characteristics and / or other features) and utilize the vertical position value of the pixel(s) corresponding to the static point 424a in combination with measured height differentials to calculate the distance / depth of the target 424 (e.g., using equation (1)). The measured height differentials of the AMSV 422 between the first time and the second time may be the baseline distance value B in equation (1). The perception algorithm may infer the height differential based on the change in vertical angles 425a, 428 and the vertical position value of the pixel(s) corresponding to the static point 424a. Additionally, or alternatively, the perception algorithm may measure the height differential using any suitable sensor or combinations thereof, such as an accelerometer, a gyroscope, a GPS (e.g., GPS communication interface 220a), and / or an inertial measurement unit (IMU).

[0102] FIGS. 4F and 4G depict a sixth example perception software analysis scenario 430 and a seventh example perception software analysis scenario 440, respectively, using any of the hardware configurations 300-530 of FIGS. 3A-3D, and in accordance with various embodiments described herein. Generally speaking, the sixth and seventh example perception software analysis scenarios 430, 440 depict the same stereovision system 432 and a target 434 at two distinct times. The sixth example perception software analysis scenarioAttorney Docket No. 33894-S003 PC(PATENT)430 may be at a first time when the target 434 is at a first distance from the stereovision system 432 and is maintained at an offset 438b from the stereovision system 432 FOV 438 optical axis 438a. The seventh example perception software analysis scenario 440 may be at a second time when the target 434 is at a second distance from the stereovision system 432 and is still maintained at the offset 438b from the stereovision system 432 FOV 438 optical axis 438a.

[0103] As previously mentioned, when tracking or otherwise locating an object, conventional systems maintain the object at / near the center of their FOV. Successive image captures of the target when using these conventional techniques may thus experience changes to the “y” coordinate value as the target moves closer or further from the imaging system, but do not typically experience changes to the “x” coordinate value because the target is maintained in a static, principal position within the FOV.

[0104] By contrast, the present techniques illustrated in FIGS. 4F and 4G maintain the target 434 in an offset position, and thereby cause successive image captures of the target 434 to reflect changes in both coordinate positions (e.g., x and y), as indicated below in equation (2). Because the AMSV maintains the target 434 in a relatively static offset 438b from the optical axis 438a, the changes in at least the “x” coordinate position may be perceived lateral movement resulting from the changes in the “y” coordinate. In other words, as the AMSV moves closer to the target 434 (or vice versa), the target 434 appears to move laterally across the FOV 438 due to the FOV’s 438 conical shape. The perception algorithm may utilize this change in “x” (and “y”) coordinate positions to determine the depth / distance to the target 434 more accurately (e.g., using equation (1)) by leveraging the covariant relationship between the perceived “x” movement and the estimated change in depth / distance. Namely, the covariant relationship may be negative because the perceived “x” movement generally increases as the depth / distance decreases. Additionally, or alternatively, the perception algorithm may utilize trigonometric principles to determine and / or infer the distance to the target 434 based on the perceived angular difference between the first angle 436 and the second angle 448.

[0105] In particular, in the sixth example perception software analysis scenario 430, the stereovision system 432 captures images of the target 434 at the offset 438b and at a first angle 436 relative to the optical axis 438a. In the captured images at the first time, the target 434 may have a first set of x and y coordinates (e.g., “(x,y)”) representing the Cartesian coordinate position of the target 434 in a coordinate plane defined for the captured image. For example, the stereovision system 432 may define a middle pixel(s) of any captured image as the origin or “(0,0),” or may define any of the pixels in an image corner as “(0,0)”.

[0106] In the seventh example perception software analysis scenario 440, the stereovision system 432 captures images of the target 434 at the offset 438b and at a second angle 448 relative to the optical axis 438a. In the captured images at the second time, the target 434 may have moved from the prior location 444 due to movement of the AMSV including the stereovision system 432 and / or of the target 434, but the AMSV may maintain the target 434 at the same offset 438b from the optical axis 438a. In so doing, the images captured by the stereovision system 432 at the second time feature the target 434 at a Cartesian position generally defined asAttorney Docket No. 33894-S003 PC(PATENT)

[0107] (x + A±,y +2) (2),

[0108] where A_1 and A_2 represent the respective differences in the target’s 434 x / y position within the captured image coordinate plane at the second time relative to the target’s 434 x / y position at the first time. These values (A_1 and A_2) may be any suitable positive or negative values, such that the target 434 may be perceived as moving away or towards the optical axis 438a. For example, the AMSV may intentionally maintain the target 434 in the offset 438b position at the first time and may subsequently rotate towards the target 434 to cause the target 434 to appear closer to the optical axis 438a than the offset 438b at the second time. In certain embodiments, the AMSV may maintain the target 434 at a similar offset from the optical axis 438a without the offset being approximately the same as the offset 438b between the first time and the second time.

[0109] Based on the perceived lateral (“x”) movement of the target 434 indicated in equation (2), the perception algorithm may constrain the depth / distance measurements generated in accordance with equation (1) based on the covariant relationship between the two values. As mentioned, the perceived lateral movement may have a negative / invariant relationship with the target 434 depth / distance. The perception algorithm can utilize this relationship to inform or check the depth / distance estimates / measurements using the stereovision system 432 images and equation (1) and thereby ensure that the error associated with the depth / distance estimates is reduced / minimized. In other words, the perception algorithm can utilize this covariant relationship to check that the depth / distance estimate from equation (1) is not significantly different from what would be expected based on the corresponding estimated change in depth / distance from the first time to the second time and the associated, known perceived change in “x” position over the same period (e.g., first time to second time). If the estimated depth / distance value differs significantly from what would be expected based on the covariant relationship, the perception algorithm may adjust the estimated depth / distance value based on the perceived lateral movement.

[0110] In many instances, the target or object of interest may be moving within the AMSV’s FOV, which can further complicate accurate depth / distance measurements. In these scenarios, the perception algorithm may utilize these changes in the target’s position to determine the target’s speed, direction, and / or other quantities to inform the subsequent guidance of the AMSV. For example, FIGS. 4H and 4I depict an eighth example perception software analysis scenario 450 and a ninth example perception software analysis scenario 460, respectively, using any of the hardware configurations 300-330 of FIGS. 3A-3D, and in accordance with various embodiments described herein.

[0111] The eighth example perception software analysis scenario 450 includes an AMSV stereovision system 452 with a target 454 within the FOV 456 at a first time. The ninth example perception software analysis scenario 460 includes the AMSV stereovision system 452 at a second time where the target 454 has moved from the first position 464 to a second position, as indicated by the displacement 466 and the lateral movement angle 468 within the FOV 456. By accounting for the AMSV’s movement in the period between theAttorney Docket No. 33894-S003 PC(PATENT) first time and the second time, the perception algorithm can utilize this change in the target’s 454 position to determine several important quantities about the target 454.

[0112] For example, the perception algorithm may determine a depth / distance value from the target 454 by comparing the first position 464 of the target 454 at the first time with the second position of the target 454 at the second time. Namely, the perception algorithm may infer the depth / distance of the target 454 from the AMSV based on trigonometric principles utilizing the lateral movement angle 468.

[0113] Additionally, or alternatively, the perception algorithm may determine (i) the target’s 454 orientation and / or (ii) the target’s 454 speed based on the movement of the target 454 within the FOV between the first time and the second time. For example, the perception algorithm may determine the target’s 454 speed at least by evaluating the estimated change in position (i.e. , distance traveled by the target 454) and dividing that estimate by the change in time between the first time and the second time. The perception algorithm may also determine / estimate the target’s 454 orientation based on the target's 454 movement in combination with the image analysis and classification / categorization described herein. For example, the perception algorithm may generally determine (via image analysis) that the target 454 is a large vessel oriented towards the right side of the FOV in a three-quarter view, such that the front of the vessel is mostly visible. In this example, the perception algorithm may supplement this image analysis with the detected movement of the vessel (e.g., between the first position 464 and the second position) to confirm that the vessel is oriented in a right-ward direction moving slightly towards and to the right of the AMSV FOV 456.

[0114] Based on any / all of these determinations regarding the target’s position / movement / etc., the perception algorithm may output or otherwise transmit data to the AMSV guidance systems (e.g., AMSV control module 230, locomotion system 218) to adjust the path planning / guidance of the AMSV. In particular, the perception algorithm may output data that causes the AMSV guidance systems to adjust (i) an AMSV orientation and / or (ii) an AMSV speed of the AMSV based on the target 454 orientation or the target 454 speed.

[0115] In general, the baseline distance between image sensors of a stereovision system plays a critical role in the resolution / accuracy of resulting depth / distance measurements, and a larger baseline distance typically yields higher resolution / accuracy. Accordingly, in certain instances, the baseline distance between individual image sensors (e.g., first IR image sensor 302a, second IR image sensor 302b) may be less than optimal to achieve high-resolution depth / distance measurements. To overcome these challenges, the present techniques described in reference to FIGS. 4J and 4K provide another method to create a synthetic baseline that greatly improves the perception algorithm’s ability to provide high accuracy / resolution depth / distance measurements.

[0116] FIGS. 4J and 4K depict a tenth example perception software analysis scenario 470 and an eleventh example perception software analysis scenario 480, respectively, using any of the hardware configurations 300-330 of FIGS. 3A-3D, and in accordance with various embodiments described herein. TheAttorney Docket No. 33894-S003 PC (PATENT) tenth example perception software analysis scenario 470 includes an AMSV 472 with an FOV 474 having an optical axis 474a and a target 476 included in the FOV 474 at a first time. In this scenario 470, the AMSV 472 is on the left side of a central line 478 and the target 476 is on the right side of the central line 478. At this first time, the AMSV 472 may capture images of the target 476 on the right side of the optical axis 474a. Further, the FOV 474 generally represents the composite FOV (e.g., composite FOVs 304C, 314C, 324C, 336) of multiple image sensors operating as part of a stereovision system / camera.

[0117] The eleventh example perception software analysis scenario 480 includes the AMSV 472 having moved from the first position 484 to the right side of the central line 478 along with the target 476 at a second time. This movement of the AMSV 472 is reflected by the lateral displacement 482 of the AMSV 472 from the first time to the second time, which generally represents the synthetic baseline the perception algorithm uses to generate high accuracy / resolution depth / distance measurements of the target 476. At the second time, the AMSV 472 may again capture images of the target 476, which in this example, features the target 476 on the left side of the optical axis 474a.

[0118] Using these two sets of image captures at the first time and the second time, the perception algorithm may determine the depth / distance to the target 476 using equation (1). Namely, the perception algorithm may generate a composite image of the target 476 using the image captures from the individual image sensors at the first time to serve as one of the images captured as part of the synthetic stereovision system having a baseline separation between imagers defined by the lateral displacement 482. The perception algorithm may repeat this process for the images captured at the second time and may thereby have a pair of images representing the target 476 captured at distinct locations separated by the lateral displacement 482 (e.g., the synthetic baseline). The perception algorithm may then account for the movement of the AMSV 472 between the first time and the second time and may then utilize equation (1) with the lateral displacement 482 serving as the baseline value B to generate a depth / distance value for the target 476.

[0119] Additionally, or alternatively, the perception algorithm may utilize any suitable combination of the captured images at the first / second times to calculate the target’s 476 depth / distance. For example, the perception algorithm may utilize the image captured by the left IR image sensor at the first time and the right IR image sensor at the second time to achieve the largest possible synthetic baseline between image captures.

[0120] In certain embodiments, the perception algorithm may analyze data from a group / plurality of AMSVs that are connected via a mesh network (e.g., mesh network 305). A host device, such as one AMSV in the group and / or a mobile control system (e.g., MCS 18) servicing the group of AMSVs, may transmit control instructions to each of the AMSVs of the group to maneuver each of them as illustrated in FIG. 4K. In particular, the host device may cause the group of AMSVs to laterally maneuver in a manner that creates a synthetic baseline between the image sensors of the respective AMSVs, resulting in a synthetic disparity between the images at the two laterally separated locations. Each AMSV may capture radiation using their respective passive sensing systems, and the perception algorithm may detect one or more objects in theAttorney Docket No. 33894-S003 PC(PATENT) image data corresponding to the captured radiation for each AMSV. The host device and / or any individual AMSV may analyze these one or more detected objects and determine and transmit further control instructions to at least a subset of the group of AMSVs to change an orientation, a geospatial location, and / or a speed of any respective AMSV of the subset.

[0121] As previously mentioned, creating a synthetic baseline generally improves the resolution / accuracy of depth / distance measurements resulting from the stereoscopic vision techniques described herein. Thus, because multiple AMSVs of the group of AMSVs are laterally maneuvered to create a respective synthetic baseline when capturing their radiation / image data, the depth estimation and corresponding object detection / identification for each laterally maneuvered AMSV is increased. When these independent high- resolution / accuracy object detections and / or identifications are analyzed in tandem and / or otherwise compared for consistency, these resolution / accuracy improvements are further compounded, as the propagation of errors during this comparative analysis can be significantly lower than when synthetic baselines (and resulting synthetic disparities) are not utilized.

[0122] In some embodiments, each AMSV may maneuver a different lateral distance and / or one or more of the group of AMSVs may laterally maneuver the same distance. In certain embodiments, not all of the group of AMSVs may be maneuvered laterally, such that only a subset of the group of AMSVs create a synthetic baseline for their image sensors.

[0123] In certain embodiments, the AMSV 472 may iteratively / repeatedly perform the lateral movement illustrated in FIG. 4K in a back-and-forth pattern to iteratively / repeatedly determine the target’s 476 depth / distance using the synthetic baseline technique described in reference to FIGS. 4J and 4K.

[0124] Example Computer-Implemented Methods

[0125] FIG. 5A depicts a first flow diagram representing an example computer-implemented method 500, in accordance with various embodiments described herein. The method 500 may be implemented by one or more processors of the AMSV 200, such as the processors 222 executing the LSA module 232 and / or other hardware / software of the AMSV 200 (e.g., passive sensing system 212), for example.

[0126] At a high level, the method 500 represents the target detection / identification process performed by an AMSV with the perception algorithm described herein. Namely, the method 500 includes capturing radiation from an external environment of the AMSV. This radiation may be or include IR radiation that is passively sensed by a passive sensing system (e.g., system 212) of the AMSV, but may be or include any suitable radiation of any suitable wavelength. As an example, the AMSV may include passive sensors configured to sense radiation in near IR, mid IR, far IR, and visible light spectra.

[0127] When the passive sensing system senses / captures the radiation from the external environment and converts the radiation into image data, the method 500 further includes analyzing the captured radiation to detect one or more objects within the data representing the radiation (block 502a). The perceptionAttorney Docket No. 33894-S003 PC (PATENT) algorithm includes instructions to perform object detection within the image data, and may include instructions to utilize any suitable methods, as described herein. For example, the perception algorithm may include instructions that cause the AMSV processors to perform image segmentation, object detection, edge detection, scale invariant feature transforms (SIFT), histogram of oriented gradients (HOG), implementing a convolutional neural network (CNN), and / or any other suitable image processing techniques or combinations thereof to detect objects within the image data. The objects identified within the image data may include any object that is determined to be distinct or otherwise separate from the external / marine environment of the AMSV (e.g., targets, rocks, etc.).

[0128] The method 500 further includes identifying targets based on the detected objects within the image data (block 502b). The image processing techniques described above to detect objects within the image data may also identify the targets from amongst the set of detected objects. For example, the perception algorithm may include instructions that cause the AMSV processors to perform image segmentation on the image data, after which, the pixels corresponding to each object in the image data may be assigned to one or more classes via an applied segmentation mask. These masks contain different labels (e.g., integer values) that correspond to different object classes / categories, and thereby associate the pixels with a known object. The perception algorithm analyzes these outputs of the image segmentation process and can readily identify targets from amongst the detected objects by determining which objects have segmentation masks corresponding with a “target” object class. Of course, in practice, the “target” object class may be labelled in accordance with any suitable target, such as the name / designation of a ship or vessel of interest.

[0129] Of course, it is to be appreciated that the actions of the method 500 may be performed any suitable number of times, and that the actions described in reference to the method 500 may be performed in any suitable order.

[0130] FIG. 5B depicts a second flow diagram representing another example computer-implemented method 510, in accordance with various embodiments described herein. The method 510 may be implemented by one or more processors of the AMSV 200, such as the processors 222 executing the LSA module 232 and / or other hardware / software of the AMSV 200 (e.g., passive sensing system 212), for example.

[0131] The method 510 includes sensing radiation from an external environment of the AMSV using a sensing system that includes at least a stereovision IR camera (block 512). The stereovision IR camera includes (i) a first IR image sensor with a first IR field of view (FOV) having a first optical axis and (ii) a second IR image sensor with a second IR FOV having a second optical axis that is not parallel with the first optical axis. The method 510 further includes applying a perception algorithm to data representing the radiation to detect one or more objects indicated by the data (block 514).

[0132] The method 510 further includes identifying the target within sensed data from the sensing system at (i) a first time instance and (ii) a second time instance that is different from the first time instance (blockAttorney Docket No. 33894-S003 PC(PATENT)516). The method 510 further includes determining a distance value of the target from the AMSV by comparing a first position of the target at the first time instance with a second position of the target at the second time instance (block 518).

[0133] The method 510 further includes determining at least one of (i) a target orientation or (ii) a target speed of the target based on identification of the target at the first time instance and the second time instance (block 520). The method 510 further includes adjusting at least one of: (i) an AMSV orientation or (ii) an AMSV speed of the AMSV based on the target orientation or the target speed (block 522).

[0134] In some aspects, the sensing system further includes at least two electro-optical (EO) image sensors including a first EO image sensor with a first EO FOV and a second EO image sensor with a second EO FOV.

[0135] In some aspects, the method 510 further includes determining that at least one object of the one or more objects indicated by the data represents a target; and orienting the AMSV to offset the target from an optical axis of a sensing system FOV of the sensing system.

[0136] In some aspects, determining that the at least one object represents the target by performing image segmentation on the data.

[0137] In some aspects, performing image segmentation on the data includes determining one or more segmentation masks associated with the one or more objects, and the method 510 further includes identifying a target within the one or more objects based on the one or more segmentation masks.

[0138] In some aspects, the method 510 further includes determining a lowest pixel associated with the target that has a lowest vertical position value of pixels corresponding to the target; and determining a distance value of the target from the AMSV based on (i) depth data derived from a disparity of the sensing system and (ii) a height differential between the lowest vertical position and a vertical position of the sensing system.

[0139] In some aspects, determining the distance value further includes determining, using a stereoscopic distance algorithm, a preliminary distance value based on at least one of the first position or the second position; determining a lateral displacement value of the target based on a perceived lateral movement of the target within the sensing system FOV between the first position and the second position; and adjusting the preliminary distance value to the distance value based on the lateral displacement value.

[0140] In some aspects, the method 510 further includes determining a lateral angular displacement value based on the lateral displacement value, wherein the lateral angular displacement value results from maintaining the offset of the target from the optical axis of the sensing system FOV at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the lateral displacement value further includes: adjusting, based on the lateral angular displacement value, theAttorney Docket No. 33894-S003 PC (PATENT) preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the lateral angular displacement value.

[0141] In some aspects, the method 510 further includes determining a vertical displacement value of the target based on a perceived vertical movement of the target within the sensing system FOV between the first position and the second position; and adjusting a preliminary distance value to the distance value based on the vertical displacement value.

[0142] In some aspects, the method 510 further includes determining a vertical angular displacement value based on the vertical displacement value, wherein the vertical angular displacement value results from water surface oscillations at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the vertical displacement value further includes: adjusting, based on the vertical angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the vertical angular displacement value.

[0143] In some aspects, the covariant relationship between the preliminary distance value and the lateral angular displacement value is a first covariant relationship, the covariant relationship between the preliminary distance value and the vertical angular displacement value is a second covariant relationship, and the method 510 further includes adjusting the preliminary distance value based on (i) the first covariant relationship and (ii) the second covariant relationship.

[0144] In some aspects, the offset is between approximately 2° to approximately 7° from the optical axis of the sensing system FOV. In some aspects, the first IR FOV represents at least 65° of visibility and the second IR FOV represents less than 35° of visibility.

[0145] In some aspects, the sensing system includes at least one monochrome image sensor and at least one multi-color sensor. Generally, removing color filters from a typical color sensor increases the total incident light by up to approximately a factor of five, which significantly improves the imaging resolution, especially at distance and in lower light conditions. Moreover, the techniques of the present disclosure may partially recover chroma information by superimposing the information from other sensors, including lower resolution sensors.

[0146] In some aspects, the at least one monochrome image sensor has a wider FOV than the at least one multi-color sensor; or the at least one monochrome image sensor has a narrower FOV than the at least one multi-color sensor.

[0147] In some aspects, a first edge of the first IR FOV is oriented to be substantially parallel with a second edge of the second IR FOV.

[0148] In some aspects, an overlap point between the first IR FOV and the second IR FOV is less than approximately ten meters from a front surface of the AMSV.Attorney Docket No. 33894-S003 PC(PATENT)

[0149] In some aspects, the method 510 further includes determining a thermal expansion value corresponding to thermal expansion of one or more materials including a support structure of the sensing system; and applying, by the one or more processors, the perception algorithm to (i) the data representing the radiation and (ii) the thermal expansion value to detect the one or more objects indicated by the data.

[0150] In some aspects, the method 510 further includes maneuvering the AMSV between a first lateral position relative to the one or more objects and a second lateral position relative to the one or more objects to create a synthetic baseline for the sensing system; and detecting the one or more objects based on a synthetic disparity resulting from the synthetic baseline.

[0151] In some aspects, the sensing system is a passive sensing system excluding any active sensing system.

[0152] Of course, it is to be appreciated that the actions of the method 510 may be performed any suitable number of times, and that the actions described in reference to the method 510 may be performed in any suitable order.

[0153] FIG. 5C depicts a third flow diagram representing another example computer-implemented method 530, in accordance with various embodiments described herein. The method 530 may be implemented by one or more processors of the AMSV 200, such as the processors 222 executing the LSA module 232 and / or other hardware / software of the AMSV 200 (e.g., passive sensing system 212), for example.

[0154] The method 530 includes sensing radiation from an external environment of the AMSV using a sensing system (block 532). The method 530 further includes applying a perception algorithm to data representing the radiation to detect one or more objects indicated by the data (block 534). The method 530 further includes determining that at least one object of the one or more objects indicated by the data represents a target (block 536). The method 530 further includes orienting the AMSV to offset the target from an optical axis of a sensing system field of view (FOV) of the sensing system (block 538).

[0155] In some aspects, the sensing system includes at least a stereovision IR camera with (i) a first IR image sensor with a first IR FOV and (ii) a second IR image sensor with a second IR FOV.

[0156] In some aspects, the first IR FOV has a first optical axis and the second IR FOV has a second optical axis that is not parallel with the first optical axis.

[0157] In some aspects, the sensing system includes at least two electro-optical (EO) image sensors including a first EO image sensor with a first EO FOV and a second EO image sensor with a second EO FOV.

[0158] In some aspects, the method 530 further includes determining that the at least one object represents the target by performing image segmentation on the data.Attorney Docket No. 33894-S003 PC(PATENT)

[0159] In some aspects, performing image segmentation on the data includes determining one or more segmentation masks associated with the one or more objects, and wherein the perception method further includes: identifying a target within the one or more objects based on the one or more segmentation masks.

[0160] In some aspects, the method 530 further includes determining a lowest pixel associated with the target that has a lowest vertical position value of pixels corresponding to the target; and determining a distance value of the target from the AMSV based on (i) depth data derived from a disparity of the sensing system and (ii) a height differential between the lowest vertical position and a vertical position of the sensing system.

[0161] In some aspects, the method 530 further includes identifying the target within sensed data from the sensing system at (i) a first time instance and (ii) a second time instance that is different from the first time instance; and determining a distance value of the target from the AMSV by comparing a first position of the target at the first time instance with a second position of the target at the second time instance.

[0162] In some aspects, the method 530 further includes determining at least one of (i) a target orientation or (ii) a target speed of the target based on identification of the target at the first time instance and the second time instance.

[0163] In some aspects, the method 530 further includes adjusting at least one of: (i) an AMSV orientation or (ii) an AMSV speed of the AMSV based on the target orientation or the target speed.

[0164] In some aspects, the method 530 further includes determining the distance value by determining, using a stereoscopic distance algorithm, a preliminary distance value based on at least one of the first position or the second position; determining a lateral displacement value of the target based on a perceived lateral movement of the target within the sensing system FOV between the first position and the second position; and adjusting the preliminary distance value to the distance value based on the lateral displacement value.

[0165] In some aspects, the method 530 further includes determining a lateral angular displacement value based on the lateral displacement value, wherein the lateral angular displacement value results from maintaining the offset of the target from the optical axis of the sensing system FOV at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the lateral displacement value further includes: adjusting, based on the lateral angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the lateral angular displacement value.

[0166] In some aspects, the method 530 further includes determining a vertical displacement value of the target based on a perceived vertical movement of the target within the sensing system FOV between the first position and the second position; and adjusting a preliminary distance value to the distance value based on the vertical displacement value.Attorney Docket No. 33894-S003 PC(PATENT)

[0167] In some aspects, the method 530 further includes determining a vertical angular displacement value based on the vertical displacement value, wherein the vertical angular displacement value results from water surface oscillations at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the vertical displacement value further includes: adjusting, based on the vertical angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the vertical angular displacement value.

[0168] In some aspects, the covariant relationship between the preliminary distance value and the lateral angular displacement value is a first covariant relationship, the covariant relationship between the preliminary distance value and the vertical angular displacement value is a second covariant relationship, and the method 530 further includes adjusting the preliminary distance value based on (i) the first covariant relationship and (ii) the second covariant relationship.

[0169] In some aspects, the offset is between approximately 2° to approximately 7° from the optical axis of the sensing system FOV. In some aspects, the first IR FOV represents at least 65° of visibility and the second IR FOV represents less than 35° of visibility. In some aspects, the sensing system includes at least one monochrome image sensor and at least one multi-color sensor.

[0170] In some aspects, the at least one monochrome image sensor has a wider FOV than the at least one multi-color sensor; or the at least one monochrome image sensor has a narrower FOV than the at least one multi-color sensor.

[0171] In some aspects, one of the first IR FOV or the second IR FOV is oriented to have an optical axis that is angularly offset from an orientation of the AMSV. In some aspects, a first orientation of the first IR FOV is different than a second orientation of the second IR FOV.

[0172] In some aspects, a first edge of the first IR FOV is oriented to be substantially parallel with a second edge of the second IR FOV. In some aspects, an overlap point between the first IR FOV and the second IR FOV is less than approximately ten meters from a front surface of the AMSV.

[0173] In some aspects, the method 530 further includes determining a thermal expansion value corresponding to thermal expansion of one or more materials including a support structure of the sensing system; and applying, by the one or more processors, the perception algorithm to (i) the data representing the radiation and (ii) the thermal expansion value to detect the one or more objects indicated by the data.

[0174] In some aspects, the method 530 further includes maneuvering the AMSV between a first lateral position relative to the one or more objects and a second lateral position relative to the one or more objects to create a synthetic baseline for the sensing system; and detecting the one or more objects based on a synthetic disparity resulting from the synthetic baseline.

[0175] In some aspects, the sensing system is a passive sensing system excluding any active sensing system.Attorney Docket No. 33894-S003 PC (PATENT)

[0176] Of course, it is to be appreciated that the actions of the method 530 may be performed any suitable number of times, and that the actions described in reference to the method 530 may be performed in any suitable order.

[0177] FIG. 5D depicts a fourth flow diagram representing another example computer-implemented method 540, in accordance with various embodiments described herein. The method 540 may be implemented by one or more processors of the AMSV 200, such as the processors 222 executing the LSA module 232 and / or other hardware / software of the AMSV 200 (e.g., passive sensing system 212), for example.

[0178] The method 540 includes applying a perception algorithm to data representing radiation sensed by a sensing system from an external environment of the AMSV to detect one or more objects indicated by the data, the sensing system being a passive sensing system excluding any active sensing system (block 542). The method 540 further includes determining that at least one object of the one or more objects indicated by the data represents a target object (block 544).

[0179] The method 540 further includes, based on determining that the at least one object represents the target object, determining an AMSV path plan configured to cause the AMSV to intercept the target object (block 546). The method 540 further includes causing the AMSV to maneuver in accordance with the AMSV path plan and intercept the target object (block 548).

[0180] In some embodiments, the method 540 further includes determining, by the one or more processors, respective AMSV path plans for a plurality of AMSVs located at a plurality of different locations relative to the target object based on radiation sensed by respective sensing systems of each AMSV of the plurality of AMSVs, wherein each respective sensing system is a passive sensing system excluding any active sensing system; and causing, by the one or more processors, each AMSV of the plurality of AMSVs to maneuver in accordance with the respective AMSV path plans to intercept the target object.

[0181] In some embodiments, causing the AMSV to maneuver in accordance with the AMSV path plan further comprises: deactivating, by the one or more processors, all radio transceivers on-board the AMSV prior to intercepting the target object.

[0182] In some embodiments, the sensing system includes a stereovision infrared (I R) camera with (i) a first IR image sensor with a first IR field of view (FOV) having a first optical axis and (ii) a second IR image sensor with a second IR FOV having a second optical axis that is not parallel with the first optical axis, the stereovision IR camera configured to sense radiation from the external environment of the AMSV.

[0183] In some embodiments, the first optical axis is angled towards the second optical axis, and the second optical axis is angled towards the first optical axis, thereby creating a reduced blind spot near a front portion of the AMSV.Attorney Docket No. 33894-S003 PC(PATENT)

[0184] In some embodiments, the sensing system includes at least two electro-optical (EO) image sensors including a first EO image sensor with a first EO FOV and a second EO image sensor with a second EO FOV.

[0185] In some embodiments, the method 540 further includes orienting, by the one or more processors, the AMSV to offset the target object from an optical axis of a sensing system FOV of the sensing system.

[0186] In some embodiments, determining that the at least one object represents the target object by performing image segmentation on the data.

[0187] In some embodiments, performing image segmentation on the data includes determining one or more segmentation masks associated with the one or more objects, and wherein the method 540 further includes identifying, by the one or more processors, the target object within the one or more objects based on the one or more segmentation masks.

[0188] In some embodiments, the method 540 further includes determining, by the one or more processors, a lowest pixel associated with the target object that has a lowest vertical position value of pixels corresponding to the target object; and determining, by the one or more processors, a distance value of the target object from the AMSV based on (i) depth data derived from a disparity of the sensing system and (ii) a height differential between the lowest vertical position and a vertical position of the sensing system.

[0189] In some embodiments, the method 540 further includes identifying, by the one or more processors, the target object within sensed data from the sensing system at (i) a first time instance and (ii) a second time instance that is different from the first time instance; and determining, by the one or more processors, a distance value of the target object from the AMSV by comparing a first position of the target object at the first time instance with a second position of the target object at the second time instance.

[0190] In some embodiments, the method 540 further includes determining, by the one or more processors, at least one of (i) a target orientation or (ii) a target speed of the target object based on identification of the target object at the first time instance and the second time instance.

[0191] In some embodiments, the method 540 further includes adjusting, by the one or more processors, at least one of: (i) an AMSV orientation, (ii) an AMSV geospatial location, or (iii) an AMSV speed of the AMSV based on the target orientation or the target speed.

[0192] In some embodiments, determining the distance value further includes determining, using a stereoscopic distance algorithm, a preliminary distance value based on at least one of the first position or the second position; determining, by the one or more processors, a lateral displacement value of the target object based on a perceived lateral movement of the target object within the sensing system FOV between the first position and the second position; and adjusting, by the one or more processors, the preliminary distance value to the distance value based on the lateral displacement value.Attorney Docket No. 33894-S003 PC(PATENT)

[0193] In some embodiments, the method 540 further includes determining, by the one or more processors, a lateral angular displacement value based on the lateral displacement value, wherein the lateral angular displacement value results from maintaining the offset of the target object from the optical axis of the sensing system FOV at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the lateral displacement value further includes adjusting, based on the lateral angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the lateral angular displacement value.

[0194] In some embodiments, the method 540 further includes determining, by the one or more processors, a vertical displacement value of the target object based on a perceived vertical movement of the target object within the sensing system FOV between the first position and the second position; and adjusting, by the one or more processors, a preliminary distance value to the distance value based on the vertical displacement value.

[0195] In some embodiments, the method 540 further includes determining, by the one or more processors, a vertical angular displacement value based on the vertical displacement value, wherein the vertical angular displacement value results from water surface oscillations at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the vertical displacement value further includes adjusting, based on the vertical angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the vertical angular displacement value.

[0196] In some embodiments, the covariant relationship between the preliminary distance value and the lateral angular displacement value is a first covariant relationship, the covariant relationship between the preliminary distance value and the vertical angular displacement value is a second covariant relationship, and wherein the method 540 further includes adjusting, by the one or more processors, the preliminary distance value based on (i) the first covariant relationship and (ii) the second covariant relationship.

[0197] In some embodiments, the offset is between approximately 2° to approximately 7° from the optical axis of the sensing system FOV.

[0198] In some embodiments, the first IR FOV represents at least 65° of visibility and the second IR FOV represents less than 35° of visibility.

[0199] In some embodiments, the sensing system includes at least one monochrome image sensor and at least one multi-color sensor.

[0200] In some embodiments, the method 540 further includes the at least one monochrome image sensor has a wider FOV than the at least one multi-color sensor; or the at least one monochrome image sensor has a narrower FOV than the at least one multi-color sensor.Attorney Docket No. 33894-S003 PC (PATENT)

[0201] In some embodiments, a first edge of the first IR FOV is oriented to be substantially parallel with a second edge of the second IR FOV.

[0202] In some embodiments, an overlap point between the first IR FOV and the second IR FOV is less than approximately ten meters from a front surface of the AMSV.

[0203] In some embodiments, the method 540 further includes determining, by the one or more processors, a thermal expansion value corresponding to thermal expansion of one or more materials comprising a support structure of the sensing system; and applying, by the one or more processors, the perception algorithm to (i) the data representing the radiation and (ii) the thermal expansion value to detect the one or more objects indicated by the data.

[0204] In some embodiments, the method 540 further includes maneuvering the AMSV between a first lateral position relative to the one or more objects and a second lateral position relative to the one or more objects to create a synthetic baseline for the sensing system; and detecting, by the one or more processors, the one or more objects based on a synthetic disparity resulting from the synthetic baseline.

[0205] FIG. 5E depicts a fifth flow diagram representing another example computer-implemented method 550, in accordance with various embodiments described herein. The method 550 may be implemented by one or more processors of the AMSV 200, such as the processors 222 executing the LSA module 232 and / or other hardware / software of the AMSV 200 (e.g., passive sensing system 212), for example.

[0206] The method 550 includes maneuvering one or more AMSVs of the group of AMSVs between a respective first lateral position relative to one or more objects located within a group FOV and a respective second lateral position relative to the one or more objects to create a respective synthetic baseline for the one or more AMSVs (block 552). The group FOV may comprise respective FOVs of each AMSV in the group of AMSVs. The method 550 further includes receiving respective indications of radiation sensed by respective sensing systems of the one or more AMSVs (block 554). The method 550 further includes applying a perception algorithm to data representing the respective indications of radiation to detect the one or more objects indicated by the data based on a respective synthetic disparity resulting from the respective synthetic baseline for the one or more AMSVs, thereby optimizing detection of the one or more objects (block 556).

[0207] The method 550 further includes, based on detecting the one or more objects, determining a respective control instruction for at least a subset of the group of AMSVs, the respective control instruction indicating a respective change to at least one of: (i) a respective orientation, (ii) a respective geospatial location, or (iii) a respective speed of each AMSV comprising the subset of the group of AMSVs (block 558). The method 550 further includes transmitting the respective control instruction to the each AMSV of the subset (block 560).

[0208] In some embodiments, the one or more AMSVs comprises a plurality of AMSVs.Attorney Docket No. 33894-S003 PC(PATENT)

[0209] In some embodiments, each respective sensing system includes a stereovision infrared (IR) camera with (i) a first IR image sensor with a first IR field of view (FOV) having a first optical axis and (ii) a second IR image sensor with a second IR FOV having a second optical axis that is not parallel with the first optical axis, the stereovision IR camera configured to sense radiation from a respective external environment of the one or more AMSVs.

[0210] In some embodiments, the first optical axis is angled towards the second optical axis, and the second optical axis is angled towards the first optical axis, thereby creating a reduced blind spot near a front portion of the one or more AMSVs.

[0211] In some embodiments, the method 550 further includes determining, by the one or more processors, respective AMSV path plans for a plurality of AMSVs located at a plurality of different locations relative to a target object of the one or more objects based on radiation sensed by respective sensing systems of each AMSV of the plurality of AMSVs, wherein each respective sensing system is a passive sensing system excluding any active sensing system; and causing, by the one or more processors, each AMSV of the plurality of AMSVs to maneuver in accordance with the respective AMSV path plans to intercept the target object.

[0212] In some embodiments, causing each AMSV to maneuver in accordance with the respective AMSV path plans further includes deactivating all radio transceivers on-board the each AMSVs prior to intercepting the target object.

[0213] In some embodiments, one or more sensing systems of the respective sensing systems include at least two electro-optical (EO) image sensors including a first EO image sensor with a first EO FOV and a second EO image sensor with a second EO FOV.

[0214] In some embodiments, the method 550 further includes orienting at least one AMSV to offset a target object of the one or more objects from an optical axis of a sensing system FOV of the respective sensing system of the at least one AMSV.

[0215] In some embodiments, the method 550 further includes determining, by the one or more processors, that at least one object represents a target object by performing image segmentation on the data.

[0216] In some embodiments, wherein performing image segmentation on the data includes determining one or more segmentation masks associated with the one or more objects, and wherein the method 550 further includes identifying, by the one or more processors, the target object within the one or more objects based on the one or more segmentation masks.

[0217] In some embodiments, the method 550 further includes determining, by the one or more processors, a lowest pixel associated with the target object that has a lowest vertical position value of pixels corresponding to the target object; and determining, by the one or more processors, a distance value of the target object from each of the respective AMSVs based on (i) depth data derived from the respective syntheticAttorney Docket No. 33894-S003 PC(PATENT) disparity of the respective sensing systems and (ii) a height differential between the lowest vertical position and a vertical position of the respective sensing systems.

[0218] In some embodiments, the method 550 further includes identifying, by the one or more processors, the target object within sensed data from the respective sensing systems at (i) a first time instance and (ii) a second time instance that is different from the first time instance; and determining, by the one or more processors, a distance value of the target object from each of the respective AMSVs by comparing a first position of the target object at the first time instance with a second position of the target object at the second time instance.

[0219] In some embodiments, the method 550 further includes determining, by the one or more processors, at least one of (i) a target orientation or (ii) a target speed of the target object based on identification of the target object at the first time instance and the second time instance.

[0220] In some embodiments, the method 550 further includes adjusting at least one of: (i) an AMSV orientation, (ii) an AMSV geospatial location, or (iii) an AMSV speed of the one or more AMSVs based on the target orientation or the target speed.

[0221] In some embodiments, determining the distance value further includes determining, using a stereoscopic distance algorithm, a preliminary distance value based on at least one of the first position or the second position; determining, by the one or more processors, a lateral displacement value of the target object based on a perceived lateral movement of the target object within the sensing system FOV between the first position and the second position; and adjusting, by the one or more processors, the preliminary distance value to the distance value based on the lateral displacement value.

[0222] In some embodiments, the method 550 further includes determining, by the one or more processors, a lateral angular displacement value based on the lateral displacement value, wherein the lateral angular displacement value results from maintaining the offset of the target object from the optical axis of the sensing system FOV at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the lateral displacement value further includes adjusting, based on the lateral angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the lateral angular displacement value.

[0223] In some embodiments, the method 550 further includes determining, by the one or more processors, a vertical displacement value of the target object based on a perceived vertical movement of the target object within the sensing system FOV between the first position and the second position; and adjusting, by the one or more processors, a preliminary distance value to the distance value based on the vertical displacement value.

[0224] In some embodiments, the method 550 further includes determining, by the one or more processors, a vertical angular displacement value based on the vertical displacement value, wherein theAttorney Docket No. 33894-S003 PC(PATENT) vertical angular displacement value results from water surface oscillations at the first time instance and the second time instance; and wherein adjusting the preliminary distance value based on the vertical displacement value further includes adjusting, based on the vertical angular displacement value, the preliminary distance value in accordance with a covariant relationship between the preliminary distance value and the vertical angular displacement value.

[0225] In some embodiments, the covariant relationship between the preliminary distance value and the lateral angular displacement value is a first covariant relationship, the covariant relationship between the preliminary distance value and the vertical angular displacement value is a second covariant relationship, and the method 550 further includes adjusting, by the one or more processors, the preliminary distance value based on (i) the first covariant relationship and (ii) the second covariant relationship.

[0226] In some embodiments, the offset is between approximately 2° to approximately 7° from the optical axis of the sensing system FOV.

[0227] In some embodiments, the first IR FOV represents at least 65° of visibility and the second IR FOV represents less than 35° of visibility.

[0228] In some embodiments, the one or more of the respective sensing systems include at least one monochrome image sensor and at least one multi-color sensor.

[0229] In some embodiments, the method 550 further includes the at least one monochrome image sensor has a wider FOV than the at least one multi-color sensor; or the at least one monochrome image sensor has a narrower FOV than the at least one multi-color sensor.

[0230] In some embodiments, a first edge of the first IR FOV is oriented to be substantially parallel with a second edge of the second IR FOV.

[0231] In some embodiments, an overlap point between the first IR FOV and the second IR FOV is less than approximately ten meters from a front surface of the AMSV.

[0232] In some embodiments, the method 550 further includes determining, by the one or more processors, a thermal expansion value corresponding to thermal expansion of one or more materials comprising a support structure of one or more of the respective sensing systems; and applying, by the one or more processors, the perception algorithm to (i) the data representing the radiation and (ii) the thermal expansion value to detect the one or more objects indicated by the data.

[0233] Example Vision Perception Optimization System

[0234] FIG. 6A is a block diagram of an example vision perception optimization system 600 which may be included in an autonomous maritime surface vehicle, such as the ASMV 100 of FIGS. 1 A-1 E and / or the AMSV 200 of FIG. 2, where the AMSV is located in or disposed on a body of water. For example, the vision perception optimization system 600 may be included in the vision system 130 shown in FIGS. 1A-1 E, theAttorney Docket No. 33894-S003 PC (PATENT) passive remote sensing system 212 and / or the AMSV control system 215 of FIG. 2, and / or in the perception hardware configurations 300, 310, 320, 330 of FIGS. 3A-3D. For ease of discussion, and not for limitation purposes, FIG. 6A is discussed below with simultaneous reference to FIGS. 1A-1 E, FIG. 2, FIGS. 3A-3D, FIGS. 4A-4K, and FIGS. 5A-5E.

[0235] In an embodiment, the vision perception optimization system 600 may include a set of computerexecutable instructions that are stored on one or more memories and executable by one or more processors to perform one or more of the methods and / or techniques disclosed herein. For example, the vision perception optimization system 600 can be stored on the memories 225 of the AMSV control system 215 and be executable by the processors 222 of the AMSV control system 215 and, in some implementations, at least part of the vision perception optimization system 600 and at least part of one or more of the modules 230, 232, 235 of the AMSV control system 215 may operate as an integral module. Additionally or alternatively, the vision perception optimization system 600 can be stored on one or more memories included in the passive remote sensing system 212 (not shown) and be executable by one or more processors of the passive remote sensing system 212 (also not shown), and / or the vision perception optimization system 600 can be stored on one or more memories and be executable by one or more processors elsewhere on the AMSV 200. In some implementations, the vision perception optimization system 600 may be implemented at least partially using firmware and / or at least partially using hardware.

[0236] As depicted in FIG. 6A, the vision perception optimization system 600 includes an image data optimizer 602. In an embodiment, the image data optimizer 602 may be the image data optimizer 260 shown in FIG. 2. FIG. 6A depicts the image data optimizer 602 as including an entropy detector 605 and a cropper 608, and it is noted that even though FIG. 6A depicts the entropy detector 602 and the cropper 605 as being separate and distinct modules, this is only for clarity of discussion. For example, the entropy detector 602 and the cropper 605 may be implemented as an integral module or unit, if desired.

[0237] Generally speaking, the image data optimizer 602 optimizes sets of image data (also referred to interchangeably herein as images, raw image data, FOV data, FOV sensed data, or FOV image data) generated by a passive remote sensing system 130, 212, 300, 310, 320, 330 of an AMSV 100, 200, and provides the optimized sets of image data to one or more vision perceptors or vision perception systems 610 on-board the AMSV 100, 200 for (further) image processing. Each of the vision perceptors 610 may execute one or more computer-vision algorithms to thereby allow computers and other computer modules (e.g., the AMSV control system 215) to comprehend, interact with, and / or act upon the image data provided by the passive remote sensing system 212. For example, the vision perceptors 610 may provide indications of the presence and location of various objects within the FOV of the capturing sensor or camera. In embodiments, one or more of the vision perceptors 610 may perform and / or provide data and information as discussed for one or more of the analyses scenarios 400, 410, 415, 421 , 427, 430, 440, 450, 460, 470, or 480 of FIGS. 4A- 4K. In embodiments, one or more of the vision perceptors 610 may perform one or more of the methods 500, 510, 530, 540, or 550 of FIGS. 5A-5E.Attorney Docket No. 33894-S003 PC (PATENT)

[0238] At any rate, examples of computer-vision algorithms which may be utilized by the vision perceptors 610 may include stereo vision, optical flow, dense matching, sparks matching, SIFT (Scale-Invariant Feature Transform) and / or other types of feature matching, template matching, filtering, CLIP (Contrastive Language- Image Pre-Training), and / or other types of computer vision techniques which are currently known or envisioned. Outputs of the vision preceptors 610 may be provided to other algorithms, modules, devices, and / or systems on-board the AMSV 200 for making decisions and executing actions of the AMSV 200 based on the received outputs. For example, the AMSV control module 230 may control the locomotion, navigation, communications, and / or other types of operations of the AMSV 200 (and in some situations, of other AMSVs operating in conjunction with the AMSV 200, e.g., during a coordinated mission) based at least partially on outputs of the vision perceptors 610. One or more vision perceptors 610 may be included in the passive remote sensing system 212 and / or the LSA module 232 of the AMSV control system 215, and / or one or more vision perceptors 610 may be implemented as respective stand-alone modules, devices, or systems on-board the AMSV 200 which are separate and distinct from either the passive remote sensing system 212 or the AMSV control system 215.

[0239] In the example use case illustrated in FIG. 6A, the image data optimizer 602 of the system 600 operates on a first set of image data A captured by a first camera or sensor on-board the AMSV 200 to generate an optimized first set of image data A’, and the image data optimizer 602 operates on a second set of image data B to generate an optimized second set of image data B’. The second set of image data B may be (or may have been) captured by the first camera or sensor (e.g., at a time different than the time at which image data A was captured), or the second set of image data B may be (or may have been) captured by a second camera or sensor which is on-board the AMSV 200 and is stereoscopically paired with the first camera or sensor. For instance, the first camera or sensor may be the image sensor 242a and the second camera or sensor may be the image sensor 242b. The image data optimizer 602 may operate sequentially and / or in parallel on multiple sets of image data A, B, and the image data optimizer 602 may provide the optimized sets of image data A’, B’ to one or more vision perceptors 610 for (further) image processing.

[0240] The systems, devices, and techniques disclosed herein optimize the image data sets generated by the passive remote sensing system 212 to thereby enable downstream vision perceptors 610 to process (e.g., image process) the image data sets more quickly and efficiently than is able to be done by current computervision techniques. Typically, current computer-vision techniques (for example, as utilized in domains such as autonomous car and trucks, robotics, medical imaging, and the like) operate on a single image or image data set in its entirety.

[0241] However, images or image data sets which are captured in maritime domains or environments (such as in or on bodies of water in which the AMSV 200 may operate) typically include much larger quantities of random noise as compared to images which are captured in non-maritime environments, due to the presence of waves, wave and wind actions on the AMSV 200, spray, water droplets on the sensor or camera, and other dynamic characteristics of maritime environments. Further, the random noise included inAttorney Docket No. 33894-S003 PC(PATENT) maritime environment images is likely to have high entropy (or, said another way, less fidelity) and, as such, computer-vision techniques applied to maritime environment images are more likely to utilize more computing resources and take longer to execute as compared to non-maritime environment images. As known to one skilled in the art, an entropy of an image is indicative of a degree, level, or measure (e.g., a statistical measure) of randomness or disorder of pixel values of the image. As such, an image having higher entropy or less fidelity generally indicates a more diverse and detailed image with a wider range of pixel values, which can reflect the higher level of detail, texture, and variations in the pixel values. On the other hand, an image having lower entropy or more fidelity (e.g., less random noise) can indicate a simpler, more uniform image. Consequently, because images captured in maritime environments typically include much larger quantities of random noise due to at least the aforementioned causes specific to maritime environments, to process such noisy images, computer-vision techniques must utilize large quantities of resources, e.g., computer resources such as memory and processing cycles as well as other types of resources such as time. However, as energy sources on-board the AMSV 200 (e.g., fuel, batteries, etc.) which power computer operations on-board the AMSV 200 (and indeed, power the entirety of operations of the AMSV 200) are limited, it is desirable to conserve as much energy as possible during AMSV 200 operations. Further, as an AMSV 200 may be operating at high speeds and, in some situations, in coordination with the movements of other AMSVs and / or target objects, it is also desirable to control AMSV 200 movements and operations (e.g., speed, orientation, navigation, among others) in as quickly and responsively as possible to allow for more granularity and accuracy of AMSV movements and behaviors as well as in the timing of said movements and behaviors.

[0242] As discussed above, the systems, devices, and techniques disclosed herein optimize the input or raw FOV images or image data sets generated by the passive remote sensing system 212 to thereby enable downstream vision perceptors 610 to process (e.g., image process) the optimized, FOV images more quickly and efficiently, e.g., with more speed and by using less computer resources. Thus, by using the systems, devices, and techniques disclosed herein, computer resources (e.g., memory, processing cycles, etc.) are conserved, and thereby energy usage on-board the AMSV 200 is also conserved. Further, the timeliness of AMSV 200 operations which are based on the content of the FOV images is not compromised and, indeed, may be improved, as overall image processing time of the vision perceptors 610 may decrease due to the optimized data sets provided to the vision perceptors 610.

[0243] FIGS. 6B-6E are reproductions of images which were obtained during the development and testing of at least some of the techniques disclosed herein, such as the example vision perception optimization system 600 of FIG. 6A and / or other techniques disclosed herein with respect to FIGS. 1 A-1 E, FIG. 2, FIGS. 3A-3D, FIGS. 4A-4K, and FIGS. 5A-5E.

[0244] FIG. 6B is a reproduction of an image 620a obtained by a first image sensor disposed on an AMSV, and FIG. 6C is a reproduction of an image 620b obtained at a by a second image sensor disposed on the AMSV. The two images 620a, 620b were obtained at a similar or same time. The first image sensor was disposed on the right side (e.g., starboard side) of the AMSV, and the second image sensor was disposed onAttorney Docket No. 33894-S003 PC(PATENT) the left side (e.g., port side) of the AMSV. As such, the two image sensors via which the images 620a, 620b were obtained are examples of the image sensors 242a, 242b disposed on the AMSV 200 of FIG. 2. In both images 620a, 620b, the bow 625 of the AMSV is visible, and an object of interest 628 (e.g., another maritime vehicle) is also visible.

[0245] FIG. 6D is a reproduction of a disparity map 630 illustrating disparities or differences between the pair of images 620a, 620b when image processed by using traditional, known image processing techniques. Each of the images 620a, 620b was processed and compared to the other image in their entireties using traditional image processing techniques, and the differences or disparities between the two images 620a, 620b are depicted by the disparity map 630. Specifically, the various colors included in the disparity map 630 (which are represented in FIG. 6D by different greyscale intensities of the depicted pixels and of the differently-sized groups of pixels) represent different levels or measures of disparities between pixels of the two images. As can be easily seen in FIG. 6D, it is difficult, if not impossible, to visually ascertain any overlap between the two images, and in particular, to visually ascertain the location of object of interest 628 within the disparity map 630, at least because the backgrounds of the two images 620a, 620b are extremely noisy due to the characteristics of the waves, water, and sky.

[0246] FIG. 6E is reproduction of a disparity map 640 which was generated by using at least some of the optimization techniques described in this disclosure on the pair of images 620a, 620b. For example, the image data optimizer 602 may have operated on the image 620a to generate an optimized image 620a’ and the image data optimizer 602 may have operated on the image 620b to generate an optimized image 620b’. The differences or disparities between the two optimized images 620a’ and 620b’ are depicted by the disparity map 640. As shown in FIG. 6E, the noisy water and sky backgrounds have been cropped or filtered out by the optimizer 602, and the remaining areas 645, 648 of relatively constant entropy are clearly ascertained and depicted in the disparity map 640. Specifically, the sub-area 645 is indicative of the location of the bow 625 of the AMSV within the images 620a’, 620b’, and the sub-area 648 is indicative of the location of the object 628 within the images 620a’, 620b’. Thus, as indicated by the disparity map 640 (and in particular, in comparison with the disparity map 630), any downstream vision perceptors 610 are able to process the optimized images 620a’, 620b’ much more quickly than the unoptimized images 620a, 620b. Indeed, during testing, image processing of the optimized images 620a’, 620b’ was measured as being at least 50% faster than image processing of the non-optimized images 620a, 620b. Accordingly, by using the techniques disclosed herein, the responsiveness of the AMSV to sensed image data can be quicker, and accordingly the AMSV can be controlled (e.g., in its movements, responsiveness to unexpected events, and the like) in a quicker and more granular and accurate manner. Moreover, on-board resources utilized for image processing (e.g., processor cycles and energy usage) are decreased when compared to those utilized by traditional image processing techniques, and consequently the techniques disclosed herein can conserve processor cycles, energy usage, and other types of resources which are limited on-board the AMSV.Attorney Docket No. 33894-S003 PC(PATENT)

[0247] Now turning back to the entropy detector 605 and the cropper 608 of FIG. 6A in more detail, the input image data sets A, B may be (or may have been) generated by the passive remote sensing system 212, and thus each input image data set A, B may include a data set or image indicative of captured electromagnetic radiation within a FOV of a respective sensor or camera of the passive remote sensing system 212. The entropy detector 605 operates on a set of image data, say, image data set A, to detect one or more sub-areas of the image data set A which have respective entropies less than the entropies of at least one other sub-area of the image data set A. For example, if the image data set A represents a FOV image which depicts mostly ocean and sky and a single maritime vessel in the distance, the pixels corresponding to the single maritime vessel would have less entropy than the pixels corresponding to the ocean and the sky. In another example, the image data set A represents an FOV image which depicts mostly ocean and sky and multiple maritime vessels. Each set of pixels corresponding to respective vessels may differ in entropy, however, generally speaking, each of the sets the pixels corresponding to the multiple maritime vessels would respectively have less entropy than the pixels corresponding to the ocean and sky. At any rate, the entropy detector 605 may detect one or more sub-areas of the image data set A having less entropy than at least one other sub-area, and / or having entropy below an entropy threshold. The entropy detector 605 may use one or more suitable techniques to detect the sub-areas of lower entropy, including (but not limited to) edge detection, color gradient, heat map, neural network, etc. Said another way, the entropy detector 605 may utilize one or more computer-vision detection techniques to filter out higher entropy sub-areas from one or more lower entropy sub-areas of FOV images.

[0248] Subsequently, the cropper 608 crops the image data set A based on the detected sub-areas of lower entropy. For example, the cropper 608 may crop the image data set A into a cropped image A’ in which multiple (or all) of the sub-areas of the image A having lesser entropies are included, or the cropper 608 may crop the image data set A into multiple cropped images A’, A”, A’”, etc., each of which includes only a single cropped sub-area of the image data set A. Thus, the optimized image data set A’ output by the cropper 608 includes only the sub-areas of the image data set A having lesser entropy (e.g., as compared to at least one or a majority of other sub-areas of the original image data set A, or entropy less than a predetermined threshold). Thus, not only is the output A’ of the cropper 608 and of the image data optimizer 602 a smaller data set, but the output A’ is also a data set which is more uniform and has higher fidelity than the original image data set A. Accordingly, the downstream vision perceptors 610 are able to process the optimized image data set A’ more quickly and efficiently to detect or obtain any information contained therein which the AMSV 200 may utilize to control its operations, movements, and / or behaviors. Said another way, image data optimizer 602 optimizes FOV image data generated by the passive remote sensing systems 212 on-board the AMSV 200 so that downstream vision perceptors 610 operate on only image data indicative of areas of interest to the AMSV 200, such as detected objects disposed in and / or associated with the maritime environment (e.g., other vessels, buoys, stations, coastlines, islands, landmarks, and the like). Thus, in a sense, the image data optimizer 602 may operate as a filter on provided image data sets A, B so that only image data indicative of areas of interest to the AMSV 200 A’, B’ are provided to the vision perceptors 610.Attorney Docket No. 33894-S003 PC (PATENT)

[0249] In some embodiments, instead of cropping the original image data set A into a smaller data set A’ for processing by the vision perceptors 610, the cropper 608 may black out or otherwise make uniform the areas of the original data set A having higher entropy, thereby optimizing the original image data set A into an optimized image data set A’ which the downstream vision perceptors 610 can more quickly and efficiently process. For ease of discussion, though, the present disclosure refers to the term “cropping” rather than the terms “blacking out” or “making uniform,” although one skilled in the art can understand that the techniques related to a cropped image A’ can be easily applied to an optimized image A’ in which sub-areas of higher entropies have been blacked-out or made uniform.

[0250] Information generated by the vision perceptors 610 may be provided to one or more on-board controllers 612 on-board the AMSV 100, where the one or more controllers 612 may utilize the information generated by the vision perceptors 610 (and optionally other information generated by one or more other onboard systems) to generate a control signal to control (e.g., responsively control) one or more operations of the AMSV 100. For example, the information generated by the vision perceptors 610 may indicate the detection of a presence of an object within the FOV of the AMSV 100, the measurement of a distance of the detected object from the AMSV 100, an orientation of the object, a movement (e.g., heading, direction, speed, trajectory, etc.) of the object, a track of the object, and the like. The controller(s) 612 may include, for example, the AMSV control module 230, the LSA module 232, the SSA module 235, and / or any other controllers which are disposed on-board the AMSV 100 and control one or more systems or devices of the AMSV 100, such as the locomotion system 218, communication systems, maintenance systems, etc. For example, the AMSV control module 230 may control the speed and / or orientation of the AMSV 100 based on the information generated by the vision perceptors 610, another controller may deactivate an active remove sensing system 248 based on the information generated by the vision perceptors 610, a charge deployment controller (not shown) may prepare payload for discharge based on the information generated by the vision perceptors 610, a communications controller (not shown) and the SSA module 235 may cooperate to send instructions to other AMSVs of a swarm in which the AMSV 100 is included based on the information generated by the vision perceptors 610, and the like.

[0251] Example Computer-Implemented Methods

[0252] FIG. 7A is a flow diagram of an example method 700 performed by a vision perception optimization system on-board an autonomous maritime surface vehicle (AMSV), such as the AMSV 100 or the AMSV 200, where the AMSV is located in or disposed on a body of water. For example, in an embodiment the example method 700 may be performed by at least a portion of the vision perception optimization system 600 of FIG. 6A. For instance, the entropy detector 605 may perform at least a portion of the method 700, the cropper 608 may perform at least a portion of the method 700, and / or one or more vision perceptors 610 may perform at least a portion of the method 700. For ease of discussion, and not for limitation purposes, the method 700 is described with simultaneous reference to FIGS. 1A-1 E, 2, and 6 (and to various elements thereof), although it is understood that any one or more portions of the method 700 may be performed in conjunction with otherAttorney Docket No. 33894-S003 PC(PATENT) embodiments of AMSVs and / or other AMSVs. Further, in embodiments, the method 700 may operate in conjunction with at least portions of one or more of the other methods, algorithms, and / or analyses described herein, and / or the method 700 may include additional and / or alternate blocks other than those described herein. For example, the method 700 may be or may be included in a method of controlling operations of maritime vehicles in a maritime environment, and / or the method 700 may be or may be included in a method of mitigating effects of noise on computer vision systems in maritime environments.

[0253] Generally speaking, the method 700 relates to controlling operations of maritime vehicles (e.g., autonomous maritime vehicles or AMSVs) in a maritime domain or environment. In embodiments, the method 700 relates to mitigating the effects of noise on on-board computer vision systems in maritime environments to thereby more efficiently control the operations of maritime vehicles in maritime domains or environments based on information generated by the on-board computer vision systems.

[0254] At a block 702, the method 700 may include detecting a first sub-area of a first image captured by a first image sensor included in a stereo vision system mounted on a maritime vehicle. For example, the first image sensor may be the sensor 242a (which may be an electro-optical image sensor or an infrared (IR) sensor), the stereo vision system may be the stereo vision system 240 or 212, and the maritime vehicle may be the AMSV 200. The first sub-area may depict at least a portion of an object located remotely from the maritime vehicle, such as another maritime vehicle or vessel, a coast line, a buoy, a platform, a dock, and / or other objects within the field-of-view of the stereo vision system and disposed on land or on water. The detecting 702 of the first sub-area of the first image may include detecting that the first sub-area has a respective entropy less than a first threshold. The first threshold can be a pre-determined threshold, in some implementations. Additionally or alternatively, the detecting 702 of the first sub-area of the first image may include detecting that the first sub-area has a respective entropy less than at least one other sub-area of the first image, or less than a majority of the other sub-areas of the first image, in some implementations. The detecting of the first sub-area of the first image having a lesser entropy than one or more other sub-areas of the first image may include utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique, such as in manners described elsewhere herein. Said another way, the detecting 702 of the first sub-area of the first image having the entropy less than a threshold or less than a majority of the other sub-areas of the first image may include using one or more computer-vision detection techniques as a filter to distinguish the first sub-area from the other sub-areas of the first image.

[0255] At a block 705, the method 700 may include detecting a second sub-area of a second image captured by a second image sensor included in the stereo vision system. For example, the second image sensor may be the sensor 242b, which may be an electro-optical image sensor or an infrared (IR) sensor, and the first sensor and the second sensor may be stereoscopically paired, in embodiments. The second sub-area may depict at least a portion of the object sensed by the first image sensor and located remotely from the maritime vehicle. The detecting 705 of the second sub-area of the second image may include detecting thatAttorney Docket No. 33894-S003 PC (PATENT) the second sub-area has a respective entropy less than a second threshold, and the second threshold can be a pre-determined threshold. The second threshold may or may not be equivalent to the first threshold. Additionally or alternatively, the detecting of the second sub-area of the second image may include detecting that the second sub-area has a respective entropy less than at least one other sub-area of the second image or less than a majority of the other sub-areas of the second image. The detecting of the second sub-area of the second image having a lesser entropy than one or more other sub-areas of the second image may include utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique, such as in manners described elsewhere herein. Said another way, the detecting 705 of the second sub-area of the second image having the entropy less than a threshold or less than a majority of the other sub-areas of the second image using one or more computer-vision detection techniques as a filter to distinguish the second sub-area from the other sub-areas of the second image.

[0256] At a block 708, the method 700 may include cropping the first image to include only the first-sub area. In alternate embodiments, cropping 708 the first image may include blacking out or otherwise making uniform (e.g., in entropy, color, or other characteristic different than the entropy, color, or other characteristic of the first sub-area) all sub-areas of the first image except the first sub-area. For example, at the block 708 the method 700 may operate to crop the image A into the cropped image A’, as shown in FIG. 6A.

[0257] At a block 710, the method 700 may include cropping the second image to include only the second sub-area. In alternate embodiments, cropping 708 the second image may include blacking out or otherwise making uniform (e.g., in entropy, color, or other characteristic different than the entropy, color, or other characteristic of the second sub-area) all sub-areas of the second image except the second sub-area. For example, at the block 708 the method 700 may operate to crop the image B into the cropped image B’, as shown in FIG. 6A.

[0258] At a block 712, the method 700 may include determining a distance between the maritime vehicle and the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image. For example, the block 712 may include determining or measuring a distance of a sensed or detected target form the maritime vehicle in manners such as those discussed with respect to the FIGS. 4A-4K, and / or 5A-5E.

[0259] At a block 715, the method 700 may include controlling an operation of the maritime vehicle based on the distance between the maritime vehicle and the object. The controlling 715 of the operation of the maritime vehicle may include adjusting at least one of a speed or an orientation of the maritime vehicle, in an embodiment. The controlling 715 of the operation of the maritime vehicle may include controlling the tracking of the object by the maritime vehicle, sending information and / or instructions to other maritime vehicles with which the maritime vehicle is coordinating operations (e.g., as a swarm), and / or changing one or more settings or parameters of one or more on-board systems and / or devices on-board the maritime vehicle. In some embodiments, the controlling 715 of the operation of the maritime vehicle may include controlling additional and / or alternate operations performed by the AMSV 200 in the maritime environment, such as butAttorney Docket No. 33894-S003 PC(PATENT) not limited to those described with respect to the methods 510, 530, 540, 550 and / or as described in priority documents U.S. Provisional Patent Application No. 63 / 698,453 and U.S. Provisional Patent Application No. 63 / 701 ,166.

[0260] In some implementations, the method 700 may be repeatedly executed by the AMSV 200 over time to thereby track a detected object over time. For example, after executing a first instance of the method 700, the AMSV 200 may execute a second instance at a subsequent time. As such, the method 700 may further include, during the second instance, detecting 702 a first subsequent sub-area of a third image captured by a first image sensor, where the first subsequent sub-area of the third image depicts a respective at least a portion of the object and the first subsequent sub-area has a respective entropy less than a respective threshold. The method 700 may further include, during the second instance, detecting 705 a second subsequent sub-area of a fourth image captured by the second image sensor, where the second subsequent sub-area depicts a respective at least a portion of the object, and the second subsequent subarea has a respective entropy less than a respective threshold. Additionally, during the second instance, the method 700 may include 708 cropping the third image to include only the first subsequent sub-area, cropping 710 the fourth image to include only the second subsequent sub-area, and determining 712 a second distance between the maritime vehicle and the object based on the cropped third image and the cropped fourth image and not based on at least one of an entirety of the third image or an entirety of the fourth image.

[0261] Further, during the second instance, the method 700 may include detecting a movement of the object based on the first distance (which was determined during the execution of the first instance of the method 700) and the second distance (which is determined during the execution of the second, subsequent instance of the method 700). For example, the method 700 may determine a speed, heading, movement trajectory, etc. of the object based on the first distance determined at a first time and the second distance determined at a second time occurring after the first time. The method 700 may include controlling 715 the operation of the maritime vehicle based on the detected movement of the object.

[0262] FIG. 7B is a flow diagram of an example method 750 performed by a vision perception optimization system on-board an autonomous maritime surface vehicle (AMSV), such as the AMSV 100 or the AMSV 200, where the AMSV is located in or disposed on a body of water. For example, in an embodiment the example method 700 may be performed by at least a portion of the vision perception optimization system 300 of FIG. 6A. For instance, the entropy detector 605 may perform at least a portion of the method 750, the cropper 608 may perform at least a portion of the method 750, and / or one or more vision perceptors 610 may perform at least a portion of the method 750. For ease of discussion, and not for limitation purposes, the method 750 is described with simultaneous reference to FIGS. 1A-1 E, 2, and 6 (and to various elements thereof), although it is understood that any one or more portions of the method 750 may be performed in conjunction with other embodiments of AMSVs and / or other AMSVs. Further, in embodiments, the method 750 may operate in conjunction with at least portions of one or more of the other methods described herein, and / or the method 750 may include additional and / or alternate blocks other than those described herein. For example, theAttorney Docket No. 33894-S003 PC(PATENT) method 750 may be or may be included in a method of controlling operations of maritime vehicles in a maritime environment, and / or the method 750 may be or may be included in a method of mitigating effects of noise on computer vision systems in maritime environments.

[0263] Generally speaking, the method 700 relates to controlling operations of maritime vehicles (e.g., autonomous maritime vehicles or AMSVs) in a maritime domain or environment. In embodiments, the method 700 relates to mitigating the effects of noise on on-board computer vision systems in maritime environments to thereby more efficiently control the operations of maritime vehicles in maritime domains or environments based on information generated by the on-board computer vision systems.

[0264] At a block 752, the method 700 may include detecting, at a first time, a first sub-area of a first image captured by a vision system mounted on a maritime vehicle. The vision system may include one or more image sensors, such as electro-optical image sensors or an infrared (IR) sensors. For example, the vision system may be the stereo vision system 240 or 212, and the maritime vehicle may be the AMSV 200. The first sub-area may depict at least a portion of an object located remotely from the maritime vehicle, such as another maritime vehicle or vessel, a coast line, a buoy, a platform, a dock, and / or other objects within the field-of-view of the stereo vision system and disposed on land or on water. The detecting 752 of the first subarea of the first image may include detecting that the first sub-area has a respective entropy less than a first threshold. The first threshold can be a pre-determined threshold, in some implementations. Additionally or alternatively, the detecting 752 of the first sub-area of the first image may include detecting that the first subarea has a respective entropy less than at least one other sub-area of the first image, or less than a majority of the other sub-areas of the first image, in some implementations. The detecting of the first sub-area of the first image having a lesser entropy than one or more other sub-areas of the first image may include utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique, such as in manners described elsewhere herein. Said another way, the detecting 752 of the first sub-area of the first image having the entropy less than a threshold or less than a majority of the other sub-areas of the first image using one or more computer-vision detection techniques as a filter to distinguish the first sub-area from the other sub-areas of the first image.

[0265] At a block 755, the method 750 includes detecting, at a second time, a second sub-area of a second image captured by the vision system. The second time may occur after the first time has occurred (e.g., subsequent to the first time), and the second sub-area may depict a respective at least a portion of the object, where the second sub-area having a respective entropy less than a second threshold. The respective portion of the object detected at the second time may be the same or different than the respective portion of the object detected at the first time, and the second threshold may be the same as or different than the first threshold. The second threshold can be a pre-determined threshold, in some implementations. Additionally or alternatively, the detecting 755 of the second sub-area of the first image may include detecting that the second sub-area has a respective entropy less than at least one other sub-area of the second image, or less than a majority of the other sub-areas of the second image, in some implementations. The detecting of theAttorney Docket No. 33894-S003 PC(PATENT) second sub-area of the first image having a lesser entropy than one or more other sub-areas of the second image may include utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique, such as in manners described elsewhere herein. Said another way, the detecting 755 of the second sub-area of the second image having the entropy less than a threshold or less than a majority of the other sub-areas of the second image using one or more computer-vision detection techniques as a filter to distinguish the second sub-area from the other sub-areas of the second image.

[0266] At a block 758, the method 750 includes cropping the first image to include only the first-sub area, and at a block 760, the method 750 includes cropping the second image to include only the second sub-area. The blocks 758 and 760 may utilize techniques similar to those discussed for the blocks 708 and 710, for example.

[0267] At a block 762, the method 750 includes detecting a movement of the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image. For example, the block 762 may include detecting a speed, heading, movement, trajectory, etc. of the detected object based on the detected movement of the object, such as in manners such as those discussed with respect to the method 510 of FIG. 5B. In embodiments, the detecting of the movement of the object may include utilizing at least one of: optical flow, dense matching, sparks matching, template matching, filtering, or another type of computer vision technique to the first cropped image and the second cropped image.

[0268] At a block 765, the method 750 includes controlling an operation of the maritime vehicle based on the detected movement of the object. The controlling 715 of the operation of the maritime vehicle may include adjusting at least one of a speed or an orientation of the maritime vehicle, in an embodiment. The controlling 715 of the operation of the maritime vehicle may include controlling the tracking of the object by the maritime vehicle, sending information and / or instructions to other maritime vehicles with which the maritime vehicle is coordinating operations (e.g., as a swarm), and / or changing one or more settings or parameters of one or more on-board systems and / or devices on-board the maritime vehicle. In some embodiments, the controlling 715 of the operation of the maritime vehicle may include controlling additional and / or alternate operations performed by the AMSV 200 in the maritime environment, such as but not limited to those described with respect to the methods 510, 530, 540, 550 and / or as described in priority documents U.S. Provisional Patent Application No. 63 / 698,453 and U.S. Provisional Patent Application No. 63 / 701 ,166.

[0269] In some embodiments (not shown), the method 750 includes detecting a presence of the object based on the cropped first image and not based on the entirety of the first cropped image. For example, the method 750 may include detecting the presence of the object within the FOV of the first image sensor based on the cropped first image (and not based on an entirety of the first cropped image). Additionally or alternatively, the method 750 may include determining a class, type, or specific identity of the detected object based on the cropper first image (and not based on an entirety of the first cropped image). For example, oneAttorney Docket No. 33894-S003 PC (PATENT) or more vision perceptors 610 may detect presences of objects and optionally determine respective classes, types, and or specific identities of detected objects based on at least one of the cropped images (and not based on any entireties of captured images). As these embodiments utilize cropped images instead of entities of captured images, the processing cycles, memory, time, and other resources required to detect presences of objects within the FOV of a maritime vehicle and identify classes, types, and / or specific identities thereof is significantly reduced as compared to traditional image processing techniques. Consequently, the responsiveness of the operations of the maritime vehicle based on the detected and identified objects (e.g., at the block 765) is significantly increased.

[0270] In some embodiments (not shown), the method 750 may apply to multiple objects disposed within the FOV of the maritime vehicle. For example, the method 750 may include detecting, at the first time, multiple objects which are located within the FOV of the maritime vehicle. For instance, the method 750 may include detecting a third sub-area of the first image, where the third sub-area depicts a respective at least a portion of a second object located remotely from the maritime vehicle and the third sub-area has a respective entropy less than a respective threshold, e.g., in manners such as previously discussed. Additionally, the method 750 may include detecting, within the FOV of the maritime vehicle, the multiple objects at a second time subsequent to the first time. For instance, the method 750 may include detecting a fourth sub-area of the first image, where the fourth sub-area depicts a respective at least a portion of the second object, and the fourth sub-area has a respective entropy less than a respective threshold, e.g., in manners such as previously discussed. In these embodiments, the method 750 may include cropping the first image to include only the third sub-area, cropping the second image to include only the fourth sub-area, and detecting a movement of the second object based on the cropped first image including only the third sub-area and the cropped second image including only the fourth sub-area, and not based on the at least one of the entirety of the first image or the entirety of the second image.

[0271] Additionally, in these embodiments, the method 750 may include detecting a movement of the second object (e.g., over time) based on the cropped first image including only the third sub-area and the cropped second image including only the fourth sub-area, and not based on the at least one of the entirety of the first image or the entirety of the second image. The method 750 may include responsively controlling one or more operations of the maritime vehicle based on the detected movement of the second object over time.

[0272] Thus, in these embodiments and generally speaking, the method 750 may detect the respective presences and movements of multiple objects in each captured image over time. The method 750 may responsively control one or more operations of the maritime vehicle based on the movements over time of at least one of the multiple detected objects, and in some cases based on the movements over time of more than one of the multiple detected objects.

[0273] Additional ConsiderationsAttorney Docket No. 33894-S003 PC(PATENT)

[0274] Further, although certain autonomous maritime surface vehicles and related systems, methods, and components have been described herein in accordance with the teachings of the present disclosure, the scope of coverage of this patent is not limited thereto. On the contrary, while the invention has been shown and described in connection with various preferred embodiments, it is apparent that certain changes and modifications, in addition to those mentioned above, may be made. This patent covers all embodiments of the teachings of the disclosure that fairly fall within the scope of permissible equivalents. Accordingly, it is the intention to protect all variations and modifications that may occur to one of ordinary skill in the art.

[0275] Still further, when implemented, any of the methods and techniques described herein or portions thereof may be performed by executing software one or more non-transitory, tangible, computer readable storage media or memories such as magnetic disks, laser disks, optical discs, semiconductor memories, biological memories, other memory devices, or other storage media, in a RAM or ROM of a computer or processor, etc.

[0276] Moreover, although the foregoing text sets forth a detailed description of numerous different embodiments, it should be understood that the scope of the patent is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only and does not describe every possible embodiment because describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims. By way of example, and not limitation, the disclosure herein contemplates at least the following aspects:

[0277] 1 . A method of controlling operations of maritime vehicles in a maritime environment, the method comprising detecting a first sub-area of a first image captured by a first image sensor included in a stereo vision system mounted on a maritime vehicle, the first sub-area depicting a respective at least a portion of an object located remotely from the maritime vehicle, and the first sub-area having a respective entropy less than a first threshold; detecting a second sub-area of a second image captured by a second image sensor included in the stereo vision system, the second sub-area depicting a respective at least a portion of the object, and the second sub-area having a respective entropy less than a second threshold; cropping the first image to include only the first sub-area; and cropping the second image to include only the second sub-area. The method additionally comprises determining a distance between the maritime vehicle and the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image; and controlling an operation of the maritime vehicle based on the distance between the maritime vehicle and the object.

[0278] 2. The method of aspect 1 , wherein the second threshold is equivalent to the first threshold.

[0279] 3. The method of aspect 1 , wherein the second threshold is not equivalent to the first threshold.

[0280] 4. The method of any one of aspects 2-3, wherein the first image sensor and the second image sensor are respective electro-optical image sensors.Attorney Docket No. 33894-S003 PC(PATENT)

[0281] 5. The method of any one of aspects 2-3, wherein the first image sensor and the second image sensor are respective infrared (IR) image sensors.

[0282] 6. The method of any one of aspects 1-5, wherein the distance between the maritime vehicle and the object is a first distance determined by executing a first instance of the method of aspect 1 at a first time; and the method further comprises, at a subsequent time:

[0283] detecting a first subsequent sub-area of a third image captured by a first image sensor, the first subsequent sub-area depicting a respective at least a portion of the object, and the first subsequent sub-area having a respective entropy less than a respective threshold; detecting a second subsequent sub-area of a fourth image captured by the second image sensor, the second subsequent sub-area depicting a respective at least a portion of the object, and the second subsequent sub-area having a respective entropy less than a respective threshold; cropping the third image to include only the first subsequent sub-area; cropping the fourth image to include only the second subsequent sub-area; determining a second distance between the maritime vehicle and the object based on the cropped third image and the cropped fourth image and not based on at least one of an entirety of the third image or an entirety of the fourth image; and detecting a movement of the object based on the first distance and the second distance;

[0284] where the controlling of the operation of the maritime vehicle is based on the detected movement of the object.

[0285] 7. The method of any one of aspects 1-6, wherein at least one of: the detecting of the first subarea of the first image having the entropy less than the first threshold or the detecting of the second sub-area of the second image having the entropy less than the second threshold includes utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique.

[0286] 8. The method of any one of aspects 1-7, wherein the controlling of the operation of the maritime vehicle includes adjusting at least one of a speed or an orientation of the maritime vehicle.

[0287] 9. A method of controlling operations of maritime vehicles in a maritime environment, the method comprising detecting, at a first time, a first sub-area of a first image captured by a vision system mounted on a maritime vehicle, the first sub-area depicting a respective at least a portion of an object located remotely from the maritime vehicle, and the first sub-area having a respective entropy less than a first threshold; detecting, at a second time, a second sub-area of a second image captured by the vision system, the second sub-area depicting a respective at least a portion of the object, and the second sub-area having a respective entropy less than a second threshold; cropping the first image to include only the first sub-area; cropping the second image to include only the second sub-area; detecting a movement of the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image; and controlling an operation of the maritime vehicle based on the detected movement of the object.Attorney Docket No. 33894-S003 PC(PATENT)

[0288] 10. The method of aspect 9, wherein the detecting of the movement of the object includes utilizing at least one of: optical flow, dense matching, sparks matching, template matching, filtering, or another type of computer vision technique to the first cropped image and the second cropped image.

[0289] 11 . The method of aspect 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing dense matching.

[0290] 12. The method of aspect 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing optical flow.

[0291] 13. The method of aspect 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing template matching.

[0292] 14. The method of aspect 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing sparks matching.

[0293] 15. The method of aspect 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing filtering.

[0294] 16. The method of any one of aspects 9-15, wherein the detecting of the sub-area of the image having the entropy less than the threshold includes utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique.

[0295] 17. The method of any one of aspects 9-16, wherein the method further comprises detecting a presence of the object based on the cropped first image and not based on the entirety of the first cropped image.

[0296] 18. The method of any one of aspects 9-17, wherein the controlling of the operation of the maritime vehicle includes adjusting at least one of a speed or an orientation of the maritime vehicle.

[0297] 19. The method of any one of aspects 9-18, wherein the controlling of the operation of the maritime vehicle includes tracking the object.

[0298] 20. The method of any one of aspects 9-19, wherein the object is a first object, and the method further comprises detecting, at the first time, a third sub-area of the first image, the third sub-area depicting a respective at least a portion of a second object located remotely from the maritime vehicle, and the third subarea having a respective entropy less than a respective threshold; detecting, at the second time, a fourth subarea of the first image, the fourth sub-area depicting a respective at least a portion of the second object, and the fourth sub-area having a respective entropy less than a respective threshold; cropping the first image to include only the third sub-area; cropping the second image to include only the fourth sub-area; and detecting a movement of the second object based on the cropped first image including only the third sub-area and the cropped second image including only the fourth sub-area, and not based on the at least one of the entirety of the first image or the entirety of the second image.Attorney Docket No. 33894-S003 PC(PATENT)

[0299] 21 . A maritime vehicle configured to perform the method of any one of aspects 1-8.

[0300] 22. A maritime vehicle configured to perform the method of any one of aspects 9-20.

[0301] 23. A maritime vehicle configured to perform the method of any one of aspects 1-8 and the method of any one of the aspects 9-20.

[0302] 24. Any one the preceding aspects in combination with any other one of the preceding aspects.

[0303] Thus, many modifications and variations may be made in the techniques, methods, and structures described and illustrated herein without departing from the spirit and scope of the present claims. Accordingly, it should be understood that the methods and apparatus described herein are illustrative only and are not limiting upon the scope of the claims.

Claims

Attorney Docket No. 33894-S003 PC(PATENT)WHAT IS CLAIMED:1 . A method of controlling operations of maritime vehicles in a maritime environment, the method comprising: detecting a first sub-area of a first image captured by a first image sensor included in a stereo vision system mounted on a maritime vehicle, the first sub-area depicting a respective at least a portion of an object located remotely from the maritime vehicle, and the first sub-area having a respective entropy less than a first threshold; detecting a second sub-area of a second image captured by a second image sensor included in the stereo vision system, the second sub-area depicting a respective at least a portion of the object, and the second sub-area having a respective entropy less than a second threshold; cropping the first image to include only the first sub-area; cropping the second image to include only the second sub-area; determining a distance between the maritime vehicle and the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image; and controlling an operation of the maritime vehicle based on the distance between the maritime vehicle and the object.

2. The method of claim 1 , wherein the second threshold is equivalent to the first threshold.

3. The method of claim 2, wherein the first image sensor and the second image sensor are respective electro-optical image sensors or respective infrared (IR) image sensors.

4. The method of claim 1 , wherein the second threshold is not equivalent to the first threshold.

5. The method of claim 4, wherein the first image sensor and the second image sensor are respective electro-optical image sensors or respective infrared (IR) image sensors.

6. The method of claim 1 , wherein: the distance between the maritime vehicle and the object is a first distance determined by executing a first instance of the method of claim 1 at a first time; the method further comprises, at a subsequent time: detecting a first subsequent sub-area of a third image captured by a first image sensor, the first subsequent sub-area depicting a respective at least a portion of the object, and the first subsequent sub-area having a respective entropy less than a respective threshold;Attorney Docket No. 33894-S003 PC(PATENT) detecting a second subsequent sub-area of a fourth image captured by the second image sensor, the second subsequent sub-area depicting a respective at least a portion of the object, and the second subsequent sub-area having a respective entropy less than a respective threshold; cropping the third image to include only the first subsequent sub-area; cropping the fourth image to include only the second subsequent sub-area; determining a second distance between the maritime vehicle and the object based on the cropped third image and the cropped fourth image and not based on at least one of an entirety of the third image or an entirety of the fourth image; and detecting a movement of the object based on the first distance and the second distance; and the controlling of the operation of the maritime vehicle is based on the detected movement of the object.

7. The method of claim 1 , wherein at least one of: the detecting of the first sub-area of the first image having the entropy less than the first threshold or the detecting of the second sub-area of the second image having the entropy less than the second threshold includes utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique.

8. The method of claim 1 , wherein the controlling of the operation of the maritime vehicle includes adjusting at least one of a speed or an orientation of the maritime vehicle.

9. A method of controlling operations of maritime vehicles in a maritime environment, the method comprising: detecting, at a first time, a first sub-area of a first image captured by a vision system mounted on a maritime vehicle, the first sub-area depicting a respective at least a portion of an object located remotely from the maritime vehicle, and the first sub-area having a respective entropy less than a first threshold; detecting, at a second time, a second sub-area of a second image captured by the vision system, the second sub-area depicting a respective at least a portion of the object, and the second sub-area having a respective entropy less than a second threshold; cropping the first image to include only the first sub-area; cropping the second image to include only the second sub-area; detecting a movement of the object based on the cropped first image and the cropped second image and not based on at least one of an entirety of the first image or an entirety of the second image; and controlling an operation of the maritime vehicle based on the detected movement of the object.

10. The method of claim 9, wherein the detecting of the movement of the object includes utilizing at least one of: optical flow, dense matching, sparks matching, template matching, filtering, or another type of computer vision technique to the first cropped image and the second cropped image.Attorney Docket No. 33894-S003 PC(PATENT)11. The method of claim 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing dense matching.

12. The method of claim 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing optical flow.

13. The method of claim 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing template matching.

14. The method of claim 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing sparks matching.

15. The method of claim 10, wherein the detecting of the movement of the object based on the first cropped image and the second cropped image includes utilizing filtering.

16. The method of claim 9, wherein the detecting of the sub-area of the image having the entropy less than the threshold includes utilizing at least one of: edge detection, a color gradient, a heat map, a neural network, or another computer vision detection technique.

17. The method of claim 9, wherein the method further comprises detecting a presence of the object based on the cropped first image and not based on the entirety of the first cropped image.

18. The method of claim 9, wherein the controlling of the operation of the maritime vehicle includes adjusting at least one of a speed or an orientation of the maritime vehicle.

19. The method of claim 9, wherein the controlling of the operation of the maritime vehicle includes tracking the object.

20. The method of claim 9, wherein the object is a first object, and the method further comprises: detecting, at the first time, a third sub-area of the first image, the third sub-area depicting a respective at least a portion of a second object located remotely from the maritime vehicle, and the third sub-area having a respective entropy less than a respective threshold; detecting, at the second time, a fourth sub-area of the first image, the fourth sub-area depicting a respective at least a portion of the second object, and the fourth sub-area having a respective entropy less than a respective threshold; cropping the first image to include only the third sub-area;Attorney Docket No. 33894-S003 PC (PATENT) cropping the second image to include only the fourth sub-area; and detecting a movement of the second object based on the cropped first image including only the third sub-area and the cropped second image including only the fourth sub-area, and not based on the at least one of the entirety of the first image or the entirety of the second image.

Images