Multi-modal sensor suite system for pilots during ROV piloting for enhanced pilot situational awareness

Abstract

The disclosed invention comprises a first predetermined plurality of cameras configured to provide a 360° bird's eye view and a second predetermined plurality of cameras configured to provide a 360° panoramic view of an area surrounding a remotely operated vehicle (ROV) (100) disposed subsea as well as a predetermined set of doppler velocity logger (DVL) / distance sensors. A multi-height calibration feature exists which can construct a regression model using data from various heights to predict matrices for operational depth, allowing the application to stream BEV and panoramic views for varying heights while piloting an ROV. A digital pan and tilt (panoramic) process involves stitching images using checkerboard corner features placed on the ground to help ensure continuity at ground level and provides a smoother transition in BEV and panoramic views for ROVs during subsea operations and can eliminate a requirement for hydraulic pan and tilt mechanisms, resulting in overall cost reduction for an ROV.

Description

MULTI-MODAL SENSOR SUITE SYSTEM FOR PILOTS DURING ROV PILOTING FOR ENHANCED PILOT SITUATIONAL AWARENESSINVENTORS: Siddharth Srivatsa; Rahul Bapurao Devangre; Sheethal Sasidharan;Amritanjanan Thakur; Ryan W. Weeden; Vemburajan YadavaCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority through India Provisional Application 202411074999 filed on October 4, 2024, incorporated herein by reference.BACKGROUND OF THE INVENTION

[0002] A multi-modal sensor suite and software package for pilots during ROV piloting for enhanced pilot situational awareness is disclosed. In embodiments, the invention comprises a multi -camera suite to visualize 360 Bird’s eye view and 360° panoramic view surrounding subsea ROV. The invention can be used to help increase situational awareness for the pilot in a subsea vehicle moving in three (3) dimensions; provide a user-friendly user interface for pilot during navigation and manipulator operations, e.g., a bird’s eye view, 360° view with pan tilt and zoom; aid with helping prevent a remotely operated vehicle (ROV) from colliding into objects; and provide for easier ROV navigation and operations in congested areas. In embodiments, the invention uses an acoustic / EMF based sensor suite to provide awareness of objects around the ROV during no light or low light conditions. The claimed invention can provide easier ROV navigation and operations in congested areas. By way of example and not limitation, in embodiments the claimed invention handles the use case for robots to operate in all three dimensions. In part, this is accomplished by stitching images together and displaying a resultant image in an intuitive way for an ROV pilot. Further, the claimed invention, using its predetermined set of sensors handles no / low light conditions by using different sensors to provide information for object detection and avoidance.{ NATAER0000066113.0} Page 1 of 12BRIEF DESCRIPTION OF DRAWINGS

[0003] Various figures are included herein which illustrate aspects of embodiments of the disclosed inventions.

[0004] Fig. 1 is a topside schematic view of an exemplary multi-modal sensor system;

[0005] Fig. 2 is a topside schematic view of an exemplary multi-modal sensor system;

[0006] Fig. 3 is a block diagram view of an exemplary multi-modal sensor system method;

[0007] Fig. 4 is a flowchart of an exemplary multi-modal sensor system method;

[0008] Fig. 5 is a flowchart of an exemplary multi-modal sensor system method;

[0009] Fig. 6 is a flowchart of an exemplary multi-modal sensor system method;

[0010] Fig. 7 is a flowchart of an exemplary multi-modal sensor system method;

[0011] Fig. 8 is a flowchart of an exemplary multi-modal sensor system method; and

[0012] Fig. 9 is a flowchart of an exemplary multi-modal sensor system method;DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0013] In a first embodiment, referring generally to Fig. 1, multi-modal sensor system 1 comprises a predetermined plurality of cameras 10, where one or more first cameras 11 of the first predetermined plurality of cameras 10 is configured to provide a 360° bird’s eye view and one or more second cameras 12 of the second predetermined plurality of cameras 10 is configured to provide a 360° panoramic view of an area surrounding remotely operated vehicle (ROV) 100 disposed subsea; a predetermined set of doppler velocity logger (DVL) / di stance sensors 20; and acoustic bumper sensor array 30. Typically, multi-modal sensor system 1 can perform under pressures associated with 4,500m water depth, 7,500 PSI.

[0014] Typically, first camera 11 comprises pan, tilt, and zoom functionality which, in embodiments, comprises digital pan, tile, and zoom functionality. The first and second{ NATAER0000066113.0} Page 2 of 12predetermined pluralities of cameras 10 may be disposed in the specific configuration illustrated in Fig. 1 to provide the 360° coverage.

[0015] The predetermined set of DVL / di stance sensors 20 are typically disposed about ROV 100 in a predetermined pattern and may comprise one or more acoustic sensors, one or more EMF sensors, or both acoustic and EMF sensors, or the like. Acoustic sensors 20 may be low-cost acoustic sensors 20. The predetermined set of DVL / distance sensors 20 are typically disposed about ROV 100 and provide sensed data representative of an object located proximate ROV 100 which may occur during a no light or low light condition for object / ob stacl e detection when visual data are not available.

[0016] Multi-modal sensor system 1 may further comprise image processor 40 operatively in communication with the predetermined set of cameras 10 and the predetermined set of DVL / distance sensors 20 to receive image data representative of an environment proximate ROV 100; and video data processor 50 configured to accept image data from image processor 40 and stitch and project the image data to provide top-down / bird’s-eye view (BEV) / digital pan, tilt and zoom such as by using data processor 50 operatively in communication with the one or more cameras 10. Typically, video data processor 50 comprises video data processor 51, system calibrator 52, and image stitcher 53.

[0017] In embodiments, haptic feedback interface 60 is present and operatively in communication with video data processor 50.

[0018] Human interface device 70 may also be present and operatively in communication with video data processor 50. Human interface device 70 may comprises a display operatively in communication with the video data processor and video data processor 50 may comprise a graphic user interface (GUI) adapted to provide visual feedback for a human operator that maximizes pilot{ NATAER0000066113.0} Page 3 of 12situational awareness and provides multiple views of environment surrounding ROV 100 such as by combining data from video, audio, and EMF sensors. Typically, the GUI can be toggled between bird’s eye view and digital pan-tilt-zoom (PTZ) views. By way of example and not limitation, displayed visual can comprise multiple popups, each showing the video of the respective camera aligned to match the physical positioning of the cameras on the ROV. Video data processor 50 can provide an ROV pilot with an ability to switch displays on or off individually or together.

[0019] Human interface device 70 may comprise a virtual reality (VR) headset or other VR display and software that processes the video data to provide a virtual panoramic view to enhance pilot experience.

[0020] In embodiments, there are at least six cameras 10, one of which being a field of view (FOV) camera, and the predetermined set of DVL / di stance sensors 20 may further comprise a field-of-view (FOV) sensor. There is typically an overlap between camera views and an overlap between sensors.

[0021] Software operative in video data processor 50 is used to go through different calibration procedures. By way of example and not limitation, this can include intrinsic calibration for each camera 10, a transformation calibration for each camera 20 to project the video on to a top-down bird’s eye view, a homography calibration between cameras 20 to stitch the views together, or the like, or a combination thereof.

[0022] Multi-modal sensor system 1 may be used to provide enhanced pilot situational awareness for and during ROV piloting by providing a digital zoom, pan and tilt option, combining visual data from cameras 10 and acoustic and EMF sensor data from DVL distance sensors 20 into an object data set, and using the object data set to detect and alert a pilot of an object that is near{ NATAER0000066113.0} Page 4 of 12ROV 100. In addition, data from acoustic bumper sensor array 30 may also be combined into the object data set.

[0023] Where multi-modal sensor system 1 further comprises human interface device 70 and / or haptic feedback interface 60, e.g., a pilot chair, visual and / or haptic feedback may be provided via human interface device 70 and / or haptic feedback interface 60 to warn an ROV pilot of a detected object where such may comprise an intensity directly proportional to distance of object from ROV 100.

[0024] In certain embodiments, it is desirable to provide a BEV image to the ROV pilot which may occur by calibrating and generating corresponding matrices that are stored in calibration files. As illustrated in Figs. 4-9, this may be accomplished by reading the matrices from the files to generate the BEV and using an intrinsic calibration workflow to get a camera intrinsic coefficient matrix. Calibration may further comprise intrinsic calibration and / or fisheye calibration.

[0025] To get a transformation matrix for getting BEV of images for each camera, a plurality of points, e.g., four, may be marked on the ground in such a way that they are visible through one or more cameras 10 to define a rectangle. These source points are typically selected directly from undistorted images displayed by or on human interface 70 and serve as a reference, where the goal is to change the perspective of the image. To achieve this, both source points (from the undistorted image) and destination points (representing a desired top-down view) are typically required. Mathematical formulations are then used to calculate the positions of these destination points and the stitching pattern stored in the form of a matrix. This matrix, known as the homography matrix, encodes the transformation needed to achieve the desired top-down perspective.{ NATAER0000066113.0} Page 5 of 12

[0026] The calibration process, excluding intrinsic calibration, may be repeated at different heights to enable image stitching across various depths as ROV 100 moves along its depth axis. This approach is known as multi-height calibration. An ROV moving in all 3-dimensions poses a unique challenge since most of existing prior art computer vision techniques are planar and in 2- dimensions. The multi-height calibration feature of the invention constructs a regression model using data from various heights. This model predicts matrices for operational depth, allowing the application to stream Bird’s Eye View (BEV) and panoramic views for varying heights. The invention stitches images and produces a bird’s eye view while piloting an ROV. Digital Pan and Tilt (panoramic) process involves stitching images using checkerboard corner features placed on the ground. This approach ensures continuity at ground level and provides a smoother transition in Bird’s Eye View (BEV) and panoramic views for remotely operated vehicles (ROVs) during subsea operations. Importantly, this technique eliminates the requirement for hydraulic pan and tilt mechanisms, resulting in overall cost reduction for the vehicle.

[0027] Once the calibration routines are completed, a single calibration file is generated and used to store the calibration parameters, e.g., for multiple heights. Subsequently, the calibration file is used to create a regression model that predicts the calibration parameters for any given height. With this model, an operator can generate a bird’s-eye view or a panoramic stitched view for any depth from the seabed. In embodiments, this is accomplished by having an application receive depth information of ROV 100 from a Doppler Velocity Log (DVL), an altimeter, or a similar sensor. This depth information is then passed to a model used by video data processor 50 to calculate the appropriate calibration parameters based on the operating depth of ROV 100. Finally, the images from each camera 10 are projected from a top-down view, creating a bird’s-eye{ NATAER0000066113.0} Page 6 of 12view for each camera or image. This process ensures accurate perspective correction and seamless stitching across different depths.

[0028] After completing the calibration process, video data processor 50 may transition into an operation mode. During this phase, video data processor 50 reads the calibration matrix from the specified files and, initially, video data processor 50 begins by performing undistortion. Following that, video data processor 50 stitches together the port side images from cameras 12a, 12c and starboard side images from cameras 12b, 12d. Finally, video data processor 50 may then apply a transformation to achieve a top-down view and may stitch all four images together to get a 360° BEV in real time.

[0029] In certain embodiments, video data processor 50 may perform a homography calibration to stitch together images captured by two or more cameras 10, e.g., port side cameras 12a, 12c and starboard side cameras 12b, 12d. A homography matrix may be used to perform panoramic stitching. In embodiments, performing homography calibration is achieved by adjusting the perspective of the images and bringing it to the same plane, such as by placing a checkerboard in a common region that is visible to both the portl camera 12a and port2 camera 12c and using a utility such as Gstreamer to provide the frames from both portl camera 12a and port2 camera 12c which are further undistorted and the checkerboard corners detected in both frames. Based on these detected features, images are stitched together into a single composite image and the stitching pattern stored in the form of a matrix within a calibration fde named as homography matrix. Further, images from portl camera 12a and port2 camera 12c may be combined to form a port side image.{ NATAER0000066113.0} Page 7 of 12

[0030] Software in video data processor 50 may be used to retrieve a desired set of image frames from the one or more cameras and a desired set of image frames seamlessly streamed for subsequent processing.

[0031] In embodiments, each image frame may be encapsulated within a container, preserving crucial timestamp data essential for synchronizing multiple camera feeds.

[0032] Images may be saved to local storage.

[0033] Additionally, panoramic stitching may be performed by stitching images from cameras 12a and 12c and cameras 12b and 12d together using checkerboard corners shared in a common area. Subsequently, these stitched images may be further combined with front and rear images, leveraging similar features. The result is a fully 360° panoramic view.

[0034] Where multi-modal sensor system 1 further comprises acoustic bumper sensor array 30 around an outer boundary of ROV 100, e.g., an acoustic sensor array, acoustic bumper sensor array 30 may be used to continuously transmit a signal and receive a distance measurement if an object is detected within its sensory cone. When an object is detected by acoustic bumper sensor array 30, video data processor 50 may provide feedback to the pilot when an object is close to ROV 100 by providing visual feedback in a quadrant in which the object is displayed to focus on that specific feed and turn off all other camera feeds. Video data processor 50 may also provide haptic feedback to haptic feedback interface 60 and / or audio feedback like a beeper to alert the pilot.

[0035] The foregoing disclosure and description of the inventions are illustrative and explanatory. Various changes in the size, shape, and materials, as well as in the details of the illustrative construction and / or an illustrative method may be made without departing from the spirit of the invention.{ NATAER0000066113.0} Page 8 of 12

Claims

CLAIMSWhat is claimed is:1) A multi-modal sensor system (1), comprising: a) a first predetermined plurality of cameras (10) configured to provide a 360° bird’s eye view and a second predetermined plurality of cameras configured to provide a 360° panoramic view of an area surrounding a remotely operated vehicle (ROV) (100) disposed subsea; and b) a predetermined set of doppler velocity logger (DVL) / di stance sensors (20).2) The multi-modal sensor system of Claim 1, further comprising an acoustic bumper sensor array (30).3) The multi-modal sensor system of Claim 1, wherein a first camera (11) of the first predetermined set of cameras (10) comprises pan, tilt, and zoom functionality.4) The multi-modal sensor system of Claim 3, wherein the pan, tilt, and zoom functionality comprises digital pan, tile, and zoom functionality.5) The multi-modal sensor system of Claim 1, wherein the first and second predetermined pluralities of cameras are disposed in a specific configuration to provide the 360° coverage, comprising: a) a predetermined set of the first predetermined plurality of cameras (10) comprising a predetermined set of port side cameras (12a, 12c); b) a first camera (11) of the first predetermined plurality of cameras (10) comprising pan, tilt, and zoom functionality; c) a predetermined set of the second predetermined plurality of cameras (10) comprising a predetermined set of starboard cameras (12b, 12d); and{ N.ATAER0000066113.0} Page 9 of 12d) a second camera (12) of the second predetermined plurality of cameras (10) configured to provide a 360° panoramic view of an area surrounding a remotely operated vehicle (ROV) 100 disposed subsea.6) The multi-modal sensor system of Claim 1, wherein the predetermined set of DVL / di stance sensors (20) are disposed about the ROV (100) in a predetermined pattern and comprises an acoustic sensor (20), an EMF sensor (20), or both an acoustic sensor and an EMF sensor.7) The multi-modal sensor system of Claim 2, wherein the predetermined set of DVL / di stance sensors (20 are disposed about the ROV (100) and configured to provide sensed data representative of an object located proximate the ROV during a no light or low light condition for object / obstacle detection when visual data are not available.8) The multi-modal sensor system of Claim 1, further comprising: a) an image processor (40) operatively in communication with the predetermined set of cameras (10) and the predetermined set of DVL / di stance sensors (20) to receive image data representative of an environment proximate the ROV (100); and b) a video data processor (50) operatively in communication with the one or more cameras (10) and configured to accept image data from the image processor (40) and stitch and project the image data to provide top-down / bird’s-eye view / digital pan, tilt and zoom, the video data processor (50) comprising: i) a video data processor (51); ii) a system calibrator (52); and iii) an image stitcher (53).9) The multi-modal sensor system of Claim 8, further comprising:{ NATAER0000066113.0} Page 10 of 12a) a haptic feedback interface (60) operatively in communication with the video data processor (50); b) a human interface device (70) operatively in communication with the video data processor (50).10) The multi-modal sensor system of Claim 9, wherein: a) the human interface device (70) comprises a display operatively in communication with the video data processor (50); and b) the video data processor (50) comprises a graphic user interface (GUI) configured to provide visuals for a human operator that maximize pilot situational awareness and to provide multiple views of environment surrounding the ROV (100) such as by combining data from video, audio and EMF sensors.11) The multi-modal sensor system of Claim 10, wherein the GUI can be toggled between a bird’s eye view and a digital pan-tilt-zoom (PTZ) view.12) The multi-modal sensor system of Claim 9, wherein the human interface device (70) comprises: a) a virtual reality (VR) headset or other VR display; and b) software configured to process video data to provide a virtual panoramic view to enhance pilot experience.13) The multi-modal sensor system of Claim 1, wherein: a) the predetermined plurality of cameras (10) comprises at least six cameras, one of which being a field of view (FOV) camera; and b) the predetermined set of DVL / di stance sensors (20) comprises an FOV sensor.{ NATAER0000066113.0} Page 11 of 12

Images