Surveying system

Abstract

A survey system for capturing digital images in an environment, the system comprising: a digital camera (20) and a light array (19), the camera and light array arranged to be mounted on a mobile device (10) in a known spatial relationship to each other, the camera having a field of view defining an imaging space within a hostile environment, the light array comprising one or more independently operable light elements oriented to illuminate different parts of the imaging space; a storage device having stored therein a look-up table associating regions of images taken by the camera with elements of the light array oriented to be able to illuminate those regions; and a data processor communicatively connected to the camera and light array and the storage device, configured to: evaluate an illumination metric for pixels corresponding to a region of interest within an image; and based on the evaluation of the illumination metric, provide a light control signal to the light array to individually adjust elements of the light array based on the look-up table to improve visibility of the region of interest to the camera.

Description

Background Technology

[0001] When surveying seabed sites, asset facilities, environments, or any other harsh environments, mobile surveying devices such as robots or remotely operated vehicles are typically used. For simplicity, these facilities will be referred to as survey sites, and such surveying devices will be referred to as mobile devices (e.g., mobile robotic vehicles).

[0002] Lighting systems on remotely operated vehicles (ROVs) are often integrated with the vehicle controller and designed for human control. Some systems include a gimbal / pitch mechanism that adjusts the direction of the lighting when the camera gimbal / pitch unit is in use. Similar arrangements are often found on unmanned aerial vehicles (UAVs) or unmanned ground vehicles (UGVs).

[0003] Human operators have limited attention spans when performing complex tasks such as driving mobile robotic vehicles, making it difficult to optimally adjust lighting for best imaging performance—especially in environments with backscattering effects and reflective particles. Ambient conditions in the air, such as dust, smoke, or fog, can also contribute to backscattering.

[0004] Furthermore, when using Autonomous Underwater Vehicles (AUVs) for surveying, it is impossible to rely on human supervision to control the lighting system. Therefore, existing UAVs heavily depend on sonar sensors, which limits the availability of high-resolution data when navigating in complex environments.

[0005] The inventors have designed a new lighting control system and method that enables efficient real-time camera-based surveying. Summary of the Invention

[0006] Underwater imaging heavily relies on discrete illumination due to the lack of ambient light at depths. The underwater depth can be arbitrary, as light begins to be absorbed once it enters the water; or the surveying equipment may be photographing objects in shadow, such as beneath a cantilever. For example, in clear tropical waters, only about 1% of visible light (primarily in the blue range) can penetrate to a depth of 100 meters. In turbid conditions, the situation can be even worse. Controlling discrete illumination for optimal imaging is challenging because water is an interacting medium for light, and excessive artificial illumination can lead to backscattering, reducing image sharpness and making it more difficult to capture images and videos of high dynamic range scenes. Furthermore, in other environments, poor lighting hinders proper assessment by surveying equipment. Subsurface environments are one such example, requiring artificial illumination and often containing light-interacting particles that cause backscattering. Therefore, closed-loop lighting control is a critical component when deploying mobile robotic vehicles with visual imaging systems, as poor lighting significantly reduces the value of any visual data.

[0007] By way of non-limiting overview, embodiments of the invention relate to a system comprising a combination of hardware and software elements. In its minimal form, the hardware system may consist of a camera and multiple controllable lights, both attached to a computer. All hardware elements may be housed within a pressure vessel to withstand underwater operating conditions. The software elements may include: an image processing module that uses semantic understanding of the scene to evaluate lighting; and a control process that can adjust the lights to improve lighting. For example, a PID controller may be used to adjust the output of the lighting system—reducing the output when a metric indicates overexposure of a semantic region of interest (ROI). The lights in the system may be steerable. One example may be one or more arrays consisting of multiple individually controllable lighting elements (e.g., LEDs). The multiple lighting elements in the array may be arranged such that their normals are not parallel.

[0008] The calibration process, performed in low-light environments (which can be a one-time calibration), establishes a correlation between the illumination of camera pixels and the adjustment of individual illumination elements. Using this correlation mapping, illumination elements that do not illuminate the ROI (as indicated by semantic scene understanding) can be darkened to reduce unwanted backscattering and overexposure.

[0009] Therefore, this system can achieve the following advantages: - High-quality data is captured by ensuring that scene lighting meets quantitative standards. By introducing a scene understanding component, the system can adjust the lighting to prioritize capturing details in the most important parts of the image.

[0010] - This system can be used in autonomous vehicles where lighting control cannot be manually intervened.

[0011] - This system can control multiple lighting variables without adding extra cognitive burden to the robot vehicle operator.

[0012] According to a first aspect of the invention, a surveying system is provided. This surveying system can be arranged to perform surveys in an environment. The environment may optionally be a harsh environment, such as a seabed environment. The surveying system for capturing digital images in an environment includes: A digital camera and a light array, arranged in a spatially known relationship, are mounted on a mobile device. The camera has a field of view that defines the imaging space within the environment. The lamp array includes one or more independently operable lamp elements, which are oriented to illuminate different portions of the imaging space; Storage device, the lookup table stores lookup tables that associate regions of an image captured by a camera with elements of a light array oriented to illuminate those regions; and The data processor, which is communicatively connected to the camera and light array as well as the storage device, is configured to: Evaluate the illumination metric of the pixels corresponding to the region of interest within the image; and Based on the evaluation of illumination metrics, lighting control signals are provided to the lamp array to individually adjust the elements of the lamp array based on a lookup table, thereby improving the visibility of the region of interest to the camera.

[0013] Therefore, the surveying system according to the first aspect of the invention includes closed-loop illumination control that can automatically adjust the illumination to produce optimal conditions for its imaging system. Unlike existing systems, the imaging hardware and illumination hardware are integrated.

[0014] The data processor can be configured to select regions of interest within an image.

[0015] The data processor can be configured to repeatedly evaluate steps and lighting control steps in a PID loop to change one or more of the following: the position or direction of the light emitted from the light array; the wavelength of the light emitted from the light array; or the intensity or brightness of the light emitted from the light array.

[0016] The survey system may also include a distance sensor, wherein a lookup table defines the relationship between a three-dimensional voxel grid corresponding to the imaging space and elements oriented to illuminate each voxel.

[0017] The distance sensor can be another camera that forms a stereo pair with the camera.

[0018] Each lamp element may have its own lighting axis, and some or all lamp elements may be mounted with non-parallel lighting axes.

[0019] Lamp components may include LEDs.

[0020] The survey system may also include a mobile device on which digital cameras and light arrays are mounted in a spatial relationship known to each other.

[0021] The data processor and / or storage device can be mounted on the mobile device. The digital camera and light array can be mounted within one or more housings to protect them from environmental influences.

[0022] The mobile device may include an ROV or an AUV, and the one or more housings are watertight pressure vessels.

[0023] The system may also include a Simultaneous Localization and Mapping (SLAM) algorithm based on data received from cameras and / or distance sensors, as well as pose data. Each of the cameras and / or distance sensors may have a pose relative to the mobile device. This pose is a combination of the position and orientation of the respective camera, distance sensor, and / or mobile device in the environment. The SLAM algorithm can use images to generate pose data. Alternatively or in combination, the pose sensor may include one or more other sensors (such as gyroscopes or accelerometers) to assist in generating the pose data.

[0024] 3D survey data can include 3D model components generated by 3D reconstruction algorithms. Therefore, mobile device operators do not need to be experts on the surveyed object or site to know how to use the mobile device for inspection.

[0025] Mobile devices can include remotely controlled or autonomous mobile platforms, such as remotely operated underwater vehicles (ROVs), autonomous underwater vehicles (AUVs), unmanned aerial vehicles (UAVs), unmanned ground vehicles (UGVs), unmanned underwater vehicles (UUVs), autonomous surface vehicles (ASVs), space probes, or satellites.

[0026] In one example, the mobile device may include a remotely controlled or autonomous vehicle comprising a propulsion system, a steering system, and a command controller arranged to control the propulsion and steering systems based on command signals provided from a control station or data processor remote from the vehicle. Optionally, the remotely controlled or autonomous vehicle may be an underwater remotely operated or autonomous vehicle.

[0027] The mobile device may also include a steering and propulsion module communicatively connected to a data processor configured to run a navigation algorithm for controlling the steering and propulsion of the mobile device. Thus, the navigation algorithm can control the position of the mobile device within the survey site. Advantageously, the mobile device can be controlled to compensate for drift and make navigation choices, thereby enabling the development of the most suitable model for a specific survey along the most efficient or fastest path, balancing detail and speed.

[0028] The navigation algorithm can be configured to control the steering and propulsion of the mobile device based on control commands received from a remote server via a first data transceiver. Advantageously, if corrections or further investigation are required, the remote server can instruct the navigation algorithm to correct the navigation. Surveying experts can also instruct the navigation algorithm to correct the navigation if they wish. Therefore, the navigation algorithm can choose to correct the navigation. Alternatively, control commands received from the remote server can override the navigation algorithm.

[0029] The optional housing may be a pressure vessel (e.g., a watertight pressure vessel) configured to withstand the pressure difference between its exterior and interior, or filled with pressure-compensating gas or fluid, thereby enabling continuous use of the camera module in harsh underwater environments (e.g., in at least two meters, five meters, 10 meters, 20 meters, 40 meters, or 100 meters of water). In other embodiments, the housing may be configured to enable continuous use of the camera module in other harsh environments (e.g., vacuum).

[0030] A data transceiver may include a wireless base station, a wired data link, or any other type of transceiver known in the art. Alternatively, a data transceiver may include separate transmitters and receivers. The data transceiver may be full-duplex or half-duplex. If the data transceiver is wireless, it may include one or more antennas capable of operating at similar or different frequencies.

[0031] Various aspects of this invention can be applied to communication when autonomous survey vehicles collaborate to survey hazardous sites. Therefore, the system can include multiple mobile devices according to the first aspect, each mobile device incrementally sending processed data to a server and / or each (partially or entirely) of the multiple mobile devices. The multiple mobile devices may alternatively send model parameters, etc., to enable collaborative site surveying. The multiple mobile devices can run the same or different 3D reconstruction algorithms. The server can receive processed data from all mobile devices to develop server-side 3D models and visualizations. The server can also transmit updates generated by algorithms on the server.

[0032] According to a second aspect of the present invention, a method for surveying the environment using a surveying system according to any of the foregoing aspects is provided, the method comprising: Evaluate the illumination metric of the pixels corresponding to the region of interest within the image; and Based on the evaluation of illumination metrics, lighting control signals are provided to the lamp array to individually adjust the elements of the lamp array based on a lookup table, thereby improving the visibility of the region of interest to the camera.

[0033] The method may also include: repeating the evaluation step and the lighting control step in the PID loop to change one or more of the following in order to minimize the error value: the position or direction of the light emitted from the lamp array; the wavelength of the light emitted from the lamp array; or the intensity or brightness of the light emitted from the lamp array.

[0034] The optional features of the first aspect can be applied to the second aspect in a similar manner. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a mobile device according to an embodiment of the present invention; Figure 2 yes Figure 1 A perspective view of the camera module of the mobile device; Figure 3a It is an axonometric view of the camera and its voxelized field of view, which defines the imaging space; Figure 3b yes Figure 3a A top view of the system, with the addition of a light array; Figure 4 A method for capturing images and adjusting the lamp elements of a lamp array is demonstrated; Figure 5 An image of a pipe is shown, which exhibits both pipe characteristics and damage features. Detailed Implementation

[0036] refer to Figure 1 According to embodiments of the present invention, a mobile device for surveying harsh environments is generally shown as 10. The mobile device 10 is arranged for deployment in harsh environments, such as underwater at depths of less than two meters, on the seabed, or in the vacuum of space, for continuous use, and may include remotely operated vehicles (ROVs), autonomous vehicles (AVs) (such as autonomous underwater vehicles (AUVs)), robotic platforms, etc.

[0037] Harsh environments place stringent demands on camera modules, including mechanical stress, radiation stress, chemical reactions, and temperature exposure. Mechanical stress can include pressure, shock, vibration, and thermal expansion coefficient (CTE) mismatch. Radiation stress can include cosmic rays and magnetic effects from electromagnetic fields. Chemicals such as salt water, moisture, fuels, and toxic gases can react with such camera modules. Harsh environments may also expose them to temperatures below -40°C and / or above 125°C.

[0038] The mobile device 10 includes a computing device 12 mounted on a remotely controlled or autonomous vehicle 14 (e.g., a mobile robotic vehicle). See below for reference. Figure 2 In more detail, the computing device 10 is housed within a protective housing 16 so that it can operate in harsh environments, such as the seabed.

[0039] The mobile device 10 includes a lamp array 19. The lamp array 19 can be connected to the exterior or interior of the protective housing 16, and / or can be mounted to a remotely controlled or autonomous vehicle 14 (e.g., within its own protective housing or without a protective housing). The lamp array 19 can include multiple independently operable lamp elements oriented to illuminate different portions of the imaging space of the camera 20. The lamp array 19 can include an array of multiple lamp elements (e.g., LEDs) arranged at different angles to each other. The lamp array 19 can allow adjustment of the power of individual lamp elements within the array 19 to allow light to be directed to different portions of the imaging space.

[0040] The computing device 12 includes a data processor 18 communicatively connected to the camera 20 to receive images from the camera 20. The camera 20 has a field of view that defines the imaging space within the survey site. The camera 20 is a digital camera capable of capturing high-resolution images or video. The camera 20 may have an internal processor for performing tasks such as setting camera parameters and synchronizing with the attached system. The data processor 18 may be connected to the camera 20 via a high-bandwidth, low-latency link.

[0041] The digital camera 20 and the lamp array 19 can be mounted on the mobile device 10 with a known spatial relationship to each other (e.g., mounted on the body of the housing 16). Because the camera 20 and the lamp array 19 have a known spatial relationship, the data processor 18 can perform a calibration process that maps the lamp elements of the lamp array 19 to the imaging space of the camera 20, which can produce light mapping.

[0042] The data processor 18 can be connected to the light array 19 via a wired or wireless data link to provide control signals.

[0043] Data processor 18 can be connected to other devices, such as a network interface (not shown) and volatile and non-volatile memory 22. Memory 22 can store images, video, light maps (which associate regions of an image captured by a digital camera with elements of an array of lights oriented to illuminate those regions), or metadata, as well as algorithms. Data processor 18 can be adapted to compute complex image processing and computer vision algorithms. For example, the data processor may include processing cores that include (but are not required to be) GPU cores and embedded processing for video codecs or AI operations, such as an Nvidia Tegra™ system-on-a-chip (SoC). Data processor 18 may include one or more communicatively connected cores and / or different processors, i.e., it can be a distributed system.

[0044] Data processor 18 is configured to analyze the content of images or videos generated by camera 20. Data processor 18 allows for the automatic identification of regions of interest (ROIs) and the evaluation of the lighting of these ROIs. Based on the evaluation of the ROI lighting, processor 18 provides control signals to lamp array 19.

[0045] The mobile device can be autonomous, such as an AUV. In the AUV, the data processor 18 can be configured to execute a 3D reconstruction algorithm to build a 3D model. To generate a 3D model from video or camera images, the 3D reconstruction algorithm can acquire, for example, keypoints that constitute the edges of objects in the images and associate these keypoints with corresponding keypoints in other images (e.g., previously captured images or images captured simultaneously by other cameras). Knowing the camera pose associated with each image, rays can be projected from the camera position through the keypoints of each image, where the intersection of rays at a particular point in 3D space, or the best-fit point, represents the 3D position of that corresponding point in the 3D model. The 3D reconstruction algorithm can be a structure from motion or any other suitable technique for generating a 3D model from video or camera images. The 3D model can, for example, include a point cloud. The 3D reconstruction algorithm can build a 3D model starting from a pair of images, or it can use the images to enhance a 3D model already being built by the data processor.

[0046] Optional distance sensor 25 may be, for example, a multibeam echo sounder (MBES), or a stereo camera arrangement including camera 20 and additional cameras. Distance sensor 25 generates distance data that can be used to identify the distance between features and distance sensor 25. Distance sensor 25 may have a known spatial arrangement / relationship relative to camera 20. For example, this spatial arrangement / relationship may be predetermined or calculated. Data from distance sensor 25 can be used to create or add to a 3D model generated by a 3D reconstruction algorithm.

[0047] To create an accurate 3D model, a 3D reconstruction algorithm requires multiple input images. In other words, to model a 3D object, multiple images taken around the object from different angles help build a more reliable 3D model of the object. If two images are taken sequentially and camera 20 has a high capture rate, the two images are likely to be very similar, and the received later image adds little information relative to the previous image. A “keyframe” can be defined by the degree of difference between two different views of an object. The 3D reconstruction algorithm can be arranged to compare the received new image with the previous image (or an earlier image) to determine the degree of difference. Due to the movement of the object and / or camera 20, the new image may contain the object from a different viewpoint relative to the previous image. Once the difference threshold is exceeded, the received image is marked as a keyframe. In other embodiments, keyframes can be identified by a data processor and / or other means different from data processor 18, such as waiting for a fixed keyframe interval (e.g., 1 second) or selecting every nth frame. Alternatively, an inertial measurement unit can be used to determine whether the camera position has moved enough to initiate a keyframe.

[0048] The data processor 18 can provide synchronization of visual SLAM data and distance data (e.g., MBES scans from acoustic sensors, or via stereo camera setup). This allows for the real-time construction of a single, combined, and optimized 3D reconstruction composed of measurements acquired by both visual and distance sensors. Furthermore, for resource-constrained computer systems used in harsh environments such as underwater or seabed, this system design allows for efficient utilization of available processing power. As data is recorded (SLAM model, keyframes, distance data, and sensor extrinsic parameters), a 3D model can be computed by applying appropriate transformations based on the one-to-one correspondence between SLAM and distance data, and known sensor geometry.

[0049] 3D reconstruction algorithms typically process all input images (i.e., the output of camera 20 determines the processing power of data processor 18) and select keyframes to add points that fit well with the model being built. Points from the keyframes are connected to form a point cloud, which can be processed into a mesh and rendered as solid objects using computer graphics. Each time a keyframe is added, the camera view is typically a unique view of the model. Unique views are well-suited as input to object detection algorithms because a complete set of keyframes provides all the views of the object needed to create a complete model. The object detection algorithm can run on mobile device 10 or remotely (e.g., on a server). Therefore, mobile device 10 runs 3D reconstruction algorithms to generate 3D survey data (e.g., point cloud or keyframe recognition) based on images captured by camera 20.

[0050] The computing device 12 may also include a pose sensor 24 arranged to determine pose data characterizing the pose of the mobile device. Pose is a combination of the mobile device's position and orientation in its environment. In the illustrated embodiment, the pose sensor 24 includes a simultaneous localization and mapping (SLAM) module (which may run on the data processor 18) that uses images from a camera to determine the pose based on features in the images. Furthermore, keyframes and pose data can be used to generate a visual SLAM map. Alternatively or in combination, the pose sensor may include other sensors (e.g., a gyroscope or accelerometer) to assist in generating the pose data.

[0051] The mobile device can carry an airborne sensor suite. The rigidly attached sensors include at least a camera system 20 and a light array 19. The vehicle can be used for surveying seabed areas or asset facilities.

[0052] The computing device 12 also includes a data transceiver 26, which may include, for example, a wireless base station. The data processor 18 is configured to incrementally transmit data and pose data to a server (not shown) via the data transceiver 26. The transmitted data may be 3D survey data, low-resolution images obtained by processing corresponding input images, or may include keyframes, which may be keyframes at their original resolution or keyframes processed to reduce resolution. The transmitted data may be a combination of real-time data and post-processed data. Depending on project requirements and the quality of available data links, only a small amount of data may be transmitted in either direction. The data processor 18 may be arranged to process the 3D survey data using an encryption key before transmission. Advantageously, this encodes the 3D survey data, making it more secure / confidential. The survey mobile device and the server may exchange encryption keys, for example, before the mobile device 10 is deployed to a hazardous area, so that all survey data communications can be secure.

[0053] The computing device 12 in the illustrated embodiment also includes an optional steering and propulsion module 28, which is arranged to control the navigation of the vehicle 14. In such an embodiment, any suitable software interface can be provided to interface with the vehicle's onboard steering and propulsion system.

[0054] In the AUV, 3D survey data can be processed by the data processor 18 and used to operate the AUV for surveying by using the steering and propulsion module 28.

[0055] Now for reference Figure 2The computing device of the illustrated embodiment may include a camera module 30 for imaging in harsh environments. The camera module 30 may include a housing 16 defining a watertight enclosure with an internal space. One end of the housing 16 may include a transparent window or lens 32. A camera 20 is mounted within the internal space and arranged to capture images of the external environment through the window 32. The internal space of the housing 16 can be accessed by removing a housing end cap 16a, which is removably connected to the body of the housing 16 via an O-ring sealing flange. A seal may be provided between the housing end cap 16a and the housing body to prevent water ingress. In this embodiment, the housing 16 is made of stainless steel and is cylindrical to structurally resist the high pressures that the camera module 30 may experience in harsh underwater environments such as deep water environments and / or subsea oil and gas infrastructure sites. In other embodiments, the material and / or shape of the housing 16 (e.g., aluminum, beryllium copper, titanium, plastic, ionomer, PEKK, carbon fiber, or ceramic) may be varied depending on the deployment environment to provide stress, chemical, and / or temperature resistance. Preferably, the material and shape of housing 16 form a robust and rigid structure. The shape of housing 16 can be any pressure-resistant shape, such as prismatic or cylindrical. Camera 20 can be part of housing 16 such that housing 16 forms a fluid seal with at least a portion of camera 20 (e.g., the lens of camera 20). Other end cap seals can be formed by methods including through bolts or threaded tubes. Some or all components of computing device 12 and lamp array 19 can be housed within housing 16. Ports P1-P5 can be provided for wired connections to other camera modules, computing devices, etc., to provide an external data link when the camera module is operating in harsh environments such as the seabed.

[0056] In any embodiment, some or all of the components of the computing device may be housed within one or more respective housings, each housing defining a sealed internal space for isolating these components from underwater or other harsh environments, such as those described in WO2020094498A1.

[0057] Figure 3a A 3D representation of camera 20 and its field of view 40 is shown, which defines the imaging space in a harsh environment. Figure 3b It shows Figure 3a A top-down view. Once the position of the illumination element is known, the camera frustum is voxelized up to the maximum working distance, such as 2m, 5m, 10m, or 15m (larger distances may be more suitable for non-seabed environments). Figure 3aAs shown, the field of view 40 can be represented by voxelized frustums. The field of view 40 is divided into a set of 64 voxels (e.g., 42a-42d, although the field of view can be divided into any number of voxels depending on the desired resolution). Similar to how pixels represent values ​​on a (typically) regular grid on a two-dimensional screen, voxels represent values ​​on a (typically) regular grid in three-dimensional space.

[0058] Figure 3b A lamp array 19 with two LEDs 44a, 44b as shown is also shown. Each LED 44a, 44b produces a corresponding beam 46a, 46b. Each lamp element can emit light, and the beam can represent 68% or 95% of the light emitted from the respective lamp element. The lamp elements can be oriented such that the beam can be focused into a narrow beam. Narrow beams (for the same total illuminance of the lamp elements) may be advantageous because they penetrate further than wider beams and are less susceptible to problematic / undesirable backscattering.

[0059] It is possible Figure 3b As seen, beams 46a and 46b illuminate different voxels to varying degrees. For example, beam 46a illuminates a portion of voxels 42a and 42b, while beam 46b illuminates a portion of voxels 42d and 42c. Therefore, there is a relationship between the illumination of the voxels and the lamp elements of the lamp array 19.

[0060] Variations in beam angle can reduce problematic / undesirable backscattering effects in the seabed environment. Therefore, the lamp elements of lamp array 19 can be mounted such that the lamp elements are oriented in multiple axial directions. For example... Figure 3b As shown, LED 44a is tilted at an angle of 'a' degrees, and LED 44b is tilted at an angle of 'b' degrees, where 'a' and 'b' are different. Therefore, Figure 3b The diagram shows beams 46a and 46b diverging (although beams can converge and cross). Beams 46a and 46b in... Figure 3b The light is shown as an angle variation in the xz plane; however, to further increase the diversity of light, the lamp elements can be arranged such that their respective beams can also vary along one or more 3D axes x, y, z.

[0061] The light map stored in memory 22 associates regions of the imaging space (e.g., images captured by digital camera 20) with elements of a lamp array oriented to illuminate those regions. This light map can be used to project the lamp elements into a voxelized frustum space, such that each voxel contains a list of lamp elements illuminating that voxel. In other words, the light map records the relationship between voxels (e.g., voxelized frustums) of the imaging space and the lamp elements of lamp array 19. For example, each voxel can be assigned a weight corresponding to the contribution of the respective lamp element. For example, for voxel K, the weight of the lamp element can be estimated as: (1) Where m is the cosine of the angle between the normal vector of the lighting element (e.g., 48a, 48b) and the light rays passing through voxel K, and V z This is the number of voxels that the light ray passes through from the lamp element to voxel K. For example, the weight of lamp element 44a to voxel 42a is 1 / 4 because the normal vector of the lamp element 48a is equal to the light ray 50 (therefore m=1), and the light ray 50 passes through 4 voxels to reach voxel K.

[0062] The normal vector of each lighting element (e.g., 48a, 48b) can be defined as the lighting axis. Therefore, some or all of the lamp elements are mounted on the lamp array 19 with non-parallel lighting axes.

[0063] The light map can be known to the survey system before the mobile device is deployed. If the poses of camera 20 and lamp array 19 are known and remain constant, the light map can be determined as part of a one-time calibration process. If the relative pose between camera 20 and lamp array 19 (or the lamp elements of the lamp array) changes, the light map can be recalculated, or a transformation can be added to the light map to compensate for the change. One such procedure for establishing the light map is described in “A Light Source Calibration Technique for Multi-Camera Inspection Apparatus” by Pistellato, M et al., published in 2022, in the proceedings of the 11th International Conference on Applications and Methods of Pattern Recognition (ICPRAM) (SciTePress, pp. 488-494, DOI 10.5220 / 0010995900003122).

[0064] Figure 4 A method for capturing images according to an embodiment of the present invention is illustrated in 60.

[0065] In step 62, a digital camera 20 with a field of view that defines the imaging space within the harsh environment is configured to capture image data (e.g., a single image, a series of images, or video). The image data can be used for SLAM algorithms, generating 3D reconstructions, and / or for navigation purposes. Figure 5 An example of image 63 is shown. Figure 5 An image of pipe 78 is shown, which has characteristic features 80 and damage features 82 of pipe 78.

[0066] In step 64, the image data is processed (e.g., by data processor 18) to identify / select Regions of Interest (ROIs) within the image data. ROI identification can be performed manually or using an artificial neural network for image segmentation (such as Mask-RCNN). The image processing software can run at or near the camera frame rate to enable the system to adapt to changing lighting conditions and / or identify keyframes. Data processor 18 can send control signals to the lighting system to improve the illumination of the ROIs. If an image segmentation algorithm is used, the output can be labels for objects or areas within the survey site. These semantic labels can be used at step 66 and / or at step 72 (if distance data is available). Furthermore, semantic labels can be used for SLAM algorithms, generating 3D reconstructions, and / or for navigation purposes. Figure 5 Example images are shown, where ROIs, such as pipe 78 (and / or characteristic feature 80, and / or damage feature 82), are identified by processing image data.

[0067] In step 66, after selecting a Region of Interest (ROI) within the digital image, an illumination metric for the pixels within the image corresponding to that ROI is evaluated. The illumination metric can be illuminance: a metric used to measure the intensity of light within a space. Illuminance is measured in foot-candles or lux—it is the amount of light (lumens) falling on a surface (any given square foot or square meter). One way to evaluate the illumination metric is to look at the proportion of high-illuminance pixels (i.e., pixels at or near saturation) within the ROI and compare it to the proportion outside the ROI. This evaluation can be part of a lighting control algorithm (e.g., a PID loop), for example, the evaluation can be used to determine the error value that the lighting control algorithm is trying to reduce. The data processor can be configured to repeat the evaluation step and the lighting control step in the PID loop to change one or more of the following: position or direction; wavelength; or intensity or luminance of the light emitted from the lamp array.

[0068] In step 68, a lighting control algorithm can be used to provide lighting control signals to the light array 19 to individually adjust the elements of the light array 19, thereby improving the visibility of the ROI to the camera 20, for example, maintaining the optimal illumination range achievable using lighting metrics. This can reduce overexposure of objects / regions of interest when imaging against a dark background—a common situation in deep-water inspections. Depending on the content of the image, contrast can simply serve as a proxy for understanding how much the object / ROI is illuminated. If the data processor 18 can estimate the scene distance (e.g., in the case of a stereo camera system), this additional distance data can inform lighting adjustment by using different sets of PID parameters for different average scene distances, thus achieving improved lighting quality conditions more quickly. For example, when the average scene distance is large, the PID saturation value can be set higher.

[0069] In step 70, if additional distance data is provided, light mapping allows the ROI identified by the segmentation algorithm to be projected onto a voxelized frustum, and quickly identifies which lamp elements of the lamp array 19 will illuminate that ROI. As previously described, the list of lamp elements for each voxel can contain weights approximating the amount of illumination contributed by each illumination element to that voxel. For example, a lookup table can define the relationship between a three-dimensional voxel grid corresponding to the imaging space and the elements of the lamp array 19 oriented to illuminate each voxel. If no additional distance data is provided, 2D light mapping can still be used based on an estimated or predetermined distance between the camera and objects in the survey site.

[0070] In an optional step 72, distance sensor 25 is provided to determine an estimated distance between camera 20 and the segmented object of interest / ROI in the survey site. Distance sensor 25 provides additional distance data, which can also be used by SLAM algorithms to generate 3D reconstructions and / or for navigation purposes.

[0071] In step 74, the data processor 18, including the lighting control algorithm, generates lighting control signals to the light array 19 to individually adjust the elements of the light array 19, thereby improving the visibility of the ROI to the camera 20 and improving the usability of the ROI when the data processor 18 builds a 3D model and / or navigates. The lighting control algorithm may be based on an evaluation of illumination metrics and / or light mapping.

[0072] In step 76, the light array 19 is operated by a light control signal to illuminate the ROI and the survey site. Optionally, the image capture method according to an embodiment of the invention (generally shown as 60) is iteratively repeated, either entirely or partially (e.g., repeating the steps of: evaluating an illumination metric for the pixels corresponding to the ROI within the image; and, based on the evaluation of the illumination metric, providing a light control signal to the light array 19 to individually adjust the elements of the light array based on a lookup table, thereby improving the visibility of the ROI to the camera), to optimally adjust the illumination from the light array 19, thereby improving imaging performance. If keyframes are identified, the image capture method according to an embodiment of the invention (generally shown as 60) can be repeated more frequently to ensure that optimal lighting conditions are applied to improve the feature data extracted from the keyframes for use by optional 3D reconstruction algorithms.

[0073] In any example, a Region of Interest (ROI) can be a large object or set of objects in the foreground of an image (e.g., pipe 78), or it can be a specific sub-part of an object or set of objects (e.g., feature 80 and / or damage feature 82 of pipe 78). Typically, an ROI can include discontinuous regions. ROIs can include non-binary memberships; for example, some regions may be assigned high priority, some medium priority, and others low priority (this may depend on the ROI algorithm used).

[0074] The mobile devices, systems, and methods according to embodiments of the present invention can be used in marine and seabed surveying, but can also be applied to any surveying system used in hazardous environments (such as space, nuclear, mining, etc.). Surveying can be conducted autonomously, semi-autonomously, manually via remote control, or manually with a minimal number of crew members to reduce risk. Dispatching crew members to hazardous locations requires safety precautions, which is inconvenient and therefore increases costs. The mobile devices can be, for example, ROVs or AUVs.

[0075] As described, the data processor 18 can be configured to repeat the evaluation step and lighting control step (66, 68, 70, 74) in a PID loop to change the intensity of each lamp element in the lamp array 19. However, it will be apparent that in any embodiment, the data processor 18 can also be configured to repeat the evaluation step and lighting control step in a PID loop to change one or more of the following: the position or direction of light emitted from the lamp array 19; the wavelength of light emitted from the lamp array 19; or the intensity or brightness of light emitted from the lamp array 19. For example, the wavelength can be changed by altering the illumination intensity between red, green, and / or blue lamp elements. The lamp elements and / or lamp array 19 used in this system can be steerable and can be adjusted according to... Figure 4 The illumination can be controlled in various ways. For example, some or all of the lamp elements can be arranged in different directions (as described above). Alternatively or additionally, some or all of the lamp elements can be connected to a steering device. Alternatively or additionally, some or all of the lamp elements can be associated with a corresponding optical element (e.g., a lens array located on top of the corresponding lamp element). This optical element can be configured to guide light from the corresponding lamp element and arranged to direct the illumination in different directions.

[0076] In one embodiment, the lamp array may include multiple independently operable lamp elements. Alternatively, the lamp array may include only a single steerable lighting element. This lighting element may be steered by mechanical, optical, or optomechanical means.

[0077] In any embodiment, the data processor 18 and / or storage device 22 may be mounted on the mobile device 10, and the data processor 18 and / or storage device 22 may be mounted within one or more housings 16 to protect the data processor 18 and / or storage device 22 from harsh environments.

[0078] In any embodiment, the system described herein may use a distance sensor to provide distance data to utilize 3D voxel light mapping. Alternatively, if a distance sensor is not present, some advantages of the system described herein may still be apparent, and light mapping may simply map the lamp elements onto a 2D pixel plane.

[0079] In any embodiment, the data processor and / or memory may be remote from the mobile device, such that images are transmitted from the mobile device 10, while the light control signals may be received by the mobile device 10.

[0080] In any embodiment, the camera may alternatively be supplemented by any means (i.e., imaging device) capable of imaging the area. Thus, the imaging device may be one or more cameras, sonar devices, or laser scanning devices, or alternatively a combination of such cameras, sonar devices, and / or laser scanning devices.

[0081] Embodiments of the present invention are extended to survey systems arranged as described above and mounted on mobile devices.

[0082] Although the invention has been described above with reference to one or more preferred embodiments, it should be understood that various changes or modifications may be made without departing from the scope of the invention as defined by the appended claims. The word "comprising" can mean "including" or "consisting of," and therefore does not exclude the presence of other elements or steps besides those listed in any claim or throughout the specification. The mere listing of certain measures in mutually different dependent claims does not imply that combinations of these measures cannot be advantageously used.

Claims

1. A surveying system for capturing digital images in an environment, the system comprising: A digital camera and a light array, arranged in a known spatial relationship and mounted on the mobile device. The camera has a field of view that defines the imaging space within the environment. The lamp array includes one or more independently operable lamp elements, which are oriented to illuminate different portions of the imaging space; A storage device that stores a lookup table that associates regions of an image captured by the camera with elements of the lamp array oriented to illuminate those regions; as well as A data processor, communicatively connected to the camera and light array and the storage device, is configured to: Evaluate the illumination metric of the pixels corresponding to the region of interest within the image; as well as Based on the evaluation of the illumination metrics, lighting control signals are provided to the light array to individually adjust the elements of the light array based on the lookup table, thereby improving the visibility of the region of interest to the camera.

2. The surveying system according to claim 1, wherein, The data processor is configured to select the region of interest within the image.

3. The surveying system according to any one of the preceding claims, wherein, The data processor is configured to repeatedly evaluate and control the lighting in a PID loop to change one or more of the following: the position or direction of the light emitted from the light array; the wavelength of the light emitted from the light array; or the intensity or brightness of the light emitted from the light array.

4. The surveying system according to any one of the preceding claims further includes a distance sensor, wherein, The lookup table defines the relationship between the three-dimensional voxel grid corresponding to the imaging space and the elements of the lamp array oriented to illuminate each voxel.

5. The surveying system according to claim 4, wherein, The distance sensor is another camera that forms a stereo pair with the camera.

6. The surveying system according to any one of the preceding claims, wherein, Each of the lamp elements has an illumination axis, and some or all of the lamp elements are mounted with non-parallel illumination axes.

7. The surveying system according to any one of the preceding claims, wherein, The lamp element is an LED.

8. The surveying system according to any one of the preceding claims further includes a mobile device, wherein, The digital camera and the light array are mounted on the mobile device in a spatially known relationship to each other.

9. The surveying system according to claim 8, wherein, The data processor and / or storage device are mounted on the mobile device, wherein the digital camera and the light array are mounted in one or more housings to protect the digital camera and the light array from the environmental influences, and the data processor and / or storage device are mounted in one or more housings to protect the data processor and / or storage device from the environmental influences.

10. The surveying system according to any one of the preceding claims, wherein, The mobile device includes an ROV or AUV, and the one or more housings are watertight pressure vessels.

11. A method for surveying an environment using a surveying system according to any one of the preceding claims, the method comprising: Evaluate the illumination metric of the pixels corresponding to the region of interest within the image; as well as Based on the evaluation of the illumination metrics, lighting control signals are provided to the light array to individually adjust the elements of the light array based on the lookup table, thereby improving the visibility of the region of interest to the camera.

12. The method of claim 11, further comprising: The evaluation and lighting control steps are repeated in the PID loop to change one or more of the following to minimize the error value: the position or direction of the light emitted from the light array; the wavelength of the light emitted from the light array; or the intensity or brightness of the light emitted from the light array.

Images