A method and system for monitoring the benthic environment under the influence of a deep-sea mining plume

Abstract

The present disclosure provides a benthic environment monitoring method and system under the influence of deep-sea mining plume, relating to the technical field of deep-sea environment monitoring, comprising: obtaining in-situ three-dimensional images of the benthic environment in the deep-sea mining area; remapping the in-situ three-dimensional image data to map the pixel coordinates to three-dimensional coordinates to obtain a depth image; calculating the contrast of the depth image and the definition of the depth image, and combining the image contrast and the image definition to quantitatively monitor the turbidity of the water in the monitoring area in real time to obtain the influence period of the mining activity; based on the depth image, extracting a biological depth image, inputting the biological depth image into a depth prediction model, and outputting the predicted biological size information; the present disclosure realizes continuous and quantitative evaluation of the ecological impact of mining activities.

Description

Technical Field

[0001] This disclosure relates to the field of deep-sea environmental monitoring technology, specifically to a method and system for monitoring the environment of benthic organisms under the influence of deep-sea mining plumes. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] With the increasing global demand for rare metals and mineral resources, the development of deep-sea mineral resources has become an important direction for development. In particular, engineering pilot mining work is gradually underway in deep-sea polymetallic nodules and sulfide-rich areas. During the mining process, human activities cause damage to the benthic environment. As mining vehicles move to collect minerals from the seabed, the stirred-up sediment is carried by the vehicle's wake, forming plumes carrying large amounts of heavy metals and adhering particles. These spreading plumes have a long-term impact on the habitat of benthic organisms. Currently, data on the types, sizes, and habits of benthic organisms in deep-sea mining areas are insufficient, and effective detection methods are lacking to address the damage to the benthic environment and the impact on benthic life caused by mining activities.

[0004] Current deep-sea monitoring methods primarily utilize platforms such as manned submersibles, remotely operated vehicles (ROVs), autonomous underwater vehicles (AUVs), seabed landers, and fixed seabed observation stations. These platforms are combined with turbidity sensors, acoustic detection equipment, underwater cameras, and environmental parameter sensors to monitor mining plume dispersion, water physicochemical parameters, and the benthic ecosystem. However, while these methods can acquire environmental information within a certain range, they still have limitations. (1) Single sensors such as turbidity and flow velocity can only obtain local environmental parameters, which are difficult to accurately reflect the relationship between the plume diffusion process and the benthic organism response; (2) On the other hand, traditional two-dimensional underwater images are difficult to use for three-dimensional biological environment information statistics and size measurement.

[0005] (3) In addition, existing monitoring equipment mostly rely on external power supply, manual recovery or real-time communication, and have limited long-term autonomous operation capabilities. They are not stable and reliable enough in the complex environment of the deep sea, and cannot meet the needs of continuous, refined and multi-dimensional monitoring of the impact range, impact cycle and benthic ecological changes of plumes during deep-sea mining. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes a method and system for monitoring the benthic environment under the influence of deep-sea mining plumes. The system utilizes a binocular camera array to acquire high-resolution three-dimensional images, supplemented by water parameters to achieve comprehensive perception of the benthic environment. The monitoring system employs a deep-sea pressure-resistant sealed shell, integrating a highly reliable power system, data processing module, and large-capacity storage device, enabling long-term autonomous operation. It also supports timed wake-up and task triggering mechanisms, allowing for multi-dimensional information collection and monitoring of the deep-sea mining area's ecosystem status without external intervention.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: A method for monitoring the benthic environment under the influence of deep-sea mining plumes, comprising: Acquire in-situ 3D images of the benthic environment in deep-sea mining areas; The in-situ 3D image data is remapped to convert pixel coordinates to 3D coordinates to obtain a depth image; Calculate depth image contrast and depth image sharpness, and combine image contrast and image sharpness to quantify the turbidity of the water body in the monitoring area in real time, thereby obtaining the impact cycle of mining activities; Based on depth images, biological depth images are extracted, and the biological depth images are input into a depth prediction model to output predicted biological size information. The deep prediction model uses field-collected data and the SmoothL1 loss function to measure prediction bias and adjust model parameters accordingly. The training of the deep prediction model is completed when the difference between the biological size output by the deep prediction model and the predicted target remains below a set threshold for a long period of time.

[0008] As one embodiment, acquiring three-dimensional images of the deep-sea environment and water parameters includes: Image acquisition is performed using a deep-sea binocular imaging sensor, which includes a deep-sea binocular equipment cabin, a deep-sea LED light source cabin, and a platform assembly frame. The deep-sea binocular equipment cabin contains a power supply module, a control module, and a data acquisition module, and is protected by a titanium alloy pressure-resistant body. The titanium alloy pressure-resistant body consists of a front observation hatch, a cylindrical pressure-resistant body, and a rear communication hatch. The front observation hatch has a set of symmetrical prototype light-transmitting holes, which are covered and protected by coated sapphire. The rear communication hatch is equipped with two watertight connectors, which are responsible for powering the deep-sea LED light source cabin and communicating with the observation platform, respectively.

[0009] As one embodiment, the power supply module consists of two miniaturized voltage regulator modules. The power supply module supplies power to the control module and the data acquisition module, and supplies power to the deep-sea LED light source compartment through the watertight connector of the rear communication hatch. The power supply module is fixed to the front observation hatch through an I-shaped aluminum alloy support plate.

[0010] In one embodiment, the control module is fixed to the front observation hatch via a Type II aluminum alloy support plate; the data acquisition module consists of two identical small-sized single-board cameras, which are connected to the control module via a data interface for power supply, control, and data transmission, and are connected to the Type I and Type II aluminum alloy support plates via an aluminum alloy support disc.

[0011] As one embodiment, the remapping of three-dimensional image data, mapping pixel coordinates to three-dimensional coordinates, includes: Using a checkerboard or dot array calibration board, corner points are extracted by taking multiple images of the calibration board from different angles, and the camera's intrinsic parameter matrix and distortion matrix are determined. By matching the corner points of the left and right cameras, the rigid body transformation between them is solved, and the most suitable rotation matrix and translation vector are found by minimizing the reprojection error. By combining the intrinsic parameter matrices, distortion parameters, rotation matrices, and translation vectors of the two cameras, a remapping table for the two cameras is obtained. The left and right images are processed separately using a remapping table, and then the reprojection matrix is ​​used to complete the mapping calculation from pixel coordinates to three-dimensional coordinates.

[0012] As one example, by monitoring image contrast and the clarity and blur of the background, the relationship between water turbidity and time is quantified, and the impact cycle of mining activities on the benthic environment and organisms is analyzed by combining the time and location parameters of the mining process of the mining truck. Read the in-situ 3D image of the main camera in each group of images, calculate the average brightness of the RGB channels, and calculate the standard deviation of the gray values ​​of all pixels relative to the average brightness to measure the dispersion of the image. The clearer the water environment, the larger the dispersion value; the more turbid the water environment, the smaller the dispersion value. Simultaneously, edge detection operators are used to detect the horizontal and vertical gradient pixel maps of the main camera image, and the root mean square of the gradients in both directions is calculated.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions: A benthic organism environmental monitoring system under the influence of deep-sea mining plumes includes: The image acquisition module is used to acquire in-situ three-dimensional images of the benthic environment in deep-sea mining areas; The image processing module is used to remap in-situ 3D image data, mapping pixel coordinates to 3D coordinates to obtain a depth image; The water body sensing module is used to calculate the depth image contrast and depth image sharpness. By combining the image contrast and image sharpness, the turbidity of the water body in the monitoring area is quantified in real time to obtain the impact cycle of mining activities. The image prediction module is used to extract biological depth images based on depth images, input the biological depth images into the depth prediction model, and output the predicted biological size information. The deep prediction model uses field-collected data and the SmoothL1 loss function to measure prediction bias and adjust model parameters accordingly. The training of the deep prediction model is completed when the difference between the biological size output by the deep prediction model and the predicted target remains below a set threshold for a long period of time.

[0014] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program that, when executed by a processor, implements the aforementioned method for monitoring the benthic organism environment under the influence of deep-sea mining plumes.

[0015] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned method for monitoring the benthic organism environment under the influence of deep-sea mining plumes.

[0016] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the method for monitoring the benthic organism environment under the influence of deep-sea mining plumes.

[0017] Compared with the prior art, the beneficial effects of this disclosure are as follows: This disclosure discloses a method for monitoring the benthic environment under the influence of deep-sea mining plumes. It utilizes a multifunctional in-situ monitoring equipment system based on binocular imaging technology, capable of long-term stable operation in ultra-deep-sea environments ranging from 5000 to 6000 meters. The system integrates optical imaging, environmental parameter acquisition, data storage, and communication functions. It is applicable to resource development areas in the deep sea, including those containing polymetallic nodules, polymetallic sulfides, or cobalt-rich crusts, for simultaneous observation and data recording of benthic community structure, ecological behavior, and environmental changes. The aim is to provide scientific, continuous, and reliable data support for ecological and environmental impact assessment and ecological protection in deep-sea mining activities.

[0018] This disclosure discloses a method for monitoring the benthic environment under the influence of deep-sea mining plumes. It combines binocular stereo imaging with an in-situ monitoring system adapted to the deep-sea environment. The binocular camera array enables high-resolution three-dimensional image acquisition and comprehensive perception of the benthic environment. The binocular device adopts a deep-sea pressure-resistant sealed shell and integrates a highly reliable power system, data processing module, and large-capacity storage device, possessing long-term autonomous operation capability. It also supports timed wake-up and task triggering mechanisms, enabling the collection of multi-dimensional information on the state of the deep-sea mining ecosystem without external intervention.

[0019] This disclosure presents a method for monitoring the benthic environment under the influence of deep-sea mining plumes. It overcomes many limitations of traditional deep-sea monitoring systems in terms of spatial imaging accuracy, data integrity, and equipment survivability. This method effectively addresses the problems of scarce data and insufficient monitoring methods for benthic ecosystems in deep-sea mining environments, and has significant application value for constructing an ecological impact monitoring system before, during, and after deep-sea mining activities. Furthermore, the proposed system scheme possesses excellent modular scalability, allowing for the integration of various types of sensors or imaging units according to mission requirements. It is also suitable for use with various deep-sea observation platforms such as seabed observation stations and underwater permanent base stations.

[0020] This disclosure discloses a method for monitoring the benthic environment under the influence of deep-sea mining plumes. By fusing image dispersion and image gradient information, it achieves automatic identification of the turbidity state and impact cycle of mining plumes, enabling real-time perception of the plume diffusion process using image information without relying on turbidity detection equipment. Combining exposure compensation, inter-frame difference, and connected component analysis, it achieves stable extraction of benthic targets in complex deep-sea environments. It utilizes binocular reconstructed depth information to complete the three-dimensional size measurement of organisms, and employs a combination of traditional geometric measurement and online self-learning of CNNs, allowing the neural network to continuously optimize model parameters based on field data. This ensures measurement accuracy while reducing long-term computational load and improving adaptability to different marine environments. Furthermore, it correlates plume dynamic changes with benthic organism quantity and size changes, achieving continuous and quantitative assessment of the ecological impact of mining activities, and providing a more intelligent, reliable, and efficient data processing method for deep-sea mining environmental monitoring. Attached Figure Description

[0021] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0022] Figure 1 This is an overall assembly diagram of the deep-sea binocular sensor according to an embodiment of this disclosure; Figure 2 This is an external view of the deep-sea binocular device cabin according to an embodiment of this disclosure; Figure 3 This is a structural diagram of the deep-sea binocular equipment cabin according to an embodiment of this disclosure; Figure 4 This is an external view of the deep-sea LED light source cabin according to an embodiment of the present disclosure; Figure 5 This is a structural diagram of the deep-sea LED light source cabin according to an embodiment of the present disclosure; Figure 6 This is a flowchart illustrating a method for monitoring the environment of benthic organisms under the influence of deep-sea mining plumes, according to an embodiment of this disclosure.

[0023] The components include: 1. Deep-sea binocular equipment compartment; 2. Deep-sea LED light source compartment; 3. Platform assembly frame; 4. Type I titanium alloy pressure-resistant hull; 5. Sapphire optical window; 6. 13-core watertight connector; 7. 4-core watertight connector; 8. Embedded computer; 9. Type I aluminum alloy support plate; 10. High-performance optical camera; 11. Aluminum alloy support disk; 12. DC-DC regulated power supply; 13. Type II aluminum alloy support disk; 14. Type II titanium alloy pressure-resistant hull; and 15. White LED light source. Detailed Implementation

[0024] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0027] Example 1 One embodiment of this disclosure provides a method for monitoring the benthic organism environment under the influence of deep-sea mining plumes, the method comprising the following steps: Step 1: Obtain in-situ 3D images of the benthic environment in the deep-sea mining area; Step 2: Remap the in-situ 3D image data, converting pixel coordinates to 3D coordinates to obtain a depth image; Step 3: Calculate the depth image contrast and depth image sharpness, and combine the image contrast and image sharpness to quantify the turbidity of the water in the monitoring area in real time, so as to obtain the impact cycle of mining activities; Step 4: Based on the depth image, extract the biological depth image, input the biological depth image into the depth prediction model, and output the predicted biological size information; The deep prediction model uses field-collected data and the SmoothL1 loss function to measure prediction bias and adjust model parameters accordingly. The training of the deep prediction model is completed when the difference between the biological size output by the deep prediction model and the predicted target remains below a set threshold for a long period of time.

[0028] As one embodiment, this disclosure discloses a method for monitoring the benthic environment under the influence of deep-sea mining plumes. It achieves high-resolution three-dimensional image acquisition through a binocular camera array, enabling comprehensive perception of the benthic environment. The system employs a deep-sea pressure-resistant sealed shell, integrating a highly reliable power system, data processing module, and large-capacity storage device, possessing long-term autonomous operation capabilities. It also supports timed wake-up and task triggering mechanisms, enabling multi-dimensional information collection on the state of the deep-sea mining area ecosystem without external intervention. The specific implementation process is as follows: Step 1: Obtain in-situ 3D images of the benthic environment in the deep-sea mining area; Specifically, this disclosure employs a deep-sea binocular imaging sensor for data acquisition. For example... Figure 1 As shown, the deep-sea binocular imaging sensor includes a deep-sea binocular equipment compartment, a deep-sea LED light source compartment, and a platform assembly frame. The deep-sea binocular equipment compartment and the deep-sea LED light source compartment are placed horizontally, with the deep-sea LED light source compartment located above the deep-sea binocular equipment compartment and fixed to the platform assembly frame by mechanical connection.

[0029] Furthermore, the external structure of the deep-sea binocular equipment cabin is as follows: Figure 2 As shown, the deep-sea binocular equipment compartment consists of a Type I titanium alloy pressure hull, front and rear circular titanium alloy pressure hull covers, a 13-core watertight connector and a 4-core watertight connector.

[0030] Furthermore, the internal structure of the deep-sea binocular equipment cabin is as follows: Figure 3 As shown, it consists of a Type I aluminum alloy support plate, a Type II aluminum alloy support disk, an embedded computer, two high-performance optical cameras, and two DC-DC regulated power supplies. The two high-performance optical cameras are placed parallel to each other, with the center of each camera corresponding to the center of the countersunk hole in the front circular titanium alloy pressure chamber. The rear end of each camera is fixed to the aluminum alloy support disk. The embedded computer is fixed to the Type I aluminum alloy support plate and placed parallel above the high-performance optical cameras. The two DC-DC regulated power supplies are fixed to the Type II aluminum alloy support disk and placed parallel below the high-performance optical cameras. The Type I aluminum alloy support plate and the Type II aluminum alloy support disk simultaneously fix the front circular titanium alloy pressure chamber cover and the aluminum alloy support disk, forming a complete support structure.

[0031] Furthermore, the external pressure-resistant structure of the deep-sea LED light source chamber, such as... Figure 4 As shown, it consists of a Type II titanium alloy pressure-resistant chamber, a front circular window pressure platform, and a rear circular titanium alloy pressure-resistant chamber cover. The front circular window pressure platform does not need to bear external pressure and is only used to fix the sapphire optical window; the rear circular titanium alloy pressure-resistant chamber cover has a through threaded hole in the center for fixing a 4-core watertight connector; notably, the height of the deep-sea LED light source chamber and the deep-sea binocular equipment chamber does not need to be exactly the same. In the implementation scheme, the height is set to be the same only for the convenience of equipment fixation.

[0032] Furthermore, the internal structure of the deep-sea LED light source cabin is as follows: Figure 5 As shown, it includes a white LED light source, which is completely fixed in the SM1 threaded hole in the center of the Type II titanium alloy pressure tank by a metal threaded fastener. There is a concave lens module in front of the white LED light source to expand the coverage area of ​​the white LED light source. The distance between the concave lens module and the white LED light source is 2mm to 10mm.

[0033] As one example, such as Figure 1 As shown, the platform assembly frame includes one front fixing plate, one rear fixing plate, four side reinforcement plates, and one platform fixing plate. The front fixing plate uses eight M6 bolts to secure the front hatch of the deep-sea binocular equipment compartment and the deep-sea LED light source compartment; similarly, the rear fixing plate uses eight M6 bolts to secure the rear hatch of the deep-sea binocular equipment compartment and the deep-sea LED light source compartment. The platform fixing plate and the rear fixing plate are connected through-hole, and the entire device is fixed to the required observation platform using four M8 bolts. The included angle of the platform fixing plate is set to 110° to 135° according to the observation requirements. After the equipment is assembled, the four side reinforcement plates are fixed around the equipment by welding or bolting to reinforce the frame.

[0034] The deep-sea binocular imaging system primarily utilizes an in-situ detection system for benthic organisms and their environment in deep-sea mining areas. This system enables long-term, real-time, in-situ observation of benthic organisms and their environment in deep-sea mining areas. Two high-performance optical cameras inside can simultaneously sample, acquiring images of the mining area with a field of view exceeding 60° and a resolution better than 1.5mm. The observable depth of field covers an area of ​​1.5-5 meters in front of the cameras, and the object-side observation field of view exceeds 3×3 meters. An embedded computer within the system analyzes and compares all acquired data, using an MTR algorithm to filter out valid data from each round of acquisition. This data is then further processed and stored on the internal hard drive. The system is pre-programmed to operate autonomously for one hour daily, generating 100MB to 200MB of data per hour. The embedded computer includes a 1TB hard drive with approximately 950GB of usable storage, allowing the system to operate autonomously in unattended deep-sea conditions for one year.

[0035] As one embodiment, the working principle of the sensor in the deep-sea binocular device is as follows: The sensor has two operating modes: command operation and autonomous operation. It receives external power through the positive and negative terminals formed by the 1 / 12 interfaces on the 13-pin watertight connector, with a power supply voltage of 5-36V. It receives and sends control commands through the RS232 port formed by the 1 / 3 / 13 interfaces on the 13-pin watertight connector. It sends monitoring data through the RJ45 port formed by the 4-11 interfaces on the 13-pin watertight connector.

[0036] Specifically, the working steps of the sensors in the deep-sea binocular device include: Step (1): After receiving external power through the 1 / 12 interface on the 13-pin watertight connector, the sensor is powered on normally and begins preparations for monitoring. After preparations are complete, the sensor sends the command $START,OK\r\n through the RS232 port to notify the host computer that the sensor is ready and waiting to receive the working command.

[0037] Step (2): The sensor waits for the work command sent by the host computer for a period of 30 seconds. Within 30 seconds after sending the command $START,OK\r\n, if the sensor receives the command $TIME[t]\r\n through the RS232 port, it starts a monitoring task with a work period of t, and simultaneously replies with the command $TIME[t],OK\r\n through the RS232 port, indicating that the sensor has started working normally according to the host computer's requirements; if the sensor does not receive any command through the RS232 port within 30 seconds after sending the command $START,OK\r\n, it defaults to starting the autonomous monitoring mode, with the working time set to 600 seconds, and simultaneously replies with $TIME600,OK\r\n to the host computer. This working time can be modified before deploying the equipment.

[0038] Step (3): The sensor automatically decides whether to save or delete the currently collected data based on the number of organisms in the monitored image and the image capture quality. The difference between two adjacent image frames is calculated, and a preset difference threshold T is set. All pixels with a difference value exceeding the preset threshold T are set to 1, and pixels with a difference value not exceeding the preset threshold T are set to 0.

[0039]

[0040] Expand all pixels exceeding a threshold T using a rectangular structure to obtain a mask for all active creatures. Then, apply the mask... D k (x, y) Applied to images I k (x, y) In this process, the gradient formula is used to calculate the square root of the gradient in the x / y direction of the current frame image, thus obtaining the gradient map of the current frame.

[0041]

[0042] Step (4) The sensor feeds back monitoring data to the host computer. During the normal working cycle of the sensor, the host computer can send a data retrieval command $GETDATA\r\n through the RS232 port. After receiving the command, the sensor checks the data stored in the database. If there is data that meets the criteria, it selects the best one and sends it to the host computer through the RJ45 port. The host computer should open the port in advance to wait for data reception. After the data is sent, the sensor sends the command $GETDATA,OK\r\n through the RS232 port to indicate that the data has been sent. When the sensor receives the command $GETDATA\r\n and finds that there is no data that meets the standard in the database, it will send the command $GETDATA,NONE\r\n through the RS232 port to indicate that there is no data that meets the standard in the current time. The RJ45 port does not send any data, and this round of data feedback ends. The host computer can continue to retrieve monitoring data after a period of time.

[0043] Step (5): Work completed, sensor powered off. After the designated working time, the sensor will automatically shut down its monitoring function and send the $CPLT,OK\r\n command to the host computer via the RS232 port, simultaneously performing a power-off operation and waiting to be activated again. If an emergency shutdown is required within the current working cycle, the host computer will send the $STOP\r\n command via the RS232 port. Upon receiving this command, the sensor will skip the timer, save all current data, and directly perform the power-off operation, while simultaneously sending the $CPLT,OK\r\n command to the host computer.

[0044] Step 2: Remap the 3D image data, converting pixel coordinates into 3D coordinates.

[0045] Specifically, the purpose of deep-sea binocular sensor calibration is to determine the intrinsic parameters, distortion parameters, and relative attitude between the two cameras for accurate 3D reconstruction. Using a checkerboard or dot array calibration board, corner points are extracted by taking multiple images of the calibration board from different angles to determine the camera's intrinsic parameter matrix and distortion matrix.

[0046] The camera's intrinsic parameter matrix and distortion parameters are:

[0047]

[0048] in, f x and f y It refers to the focal length in the horizontal and vertical directions. cx and c y These are the center coordinates of the two cameras. This represents the first-order radial distortion, second-order radial distortion, and third-order image distortion of the camera. To represent the tangential distortion of the camera, radial distortion correction, tangential x-axis distortion correction, and tangential y-axis distortion correction for a single image are performed using the following methods:

[0049] Then, by matching the corner points of the left and right cameras, the rigid body transformation between them is solved, and the most suitable rotation matrix is ​​found by minimizing the reprojection error. R Translation vector T , so that:

[0050] By combining the intrinsic parameter matrices R1, R2, distortion parameters D1, D2, rotation matrix R, and translation vector T of the two cameras, the remapping table R1, R2, P1, P2, and Q of the two cameras can be obtained. R1 and R2 are the rotation correction matrices of the left and right cameras, P1 and P2 are the projection matrices of the left and right cameras, and Q is the reprojection matrix of the two cameras, which is used for parallax to 3D coordinate reconstruction.

[0051] Furthermore, after data acquisition, the remapping tables R1, R2, P1, and P2 obtained during camera calibration are used to process the left and right images respectively:

[0052] Then, the reprojection matrix Q is used to complete the mapping calculation from pixel coordinates (u, v) to 3D coordinates (X, Y, Z) to obtain the depth image: .

[0053] Step 3: Calculate the depth image contrast and depth image sharpness, and combine the image contrast and image sharpness to quantify the turbidity of the water in the monitoring area in real time, so as to obtain the impact cycle of mining activities; Specifically, read the in-situ 3D image of the main camera in each group of images, and calculate the average brightness of the RGB channels using the following set of formulas:

[0054] The dispersion r of the image is measured by calculating the standard deviation of all pixel grayscale values ​​relative to the average brightness.

[0055] The dispersion value r is a numerical value that evaluates the clarity of the water environment by the image contrast. The clearer the water environment, the larger the dispersion value r of the image set; the more turbid the water environment, the smaller the dispersion value r of the image set.

[0056] Then calculate the average gradient value of the image, using K. x The convolution kernel is convolved with the image, and the first derivative of the image is calculated along the horizontal direction x, yielding the gradient distribution G in the horizontal direction. x :

[0057]

[0058] Using K y The convolution kernel is convolved with the image, and the first derivative of the image is calculated along the vertical direction y to obtain the gradient distribution G in the horizontal direction. y :

[0059]

[0060] Then use G x With G y The gradient magnitude matrix of the original image from the main camera is obtained. G :

[0061] G is a two-dimensional matrix with the same dimensions as the images, and it is used to obtain the sharpness score g of the set of images:

[0062] The average gradient g is a numerical value that evaluates the clarity of the water environment by the sharpness of the images. The clearer the water environment, the larger the average gradient g of the set of images; the more turbid the water environment, the smaller the average gradient g of the set of images.

[0063] By combining image contrast and image clarity, the turbidity of the monitored area can be converted into computer-readable data. During mining activities, both the dispersion r and the average gradient g decrease instantaneously, at which point the equipment captures the abnormal time point. By comparing the time when the deep-sea miner starts mining, the time required for the plume's influence to reach the equipment deployment area can be obtained; the duration of the abnormal time is the length of time the plume affects the benthic environment. This function can also monitor the start-up and shutdown / standby times of the deep-sea miner when not communicating with it.

[0064] Step 4: Based on the depth image, extract the biological depth image, input the biological depth image into the depth prediction model, and output the predicted biological size information; Specifically, the system reads the original image from the main camera and the previous frame from the main camera in each image set. A homomorphic filter is used to correct the exposure differences between adjacent frames, and the pixel differences between adjacent frames are obtained using the difference method. When no organisms pass through the monitoring area, the entire image shows only slight exposure differences, and the mask data is averaged; in this case, the currently acquired image is considered to contain only benthic environments. When an organism passes through the monitoring area, the pixel difference value of the organism's edge will be higher than that captured by the algorithm, resulting in a clear, sharp edge in the image. This sharp edge represents the edge region of the active benthic organism. A connectivity algorithm is used to penetrate the entire connected region to obtain the current location of the organism, generating a organism location mask. M ; Read the depth image from the obtained in-situ 3D images D img Apply a biological location mask M Depth data outside the biological region is removed, and only the depth data of all pixels in the biological region is retained to obtain the biological depth image. D bio .

[0065] Processing biological depth images D bio The average distance-depth from the active organism to the sensor is obtained by averaging all non-zero pixel data using the mean method. d bio , In the world coordinate system, based on the triangle conversion relationship, the average distance depth from the organism to the sensor, the pixel length occupied by the organism in the camera coordinate system, and the length corresponding to the projection of each pixel in the pixel coordinate system onto the world coordinate system are used to obtain the size information L1 of the organism. The size L of the organism is represented by the length L1 and recorded in the database along with the original image and the shooting time and saved.

[0066] Simultaneously, biological depth images D bio The data is input into a pre-trained depth prediction model, which is a CNN deep network. The output is the predicted biological size information L2. The input to this depth prediction model is a two-dimensional matrix with the same dimensions as the original image, which in this case is the input biological depth image. D bio The predicted length L2 of the output biological size information is .

[0067] This CNN network consists of 6 convolutional layers and 1 fully connected layer. The first convolutional layer has 1 input channel. The number of input channels for the second to sixth convolutional layers is set according to the actual situation. The fully connected layer then uses the Softplus activation function to output size prediction information. The activation function is as follows:

[0068] L1 is the biological size obtained by traditional algorithms, which has high accuracy but a complex calculation model and requires a long time. L2 is the biological size predicted by the CNN neural network from the image, which requires a large amount of data to train and achieve good accuracy. Considering that the benthic organisms will vary depending on the placement location of each device, we prioritize using the biological size obtained by traditional algorithms to train the CNN neural network on-site. The prediction bias is measured by the SmoothL1 loss and the model parameters are adjusted accordingly. β is the threshold value, set to 0.1.

[0069] When the difference between the predicted L2 size and L1 size output by the CNN neural network remains less than 1e-3 for a long period of time, the network is considered to have good predictive ability. At this time, the device will cut off the traditional algorithm and use the L2 length predicted by the CNN network to represent the biological size L, which is recorded in the database and saved along with the original image and the shooting time.

[0070] Example 2 One embodiment of this disclosure provides a benthic organism environmental monitoring system under the influence of deep-sea mining plumes, comprising: The image acquisition module is used to acquire in-situ three-dimensional images of the benthic environment in deep-sea mining areas; The image processing module is used to remap in-situ 3D image data, mapping pixel coordinates to 3D coordinates to obtain a depth image; The water body sensing module is used to calculate the depth image contrast and depth image sharpness. By combining the image contrast and image sharpness, the turbidity of the water body in the monitoring area is quantified in real time to obtain the impact cycle of mining activities. The image prediction module is used to extract biological depth images based on depth images, input the biological depth images into the depth prediction model, and output the predicted biological size information. The deep prediction model uses field-collected data and the SmoothL1 loss function to measure prediction bias and adjust model parameters accordingly. The training of the deep prediction model is completed when the difference between the biological size output by the deep prediction model and the predicted target remains below a set threshold for a long period of time.

[0071] Example 3 One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for monitoring the benthic organism environment under the influence of deep-sea mining plumes.

[0072] Example 4 One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the method for monitoring the benthic organism environment under the influence of deep-sea mining plumes.

[0073] Example 5 One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the method for monitoring the benthic organism environment under the influence of deep-sea mining plumes.

[0074] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for monitoring the benthic organism environment under the influence of deep-sea mining plumes, characterized in that, include: Acquire in-situ 3D images of the benthic environment in deep-sea mining areas; The in-situ 3D image data is remapped to convert pixel coordinates to 3D coordinates to obtain a depth image; Calculate depth image contrast and depth image sharpness, and combine image contrast and image sharpness to quantify the turbidity of the water body in the monitoring area in real time, thereby obtaining the impact cycle of mining activities; Based on depth images, biological depth images are extracted, and the biological depth images are input into a depth prediction model to output predicted biological size information. The deep prediction model uses field-collected data and the SmoothL1 loss function to measure prediction bias and adjust model parameters accordingly. The training of the deep prediction model is completed when the difference between the biological size output by the deep prediction model and the predicted target remains below a set threshold for a long period of time.

2. The method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in claim 1, characterized in that, The acquisition of three-dimensional images of the deep-sea environment includes: Image acquisition is performed using a deep-sea binocular imaging sensor, which includes a deep-sea binocular equipment cabin, a deep-sea LED light source cabin, and a platform assembly frame. The deep-sea binocular equipment cabin contains a power supply module, a control module, and a data acquisition module, and is protected by a titanium alloy pressure-resistant body. The titanium alloy pressure-resistant body consists of a front observation hatch, a cylindrical pressure-resistant body, and a rear communication hatch. The front observation hatch has a set of symmetrical prototype light-transmitting holes, which are covered and protected by coated sapphire. The rear communication hatch is equipped with two watertight connectors, which are responsible for powering the deep-sea LED light source cabin and communicating with the observation platform, respectively.

3. The method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in claim 2, characterized in that, The power supply module consists of two miniaturized voltage regulator modules. The power supply module supplies power to the control module and the data acquisition module, and supplies power to the deep-sea LED light source compartment through the watertight connector of the rear communication hatch. The power supply module is fixed to the front observation hatch through an I-shaped aluminum alloy support plate.

4. The method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in claim 2, characterized in that, The control module is fixed to the front observation hatch via a Type II aluminum alloy support plate; the data acquisition module consists of two identical small-sized single-board cameras. The data acquisition module is connected to the control module via a data interface for power supply, control, and data transmission, and is connected to the Type I and Type II aluminum alloy support plates via an aluminum alloy support disc.

5. The method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in claim 1, characterized in that, The remapping of 3D image data, mapping pixel coordinates to 3D coordinates, includes: Using a checkerboard or dot array calibration board, corner points are extracted by taking multiple images of the calibration board from different angles, and the camera's intrinsic parameter matrix and distortion matrix are determined. By matching the corner points of the left and right cameras, the rigid body transformation between them is solved, and the most suitable rotation matrix and translation vector are found by minimizing the reprojection error. By combining the intrinsic parameter matrices, distortion parameters, rotation matrices, and translation vectors of the two cameras, a remapping table for the two cameras is obtained. The left and right images are processed separately using a remapping table, and then the reprojection matrix is ​​used to complete the mapping calculation from pixel coordinates to three-dimensional coordinates.

6. The method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in claim 1, characterized in that, By monitoring image contrast and the clarity and blur of the background, the relationship between water turbidity and time is quantified. Combined with the time and location parameters of the mining process of the mining truck, the impact cycle of mining activities on the benthic environment and organisms is analyzed. Read the in-situ 3D image of the main camera in each group of images, calculate the average brightness of the RGB channels, and calculate the standard deviation of the gray values ​​of all pixels relative to the average brightness to measure the dispersion of the image. The clearer the water environment, the larger the dispersion value; the more turbid the water environment, the smaller the dispersion value. Simultaneously, edge detection operators are used to detect the horizontal and vertical gradient pixel maps of the main camera image, and the root mean square of the gradients in both directions is calculated.

7. A benthic organism environmental monitoring system under the influence of deep-sea mining plumes, characterized in that, include: The image acquisition module is used to acquire in-situ three-dimensional images of the benthic environment in deep-sea mining areas; The image processing module is used to remap in-situ 3D image data, mapping pixel coordinates to 3D coordinates to obtain a depth image; The water body sensing module is used to calculate the depth image contrast and depth image sharpness. By combining the image contrast and image sharpness, the turbidity of the water body in the monitoring area is quantified in real time to obtain the impact cycle of mining activities. The image prediction module is used to extract biological depth images based on depth images, input the biological depth images into the depth prediction model, and output the predicted biological size information. The deep prediction model uses field-collected data and the SmoothL1 loss function to measure prediction bias and adjust model parameters accordingly. The training of the deep prediction model is completed when the difference between the biological size output by the deep prediction model and the predicted target remains below a set threshold for a long period of time.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in any one of claims 1-6.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in any one of claims 1-6.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a method for monitoring the benthic organism environment under the influence of deep-sea mining plumes as described in any one of claims 1-6.

Images