Submarine object monitoring method, device, robot and storage medium

Abstract

The present invention relates to the field of image processing technology, and discloses a method, device, robot and storage medium for monitoring submarine objects. The method obtains submarine images, sonar data and environmental parameters from the seabed; determines the visibility of the seabed according to the environmental parameters, and performs rain removal operation on the submarine images based on the visibility using a preset expansion filter to obtain a clear submarine image; extracts multi-scale features from the clear submarine image using a deep learning model to obtain a submarine environmental feature set; models the submarine environmental feature set and sonar data in time series, and uses a long short-term memory network to analyze the time series obtained by modeling to obtain an object recognition result. The method solves the problem of low monitoring accuracy in existing submarine monitoring solutions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, robot and storage medium for monitoring submarine objects. Background Art

[0002] In recent years, with the continuous development and utilization of marine resources by humans, the construction scale and number of marine buildings have gradually increased. Marine buildings refer to various fixed or floating buildings built in the ocean, such as offshore oil platforms, submarine cables, and offshore wind power plants. These marine buildings face various challenges from the marine environment, such as waves, sea breezes, and submarine geological environments. Therefore, the research on intelligent monitoring and maintenance technology for marine buildings has important practical significance.

[0003] Monitoring and maintenance of marine structures is an important part of ensuring their safe and reliable operation. Traditional monitoring methods usually rely on manual diving or simple monitoring equipment, which are inefficient and costly. At the same time, due to the poor visibility in the seabed environment, ordinary image processing technology is limited in practical applications and cannot accurately identify buildings. Therefore, there is an urgent need for an efficient underwater robot monitoring solution that combines deep learning and intelligent control. Summary of the invention

[0004] The present invention provides a method, device, robot and storage medium for monitoring a submarine object, so as to solve the problem of low monitoring accuracy in existing submarine monitoring solutions.

[0005] A first aspect of the present invention provides a method for monitoring seabed objects, which is applied to an underwater robot, wherein the underwater robot is provided with a plurality of different types of sensors; the method comprises: obtaining seabed environmental data collected by each of the sensors, wherein the seabed environmental data comprises seabed images, sonar data and environmental parameters; using an ocean optical model, calculating the mixture ratio in the water based on the environmental parameters to obtain water transparency, and determining the visibility of the seabed based on the degree of light attenuation by the water transparency; determining the parameters of a preset expansion filter based on the visibility, and performing a rain removal operation on the seabed image based on the expansion filter after the parameters are determined to obtain a clear seabed image; performing multi-scale feature extraction on the clear seabed image through a deep learning model to obtain a seabed environmental feature set, wherein the deep learning model comprises a convolutional neural network and a Transformer network; modeling the seabed environmental feature set and the sonar data as a time series, and using a long short-term memory network to analyze the time series obtained by the modeling for recognition, to obtain an object recognition result.

[0006] In an optional embodiment, the method of determining parameters of a preset expansion filter based on the visibility, and performing a rain removal operation on the seabed image based on the expansion filter after the parameters are determined to obtain a clear seabed image includes: mapping the visibility into a preset model parameter table, matching the model parameters of the expansion filter, the model parameters being the expansion rate that determines the size of the filter's perceived neighborhood on the image; and performing a rain removal operation on the seabed image based on the expansion rate using a preset expansion filter to obtain a clear seabed image.

[0007] In an optional embodiment, based on the expansion rate, a preset expansion filter is used to perform a rain removal operation on the seabed image to obtain a clear seabed image, including: adjusting the parameters of the enhancement network layer, the convolution layer and the output layer in the preset expansion filter based on the expansion rate to obtain a target expansion filter; inputting the seabed image into the enhancement network layer, performing image enhancement and noise reduction processing on the seabed image to obtain an enhanced seabed image; inputting the enhanced seabed image into the convolution layer, performing multi-dimensional convolution calculation on the enhanced seabed image, and fusing the multi-dimensional convolved images to obtain a rain-free seabed image; inputting the rain-free seabed image into the output layer, and using preset weighting coefficients to adjust the geometric, color and texture errors of the rain-free seabed image to obtain a clear seabed image.

[0008] In an optional implementation, the step of inputting the enhanced seabed image into the convolution layer, performing multi-dimensional convolution calculation on the enhanced seabed image, and fusing the multi-dimensional convolution images to obtain a rain-free seabed image includes: inputting the enhanced seabed image into the convolution layer, performing convolution operations of different scales on the enhanced seabed image, predicting a suitable kernel for each pixel, and obtaining a rain-free seabed image; wherein the formula for the convolution operation is: ,in, is the output value after the expansion filter acts; The coordinates of the input image x after the dilation operation The pixel value at ; is the convolution kernel weight; r is the expansion rate, which determines the extent of the expansion of the convolution kernel range; It is the index variable for traversing the convolution kernel.

[0009] In an optional embodiment, the rain-free seabed image is input into the output layer, and the geometric, color and texture errors of the rain-free seabed image are adjusted using a preset weighting coefficient to obtain a clear seabed image, including: using a difference comparison method to calculate the geometric, color and texture errors between the seabed image and the rain-free seabed image; based on the preset weighting coefficient, the average value of the geometric, color and texture errors is calculated to obtain a total loss, and the rain-free seabed image is adjusted based on the total loss to obtain a clear seabed image; wherein the calculation formula of the total loss is: ,in, , , is the weighting coefficient of different loss terms; is the geometric error; is the color error; is the texture error.

[0010] In an optional embodiment, the deep learning model is used to perform multi-scale feature extraction on the clear seabed image to obtain a seabed environment feature set, including: performing local feature extraction on the clear seabed image through a convolutional neural network to obtain multiple local feature sets; based on local positional relationships, using a Transformer network to perform global information modeling on all local feature sets to obtain a seabed environment feature set, wherein the seabed environment feature set is embodied in the form of a feature map.

[0011] In an optional embodiment, the seabed environment feature set and the sonar data are modeled as a time series, and the time series obtained by the modeling is analyzed using a long short-term memory network to perform identification to obtain an object identification result, including: arranging the seabed environment feature set and the sonar data in chronological order, marking the timestamp of each data point, and constructing a seabed three-dimensional model of the arranged data based on the timestamp; using a long short-term memory network model, identifying object pixel points in the seabed three-dimensional model and marking them; using a neighborhood search algorithm to extract continuous pixel points from the marked object pixel points and draw the object contour; and determining the type of object within the current position range of the underwater robot based on the object contour.

[0012] According to a second aspect of the present invention, there is provided a device for monitoring seabed objects, which is applied to an underwater robot, wherein the underwater robot is provided with a plurality of different types of sensors; the device comprises: an acquisition module, which is used to acquire seabed environmental data collected by each of the sensors, wherein the seabed environmental data comprises seabed images, sonar data and environmental parameters; a calculation module, which is used to calculate the proportion of mixtures in the water based on the environmental parameters using an ocean optical model to obtain water transparency, and determine the visibility of the seabed based on the degree of light attenuation by the water transparency; a rain removal module, which is used to determine the parameters of a preset expansion filter based on the visibility, and perform a rain removal operation on the seabed image based on the expansion filter after the parameters are determined, to obtain a clear seabed image; an extraction module, which is used to perform multi-scale feature extraction on the clear seabed image through a deep learning model to obtain a seabed environmental feature set, wherein the deep learning model comprises a convolutional neural network and a Transformer network; and a recognition module, which is used to perform time series modeling on the seabed environmental feature set and the sonar data, and perform recognition using a long short-term memory network to analyze the time series obtained by the modeling, to obtain an object recognition result.

[0013] A third aspect of the present invention provides a robot, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the robot executes the above-mentioned seabed object monitoring method.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned method for monitoring seabed objects.

[0015] In the technical solution provided by the present invention, seabed images, sonar data and environmental parameters are obtained from the seabed; the visibility of the seabed is determined according to the environmental parameters, and based on the visibility, a preset expansion filter is used to remove rain from the seabed image to obtain a clear seabed image; multi-scale feature extraction is performed on the clear seabed image through a deep learning model to obtain a seabed environmental feature set; the seabed environmental feature set and the sonar data are modeled as a time series, and the time series obtained by the modeling is analyzed using a long short-term memory network to identify the object, thereby obtaining an object recognition result. In the present application, an expansion filter is used to remove rain from the seabed image, and a deep learning model is used to identify objects in the seabed image after rain removal. This method improves the accuracy of seabed data collection, thereby improving the recognition accuracy of seabed objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an embodiment of the method for monitoring seabed objects in the present application;

[0017] Figure 2 This is a schematic diagram of a second embodiment of the method for monitoring submarine objects in this application;

[0018] Figure 3 A schematic diagram of an embodiment of a submarine object monitoring device in the present application;

[0019] Figure 4 It is a schematic diagram of another embodiment of the submarine object monitoring device in the present application;

[0020] Figure 5 This is a schematic diagram of an embodiment of the robot in this application. DETAILED DESCRIPTION

[0021] The present invention provides a method, device, robot and storage medium for monitoring a submarine object, which are used to solve the problem of low monitoring accuracy in existing submarine monitoring solutions.

[0022] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] See also Figure 1 , an embodiment of the method for monitoring seabed objects in the present application includes:

[0024] 101. Acquire seabed environmental data collected by each sensor, where the seabed environmental data includes seabed images, sonar data, and environmental parameters.

[0025] It is understandable that the execution subject of the present invention may be a submarine object monitoring device, or a terminal, a robot or a server, which is not limited here. The embodiment of the present invention is described by taking a robot as the execution subject.

[0026] In this embodiment, the robot refers to a terminal device that can move underwater, such as an underwater operation robot, and the robot is equipped with a variety of sensors, specifically but not limited to a high-resolution camera, a sonar, and a water quality sensor.

[0027] The camera is used to capture images of the seabed within the range of the robot's location, the sonar instrument is used to collect sonar data within the range of the location, and the water quality sensor is used to collect environmental parameters such as water temperature, acidity, alkalinity, mixture, etc.

[0028] 102. Using the ocean optical model, the mixture ratio in the water is calculated based on environmental parameters to obtain water transparency, and the visibility of the seabed is determined based on the degree of light attenuation caused by water transparency;

[0029] 103. Determine parameters of a preset expansion filter based on the visibility, and perform a rain removal operation on the seabed image based on the expansion filter with the determined parameters to obtain a clear seabed image.

[0030] In this implementation, the visibility can be calculated based on environmental parameters by calling existing optical principles. Specifically, the mixture ratio in the water is calculated by environmental parameters to obtain water transparency, and the visibility of the seabed is determined based on the degree of light attenuation caused by water transparency. Then, the parameters of the preset expansion filter are configured based on the visibility, and the rain removal operation is performed on the seabed image based on the configured expansion filter to obtain a clear seabed image.

[0031] It should be noted that this step specifically includes the following implementation steps:

[0032] Step 1: Analyze environmental parameters;

[0033] Collect key environmental parameters from marine environmental monitoring equipment or historical data, including but not limited to water temperature, salinity, water flow speed, lighting conditions (especially the degree of natural light penetration into seawater), and possible suspended particulate matter concentrations.

[0034] Using these parameters, the current visibility of the seabed can be estimated through existing ocean optical models or empirical formulas. Visibility is usually inversely proportional to the attenuation of light in seawater. The more severe the attenuation, the lower the visibility.

[0035] Step 2: Mapping of visibility to dilation filter parameters;

[0036] Based on the estimated seafloor visibility, a mapping rule or lookup table is designed to map the visibility range to different parameter settings of the preset dilation filter. The dilation filter parameter here mainly refers to the dilation rate, which determines the size of the neighborhood perceived by the filter on the image.

[0037] Generally speaking, the lower the visibility (i.e., the more severe the light attenuation is, which may mean more rain streaks or suspended particles), the higher the dilation rate needs to be set to capture a larger range of contextual information in order to more effectively remove rain streaks or blur effects.

[0038] Step 3, applying the expansion filter to perform rain removal operation;

[0039] The seabed image is input into the pre-processing module, which may include steps such as image enhancement and noise reduction to improve the image quality and lay a good foundation for subsequent processing.

[0040] Based on the dilation filter parameters determined in step 2, a dilation filter is selected or adjusted and applied to the seafloor image. This step can be achieved through a convolution operation, where the dilation filter acts as a convolution kernel, slides over the image and calculates the output.

[0041] The application of the dilation filter is an iterative process, and after each iteration, a decision is made whether to continue based on the current image quality (for example, by calculating indicators such as PSNR or SSIM) and a preset stopping condition (such as reaching the maximum number of iterations or the indicator improvement is no longer significant).

[0042] Step 4: Post-processing and optimization;

[0043] The image processed by the dilation filter is post-processed, which may include steps such as image sharpening and color correction to further restore the details and colors of the image.

[0044] If necessary, a multi-scale loss function can be introduced to guide the optimization of the entire processing process to ensure that as many image details and textures as possible are retained while removing rain marks.

[0045] 104. Multi-scale feature extraction is performed on clear seabed images through deep learning models to obtain the seabed environment feature set.

[0046] It should be noted that the deep learning model includes a convolutional neural network and a Transformer network. Specifically, a feature extraction network, i.e. a deep learning model, is constructed, and then the model is used to extract multi-scale features, and finally the extracted features are classified, sorted and integrated to obtain a seabed environment feature set.

[0047] For building feature extraction networks: Use a combination of convolutional neural networks (CNNs) and Transformer networks as feature extractors. You can use pre-trained models (such as VGG, ResNet, etc.) for fine-tuning, or train from scratch.

[0048] For multi-scale feature extraction: Use image pyramid technology to scale images to different scales and extract features from each layer of images; use convolution kernels of different scales to capture features of different scales in convolutional neural networks. For example, low-level convolution kernels have a smaller receptive field and are suitable for extracting features of small targets; high-level convolution kernels have a larger receptive field and are suitable for extracting features of large targets; use structures such as FPN (feature pyramid network) to fuse features of different scales to improve the accuracy and robustness of feature extraction; build a target detection and recognition network based on the feature extraction network. Faster R-CNN, SSD and other target detection algorithms can be used to identify and locate targets in seabed images, and all detected and identified features can be used to construct a seabed environment feature set.

[0049] 105. The seabed environment feature set and sonar data are modeled as a time series, and the time series obtained by the long short-term memory network analysis model is used for recognition to obtain the object recognition result.

[0050] In this embodiment, the seabed environment feature set and sonar data are synchronously processed, and the LSTM network is used to perform time series analysis on the data to extract the time series features of the data; the time series data is weighted averaged to obtain the comprehensive features of each time step; and object recognition is performed based on the comprehensive features to obtain the type of the object.

[0051] For time series modeling: first, arrange the seabed environment feature set and sonar data in chronological order and mark the timestamp of each data point; then, according to the timestamp, synchronize the seabed environment feature set and sonar data to form a time series data set containing multiple data types.

[0052] For object recognition using LSTM network time series analysis: first build an LSTM (long short-term memory) network model, set appropriate parameters such as the number of network layers, number of neurons, and learning rate according to the characteristics of the time series data; then use the trained LSTM network to perform time series analysis on the time series data set, extract the time series features of the data, and determine the object type based on the time series features.

[0053] It should be noted that after extracting the features from the time series data, the extracted features are further fused using weighted coefficients, which specifically includes the following steps:

[0054] Step a: Sort the time series features extracted by the LSTM network to ensure that each time step contains feature vectors from different data sources.

[0055] Step b: Assign a corresponding weight to each feature vector based on its importance or relevance. The weight assignment can be based on expert experience, data quality, correlation analysis between features, etc.

[0056] Step c: Perform weighted averaging on the eigenvectors of each time step, that is, multiply each eigenvalue by the corresponding weight, and then sum them up to obtain the comprehensive eigenvalue of the time step.

[0057] Step d: Repeat step c until a comprehensive feature set of all time steps is obtained.

[0058] Step e: Apply the obtained comprehensive feature set to subsequent tasks such as marine environment monitoring, ecological protection, and resource exploration to verify the effectiveness and practicality of the data fusion method.

[0059] Step f: According to the actual application effect, the weighted average method is improved and optimized to improve the accuracy and reliability of data fusion.

[0060] In this embodiment, the seabed image, sonar data and environmental parameters are obtained; the visibility of the seabed is determined according to the environmental parameters, and the seabed image is subjected to rain removal operation based on the visibility using a preset expansion filter to obtain a clear seabed image; multi-scale feature extraction is performed on the clear seabed image using a deep learning model to obtain a seabed environmental feature set; the seabed environmental feature set and sonar data are modeled as a time series, and the time series obtained by modeling is analyzed using a long short-term memory network to identify the object, thereby solving the problem of low monitoring accuracy in existing seabed monitoring solutions.

[0061] See also Figure 2 The second embodiment of the method for monitoring seabed objects in the present application includes:

[0062] 201. Acquire seabed environmental data collected by each sensor, where the seabed environmental data includes seabed images, sonar data, and environmental parameters.

[0063] This step mainly includes two steps: one is data collection; the other is preprocessing of the seabed image. Specifically, firstly, the underwater robot is controlled to move to the target position, and the sensors installed on it collect real-time data of the surrounding area of ​​the position, that is, the seabed environmental data is obtained, which includes seabed images, sonar data and environmental parameters, and the environmental parameters include water depth, temperature, salinity, granularity, etc. Then the collected seabed image is preprocessed, which includes denoising, standardization and brightness enhancement to ensure that the image quality is suitable for subsequent analysis.

[0064] Specific: Seabed image acquisition: The underwater robot is equipped with a high-definition camera (such as a 4K high-definition or higher resolution camera), which controls the robot to move around the seabed structure to capture images of each location. The seabed images will be transmitted to the onshore control center in real time for subsequent processing and analysis.

[0065] Sonar data collection: underwater robots use sonar equipment to obtain reflected sound wave data, which can be converted into images to show the shape and distance of objects. Commonly used sonar systems include multi-beam sonar or side-scan sonar, which generate sonar images reflecting the structure of the seabed.

[0066] Environmental parameter collection: Sensors are used to monitor environmental changes in real time, such as water depth, temperature, salinity, etc. The data will be combined with image and sonar data to provide multi-dimensional monitoring information.

[0067] 202. The ocean optical model is used to calculate the proportion of mixtures in the water based on environmental parameters to obtain water transparency, and the visibility of the seabed is determined based on the degree of light attenuation caused by water transparency.

[0068] It should be noted that the visibility is inversely proportional to the attenuation of light in seawater. The more severe the attenuation, the lower the visibility. The ocean optical model is an optical model obtained by pre-construction, which specifically includes the following steps:

[0069] Step 1: Data collection and preprocessing.

[0070] Environmental parameter collection: Collect water quality parameters (such as suspended matter concentration, particle size, water temperature, salinity, etc.), lighting conditions (such as wavelength and intensity of incident light, etc.) and water depth information at the current location.

[0071] Robot status data: Get the location information of the underwater robot (such as longitude, latitude, water depth, etc.).

[0072] Step 2: Establish the optical model.

[0073] Absorption of light: Based on the absorption characteristics of seawater, calculate the absorption coefficient of light in seawater. This is mainly affected by water molecules and dissolved and particulate organic matter. In the visible light band, blue light is more easily absorbed than red light.

[0074] Light scattering: Calculate the scattering coefficient based on the wavelength and propagation direction of the scattered light. Scattering is mainly caused by dissolved and suspended particles, including Rayleigh scattering, Mie scattering and non-selective scattering.

[0075] Transmission of light: Calculate the transmission of light in seawater based on the absorption and scattering coefficients.

[0076] Step 3, calculate visible light.

[0077] Input environmental parameters: including water quality parameters, lighting conditions, water depth, etc.

[0078] Calculate the absorption coefficient: Calculate the absorption coefficient A2 based on water quality parameters (such as suspended matter concentration, water temperature, salinity, etc.) and the wavelength of light.

[0079] Calculate the scattering coefficient: Calculate the scattering coefficient A3 based on the water quality parameters and the wavelength of light.

[0080] Calculate the attenuation coefficient: Add the absorption coefficient A2 and the scattering coefficient A3 to get the attenuation coefficient A1.

[0081] Calculate light intensity: Calculate the light intensity I reaching the observation point based on the intensity of the incident light and the distance attenuation.

[0082] Calculate visibility: Use the formula V=I / 2*A1 to calculate visibility.

[0083] 203. Map the visibility to a preset model parameter table to match the model parameter of the dilation filter, where the model parameter is a dilation rate that determines the size of the filter's perceived neighborhood on the image.

[0084] Specifically, according to the calculated visibility, a mapping rule or a lookup table is designed to map the visibility range to different parameter settings of the preset dilation filter.

[0085] Generally speaking, the lower the visibility (i.e., the more severe the light attenuation is, which may mean more rain streaks or suspended particles), the higher the dilation rate needs to be set to capture a larger range of contextual information in order to more effectively remove rain streaks or blur effects.

[0086] Based on the expansion rate, a preset expansion filter is used to perform a rain removal operation on the seabed image to obtain a clear seabed image.

[0087] Specifically, the parameters of the enhancement network layer, the convolution layer and the output layer in the preset expansion filter are adjusted based on the expansion rate to obtain a target expansion filter; the seabed image is input into the enhancement network layer, and image enhancement and noise reduction processing are performed on the seabed image to obtain an enhanced seabed image; the enhanced seabed image is input into the convolution layer, and multi-dimensional convolution calculation is performed on the enhanced seabed image, and the images after the multi-dimensional convolution are fused to obtain a rain-free seabed image; the rain-free seabed image is input into the output layer, and the errors of geometry, color and texture of the rain-free seabed image are adjusted by using preset weighting coefficients to obtain a clear seabed image.

[0088] The seabed image is input into the enhancement network layer, and image enhancement and noise reduction are performed on the seabed image to obtain an enhanced seabed image. Specifically, image filtering technology, such as Gaussian filter or mean filter, is used to normalize the seabed image so that its mean is 0 and its variance is 1, which is convenient for subsequent deep learning model processing. An adaptive brightness enhancement algorithm is used to improve image clarity. The formula for its normalization processing is: , where mean(X) is the average value of all pixel values ​​in the seabed image; is the standard deviation of all pixel values ​​in the seafloor image; is a normalized seabed image, whose pixel values ​​are adjusted to satisfy the distribution with a mean of 0 and a variance of 1.

[0089] It can be understood that the step of inputting the enhanced seabed image into the convolution layer, performing multi-dimensional convolution calculation on the enhanced seabed image, and fusing the multi-dimensional convolution images to obtain a rain-free seabed image includes: inputting the enhanced seabed image into the convolution layer, performing convolution operations of different scales on the enhanced seabed image, predicting a suitable kernel for each pixel, and obtaining a rain-free seabed image; wherein the formula of the convolution operation is: ,in, is the output value after the expansion filter acts; The coordinates of the input image x after the dilation operation The pixel value at ; is the convolution kernel weight; r is the expansion rate, which determines the extent of the expansion of the convolution kernel range; The index variable for traversing the convolution kernel. According to the real-time seabed environmental data (such as water turbidity or water temperature changes), the expansion rate value is dynamically adjusted to ensure that the visual errors caused by different types of noise (such as water surface fluctuations and object interference) are removed.

[0090] In another embodiment, the rain-free seabed image is input into the output layer, and the error adjustment of geometry, color and texture of the rain-free seabed image is performed using a preset weighting coefficient to obtain a clear seabed image, including: using a difference comparison method to calculate the geometry, color and texture errors between the seabed image and the rain-free seabed image; based on the preset weighting coefficient, the average value of the geometry, color and texture errors is calculated to obtain a total loss, and the rain-free seabed image is adjusted based on the total loss to obtain a clear seabed image; wherein the calculation formula of the total loss is: ,in, , , is the weighting coefficient of different loss terms; is the geometric error; is the color error; is the texture error.

[0091] It should be noted that the geometric error is a measure of the gradient difference between the collected seabed image and the target image, and its calculation formula is: ,in, and are the gradient values ​​of the target image and the seabed image at pixel p respectively. This formula is used to detect edges and contours in the image, thereby improving the detection accuracy of the target area.

[0092] Color error: Calculates the absolute difference in image pixel values ​​to optimize color matching and correction of lighting conditions. The calculation formula is: , where y is the target image, For seabed images, this formula is used to eliminate color deviation caused by changes in the underwater environment or the image acquisition angle.

[0093] The texture error: The texture features of the image are extracted through a convolutional neural network to optimize the details of the image. The calculation formula is: ,in represents the features extracted by the convolution layer, and F is the feature set of the convolution layer. This formula is used to restore the detail features blurred by water quality and enhance the texture performance of the image.

[0094] Multi-scale loss function: Combine the above loss functions and obtain the final loss function through weighted summation.

[0095] In another feasible implementation, after obtaining a clear seabed image, it also includes: determining the current position of the underwater robot, and extracting a real seabed image of the position from historical records, wherein the real seabed image is a seabed image when the visibility of the seabed meets a preset threshold; inputting the clear seabed image and the real seabed image into a clarity evaluation model to evaluate the rain removal effect; the evaluation can be performed by combining quantitative indicators (such as PSNR, SSIM) and qualitative evaluation (such as expert scoring, user satisfaction survey)); based on the evaluation results, determining whether to perform a secondary rain removal operation on the seabed image.

[0096] 205. Multi-scale feature extraction is performed on clear seabed images through deep learning models to obtain the seabed environment feature set.

[0097] In this embodiment, local features of the clear seabed image are extracted by a convolutional neural network to obtain multiple local feature sets. Based on the local positional relationship, the Transformer network is used to perform global information modeling on all local feature sets to obtain a seabed environment feature set, wherein the seabed environment feature set is embodied in the form of a feature map.

[0098] In practical applications, first, choose a deep learning framework: choose a deep learning framework such as TensorFlow, PyTorch, etc. to build and train models.

[0099] Then, build a multi-scale feature extraction network: use a convolutional neural network (CNN) as a feature extractor. You can use a pre-trained model (such as VGG, ResNet, etc.) for fine-tuning, or train from scratch.

[0100] Furthermore, multi-scale feature extraction is performed: the image is scaled to different scales using the image pyramid technology, and features are extracted from each layer of the image; in the convolutional neural network, convolution kernels of different scales are used to capture features of different scales. For example, the receptive field of the low-level convolution kernel is small, which is suitable for extracting features of small targets; the receptive field of the high-level convolution kernel is large, which is suitable for extracting features of large targets; using structures such as FPN (feature pyramid network), features of different scales are fused to form a seabed environment feature set to improve the accuracy and robustness of feature extraction.

[0101] Finally, the target detection and recognition network is constructed: Based on the feature extraction network, the target detection and recognition network is constructed. Target detection algorithms such as Faster R-CNN and SSD can be used to identify and locate targets in seabed images.

[0102] Furthermore, the extracted seabed environment feature set can be visualized, such as drawing feature maps, histograms, etc.

[0103] 206. The seabed environment feature set and sonar data are modeled as a time series, and the time series obtained by the modeling is analyzed using a long short-term memory network to identify the object and obtain the object recognition result.

[0104] Specifically, the seabed environment feature set and the sonar data are arranged in chronological order, and the timestamp of each data point is marked, and a three-dimensional seabed model is constructed based on the timestamp of the arranged data; the object pixel points in the three-dimensional seabed model are identified and marked using a long short-term memory network model; continuous pixel points in the marked object pixel points are extracted using a neighborhood search algorithm, and the object contour is drawn; and the type of object within the current position range of the underwater robot is determined based on the object contour.

[0105] In another embodiment, after the object type is identified, it also includes: determining the normal wear and tear pattern of the object based on the object type; evaluating the structural health, degree of corrosion and damage based on the time series obtained by modeling to obtain an evaluation result; performing early warning analysis based on the evaluation result and the normal wear and tear pattern to generate early warning information.

[0106] Specifically, the structural health, degree of corrosion, degree of damage, etc. of the submarine buildings are evaluated and a risk assessment report is generated. Based on the risk assessment results, early warning information is generated. When the risk index exceeds the predetermined threshold, the early warning is triggered and output to the control center. Based on the early warning information and risk assessment results, the path of the underwater robot is automatically adjusted or the corresponding maintenance mechanism is initiated.

[0107] Among them, the structural health assessment is specifically based on the analysis results of the seabed environmental characteristic set and time series data to evaluate the structural health, corrosion level and damage of the seabed buildings and generate corresponding structural health indicators.

[0108] The risk assessment report calculates risk assessment indicators based on the results of the structural health assessment, including multiple factors such as structural integrity, degree of corrosion, degree of damage, etc., and conducts risk assessment based on these indicators.

[0109] When generating early warning information, when the risk assessment results exceed the preset threshold, the early warning system is automatically triggered to generate high-priority early warning information, and the risk assessment results and real-time monitoring reports are transmitted to the onshore control center in real time through the communication system and provided to maintenance personnel for reference.

[0110] In another embodiment, it also includes generating an automatic path based on the warning information to achieve maintenance: specifically, according to the warning information and risk assessment results, the path or working state of the underwater robot is automatically adjusted so that detailed inspection or emergency response can be carried out when potential risks are found. When potential risks such as damage or corrosion are detected, the underwater robot is started to perform automated maintenance and repair operations, or the robot is fed back to the control center to request manual intervention. Report generation and storage, a complete monitoring report including image analysis results, risk assessment results, warning information and maintenance recommendations is generated and stored in the database for subsequent analysis and reference.

[0111] Furthermore, the monitoring system can be continuously optimized based on monitoring results and maintenance feedback, including model training, data collection and adjustment of processing procedures, so as to continuously improve monitoring accuracy and response speed.

[0112] In this embodiment, the seabed image, sonar data and environmental parameters are obtained from the seabed; the visibility of the seabed is determined according to the environmental parameters, and based on the visibility, a preset expansion filter is used to remove rain from the seabed image to obtain a clear seabed image; a deep learning model is used to extract multi-scale features of the clear seabed image to obtain a seabed environmental feature set; the seabed environmental feature set and the sonar data are modeled as a time series, and the time series obtained by the long short-term memory network analysis model is used for identification to obtain the object identification result. In this application, an expansion filter is used to remove rain from the seabed image, and a deep learning model is used to identify objects in the seabed image after rain removal. This method improves the accuracy of seabed data collection, thereby improving the accuracy of seabed object identification.

[0113] The above describes the method for monitoring submarine objects in the present application. The following describes the device for monitoring submarine objects in the present application. Figure 3 , an embodiment of the submarine object monitoring device in the present application includes:

[0114] An acquisition module 310 is used to acquire the seabed environmental data collected by each of the sensors, wherein the seabed environmental data includes seabed images, sonar data and environmental parameters;

[0115] A calculation module 320 is used to calculate the mixture ratio in the water based on the environmental parameters using an ocean optical model to obtain water transparency, and determine the visibility of the seabed based on the light attenuation degree of the water transparency;

[0116] A rain removal module 330 is used to determine the parameters of a preset expansion filter based on the visibility, and perform a rain removal operation on the seabed image based on the expansion filter after the parameters are determined to obtain a clear seabed image;

[0117] An extraction module 340 is used to perform multi-scale feature extraction on the clear seabed image through a deep learning model to obtain a seabed environment feature set, wherein the deep learning model includes a convolutional neural network and a Transformer network;

[0118] The recognition module 350 is used to model the seabed environment feature set and the sonar data in time series, and use the long short-term memory network to analyze the time series obtained by the modeling to perform recognition, so as to obtain the recognition result of the object.

[0119] In this embodiment, an expansion filter is used to remove rain from the seabed image, and a deep learning model is used to identify objects in the seabed image after rain removal. This method improves the accuracy of seabed data collection, thereby improving the accuracy of seabed object recognition.

[0120] See also Figure 4 Another embodiment of the submarine object monitoring device in the present application includes:

[0121] An acquisition module 310 is used to acquire the seabed environmental data collected by each of the sensors, wherein the seabed environmental data includes seabed images, sonar data and environmental parameters;

[0122] A calculation module 320 is used to calculate the mixture ratio in the water based on the environmental parameters using an ocean optical model to obtain water transparency, and determine the visibility of the seabed based on the light attenuation degree of the water transparency;

[0123] A rain removal module 330 is used to determine the parameters of a preset expansion filter based on the visibility, and perform a rain removal operation on the seabed image based on the expansion filter after the parameters are determined to obtain a clear seabed image;

[0124] An extraction module 340 is used to perform multi-scale feature extraction on the clear seabed image through a deep learning model to obtain a seabed environment feature set, wherein the deep learning model includes a convolutional neural network and a Transformer network;

[0125] The recognition module 350 is used to model the seabed environment feature set and the sonar data in time series, and use the long short-term memory network to analyze the time series obtained by the modeling to perform recognition, so as to obtain the recognition result of the object.

[0126] Optionally, the rain removal module 330 includes:

[0127] A matching unit 331 is used to map the visibility into a preset model parameter table to match the model parameters of the dilation filter, where the model parameters are dilation rates that determine the size of the filter's perceived neighborhood on the image;

[0128] The rain removal unit 332 is used to perform a rain removal operation on the seabed image based on the expansion rate by using a preset expansion filter to obtain a clear seabed image.

[0129] Optionally, the rain removal unit 332 is specifically used for:

[0130] Adjusting the parameters of the enhancement network layer, the convolution layer and the output layer in the preset dilation filter based on the dilation rate to obtain a target dilation filter;

[0131] Inputting the seabed image into the enhancement network layer, performing image enhancement and noise reduction processing on the seabed image to obtain an enhanced seabed image;

[0132] Inputting the enhanced seabed image into the convolution layer, performing multi-dimensional convolution calculation on the enhanced seabed image, and fusing the multi-dimensional convolution images to obtain a rain-free seabed image;

[0133] The rain-free seabed image is input into the output layer, and the errors of geometry, color and texture of the rain-free seabed image are adjusted by using preset weighting coefficients to obtain a clear seabed image.

[0134] Optionally, the rain removal unit 332 is specifically used for:

[0135] Inputting the enhanced seabed image into the convolution layer, performing convolution operations of different scales on the enhanced seabed image, predicting a suitable kernel for each pixel point, and obtaining a rain-free seabed image;

[0136] The formula for the convolution operation is:

[0137] ,in, is the output value after the expansion filter acts; The coordinates of the input image x after the dilation operation The pixel value at ; is the convolution kernel weight; r is the expansion rate, which determines the extent of the expansion of the convolution kernel range; It is the index variable for traversing the convolution kernel.

[0138] Optionally, the rain removal unit 332 is specifically used for:

[0139] Calculating the geometric, color and texture errors between the seabed image and the rain-free seabed image by using a difference comparison method;

[0140] Based on a preset weighting coefficient, an average value of geometric, color and texture errors is calculated to obtain a total loss, and the rain-free seabed image is adjusted based on the total loss to obtain a clear seabed image;

[0141] The calculation formula of the total loss is: , , , is the weighting coefficient of different loss terms; is the geometric error; is the color error; is the texture error.

[0142] Optionally, the extraction module 340 includes:

[0143] An extraction unit 341 is used to extract local features from the clear seabed image through a convolutional neural network to obtain multiple local feature sets;

[0144] The modeling unit 342 is used to perform global information modeling on all local feature sets based on local positional relationships using a Transformer network to obtain a seabed environment feature set, wherein the seabed environment feature set is embodied in the form of a feature map.

[0145] Optionally, the identification module 350 includes:

[0146] A construction unit 351 is used to arrange the seabed environment feature set and the sonar data in chronological order, mark the timestamp of each data point, and construct a seabed three-dimensional model with the arranged data based on the timestamp;

[0147] The recognition unit 352 is used to recognize and mark pixel points of objects in the three-dimensional seabed model using a long short-term memory network model;

[0148] A drawing unit 353 is used to extract continuous pixel points from the marked object pixel points and draw the object contour using a neighborhood search algorithm;

[0149] The determination unit 354 is configured to determine the type of object within the current position range of the underwater robot based on the object contour.

[0150] In this embodiment, the seabed image, sonar data and environmental parameters are obtained; the visibility of the seabed is determined according to the environmental parameters, and the seabed image is subjected to rain removal operation based on the visibility using a preset expansion filter to obtain a clear seabed image; multi-scale feature extraction is performed on the clear seabed image using a deep learning model to obtain a seabed environmental feature set; the seabed environmental feature set and sonar data are modeled as a time series, and the time series obtained by modeling is analyzed using a long short-term memory network to identify the object, thereby solving the problem of low monitoring accuracy in existing seabed monitoring solutions.

[0151] above Figure 3 and Figure 4 The seabed object monitoring device in the present application is described in detail from the perspective of modular functional entities, and the robot in the present application is described in detail from the perspective of hardware processing.

[0152] See also Figure 5 As shown, the robot includes a processor 500 and a memory 501 . The memory 501 stores machine executable instructions that can be executed by the processor 500 . The processor 500 executes the machine executable instructions to implement the above-mentioned seabed object monitoring method.

[0153] Further, Figure 5 The robot shown also includes a bus 502 and a communication interface 503 , and the processor 500 , the communication interface 503 and the memory 501 are connected via the bus 502 .

[0154] Among them, the memory 501 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), for example, at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 503 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 502 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0155] The processor 500 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 500. The above processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 501, and the processor 500 reads the information in the memory 501 and completes the steps of the seabed object monitoring method provided in the above-mentioned embodiment in combination with its hardware.

[0156] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the seabed object monitoring method.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0159] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring seabed objects, applied to an underwater robot, characterized in that: The underwater robot is provided with a plurality of different types of sensors; the method comprises: Acquiring seabed environmental data collected by each of the sensors, wherein the seabed environmental data includes seabed images, sonar data, and environmental parameters; Using an ocean optical model, the mixture ratio in the water is calculated based on the environmental parameters to obtain water transparency, and the visibility of the seabed is determined based on the degree of light attenuation by the water transparency; Determining parameters of a preset expansion filter based on the visibility, and performing a rain removal operation on the seabed image based on the expansion filter after the parameters are determined to obtain a clear seabed image; Performing multi-scale feature extraction on the clear seabed image through a deep learning model to obtain a seabed environment feature set, wherein the deep learning model includes a convolutional neural network and a Transformer network; Modeling the seabed environment feature set and the sonar data in time series, and using the long short-term memory network to analyze the time series obtained by the modeling to identify the object, so as to obtain the object identification result; The step of determining the parameters of a preset expansion filter based on the visibility, and performing a rain removal operation on the seabed image based on the expansion filter after the parameters are determined to obtain a clear seabed image includes: Mapping the visibility into a preset model parameter table to match the model parameters of the dilation filter, wherein the model parameters are dilation rates that determine the size of the filter's perceived neighborhood on the image; Adjusting the parameters of the enhancement network layer, the convolution layer and the output layer in the preset dilation filter based on the dilation rate to obtain a target dilation filter; Inputting the seabed image into the enhancement network layer, performing image enhancement and noise reduction processing on the seabed image to obtain an enhanced seabed image; Inputting the enhanced seabed image into the convolution layer, performing multi-dimensional convolution calculation on the enhanced seabed image, and fusing the multi-dimensional convolution images to obtain a rain-free seabed image; The rain-free seabed image is input into the output layer, and the errors of geometry, color and texture of the rain-free seabed image are adjusted by using preset weighting coefficients to obtain a clear seabed image.

2. The method for monitoring submarine objects according to claim 1, characterized in that: The step of inputting the enhanced seabed image into the convolution layer, performing multi-dimensional convolution calculation on the enhanced seabed image, and fusing the multi-dimensional convolution images to obtain a rain-free seabed image includes: Inputting the enhanced seabed image into the convolution layer, performing convolution operations of different scales on the enhanced seabed image, predicting a suitable kernel for each pixel point, and obtaining a rain-free seabed image; The formula for the convolution operation is: ,in, is the output value after the expansion filter acts; The coordinates of the input image x after the dilation operation The pixel value at ; is the convolution kernel weight; r is the expansion rate, which determines the extent of the expansion of the convolution kernel range; It is the index variable for traversing the convolution kernel.

3. The method for monitoring submarine objects according to claim 1, characterized in that: The step of inputting the rain-free seabed image to the output layer and adjusting the geometric, color and texture errors of the rain-free seabed image using a preset weighting coefficient to obtain a clear seabed image includes: Calculating the geometric, color and texture errors between the seabed image and the rain-free seabed image by using a difference comparison method; Based on a preset weighting coefficient, an average value of geometric, color and texture errors is calculated to obtain a total loss, and the rain-free seabed image is adjusted based on the total loss to obtain a clear seabed image; The calculation formula of the total loss is: ,in, , , is the weighting coefficient of different loss terms; is the geometric error; is the color error; is the texture error.

4. The method for monitoring submarine objects according to claim 1, characterized in that: The deep learning model is used to extract multi-scale features from the clear seabed image to obtain a seabed environment feature set, including: Extracting local features of the clear seabed image by using a convolutional neural network to obtain multiple local feature sets; Based on the local positional relationship, the Transformer network is used to perform global information modeling on all local feature sets to obtain a seabed environment feature set, wherein the seabed environment feature set is embodied in the form of a feature map.

5. The method for monitoring submarine objects according to claim 1, characterized in that: The seabed environment feature set and the sonar data are modeled as time series, and the time series obtained by the long short-term memory network analysis model is used for recognition to obtain the recognition result of the object, including: Arrange the seabed environment feature set and the sonar data in chronological order, mark the timestamp of each data point, and construct a seabed three-dimensional model with the arranged data based on the timestamp; Using a long short-term memory network model, identifying pixel points of objects in the three-dimensional seabed model and marking them; Using the neighborhood search algorithm, continuous pixels from the marked object pixels are extracted and the object outline is drawn; The type of the object within the current position range of the underwater robot is determined based on the object contour.

6. A submarine object monitoring device, applied to an underwater robot, characterized in that: The underwater robot is provided with a plurality of different types of sensors; the device comprises: An acquisition module, used to acquire the seabed environmental data collected by each of the sensors, wherein the seabed environmental data includes seabed images, sonar data and environmental parameters; A calculation module, for calculating the mixture ratio in the water based on the environmental parameters using an ocean optical model, obtaining water transparency, and determining the visibility of the seabed based on the attenuation degree of light by the water transparency; A rain removal module, used to determine the parameters of a preset expansion filter based on the visibility, and perform a rain removal operation on the seabed image based on the expansion filter after the parameters are determined, so as to obtain a clear seabed image; An extraction module, used to perform multi-scale feature extraction on the clear seabed image through a deep learning model to obtain a seabed environment feature set, wherein the deep learning model includes a convolutional neural network and a Transformer network; A recognition module is used to model the seabed environment feature set and the sonar data in a time series, and use the long short-term memory network to analyze the time series obtained by the modeling to perform recognition, so as to obtain a recognition result of the object; The rain removal module comprises: A matching unit, used to map the visibility into a preset model parameter table to match the model parameters of the dilation filter, wherein the model parameters are dilation rates that determine the size of the filter's perceived neighborhood on the image; The rain removal unit is used to adjust the parameters of the enhancement network layer, the convolution layer and the output layer in the preset expansion filter based on the expansion rate to obtain a target expansion filter; input the seabed image to the enhancement network layer, perform image enhancement and noise reduction processing on the seabed image to obtain an enhanced seabed image; input the enhanced seabed image to the convolution layer, perform multi-dimensional convolution calculation on the enhanced seabed image, and fuse the multi-dimensional convolution images to obtain a rain-free seabed image; input the rain-free seabed image to the output layer, use preset weighting coefficients to adjust the geometric, color and texture errors of the rain-free seabed image to obtain a clear seabed image.

7. A robot, characterized in that: The robot comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the robot to execute the underwater robot-based seabed object monitoring method as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are read and executed, the method for monitoring seabed objects based on an underwater robot as described in any one of claims 1 to 5 is executed.

Images