Freshwater obstacle identification method and system based on deep learning

Abstract

The present invention discloses a method and system for identifying freshwater obstacles based on deep learning, and relates to the field of deep learning technology. The method comprises: integrating sonar, laser and camera equipment into the same detection equipment for obstacle detection; collecting original sample data and performing model training based on deep learning to generate an obstacle recognition model; detecting the freshwater environment and obtaining water body parameters; collecting real-time detection data to obtain the actual parameters and obstacle information of the obstacle; using the detection equipment to obtain identification verification data to verify and correct the position of the obstacle. The present invention uses camera technology to perform shape imaging and type recognition of obstacles in freshwater, and eliminates the influence of the underwater environment on imaging and image recognition through deep learning, and improves the accuracy and efficiency of underwater obstacle recognition by combining multiple deep learning technologies; sonar and laser detection are also used to locate and correct the position of obstacles, thereby improving the accuracy of obstacle recognition and positioning.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a freshwater obstacle recognition method and system based on deep learning. Background Art

[0002] Obstacle recognition is the process of identifying obstacles in the environment through sensor and algorithm detection. It is usually used in fields such as autonomous driving and robot navigation to help the system avoid obstacles and ensure safe and effective operation. Freshwater obstacle recognition is the process of applying obstacle recognition technology to freshwater environments and identifying underwater obstacles in freshwater environments (such as rivers and lakes). This recognition usually relies on sensors such as sonar, radar, and cameras to help ships, unmanned submersibles and other equipment avoid underwater obstacles and ensure safe navigation or operation.

[0003] However, due to the complex underwater environment, the refraction and absorption of light in the water will cause the underwater image to be blurred and the details to be difficult to identify. The turbidity of the water and floating matter will affect visibility and increase the difficulty of obstacle detection. In addition, natural environmental factors such as water flow, waves and sediments may also change the shape or position of obstacles, resulting in deviations in the identification and positioning of obstacles. Therefore, there are still many problems that need to be solved in obstacle identification in freshwater environments. Summary of the invention

[0004] The purpose of the present invention is to provide a freshwater obstacle identification method and system based on deep learning, which can be achieved through the following technical solutions:

[0005] In a first aspect, an embodiment of the present application provides a method for identifying freshwater obstacles based on deep learning, comprising the following steps:

[0006] S1, integrating sonar equipment, laser equipment and camera equipment into the same detection equipment to detect the shape and position of obstacles;

[0007] S2, using the detection equipment to collect sonar echo data and camera image data and use the two as original sample data;

[0008] S3, based on deep learning, the collected original sample data is processed to generate a feature data set, specifically:

[0009] Using a deep generative adversarial network to preprocess the original sample data and generate a first data set;

[0010] Using a convolutional neural network to reprocess the first data set and generate a fused data set;

[0011] Performing feature extraction based on the fused data set to obtain obstacle features and thereby generate a feature data set;

[0012] S4, using the YOLO algorithm and performing model training according to the feature data set to generate an obstacle recognition model;

[0013] S5, using an environmental detection device to detect the freshwater environment and obtain water parameters;

[0014] S6, using the detection equipment to collect real-time sonar echo data and camera image data and use the two as real-time detection data; inputting the real-time detection data into the obstacle recognition model to obtain actual parameters of the obstacle; using the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtain obstacle information;

[0015] S7, further using the detection equipment to obtain laser detection data and using it together with the sonar echo data as identification verification data to verify the position of the obstacle in the obstacle information, and performing position correction according to the verification result;

[0016] Among them, in step S3, a deep generative adversarial network is used to perform data preprocessing, which specifically includes the following contents:

[0017] Image selection: Select underwater images based on original sample data and adjust the light energy of underwater images according to the imaging quality of underwater images;

[0018] Image enhancement: Based on the U-Net network structure, the generator in the deep generative adversarial network is improved. Specifically, a multi-directional convolution module, an embedded attention module, and a brightness equalization module combined with full convolution are added to the generator.

[0019] Image denoising: The cycle consistency loss function, the perceptual loss function and the multi-scale structural similarity index loss function are integrated to generate a joint loss function, and the joint loss function is used as the loss function of the generator for image denoising.

[0020] Preferably, regarding image selection: the light energy of the underwater image It is composed of forward scattering, backscattering and direct irradiation, expressed as:

[0021] ;

[0022] In the formula, represents the direct irradiation component; represents the forward scattering component; represents the backscattering component;

[0023] Among them, the forward scattering component It is used to reflect the small-angle scattering of light reflected from obstacles. After being absorbed and scattered by the water, the light is emitted from the obstacle and reaches the camera device and is received by it.

[0024] Direct exposure It is formed by the attenuation of the reflected light from the obstacle when it propagates in the water. It describes the energy attenuation of light during propagation and is expressed as: ;

[0025] In the formula, It represents the energy of light on the surface of the obstacle; represents the transmission efficiency, and , η is the attenuation coefficient, is the distance from the obstacle to the shooting location;

[0026] Backscatter component It represents the light energy received by the camera after encountering the suspended particles and undergoing a large angle change, expressed as:

[0027] ; B is related to distant background light.

[0028] Preferably, the image enhancement includes the following contents:

[0029] The brightness equalization module is used to locate the low-light area in the underwater image, and specifically includes:

[0030] First, a large 9×9 convolution kernel is used to capture the global feature information of the input underwater image;

[0031] Secondly, five consecutive 3×3 convolutional layers are used to gradually extract the position information of the low-light area and output the low-light attention map;

[0032] The multi-directional convolution module is used to perform convolution operations in multiple directions to capture feature information of underwater images, specifically including:

[0033] Divide and transform the input underwater image according to the channel dimension;

[0034] Use multi-directional convolution kernels for convolution operations;

[0035] Use activation functions for nonlinear transformation;

[0036] Reduce data dimensions and extract key features through pooling;

[0037] The attention module is used for dynamic weight allocation and selective focusing of information, including a channel attention submodule and a spatial attention submodule, specifically:

[0038] The channel attention submodule is used to adaptively adjust the feature weight according to the channel relationship, specifically including:

[0039] The input underwater image is processed at the channel level using average pooling and maximum pooling functions;

[0040] The number of channels is compressed into a one-dimensional vector through a fully connected layer and a ReLU activation function and then processed again to obtain a two-dimensional vector;

[0041] The output is converted into a probability distribution of weight values ​​in the range of 0 to 1 through the Sigmoid activation function;

[0042] The spatial attention submodule is used to perform convolution operations on underwater images, extract feature representations of each region, and use the attention mechanism to perform weighted averaging on the extracted features, specifically including:

[0043] Average pooling and maximum pooling values ​​are calculated to capture different features of the spatial dimension;

[0044] The spatial attention information is extracted through the convolution layer, and then the output is mapped to between 0 and 1 through the Sigmoid activation function to generate a spatial attention weight map;

[0045] A weighted average operation is performed based on the spatial attention weight map and the spatial positions of different features are output.

[0046] Preferably, in step S3, a convolutional neural network is used to perform data reprocessing, which specifically includes the following contents:

[0047] Data synchronization: obtain the timestamp recorded by each sensor in the detection equipment when collecting data; calibrate the timestamp to correct the time error; use the calibrated timestamp to align the data of each sensor;

[0048] Data fusion: Use convolutional neural networks to fuse sonar echo data and camera image data to generate comprehensive information and form a fused data set, including:

[0049] Extract sonar features from sonar echo data;

[0050] Extract visual features from camera image data;

[0051] Adjust the feature dimensions of sonar features and visual features and map them to the same dimension;

[0052] interpolating or transforming sonar and visual features so that they are spatially and semantically aligned;

[0053] Sonar features and visual features are spliced ​​and weightedly fused in feature dimensions;

[0054] The data set obtained after weighted fusion is called feature data set.

[0055] Preferably, in step S4, the YOLO algorithm is used for model training, which specifically includes the following contents:

[0056] Annotate based on the feature data set to mark the category and location of the obstacle;

[0057] Configure network structure and network parameters based on YOLO algorithm;

[0058] Combine the losses of position regression, confidence prediction, and category prediction to form a custom loss function;

[0059] Use the training set in the feature dataset to train the model and optimize the network weights through the back-propagation algorithm;

[0060] Use the test set in the feature data set to evaluate the performance indicators of the model and adjust the network parameters and network structure, and save the evaluated obstacle recognition model.

[0061] Preferably, in step S5, the water body parameters include water turbidity, water flow velocity, water flow direction and underwater topography.

[0062] Preferably, in step S6, obtaining obstacle information specifically includes:

[0063] Obtaining the type of obstacle based on the obstacle recognition model;

[0064] Acquire the pixel height of the obstacle and the distance between the obstacle and the camera device based on the feature data set;

[0065] Using a monocular ranging algorithm, the actual height of the obstacle is obtained based on pixel height and distance;

[0066] The type of obstacle, the actual height of the obstacle, and the distance between the obstacle and the camera device are collected as obstacle information.

[0067] Preferably, in step S7, verifying and correcting the position of the obstacle in the obstacle information specifically includes:

[0068] S71, acquiring laser detection data and calculating the laser distance of the obstacle;

[0069] S72, obtaining sonar echo data and calculating the sonar distance of the obstacle;

[0070] S73, obtaining a camera distance between the obstacle and the camera device from the obstacle information;

[0071] S74, comparing the laser distance and the sonar distance with the camera distance, obtaining the distance error and calculating the actual distance of the obstacle;

[0072] S75, correcting the position of the obstacle according to the actual distance of the obstacle and updating the obstacle recognition model according to the corrected data.

[0073] Preferably, in step S71, calculating the laser distance of the obstacle specifically includes:

[0074] S711, confirming the water quality level according to the water body parameters, and emitting a laser with a preset wavelength and a preset power according to the water quality level;

[0075] S712, receiving the reflected laser and calculating the laser distance of the obstacle according to the phase difference between the transmitted laser and the reflected laser and the preset wavelength;

[0076] In step S72, the sonar distance of the obstacle is calculated, specifically including:

[0077] S721, the sonar device transmits multiple sonar signals and obtains echo signals of suspected targets;

[0078] S722, collecting multiple echo signals to form beam domain data and obtaining the echo arrival time of the suspected target through processing;

[0079] S723, establishing a polar coordinate system and obtaining the position information of each suspected target based on the echo arrival time of each suspected target;

[0080] S724, tracking and confirming obstacles based on the position information of multiple suspected targets obtained according to the preset tracking gate, and obtaining the sonar distance of the obstacles.

[0081] In a second aspect, an embodiment of the present application provides a freshwater obstacle recognition system based on deep learning, which is applied to the freshwater obstacle recognition method as described above, including:

[0082] A detection equipment integration module is used to integrate sonar equipment, laser equipment and camera equipment into the same detection equipment to detect the shape and position of obstacles;

[0083] The data acquisition module uses the detection equipment to collect sonar echo data and camera image data and uses the two as original sample data;

[0084] The data processing module processes the collected raw sample data based on deep learning to generate a feature data set, specifically:

[0085] Using a deep generative adversarial network to preprocess the original sample data and generate a first data set;

[0086] Using a convolutional neural network to reprocess the first data set and generate a fused data set;

[0087] Performing feature extraction based on the fused data set to obtain obstacle features and thereby generate a feature data set;

[0088] A model training module, which uses the YOLO algorithm and performs model training according to the feature data set to generate an obstacle recognition model;

[0089] A water environment detection module uses an environmental detection device to detect the freshwater environment and obtain water parameters;

[0090] The obstacle recognition module collects real-time sonar echo data and camera image data using detection equipment and uses the two as real-time detection data; inputs the real-time detection data into the obstacle recognition model to obtain actual parameters of the obstacle; uses the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtains obstacle information;

[0091] The recognition result verification module uses the detection equipment to obtain laser detection data and uses it and the sonar echo data as recognition verification data to verify the position of the obstacle in the obstacle information, and performs position correction based on the verification result.

[0092] The beneficial effects of the present invention are:

[0093] (1) This application uses camera technology to perform shape imaging and type recognition of obstacles in fresh water, and eliminates the influence of the underwater environment on imaging and image recognition through deep learning. Specifically, when processing underwater images, deep generative adversarial networks (GANs) are used to perform image enhancement and denoising to improve image quality, thereby improving the effect of subsequent obstacle recognition; convolutional neural networks (CNNs) are used to process blurred and low-contrast images; the features extracted by CNN are input into the YOLO (You Only LookOnce) algorithm for real-time target detection, and obstacles are located and recognized based on the above features and bounding box information is provided; by combining the image optimization capability of GANs, the feature extraction capability of CNN and the real-time detection capability of the YOLO algorithm, the accuracy and efficiency of underwater obstacle recognition are improved; sonar detection technology and laser detection technology are also used to locate and correct the position of obstacles, eliminating the deviation in obstacle recognition and positioning caused by natural environmental factors such as water flow, waves and sediments, and significantly improving the effect of obstacle recognition and classification and the accuracy of positioning.

[0094] (2) The present application first integrates sonar equipment, laser equipment and camera equipment into the same detection equipment and uses them to detect the shape and position of obstacles; secondly, the detection equipment collects sonar echo data and camera image data and uses them as original sample data; then the original sample data is processed by deep learning to generate a feature data set; then model training is performed based on the feature data set and an obstacle recognition model is generated; then the detection equipment is used to collect real-time sonar echo data and camera image data and uses them as real-time detection data and inputs them into the obstacle recognition model to obtain the actual parameters of the obstacle; an environmental detection device is used to detect the freshwater environment to obtain water body parameters and use the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtain obstacle information; finally, the detection equipment is used to obtain laser detection data and uses it together with the sonar echo data as identification verification data to verify the position of the obstacle in the obstacle information, and the position is corrected based on the verification result to determine the accuracy of obstacle recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] For better understanding and implementation, the technical solution of the present application is described in detail below with reference to the accompanying drawings.

[0096] Figure 1 A flowchart of a method for identifying freshwater obstacles based on deep learning provided in an embodiment of the present application;

[0097] Figure 2 A flowchart of the steps of using a convolutional neural network for data fusion provided in an embodiment of the present application;

[0098] Figure 3 A flowchart of the steps of using the YOLO algorithm to perform model training in an embodiment of the present application;

[0099] Figure 4 A flowchart of the steps for obtaining obstacle information provided in an embodiment of the present application;

[0100] Figure 5 A schematic diagram of the structure of a freshwater obstacle recognition system based on deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0101] In order to further explain the technical means and effects taken by the present invention to achieve the predetermined invention purpose, exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of methods and systems consistent with some aspects of the present application as detailed in the attached claims.

[0102] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to any or all possible combinations of one or more associated listed items.

[0103] The specific implementation methods, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0104] Example 1

[0105] See also Figure 1 The present application embodiment provides a freshwater obstacle recognition method based on deep learning, comprising the following steps:

[0106] S1, integrating sonar equipment, laser equipment and camera equipment into the same detection equipment to detect the shape and position of obstacles;

[0107] S2, using the detection equipment to collect sonar echo data and camera image data and use the two as original sample data;

[0108] S3, based on deep learning, performs data processing on the collected original sample data to generate a feature data set, specifically:

[0109] Using a deep generative adversarial network to preprocess the original sample data and generate a first data set;

[0110] Using a convolutional neural network to reprocess the first data set and generate a fused data set;

[0111] Performing feature extraction based on the fused data set to obtain obstacle features and thereby generate a feature data set;

[0112] S4, using the YOLO algorithm and performing model training according to the feature data set to generate an obstacle recognition model;

[0113] S5, using an environmental detection device to detect the freshwater environment and obtain water parameters;

[0114] S6, using the detection equipment to collect real-time sonar echo data and camera image data and use the two as real-time detection data; inputting the real-time detection data into the obstacle recognition model to obtain actual parameters of the obstacle; using the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtain obstacle information;

[0115] S7, also using the detection equipment to obtain laser detection data and using it together with the sonar echo data as identification verification data to verify the position of the obstacle in the obstacle information, and performing position correction based on the verification result.

[0116] Specifically, in order to solve the problem of identifying obstacles in freshwater, the present application first integrates sonar equipment, laser equipment and camera equipment into the same detection equipment and uses them to detect the shape and position of obstacles; secondly, the detection equipment collects sonar echo data and camera image data and uses the two as original sample data; then the above original sample data is processed by deep learning to generate a feature data set; then model training is performed based on the feature data set and an obstacle recognition model is generated; then the detection equipment is used to collect real-time sonar echo data and camera image data and uses the two as real-time detection data and inputs them into the above obstacle recognition model to obtain the actual parameters of the obstacle; the environmental detection device is also used to detect the freshwater environment to obtain water body parameters and use the water body parameters as reference factors to evaluate the actual parameters of the obstacle, and thereby obtain obstacle information; finally, the detection equipment is used to obtain laser detection data and uses it and sonar echo data as identification verification data to verify the position of the obstacle in the obstacle information, and the position is corrected based on the verification result, thereby judging the accuracy of obstacle recognition. The present application uses camera technology to perform shape imaging and type recognition of obstacles in fresh water, and eliminates the influence of underwater environment on imaging and image recognition through deep learning. Specifically, when processing underwater images, deep generative adversarial networks (GANs) are used to perform image enhancement and denoising to improve image quality, thereby improving the effect of subsequent obstacle recognition; convolutional neural networks (CNNs) are used to process blurred and low-contrast images; the features extracted by CNN are input into the YOLO model through the YOLO (You Only Look Once) algorithm for real-time target detection, and obstacles are located and recognized based on the above features and bounding box information is provided; the accuracy and efficiency of underwater obstacle recognition are improved by combining the image optimization capability of GANs, the feature extraction capability of CNN and the real-time detection capability of the YOLO algorithm; sonar detection technology and laser detection technology are also used to locate and correct the position of obstacles, eliminating the deviation in the recognition and positioning of obstacles caused by natural environmental factors such as water flow, waves and sediments, and significantly improving the effect of obstacle recognition and classification and the accuracy of positioning.

[0117] It should be noted that the above-mentioned sonar equipment includes but is not limited to sonar transmitters (transducers): responsible for emitting sound waves into the water, and its types include single-frequency transmitters and multi-frequency transmitters; sonar receivers (transducers): receiving reflected sound waves, usually integrated with the transmitter to form a composite transducer; multi-beam sonar system: capable of emitting multiple sound beams at the same time, providing a wide underwater field of view and high-resolution images; synthetic aperture sonar (SAS): synthesizing high-resolution images through a sonar array on a mobile platform, suitable for high-precision detection; side-scan sonar: installed on a ship or unmanned submersible, used to scan underwater terrain and produce high-resolution side-view images; echo sounder: used to measure water depth and detect the presence of underwater objects, working on the principle of sound wave reflection; underwater sonar system: integrating sonar detection and navigation functions, usually used in submersibles and underwater robots. The above-mentioned sonar equipment can be used alone or in combination to meet different detection needs.

[0118] Laser equipment includes but is not limited to underwater laser scanners: used in underwater environments, generating high-resolution images of underwater structures through laser scanning; underwater laser rangefinders: calculating distance by emitting laser pulses and measuring their reflection time; although the above-mentioned laser equipment can be used for underwater detection, its effectiveness will be affected by water depth and water quality. Therefore, this application combines sonar detection and laser detection together to detect underwater obstacles, so that they can compensate for each other and avoid problems such as inaccurate distance detection.

[0119] The camera equipment includes but is not limited to underwater cameras: specially designed for underwater environments, with waterproof and corrosion-resistant properties, capable of capturing images or videos under different water depths and lighting conditions; underwater robot cameras: cameras mounted on remote-controlled underwater robots (ROVs), which can be remotely operated underwater, providing real-time video and images for detecting and identifying obstacles; underwater laser camera systems: combining laser scanning and camera technology to generate high-resolution underwater images and three-dimensional models, and enhance the ability to identify obstacles; underwater stereo vision systems: using two or more cameras to create three-dimensional stereo images to improve the recognition and depth perception of underwater obstacles; underwater high-definition cameras: providing high-resolution videos and images, capable of identifying and detecting underwater obstacles with high clarity; underwater multispectral cameras: capable of capturing images in different spectral ranges, helping with underwater detection and identification under different lighting conditions; underwater fiber optic cameras: transmitting high-quality video signals through optical fibers, suitable for detection and identification in deep water environments. This application combines sonar detection and camera detection to detect and identify underwater obstacles, so that they can complement each other and avoid problems such as inaccurate identification.

[0120] In one embodiment provided in the present application, in step S3, a deep generative adversarial network is used to perform data preprocessing, which specifically includes the following contents:

[0121] Image selection: Underwater images are selected based on the original sample data and the light energy of the underwater images is adjusted according to the imaging quality of the underwater images; thereby improving and enhancing the quality of the selected images.

[0122] Light energy in underwater images It is composed of forward scattering, backscattering and direct irradiation, expressed as:

[0123] ;

[0124] in, represents the direct irradiation component; represents the forward scattering component; represents the backscattering component;

[0125] Specifically, the forward scattered component It is used to reflect the small-angle scattering of light reflected by obstacles. It is emitted by obstacles and then absorbed and scattered by water to reach the camera and be received by it. The forward scattering component can be approximated as the convolution of the direct illumination component and the point spread function, which is affected by the distance between the obstacle and the camera. The point spread function is h(x,y), which describes the blurred shape of an ideal point light source after being imaged by the optical system. It reflects the resolution and blur characteristics of the imaging system. The convolution of an image I(x,y) containing the direct illumination component and the point spread function h(x,y) can simulate the blur effect of the imaging system. The calculation process is as follows:

[0126] ;in is an image containing the direct illumination component; * indicates a convolution operation; This is the blurred image.

[0127] Direct exposure It is formed by the attenuation of the reflected light from the obstacle when it propagates in the water. It describes the energy attenuation of light during propagation and is used to reflect the effect of the reflected light from the obstacle. It is expressed as:

[0128] ;

[0129] in, It represents the energy of light on the surface of the obstacle; represents the transmission efficiency, and , η is the attenuation coefficient, is the distance from the obstacle to the shooting location; the direct illumination component causes the image color to decay due to attenuation, so adjusting the direct illumination component can slow down the color decay of the image and improve the image contrast.

[0130] Backscatter component It indicates the light energy received by the camera after encountering suspended particles and undergoing a large angle change; it occurs during the entire imaging process and is affected by the distance between the camera and the obstacle. The longer the distance, the greater the probability of backscattering, which has a greater impact on image quality degradation, while the impact of forward scattering is relatively small and can be ignored. Therefore, by adjusting the backscattering component, the impact on image quality can be avoided, which can be expressed as:

[0131] ; B is related to the distant background light;

[0132] In summary, after ignoring forward scattering, the light energy of the underwater image can be simplified as:

[0133] .

[0134] Specifically, after light enters the water, it is absorbed and scattered by the water, and its energy will gradually weaken; suspended substances in the water will absorb light energy and cause scattering, thereby changing the light propagation path; the difference between different water qualities will lead to random influences on the underwater imaging quality; specifically, scattering can be divided into forward scattering and backward scattering. Forward scattering will cause the light received by the underwater camera to deviate at a small angle, resulting in loss of image details and reduced clarity; backward scattering is the light emitted by the light source being scattered by suspended particles in the water to the camera, which will reduce the contrast of the underwater image and further affect the imaging effect; therefore, in order to avoid the above factors affecting the final imaging quality of the underwater image, this embodiment establishes an underwater imaging model and adjusts the backscattering component and the direct illumination component by adjusting the light energy of the underwater image, thereby improving the image quality of the underwater imaging.

[0135] In an embodiment provided in the present application, in step S3, data preprocessing is performed using a deep generative adversarial network, further comprising:

[0136] Image enhancement: Based on the U-Net network structure, the generator in the deep generative adversarial network is improved. Specifically, a multi-directional convolution module, an embedded attention module, and a brightness equalization module combined with full convolution are added to the generator.

[0137] The brightness equalization module is used to locate the low-light area in the underwater image, including:

[0138] First, a large 9×9 convolution kernel is used to capture the global feature information of the input underwater image;

[0139] Secondly, five consecutive 3×3 convolutional layers are used to gradually extract the position information of the low-light area and output the low-light attention map;

[0140] The multi-directional convolution module is used to perform convolution operations in multiple directions to capture richer feature information of underwater images, including:

[0141] The input underwater image is divided and transformed according to the channel dimension; four different inputs are obtained through four different function operations, aiming to extract richer and more diverse feature information.

[0142] Convolution operations are performed using multi-directional convolution kernels to capture features in different directions. In the convolution stage, four convolution layers are defined to increase the network depth and convert the number of channels.

[0143] Use activation functions to perform nonlinear transformations to enhance network expression capabilities;

[0144] Reduce data dimensions through pooling to reduce computation and extract key features;

[0145] The attention module is used for dynamic weight allocation and selective focus of information, including channel attention submodule and spatial attention submodule, specifically:

[0146] The channel attention submodule is used to adaptively adjust the feature weight according to the channel relationship, specifically including:

[0147] The input underwater image is processed at the channel level using average pooling and maximum pooling functions, aiming to scale features and remove noise.

[0148] The number of channels is compressed into a one-dimensional vector through a fully connected layer and a ReLU activation function and then processed again to obtain a two-dimensional vector;

[0149] The output is converted into a probability distribution of weight values ​​in the range of 0 to 1 through the Sigmoid activation function;

[0150] The spatial attention submodule is used to perform convolution operations on underwater images, extract feature representations of each region, and use the attention mechanism to perform weighted averaging on the extracted features, specifically including:

[0151] Average pooling and maximum pooling values ​​are calculated to capture different features of the spatial dimension;

[0152] The spatial attention information is extracted through the convolution layer, and then the output is mapped to between 0 and 1 through the Sigmoid activation function to generate a spatial attention weight map;

[0153] Perform weighted averaging based on the spatial attention weight map and output the spatial positions of different features;

[0154] Image denoising: The cycle consistency loss function, the perceptual loss function and the multi-scale structural similarity index loss function are integrated to generate a joint loss function, and the joint loss function is used as the loss function of the generator for image denoising; Joint loss function The formula is as follows:

[0155] ;

[0156] in, represents the cycle consistency loss function; represents the perceptual loss function; represents the multi-scale structural similarity index loss function; and Both represent scale factors.

[0157] Specifically, the multi-directional convolution module can effectively extract features from multiple directions and scales, enhance the network's expressiveness and robustness; it can obtain different flip feature maps and multi-scale semantic information, enrich the local feature maps of underwater images, and improve the diversity and possibility of extracting key features, which better promotes the image enhancement effect. The attention module adaptively adjusts the feature weights according to the channel relationship through the channel attention submodule to enhance the focus on key channel features; through the spatial attention submodule, it focuses on specific areas of the image, accurately analyzes spatial data, realizes multi-level attention, captures complex structures and patterns, and improves the image representation effect. Combining the above two submodules, the feature extraction ability of the neural network is comprehensively enhanced, the performance of the generator is significantly improved, and the problem of blurred texture details in the image after underwater enhancement is effectively solved. This embodiment compensates for the shortcomings in color and details after underwater image enhancement through the above-mentioned joint loss function, solves the problems of blurred texture details and color distortion in underwater images, and significantly improves the quality of underwater images.

[0158] It should be noted that the above-mentioned weak light area refers to an area with weaker brightness selected after analyzing the global feature information. This embodiment does not specifically limit strong light and weak light, and is only customized according to the operation process in the algorithm steps.

[0159] In summary, through the above improved deep generative adversarial network, adding a multi-directional convolution module enhances the richness of the local feature map of the underwater image and improves the diversity and accuracy of feature extraction; adding an attention mechanism enables the deep generative adversarial network to focus more on the key features of the image; adding a brightness equalization module optimizes the brightness and contrast of the image and highlights the detail information. The joint loss function effectively improves the quality of the generated image while also speeding up the model convergence, stabilizing the GAN training process and improving the authenticity and diversity of the generated samples.

[0160] like Figure 2 As shown, in one embodiment provided in the present application, in step S3, a convolutional neural network is used to perform data reprocessing, which specifically includes the following contents:

[0161] Data synchronization: obtain the timestamp recorded by each sensor in the detection equipment when collecting data; calibrate the timestamp to correct the time error; use the calibrated timestamp to align the data of each sensor;

[0162] Data fusion: Use convolutional neural networks to fuse sonar echo data and camera image data to generate comprehensive information and form a fused data set, including:

[0163] Extract sonar features from sonar echo data;

[0164] Extract visual features from camera image data;

[0165] Adjust the feature dimensions of sonar features and visual features and map them to the same dimension;

[0166] interpolating or transforming sonar and visual features so that they are spatially and semantically aligned;

[0167] Sonar features and visual features are spliced ​​and weightedly fused in feature dimensions;

[0168] The data set obtained after weighted fusion is called feature data set.

[0169] Specifically, this embodiment uses a convolutional neural network to fuse the sonar features in the sonar echo data with the visual features in the camera data, ensuring that the sonar data and the visual data can be effectively combined, thereby providing more accurate and comprehensive data support for subsequent model training.

[0170] like Figure 3 As shown, in one embodiment provided in the present application, in step S4, the YOLO algorithm is used to perform model training, which specifically includes the following contents:

[0171] Annotate based on the feature data set to mark the category and location of the obstacle;

[0172] Configure network structure and network parameters based on YOLO algorithm;

[0173] Combine the losses of position regression, confidence prediction, and category prediction to form a custom loss function;

[0174] Use the training set in the feature dataset to train the model and optimize the network weights through the back-propagation algorithm;

[0175] Use the test set in the feature data set to evaluate the performance indicators of the model and adjust the network parameters and network structure, and save the evaluated obstacle recognition model. Performance indicators include detection accuracy, recall rate, and inference speed.

[0176] Specifically, this embodiment adopts the YOLO algorithm for model training, regards the target detection problem as a regression problem, and the entire process is completed through a single network, which makes the detection speed very fast and suitable for real-time applications; at the same time, the YOLO algorithm also effectively captures the contextual information in the image through global feature extraction and complex convolutional network structure, thereby improving the detection accuracy.

[0177] It should be noted that, in step S3 and step S4, the present application respectively adopts three different deep learning technologies, namely, deep generative adversarial networks, convolutional neural networks and YOLO algorithms, and combines the three to apply to obstacle recognition and detection of the present application. First, data enhancement and denoising are performed through deep generative adversarial networks (GANs): GANs are used to improve the quality of underwater images by generating clearer images or removing noise to improve data quality; secondly, feature extraction is performed through convolutional neural networks (CNNs): the images processed by GANs are input into CNN to extract deep features, which help capture the complex patterns and details of obstacles; finally, target detection is performed through the YOLO algorithm (target detection algorithm): the features extracted by CNN are input into the YOLO model for real-time target detection. YOLO will locate and identify obstacles based on the above features and provide bounding box information. This embodiment combines the image optimization capability of GANs, the feature extraction capability of CNN and the real-time detection capability of YOLO, thereby improving the accuracy and efficiency of underwater obstacle recognition.

[0178] In an embodiment provided in the present application, in step S5, the water body parameters include but are not limited to water turbidity, water flow velocity, water flow direction and bottom topography.

[0179] It should be noted that this embodiment does not specifically limit the environmental detection device. Any device that can obtain the above-mentioned water parameters can be used as the environmental detection device of this embodiment, such as a water quality sensor for obtaining water turbidity and a water flow meter for measuring water flow velocity and direction.

[0180] like Figure 4 As shown, in one embodiment provided in the present application, in step S6, obtaining obstacle information specifically includes:

[0181] Obtaining the type of obstacle based on the obstacle recognition model;

[0182] Acquire the pixel height of the obstacle and the distance between the obstacle and the camera device based on the feature data set;

[0183] Using a monocular ranging algorithm, the actual height of the obstacle is obtained based on pixel height and distance;

[0184] The type of obstacle, the actual height of the obstacle, and the distance between the obstacle and the camera device are collected as obstacle information.

[0185] In an embodiment provided in the present application, in step S7, verifying and correcting the position of the obstacle in the obstacle information specifically includes:

[0186] S71, acquiring laser detection data and calculating the laser distance of the obstacle;

[0187] S72, obtaining sonar echo data and calculating the sonar distance of the obstacle;

[0188] S73, obtaining a camera distance between the obstacle and the camera device from the obstacle information;

[0189] S74, comparing the laser distance and the sonar distance with the camera distance, obtaining the distance error and calculating the actual distance of the obstacle;

[0190] S75, correcting the position of the obstacle according to the actual distance of the obstacle and updating the obstacle recognition model according to the corrected data.

[0191] Specifically, this embodiment calculates the obstacle distances obtained by the two ranging technologies based on the data obtained by laser detection and the data obtained by sonar detection, and compares the above distances with the obstacle distances obtained by the camera and the obstacle recognition model, obtains the errors therein and calculates the actual distances of the obstacles, thereby correcting the distances according to the errors and updating the obstacle recognition model, thereby improving the accuracy of obstacle distance detection and the accuracy of subsequent recognition and positioning.

[0192] In an embodiment provided in the present application, in step S71, calculating the laser distance of the obstacle specifically includes:

[0193] S711, confirming the water quality level according to the water body parameters, and emitting a laser with a preset wavelength and a preset power according to the water quality level;

[0194] S712, receiving the reflected laser and calculating the laser distance of the obstacle according to the phase difference between the transmitted laser and the reflected laser and the preset wavelength;

[0195] In step S72, the sonar distance of the obstacle is calculated, specifically including:

[0196] S721, the sonar device transmits multiple sonar signals and obtains echo signals of suspected targets;

[0197] S722, collecting multiple echo signals to form beam domain data and obtaining the echo arrival time of the suspected target through processing;

[0198] S723, establishing a polar coordinate system and obtaining the position information of each suspected target based on the echo arrival time of each suspected target;

[0199] S724, tracking and confirming obstacles based on the position information of multiple suspected targets obtained according to the preset tracking gate, and obtaining the sonar distance of the obstacles.

[0200] Specifically, since laser ranging is not very suitable for underwater obstacle detection, this embodiment combines laser ranging with sonar ranging to measure and calculate the obstacle distance, thereby ensuring the accuracy of obstacle position calculation.

[0201] Example 2

[0202] See also Figure 5 The embodiment of the present application provides a freshwater obstacle recognition system based on deep learning, which is applied to the freshwater obstacle recognition method as described above, including:

[0203] A detection equipment integration module is used to integrate sonar equipment, laser equipment and camera equipment into the same detection equipment to detect the shape and position of obstacles;

[0204] The data acquisition module uses the detection equipment to collect sonar echo data and camera image data and uses the two as original sample data;

[0205] The data processing module processes the collected raw sample data based on deep learning to generate a feature data set, specifically:

[0206] Using a deep generative adversarial network to preprocess the original sample data and generate a first data set;

[0207] Using a convolutional neural network to reprocess the first data set and generate a fused data set;

[0208] Performing feature extraction based on the fused data set to obtain obstacle features and thereby generate a feature data set;

[0209] A model training module, which uses the YOLO algorithm and performs model training according to the feature data set to generate an obstacle recognition model;

[0210] A water environment detection module uses an environmental detection device to detect the freshwater environment and obtain water parameters;

[0211] The obstacle recognition module collects real-time sonar echo data and camera image data using detection equipment and uses the two as real-time detection data; inputs the real-time detection data into the obstacle recognition model to obtain actual parameters of the obstacle; uses the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtains obstacle information;

[0212] The recognition result verification module uses the detection equipment to obtain laser detection data and uses it and the sonar echo data as recognition verification data to verify the position of the obstacle in the obstacle information, and performs position correction based on the verification result.

[0213] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0215] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0216] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A freshwater obstacle identification method based on deep learning, characterized in that: The steps include: S1, integrating sonar equipment, laser equipment and camera equipment into the same detection equipment to detect the shape and position of obstacles; S2, using the detection equipment to collect sonar echo data and camera image data and generate original sample data; S3, based on deep learning, the collected original sample data is processed to generate a feature data set, specifically: Using a deep generative adversarial network to preprocess the original sample data and generate a first data set; Using a convolutional neural network to reprocess the first data set and generate a fused data set; Performing feature extraction based on the fused data set to obtain obstacle features and thereby generate a feature data set; S4, using the YOLO algorithm and performing model training according to the feature data set to generate an obstacle recognition model; S5, using an environmental detection device to detect the freshwater environment and obtain water parameters; S6, using the detection equipment to collect real-time sonar echo data and camera image data and use the two as real-time detection data; Inputting the real-time detection data into the obstacle recognition model to obtain actual parameters of the obstacle; using the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtain obstacle information; S7, using a detection device to obtain laser detection data and using the data and sonar echo data as identification verification data to verify the position of the obstacle in the obstacle information, and performing position correction according to the verification result; Among them, in step S3, the original sample data is preprocessed using a deep generative adversarial network, which specifically includes the following contents: Image selection: Select underwater images based on original sample data and adjust the light energy of underwater images according to the imaging quality of underwater images; Image enhancement: Based on the U-Net network structure, the generator in the deep generative adversarial network is improved. Specifically, a multi-directional convolution module, an embedded attention module, and a brightness equalization module combined with full convolution are added to the generator. Image denoising: The cycle consistency loss function, the perceptual loss function and the multi-scale structural similarity index loss function are integrated to generate a joint loss function, and the joint loss function is used as the loss function of the generator for image denoising.

2. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: About image selection: Light energy in underwater images It is composed of forward scattering, backscattering and direct irradiation, expressed as: ； In the formula, represents the direct irradiation component; represents the forward scattering component; represents the backscattering component; Among them, the forward scattering component It is used to reflect the small-angle scattering of light reflected from obstacles. After being absorbed and scattered by the water, the light is emitted from the obstacle and reaches the camera device and is received by it. Direct exposure It is formed by the attenuation of the reflected light from the obstacle when it propagates in the water. It describes the energy attenuation of light during propagation and is expressed as: ; In the formula, It represents the energy of light on the surface of the obstacle; represents the transmission efficiency, and , η is the attenuation coefficient, is the distance from the obstacle to the shooting location; Backscatter component It represents the light energy received by the camera after encountering the suspended particles and undergoing a large angle change, expressed as: ; B is related to distant background light.

3. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: Regarding image enhancement, the following contents are included: The brightness equalization module is used to locate the low-light area in the underwater image, and specifically includes: First, a large 9×9 convolution kernel is used to capture the global feature information of the input underwater image; Secondly, five consecutive 3×3 convolutional layers are used to gradually extract the position information of the low-light area and output the low-light attention map; The multi-directional convolution module is used to perform convolution operations in multiple directions to capture feature information of underwater images, specifically including: Divide and transform the input underwater image according to the channel dimension; Use multi-directional convolution kernels for convolution operations; Use activation functions for nonlinear transformation; Reduce data dimensions and extract key features through pooling; The attention module is used for dynamic weight allocation and selective focusing of information, including a channel attention submodule and a spatial attention submodule, specifically: The channel attention submodule is used to adaptively adjust the feature weight according to the channel relationship, specifically including: The input underwater image is processed at the channel level using average pooling and maximum pooling functions; The number of channels is compressed into a one-dimensional vector through a fully connected layer and a ReLU activation function and then processed again to obtain a two-dimensional vector; The output is converted into a probability distribution of weight values ​​in the range of 0 to 1 through the Sigmoid activation function; The spatial attention submodule is used to perform convolution operations on underwater images, extract feature representations of each region, and use the attention mechanism to perform weighted averaging on the extracted features, specifically including: Average pooling and maximum pooling values ​​are calculated to capture different features of the spatial dimension; The spatial attention information is extracted through the convolution layer, and then the output is mapped to between 0 and 1 through the Sigmoid activation function to generate a spatial attention weight map; A weighted average operation is performed based on the spatial attention weight map and the spatial positions of different features are output.

4. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: In step S3, a convolutional neural network is used to perform data reprocessing, which specifically includes the following contents: Data synchronization: obtain the timestamp recorded by each sensor in the detection equipment when collecting data; calibrate the timestamp to correct the time error; use the calibrated timestamp to align the data of each sensor; Data fusion: Use convolutional neural networks to fuse sonar echo data and camera image data to generate comprehensive information and form a fused data set, including: Extract sonar features from sonar echo data; Extract visual features from camera image data; Adjust the feature dimensions of sonar features and visual features and map them to the same dimension; interpolating or transforming sonar and visual features so that they are spatially and semantically aligned; Sonar features and visual features are spliced ​​and weightedly fused in feature dimensions; The data set obtained after weighted fusion is called feature data set.

5. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: In step S4, the YOLO algorithm is used to perform model training, which specifically includes the following contents: Annotate based on the feature data set to mark the category and location of the obstacle; Configure network structure and network parameters based on YOLO algorithm; Combine the losses of position regression, confidence prediction, and category prediction to form a custom loss function; Use the training set in the feature dataset to train the model and optimize the network weights through the back-propagation algorithm; Use the test set in the feature data set to evaluate the performance indicators of the model and adjust the network parameters and network structure, and save the evaluated obstacle recognition model.

6. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: In step S5, the water body parameters include water turbidity, water flow velocity, water flow direction and underwater topography.

7. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: In step S6, obtaining obstacle information specifically includes: Obtaining the type of obstacle based on the obstacle recognition model; Acquire the pixel height of the obstacle and the distance between the obstacle and the camera device based on the feature data set; Using a monocular ranging algorithm, the actual height of the obstacle is obtained based on pixel height and distance; The type of obstacle, the actual height of the obstacle, and the distance between the obstacle and the camera device are collected as obstacle information.

8. The method for identifying freshwater obstacles based on deep learning according to claim 1, characterized in that: In step S7, the position of the obstacle in the obstacle information is verified and corrected, specifically including: S71, acquiring laser detection data and calculating the laser distance of the obstacle; S72, obtaining sonar echo data and calculating the sonar distance of the obstacle; S73, obtaining a camera distance between the obstacle and the camera device from the obstacle information; S74, comparing the laser distance and the sonar distance with the camera distance, obtaining the distance error and calculating the actual distance of the obstacle; S75, correcting the position of the obstacle according to the actual distance of the obstacle and updating the obstacle recognition model according to the corrected data.

9. A method for identifying freshwater obstacles based on deep learning according to claim 8, characterized in that: In step S71, the laser distance of the obstacle is calculated, specifically including: S711, confirming the water quality level according to the water body parameters, and emitting a laser with a preset wavelength and a preset power according to the water quality level; S712, receiving the reflected laser and calculating the laser distance of the obstacle according to the phase difference between the transmitted laser and the reflected laser and the preset wavelength; In step S72, the sonar distance of the obstacle is calculated, specifically including: S721, the sonar device transmits multiple sonar signals and obtains echo signals of suspected targets; S722, collecting multiple echo signals to form beam domain data and obtaining the echo arrival time of the suspected target through processing; S723, establishing a polar coordinate system and obtaining the position information of each suspected target based on the echo arrival time of each suspected target; S724, tracking and confirming obstacles based on the position information of multiple suspected targets obtained according to the preset tracking gate, and obtaining the sonar distance of the obstacles.

10. A freshwater obstacle recognition system based on deep learning, applied to the freshwater obstacle recognition method according to any one of claims 1 to 9, characterized in that: include: A detection equipment integration module is used to integrate sonar equipment, laser equipment and camera equipment into the same detection equipment to detect the shape and position of obstacles; The data acquisition module uses the detection equipment to collect sonar echo data and camera image data and uses the two as original sample data; The data processing module processes the collected raw sample data based on deep learning to generate a feature data set, specifically: Using a deep generative adversarial network to preprocess the original sample data and generate a first data set; Using a convolutional neural network to reprocess the first data set and generate a fused data set; Performing feature extraction based on the fused data set to obtain obstacle features and thereby generate a feature data set; A model training module, which uses the YOLO algorithm and performs model training according to the feature data set to generate an obstacle recognition model; A water environment detection module uses an environmental detection device to detect the freshwater environment and obtain water parameters; The obstacle recognition module uses detection equipment to collect real-time sonar echo data and camera image data and uses the two as real-time detection data; Inputting the real-time detection data into the obstacle recognition model to obtain actual parameters of the obstacle; using the water body parameters as reference factors to evaluate the actual parameters of the obstacle and thereby obtain obstacle information; The recognition result verification module uses the detection equipment to obtain laser detection data and uses it and the sonar echo data as recognition verification data to verify the position of the obstacle in the obstacle information, and performs position correction based on the verification result.

Images