Cross-modal intelligent identification method for aquatic organisms in complex aquatic environments

Abstract

This invention discloses a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments. The invention relates to the field of intelligent aquatic organism identification technology, and involves collecting optical and acoustic image data of aquatic organisms, establishing pixel mapping relationships between the optical and acoustic data, and using a cross-attention cross-modal feature extraction network with spatial and channel branches for fusion feature extraction. The identification model is then subjected to structured pruning, quantized perceptual training, and knowledge distillation to achieve lightweight edge deployment. The advantages of this invention are: by using a cross-modal alignment method based on sonar-like transformation and adaptive Kalman filtering, it effectively solves the differences between optical images and acoustic data in coordinate systems, data formats, and time synchronization, providing a high-quality aligned data foundation for subsequent multimodal fusion; and designing a cross-attention cross-modal feature extraction network with spatial and channel branches.

Description

Technical Field

[0001] This invention relates to the field of intelligent aquatic organism identification technology, specifically a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments. Background Technology

[0002] With the acceleration of industrialization and the intensification of human activities, problems such as water pollution and habitat destruction have become increasingly prominent. Complex aquatic environments (such as turbid water, uneven lighting, partial obstruction, and long-distance monitoring scenarios) are widespread, posing a severe challenge to the identification of aquatic organisms. Against this backdrop, achieving accurate and efficient identification of aquatic organisms in complex aquatic environments has important practical significance and application value for aquatic ecological environment monitoring, protection of rare aquatic organisms, rational regulation of fishery resources, and early warning of ecological risks. Common cross-modal intelligent identification methods for aquatic organisms often rely on single optical imaging, which can lead to inaccuracies in turbid water, uneven lighting, and partial occlusion. Optical and acoustic data are difficult to fuse directly due to differences in coordinate systems, sampling rates, and timestamps, resulting in insufficient feature complementarity. Traditional networks often focus on either local details or global semantics, which are insufficient for discriminative representation in complex scenarios, thus limiting the accuracy and robustness of identification. To address these issues, we propose a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments. Summary of the Invention

[0003] The purpose of this invention is to provide a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments, comprising the following steps: S100: Simultaneously collects optical image data and acoustic data of aquatic organisms through multi-source sensors carried by the underwater robot; S200: Perform cross-modal alignment processing on the optical image data and the acoustic data to obtain aligned multimodal data; S300. Perform adaptive enhancement processing based on generative adversarial networks on the optical image data to obtain an enhanced image; S400. Input the aligned multimodal data into a cross-modal feature extraction network based on a cross-attention mechanism to extract cross-modal fusion features; S500: The cross-modal fusion features are fused with the features of the enhanced image, and the aquatic organisms are identified through the recognition network, and the species, quantity and size information of the aquatic organisms are output.

[0005] As a further aspect of the present invention, the cross-modal alignment process includes: A mapping relationship between the pixel coordinates of the optical image data and the distance-azimuth coordinates of the acoustic data is established based on sonar-like transformation; The homography matrix is ​​calculated based on the mapping relationship. The optical image data and the acoustic data are then transformed into a unified image coordinate space using the homography matrix to achieve spatial alignment. An adaptive Kalman filter is used to take acoustic distance information, optical feature point coordinate information and inertial navigation pose data as multi-source observation inputs, and to perform spatiotemporal joint alignment of the optical image data and the acoustic data. The measurement noise covariance matrix of the adaptive Kalman filter is dynamically adjusted according to the sensor signal quality assessment results at the current moment.

[0006] As a further embodiment of the present invention, the cross-modal feature extraction network based on the cross-attention mechanism includes spatial branches and channel branches; The spatial branch takes features from one modality as query input and features from another modality as key and value input, performs cross-attention calculation in the spatial dimension, captures local spatial features in cross-modal data, and outputs a spatial attention feature map.

[0007] As a further aspect of the present invention: the channel branch takes the channel features of one modality as the query input and the channel features of another modality as the key and value input, performs cross-attention calculation in the channel dimension, establishes global semantic association between cross-modal features, and outputs a channel attention feature map; The spatial attention feature map and the channel attention feature map are spliced ​​and fused to obtain the cross-modal fusion feature.

[0008] As a further aspect of the present invention: in the adaptive enhancement processing based on generative adversarial networks, the generator adopts a U-Net structure, including a symmetrically designed encoder and decoder. A skip connection is set between the corresponding layers of the encoder and the decoder to directly transmit the low-frequency features extracted by the encoder to the decoder for feature reuse. The generator takes the original underwater optical image as input and outputs an enhanced underwater image.

[0009] As a further aspect of the present invention: the discriminator of the generative adversarial network includes an adversarial branch and a scene classification branch; The adversarial branch is used to determine the probability that the input image is a real image; The scene classification branch is used to classify and determine the underwater scene category to which the input image belongs.

[0010] As a further aspect of the present invention, the adaptive enhancement processing further includes: pre-generating a reference image using a preset image processing algorithm, using the difference between the reference image and the enhanced image output by the generator as a reference loss, and guiding the generative adversarial network to achieve a dynamic balance between color restoration and texture enhancement.

[0011] As a further aspect of the present invention, it also includes lightweighting the recognition network, the lightweighting process comprising: The importance of each channel in the convolutional layer of the recognition network is evaluated, and convolutional channels with importance scores below a preset threshold are removed to complete structured pruning. Quantization perception training is performed on the pruned network to convert the floating-point parameters in the network into low-ratio fixed-point parameters, and the recognition accuracy is restored by fine-tuning with mixed precision. The original recognition network before pruning and quantization is used as the teacher model, and the lightweight network is used as the student model. Knowledge distillation training is performed through feature similarity constraints to make the student model approximate the feature representation ability of the teacher model.

[0012] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows: 1. This invention effectively solves the problem of differences between optical images and acoustic data in terms of coordinate system, data format and time synchronization by adopting a cross-modal alignment method based on sonar-like transformation and adaptive Kalman filtering. It provides a high-quality aligned data foundation for subsequent multimodal fusion. The designed cross-attention cross-modal feature extraction network, which includes spatial and channel branches, can simultaneously capture local spatial details and global semantic association features in cross-modal data. It can still effectively extract the discriminative features of aquatic organisms under complex conditions such as partial occlusion, turbid water, and long distance, significantly improving the accuracy and robustness of recognition. 2. This invention, through a dual-branch discriminator and a reference image guidance mechanism, can adaptively adjust the enhancement strategy according to the degradation characteristics of different underwater scenes, achieving a dynamic balance between color restoration and texture enhancement, and effectively improving the visual quality of optical images in complex underwater environments; 3. This invention employs a three-stage lightweighting process—structured pruning, quantitative perception training, and knowledge distillation—to significantly reduce the number of model parameters and computational complexity while maintaining recognition accuracy, enabling real-time inference deployment on an embedded edge computing platform mounted on an underwater robot. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method steps in an embodiment of the present invention. Detailed Implementation

[0014] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0015] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0016] This invention relates to a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments.

[0017] Therefore, in order to effectively solve the above problems, this application proposes a cross-modal intelligent identification method for aquatic organisms in complex aquatic environments, as shown in the accompanying drawings. Figure 1 As shown; Example 1, Overall Method Overview: First, optical image data and acoustic data of aquatic organisms are collected synchronously by multi-source sensors carried by the underwater robot. The multi-source sensors include an underwater optical camera and an acoustic detector. The optical camera is used to acquire high-resolution color images of aquatic organisms, and the acoustic detector is used to acquire distance and orientation information of aquatic organisms. The two sensors are synchronized at the hardware level to ensure the time consistency of the collected data. Then, cross-modal alignment processing is performed on the acquired optical image data and acoustic data. This processing establishes a spatial mapping relationship between optical and acoustic data based on sonar-like transformation, achieves spatial alignment through homography matrix, and completes spatiotemporal joint alignment by fusing multi-source information through adaptive Kalman filter. Meanwhile, adaptive enhancement processing based on generative adversarial networks is applied to optical image data to improve the degradation problem of optical images in complex aquatic environments and enhance the effectiveness of subsequent feature extraction. Next, the aligned multimodal data is input into a cross-modal feature extraction network based on a cross-attention mechanism. This network extracts local spatial features and global semantic features through spatial branches and channel branches, respectively, and then concatenates and fuses the two types of features to form cross-modal fusion features. Finally, the cross-modal fusion features are fused with the features of the enhanced image, and the aquatic organisms are classified and located by the recognition network. The information on the species, quantity and size of the aquatic organisms is output. In actual deployment, the recognition network is lightweighted and runs on the embedded edge computing platform of the underwater robot.

[0018] Example 2, Cross-modal data alignment: The cross-modal data alignment process of this invention specifically includes the following steps: Spatial alignment: The image data acquired by the optical camera is located in a two-dimensional pixel coordinate system, while the data acquired by the acoustic detector is located in a range-azimuth polar coordinate system. The two have fundamental differences in coordinate systems. This invention adopts a sonar-like transformation method to map the range-azimuth coordinates of the acoustic data to the same pixel coordinate space as the optical image. Specifically, the sonar-like transformation generates a set of matching point pairs by establishing a geometric correspondence between the distance and azimuth angle of each data point in the acoustic data and the pixel coordinates of the optical image. Based on these matching point pairs, the homography matrix is ​​solved using the direct linear transformation method. The homography matrix describes the projection transformation relationship between the two coordinate spaces, and its transformation formula is as follows: ; in, This represents the initial mapped coordinates of the acoustic data after sonar-like transformation. This represents the alignment coordinates in the optical image coordinate space after homography matrix transformation. The scale factor for homogeneous coordinates. for The homography matrix; Through this transformation, each data point in the acoustic data is projected into the pixel coordinate space of the optical image, achieving spatial alignment of the two modal data. Spatiotemporal joint alignment step: Based on the completion of spatial alignment, this invention uses an adaptive Kalman filter to further fuse acoustic distance information, optical feature point coordinate information and inertial navigation pose data to eliminate time asynchrony errors caused by differences in sensor sampling rates and underwater robot motion; The core recursive process of the adaptive Kalman filter is as follows: State prediction phase: ; ; Status update phase: ; ; ; The meanings of the symbols in the above formula are as follows: Indicates the first The state prediction vector at time step 1, which contains the target's position coordinates and velocity components in the aligned coordinate space. Indicates the first The optimal state estimation vector at time t. Indicates the first The state transition matrix at time t describes the change in system state from the t-th time step. Time to the The dynamic evolutionary relationship at any given moment. The covariance matrix representing the state prediction. Indicates the first The covariance matrix of the state estimation at time step. The process noise covariance matrix is ​​used to describe the uncertainties in the system modeling process. Indicates the first The Kalman gain matrix at time 10:00. Indicates the first The observation matrix at time step is used to map the state space to the observation space. Indicates the first The measurement noise covariance matrix at time step. Indicates the first The observation vector at any given time is composed of the acoustic distance measurement, the coordinates of the optical feature points, and the inertial navigation pose increment. Represents the identity matrix with the same dimension as the state vector; The adaptive characteristics of the Kalman filter in this invention are reflected in the measurement noise covariance matrix. Regarding the dynamic adjustment mechanism, specifically, at each moment... The system adjusts the signal-to-noise ratio of the current acoustic signal and the sharpness of the optical image based on these parameters. Adjust the corresponding components in: When the signal quality of a sensor deteriorates, increasing the value of the corresponding observed component of that sensor... The variance value in the data is used to reduce the weight of the sensor data in the fusion process; Conversely, when the signal quality is good, the corresponding variance value is reduced and its fusion weight is increased. Through this adaptive mechanism, the filter can automatically adjust the fusion ratio of each sensor data according to the real-time changes in the underwater environment, thereby improving the robustness of cross-modal alignment.

[0019] Example 3: Cross-modal feature extraction based on cross-attention mechanism: The cross-modal feature extraction network designed in this invention adopts a cross-attention mechanism, which includes two parallel processing paths: spatial branch and channel branch. The core task of spatial branching is to capture local spatial features in cross-modal data. In spatial branching, a feature map from one modality is used as the query input, and a feature map from another modality is used as the key and value input. Cross-attention calculation is performed in the spatial dimension. The formula for cross-attention calculation is as follows: ; in, This is the spatial query matrix obtained by linear projection of optical modal features. The spatial bond matrix is ​​obtained by linear projection of acoustic modal features. The spatial value matrix is ​​obtained by linear projection of acoustic modal features. The dimension of the spatial key vector is used to scale the dot product to stabilize the gradient. Spatial branching enables the optical modality to focus on the corresponding region information in the acoustic modality at the spatial location level, especially in complex environments, and can maintain sensitivity to details, outputting a spatial attention feature map. Channel branching focuses on establishing global semantic associations between cross-modal features. In channel branching, each channel of the feature map is treated as a semantic feature dimension. A channel feature from one modality is used as the query input, and a channel feature from another modality is used as the key and value input. Cross-attention calculation is performed along the channel dimension. The cross-attention calculation formula for channel branching is as follows: ; in, This is the channel query matrix obtained by reorganizing optical modal features through channel dimensions and linear projection. The channel key matrix is ​​obtained by reorganizing the acoustic modal features through channel dimensions and linear projection. The channel value matrix is ​​obtained by reorganizing the acoustic modal features through channel dimensions and linear projection. Given the dimension of the channel key vector, through cross-attention interaction between channels, channel branches can establish global contextual dependencies between optical and acoustic features at the semantic level, and output channel attention feature maps. The spatial attention feature map output from the spatial branch and the channel attention feature map output from the channel branch are concatenated along the channel dimension, and then... The convolutional layer performs channel number compression and feature integration to obtain cross-modal fusion features. These fusion features contain both local spatial detail information and global semantic association information, providing rich discriminative feature representations for subsequent recognition.

[0020] Example 4: Adaptive Image Enhancement Based on Generative Adversarial Networks: This invention employs generative adversarial networks to adaptively enhance underwater optical images, thereby addressing the problem of image quality degradation in complex aquatic environments. Generator Design: The generator adopts a U-Net structure, consisting of a symmetrical encoder and decoder. The encoder is composed of multiple stacked convolutional downsampling modules, used to extract multi-scale features of the input underwater image layer by layer. The decoder is composed of multiple stacked deconvolutional upsampling modules, used to restore the spatial resolution of the image layer by layer. Skip connections are set between corresponding layers of the encoder and decoder, so that the low-frequency structural features extracted by the encoder are directly passed to the corresponding layer of the decoder and concatenated with the upsampling features of the decoder. This allows the generator to maintain the overall structural consistency of the image while reconstructing high-frequency details. The input of the generator is the original degraded underwater image, and the output is the enhanced underwater image. Discriminator Design: The discriminator adopts a dual-branch structure, including an adversarial branch and a scene classification branch. The adversarial branch determines whether the input image is a real, clear image or an enhanced image output by the generator, and its output is a scalar probability value. The scene classification branch classifies the underwater scene category to which the input image belongs. Different scene categories correspond to different underwater degradation characteristics, such as shallow water with strong light, deep water with weak light, and turbid water. The two branches share the feature extraction convolutional layer at the bottom of the discriminator, and each outputs independently at the higher levels. Adaptive enhancement mechanism: This invention pre-generates a reference image by using a preset image processing algorithm (including a cascade of conventional processing such as white balance correction, histogram equalization and contrast stretching). The reference image represents the basic reference for the ideal enhancement direction. During training, the pixel-level difference between the enhanced image output by the generator and the reference image is used as the reference loss, which together with the adversarial loss and scene classification loss constitutes the total loss function. The total loss function is: ; in, This represents the adversarial loss, used to drive the generator to output an enhanced result that visually approximates a realistic and clear image. This represents the scene classification loss, used to guide the discriminator and generator to perceive the degradation type of the current underwater scene. The reference loss is calculated using the pixel-wise absolute error between the generator's output image and the pre-generated reference image. The weighting coefficient to counteract the loss is set to 1.0. The weighting coefficient for the scene classification loss is 0.5. The weighting coefficient for the reference loss is set to 0.3. By jointly optimizing the three losses mentioned above, the generative adversarial network can adaptively achieve a dynamic balance between color restoration and texture enhancement according to the degradation characteristics of different underwater scenes. In scenes with severe color distortion, the network focuses on color restoration, while in scenes with blurred textures, the network focuses on texture enhancement.

[0021] Example 5: Lightweighting of the recognition model: To enable real-time deployment of the recognition network on an embedded edge computing platform for underwater robots, this invention performs three-level lightweighting on the trained recognition network. Structured pruning: The importance of each channel in each convolutional layer of the recognition network is evaluated. Specifically, the absolute value of the scaling factor in the batch normalization layer corresponding to each channel is used as the importance score of that channel. The importance scores of all channels are globally sorted, and channels with scores below a preset threshold are removed from the network to obtain a compact network structure after pruning. The pruned network has significantly reduced parameters and computational cost, while retaining key channels that contribute significantly to the recognition task. Quantization-aware training: Quantization is performed on the pruned network to convert the 32-bit floating-point precision network parameters and activation values ​​into 8-bit fixed-point integer representations. To mitigate the precision loss caused by quantization, quantization-aware training is introduced during the quantization process. This involves simulating quantization errors during forward propagation and using a pass-through estimator to transmit gradients during backpropagation. Furthermore, a mixed precision strategy is adopted for different layers in the network: higher precision is maintained for layers with high sensitivity in the recognition task, while lower bit precision is used for layers with low sensitivity, achieving a balance between precision preservation and computational efficiency. Knowledge distillation: The original recognition network before pruning and quantization is used as the teacher model, and the lightweight network is used as the student model. Knowledge distillation training is performed. The core idea of ​​knowledge distillation is to make the feature representation output by the intermediate layer of the student model approximate the feature representation of the corresponding layer of the teacher model through feature similarity constraints. Specifically, feature mapping pairs are established between several key intermediate layers of the teacher model and the student model. The training of the student model is guided by minimizing the mean square error between the corresponding feature mapping pairs. Through knowledge distillation, the lightweight student model can inherit the feature expression ability of the teacher model, and maintain the recognition accuracy close to that of the original model while significantly reducing the model complexity. After the above three-level lightweighting process, the number of parameters and computational load of the recognition model are greatly reduced, enabling real-time reasoning on the embedded edge computing platform carried by the underwater robot, thereby completing the on-site real-time identification and output of aquatic organism species, quantity and size information; While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any variations and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the invention, fall within the protection scope defined by the claims of the present invention.

Claims

1. A cross-modal intelligent identification method for aquatic organisms in complex aquatic environments, characterized in that, Includes the following steps: S100: Simultaneously collects optical image data and acoustic data of aquatic organisms through multi-source sensors carried by the underwater robot; S200: Perform cross-modal alignment processing on the optical image data and the acoustic data to obtain aligned multimodal data; S300. Perform adaptive enhancement processing based on generative adversarial networks on the optical image data to obtain an enhanced image; S400. Input the aligned multimodal data into a cross-modal feature extraction network based on a cross-attention mechanism to extract cross-modal fusion features; S500: The cross-modal fusion features are fused with the features of the enhanced image, and the aquatic organisms are identified through a recognition network, outputting the species, quantity, and size information of the aquatic organisms.

2. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 1, characterized in that, The cross-modal alignment process includes: A mapping relationship between the pixel coordinates of the optical image data and the distance-azimuth coordinates of the acoustic data is established based on sonar-like transformation; The homography matrix is ​​calculated based on the mapping relationship. The optical image data and the acoustic data are then transformed into a unified image coordinate space using the homography matrix to achieve spatial alignment. An adaptive Kalman filter is used to take acoustic distance information, optical feature point coordinate information and inertial navigation pose data as multi-source observation inputs, and to perform spatiotemporal joint alignment of the optical image data and the acoustic data. The measurement noise covariance matrix of the adaptive Kalman filter is dynamically adjusted according to the sensor signal quality assessment results at the current moment.

3. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 1, characterized in that, The cross-modal feature extraction network based on the cross-attention mechanism includes spatial branches and channel branches; The spatial branch takes features from one modality as query input and features from another modality as key and value input, performs cross-attention calculation in the spatial dimension, captures local spatial features in cross-modal data, and outputs a spatial attention feature map.

4. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 3, characterized in that: The channel branch takes channel features from one modality as query input and channel features from another modality as key and value input, performs cross-attention calculation on the channel dimension, establishes global semantic associations between cross-modal features, and outputs a channel attention feature map; The spatial attention feature map and the channel attention feature map are spliced ​​and fused to obtain the cross-modal fusion feature.

5. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 1, characterized in that: In the adaptive enhancement processing based on generative adversarial networks, the generator adopts a U-Net structure, including a symmetrically designed encoder and decoder. Skip connections are set between corresponding layers of the encoder and the decoder to directly pass the low-frequency features extracted by the encoder to the decoder for feature reuse. The generator takes the original underwater optical image as input and outputs an enhanced underwater image.

6. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 5, characterized in that: The discriminator of the generative adversarial network includes an adversarial branch and a scene classification branch; The adversarial branch is used to determine the probability that the input image is a real image; The scene classification branch is used to classify and determine the underwater scene category to which the input image belongs.

7. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 6, characterized in that, The adaptive enhancement process further includes: pre-generating a reference image using a preset image processing algorithm, using the difference between the reference image and the enhanced image output by the generator as a reference loss, and guiding the generative adversarial network to achieve a dynamic balance between color restoration and texture enhancement.

8. The cross-modal intelligent identification method for aquatic organisms in complex aquatic environments according to claim 1, characterized in that: It also includes lightweighting the recognition network, the lightweighting process including: The importance of each channel in the convolutional layer of the recognition network is evaluated, and convolutional channels with importance scores below a preset threshold are removed to complete structured pruning. Quantization perception training is performed on the pruned network to convert the floating-point parameters in the network into low-ratio fixed-point parameters, and the recognition accuracy is restored by fine-tuning with mixed precision. The original recognition network before pruning and quantization is used as the teacher model, and the lightweight network is used as the student model. Knowledge distillation training is performed through feature similarity constraints to make the student model approximate the feature representation ability of the teacher model.

Images