Multi-mode underwater biological intrusion detection method and device

Abstract

The invention provides a multi-mode underwater biological intrusion detection method and device, and the method comprises the steps: carrying out the sampling of each sampling grid of a target water area through acoustic equipment and image equipment, and obtaining the acoustic data and image data of each sampling grid; extracting and fusing the acoustic data and the image data of each sampling grid to obtain multi-modal features of each sampling grid, taking the multi-modal features as input of a discriminator in the generative adversarial network, and obtaining a discrimination result output by the discriminator as existence of invasive organisms or absence of the invasive organisms in the sampling grids; under the condition that the invasive organism exists in the sampling grid, sampling and detecting the eDNA environment sample on the sampling grid, and determining whether the invasive organism exists in the sampling grid or not based on a detection result. According to the method, rapid identification is carried out by utilizing multi-modal information and acousto-optic characteristics, the identification accuracy is improved by utilizing eDNA, the defects of a single method are overcome, and the efficiency of monitoring invasive species is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater biometric identification, and particularly to a multi-modal underwater biological invasion detection method and device. Background Art

[0002] Due to reasons such as human introduction and commercial spread, many water areas are facing the threat of invasion by alien species. Alien invasive species can lead to a decline in local biodiversity and the destruction of ecological balance, causing huge losses to fishery resources and the economy. Therefore, the development of efficient and accurate underwater alien invasive species monitoring methods is crucial for ecological protection.

[0003] Currently, the monitoring means for underwater biological invasion are still relatively limited. Traditional monitoring methods mainly rely on manual fishing and observation, underwater sonar detection, and environmental DNA (eDNA) analysis. However, these methods have their respective limitations: Although detecting the DNA of target species through water samples can confirm the presence of the species, it is difficult to provide accurate location information and is vulnerable to DNA degradation and environmental noise interference; Although manual fishing and diving surveys can obtain direct species data, they are costly, time-consuming, and difficult to cover vast water areas.

[0004] The limitations of single monitoring means make it difficult to meet the monitoring requirements for invasive species in terms of high precision, real-time performance, and large-scale coverage. Summary of the Invention

[0005] The present invention provides a multi-modal underwater biological invasion detection method and device to solve the defect that the limitations of single monitoring means in the prior art are difficult to meet the monitoring requirements for invasive species, and to realize a multi-modal and high-precision underwater invasive biological detection method.

[0006] The present invention provides a multi-modal underwater biological invasion detection method, including: Sampling respectively through an acoustic device and an image device in each sampling grid of a target water area to obtain the acoustic data and image data of each sampling grid; Based on the acoustic data and image data of each sampling grid, respectively extract and fuse to obtain the multi-modal features of each sampling grid, and use the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network to obtain the discriminant result output by the discriminator that there is an invasive organism or no invasive organism in the sampling grid. Among them, in the training process of the generative adversarial network, its generator generates images for simulating native organisms, and its discriminator is used to discriminate whether the generated images are native organism images or invasive organism images; In the case where the discriminant result of the discriminator is that there is an invasive organism in the sampling grid, sample and detect the eDNA environmental sample of this sampling grid, and determine whether there is an invasive organism in this sampling grid based on the detection result.

[0007] A multimodal underwater biological invasion detection method provided by the present invention. The step of respectively sampling through an acoustic device and an image device in each sampling grid of the target water area to obtain the acoustic data and image data of each sampling grid specifically includes: Determine the size of the sampling grid according to the coverage range of the acoustic device, and divide the target water area into a plurality of the sampling grids as acoustic sampling grids; According to the coverage range of the optical device, divide a plurality of optical sampling grids in each sampling grid; In each sampling grid, use the acoustic device for sampling, and use the optical device for sampling in its corresponding optical sampling grid, and take the sampled data as the acoustic data and image data of each sampling grid.

[0008] A multimodal underwater biological invasion detection method provided by the present invention. The step of respectively extracting and fusing the multimodal features of each sampling grid based on the acoustic data and image data of each sampling grid specifically includes: After preprocessing the acoustic data and image data of each sampling grid, use a pre-constructed local biological feature database for filtering to filter the acoustic data and image data with a similarity to the biological features stored in the local biological database of not less than 90%; Adopt a cross-attention mechanism to perform feature fusion on the filtered acoustic data and optical data in each sampling grid to obtain the multimodal features of each sampling grid.

[0009] A multimodal underwater biological invasion detection method provided by the present invention. The step of preprocessing the acoustic data and image data of each sampling grid specifically includes: Construct a measurement matrix to extract the observation data reflecting the target physical characteristics in the acoustic data of each sampling grid, and process it into the form of a standard normal distribution to obtain the preprocessed acoustic data of each sampling grid; Perform filtering and enhancement processing on the image data of each sampling grid to obtain the preprocessed image data of each sampling grid.

[0010] A multimodal underwater biological invasion detection method provided by the present invention. Before the step of taking the multimodal features of each sampling grid as the input of the discriminator in the generative adversarial network and obtaining the discrimination result output by the discriminator that there are invasive organisms or non-invasive organisms in the sampling grid, it further includes: Construct a local organism dataset based on the acoustic and image features of local organisms in the target water area, and train a generative adversarial network on the local organism dataset, so that the generator of the generative adversarial network generates adversarial samples similar to the local organism features that can deceive the discriminator.

[0011] According to a multimodal underwater biological invasion detection method provided by the present invention, before the step of training the generative adversarial network on the local organism dataset, it further includes: Train a basic model based on an invasive species database, where the basic model takes the multimodal features of invasive species as input and the class labels of invasive species as output; Migrate the feature extraction network of the basic model to the discriminator as the feature extraction network of the discriminator.

[0012] The present invention also provides a multimodal underwater biological invasion detection device, including: An acquisition module, configured to sample through an acoustic device and an image device in each sampling grid of the target water area to obtain acoustic data and image data of each sampling grid; A screening module, configured to respectively extract and fuse the multimodal features of each sampling grid based on the acoustic data and image data of each sampling grid, use the multimodal features of each sampling grid as the input of the discriminator in the generative adversarial network, and obtain the discriminant result output by the discriminator that there are invasive organisms or non-invasive organisms in the sampling grid. Among them, during the training process of the generative adversarial network, its generator generates images for simulating local organisms, and its discriminator is used to discriminate whether the generated images are local organism images or invasive organism images; A determination module, configured to, when the discriminant result of the discriminator is that there are invasive organisms in the sampling grid, sample and detect the eDNA environmental samples of the sampling grid, and determine whether there are invasive organisms in the sampling grid based on the detection results.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the multimodal underwater biological invasion detection method as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the multimodal underwater biological invasion detection method as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the multimodal underwater biological invasion detection method as described in any one of the above.

[0016] The multi-modal underwater biological invasion detection method and device provided by the present invention obtain acoustic and optical data by sampling each sampling grid in the target water area, and preliminarily determine whether the target water area contains invasive organisms based on multi-modal features characterizing acoustic and optical characteristics and a pre-trained generative adversarial network. Then, eDNA sampling is carried out on the sampling grids where the determination result indicates the existence of invasive organisms, so as to make a joint decision in combination with eDNA to obtain a more accurate underwater invasive organism recognition result. By utilizing multi-modal information, rapid recognition is carried out using acoustic and optical features, and the recognition accuracy is improved using eDNA, making up for the deficiencies of a single method and improving the efficiency of invasive species monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 is one of the flow schematic diagrams of the multi-modal underwater biological invasion detection method provided by the present invention; Figure 2 is another flow schematic diagram of the multi-modal underwater biological invasion detection method provided by the present invention; Figure 3 is the structural schematic diagram of the multi-modal underwater biological invasion detection device provided by the present invention; Figure 4 is the structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0020] The following will be combined with Figure 1 and Figure 2 to introduce the multi-modal underwater biological invasion detection method of the present invention. As Figure 1 shown, it includes: Step 101: Sampling respectively through an acoustic device and an image device in each sampling grid of the target water area to obtain the acoustic data and image data of each sampling grid; Sampling grids are pre-divided in the target water area.

[0021] Optionally, the sampling grid can be divided according to factors such as research requirements and the detection range of the device, as long as the obtained sampling grid facilitates the acquisition of acoustic data and image data.

[0022] For each sampling grid, an acoustic device such as a sonar is used to obtain the sonar echo of the sampling grid as acoustic data. A number of underwater images of the area are collected using an image device such as a camera as the image data of the sampling grid.

[0023] The acoustic data and a number of image data of each sampling grid are sorted and organized into a set of data according to a preset rule as the acoustic data and image data of each sampling grid.

[0024] It can be understood that the preset rule is only to sort and organize the data obtained in different modalities and at different times according to a certain rule, and it can be preset based on the type and quantity of the collected data.

[0025] Step 102, based on the acoustic data and image data of each sampling grid, extract and fuse to obtain the multi-modal features of each sampling grid, and use the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network, and the discriminant result output by the discriminator is that there is an invasive organism or no invasive organism in the sampling grid. Among them, in the training process of the generative adversarial network, its generator generates images for simulating native organisms, and its discriminator is used to discriminate whether the generated images are native organism images or invasive organism images; Feature extraction is performed on the acoustic data of each sampling grid to obtain acoustic features characterizing the underwater biological acoustic characteristics in the sampling grid.

[0026] Feature extraction is performed on the image data of each sampling grid to obtain image features characterizing the underwater biological optical characteristics in the sampling grid.

[0027] After fusing the acoustic features and image features, multi-modal features characterizing the underwater biological acoustic-optical characteristics in the sampling grid are obtained.

[0028] Usually, in order to implement the recognition task based on multi-modal features, the extracted multi-modal features are usually selected to be input into a pre-trained softmax classifier, and the probability of each task is output based on the preset classification task.

[0029] However, in the process of pre-training the softmax classifier, since the number of samples of invasive organisms is much smaller than the number of samples of native organisms, such imbalance easily leads to poor generalization ability of the trained classifier, thus affecting the recognition result of the classifier; at the same time, the softmax classifier only classifies based on the training data, further resulting in its recognition result being easily affected by the local native data distribution deviation.

[0030] To this end, a pre-trained generative adversarial network is introduced in this embodiment. During the training process of the generative adversarial network, its generator generates images simulating local organisms to simulate local organism interference, enhancing the model's ability to distinguish real intrusion signals. The discriminator discriminates the images generated by the generator to adapt to different environments, improving the discriminator's ability to distinguish real intrusion signals.

[0031] After training is completed, the discriminator in the generative adversarial network is fine-tuned and used to classify the multimodal features of each sampling grid, identifying whether the multimodal features of each sampling grid are the features of invasive organisms or non-invasive organisms, that is, whether there are invasive organisms in the sampling grid.

[0032] In a specific embodiment, the process of fine-tuning the discriminator on local training data is as follows: ; In the formula, represents the fused multimodal features, represents using the discriminator of the trained generative adversarial network for identification, represents the final output category, that is, the presence / absence of invasive organisms.

[0033] Its loss function is: ; In the formula, represents the true label, represents the predicted class probability.

[0034] And the following formula is used for iterative optimization to minimize the loss function: ; In the formula, are the model parameters, and are the first and second moments of the gradient respectively, is the learning rate, is a constant to prevent division by zero, such as 10 -8 . The parameter update amplitude is dynamically adjusted through an adaptive optimization strategy to balance the convergence speed and stability to adapt to the complex gradient distribution of multimodal data and improve the training efficiency.

[0035] For each input object, the discriminator outputs a decision result of the presence / absence of invasive organisms and outputs the probability P1 of this category.

[0036] It can be understood that the process of using the fine-tuned discriminator is similar to the above process. By inputting the multimodal features into the fine-tuned discriminator, the classification result output by the discriminator can be obtained.

[0037] Optionally, when the classification result output by the discriminator indicates the presence of invasive organisms, primary warning information is simultaneously output, indicating that there is a certain possibility of invasive organisms in the sampling grid, and the information of the original sampling is determined according to the source of the input multimodal data, and the species, location depth, optical image, etc. are output.

[0038] For example, when the output result of the current sampling grid indicates the presence of invasive organisms, the location depth is determined according to the acoustic data, and the image of the current sampling grid is output as the optical image.

[0039] Step 103, when the discrimination result of the discriminator is that the sampling grid has invasive organisms, sample and detect the eDNA environmental sample of the sampling grid, and determine whether there are invasive organisms in the sampling grid based on the detection result.

[0040] It can be understood that if eDNA sampling and detection are performed on the entire target water area, the time required for sampling and detection is relatively long, the overall workload is large, and after all the sampling data is processed, the deviation of the invasive organism location information obtained is also relatively large.

[0041] Therefore, in this embodiment, by fusing acoustic and optical data to obtain multimodal features for recognition, the preliminary warning result of each sampling grid can be obtained to preliminarily identify whether there are invasive organisms in each sampling grid.

[0042] It can be understood that since the preliminary identification is implemented using a pre-trained model, the identification speed is relatively fast, and the preliminary identification result of whether there are invasive organisms in the sampling grid can be obtained in a short time. On this basis, for the sampling grid with the identification result of the presence of invasive organisms, eDNA environmental sample sampling and detection are performed, and the types of organisms contained in the sampling grid are directly identified through the detection result of eDNA sampling, and it is determined whether there are invasive organisms in the sampling grid according to the types of organisms, so as to further obtain a more accurate identification result.

[0043] Optionally, it is also possible to expand a certain range based on the sampling grid of invasive organisms, and sample and detect the eDNA environmental samples of the sampling grids within the range to simulate the movement of invasive organisms and obtain a more accurate identification result.

[0044] In a specific embodiment, a suspicious area is determined based on the sampling grid with invasive organisms. After using GPS or other positioning devices to record the specific location and depth of the sampling point, eDNA sampling is performed.

[0045] After sampling, through a special filtering device, such as 0.45 μmThe water body obtained by filtering and sampling with a filter membrane captures tiny DNA particles in the water; the filtered DNA particles are stored using liquid nitrogen or refrigeration to prevent DNA degradation.

[0046] The eDNA particles filtered from the water sample are extracted by methods such as an eDNA extraction kit, and the concentration and purity of the extracted DNA are confirmed by a spectrophotometer (such as NanoDrop) or fluorescence quantification (such as Qubit).

[0047] Furthermore, specific gene regions (such as the COI gene, 16S rRNA, etc.) are amplified by polymerase chain reaction (PCR) for gene barcode analysis to identify the species that may exist in the water.

[0048] Specifically, the following gene barcode segments are selected: COI gene: Widely used for the identification of animal species.

[0049] 16S rRNA gene: Suitable for the identification of bacteria and some invertebrates.

[0050] Specifically, specific primers are used to amplify the target gene region. The primers are selected from gene sequences related to the target species or species group.

[0051] On this basis, the amplified gene sequence needs to be compared with the detected suspected organisms to confirm whether the water sample contains the DNA of invasive species. Commonly used databases are: GenBank: A global gene sequence database containing gene sequences of a large number of known species.

[0052] BOLD: A library dedicated to barcode data, containing a large number of COI gene sequences.

[0053] FishBase: A database of aquatic species, containing various data on fish.

[0054] The comparison process is as follows: Clean the low-quality sequences and adapter sequences in the PCR products; upload the cleaned sequences to one of the above databases for sequence comparison; obtain the comparison results and view the similarity scores and matching species. According to the comparison results, determine the most likely species. Judge whether the matching species is an invasive biological species. If so, output high-level warning information, including the species type of the invasive organism, the percentage similarity, the description information of the matching species, etc.; if not, retain the primary warning information.

[0055] In the present invention, acoustic and optical data are obtained by sampling each sampling grid in the target water area. Based on multimodal features characterizing acoustic and optical properties and a pre-trained generative adversarial network, a preliminary determination is made on whether the target water area contains invasive organisms. For the sampling grids with a determination result of the existence of invasive organisms, eDNA sampling is carried out, so as to make a joint decision in combination with eDNA to obtain a more accurate underwater invasive organism recognition result. By utilizing multimodal information, rapid recognition is carried out using acoustic and optical features, and the recognition accuracy is improved using eDNA, making up for the deficiencies of a single method and improving the efficiency of invasive species monitoring.

[0056] In the multimodal underwater biological invasion detection method of the present invention, the step of respectively sampling each sampling grid in the target water area through an acoustic device and an image device to obtain the acoustic data and image data of each sampling grid specifically includes: Determine the size of the sampling grid according to the coverage range of the acoustic device, and divide the target water area into a plurality of the sampling grids as acoustic sampling grids; Compared with an optical image acquisition device, the coverage range of an acoustic device is usually larger, usually reaching dozens of meters to hundreds of meters. Therefore, in this embodiment, first, based on the coverage range of the acoustic device, the size of the sampling grid is determined, and the sampling grids of the target water area are divided.

[0057] Optionally, in this embodiment, sampling grid units with side lengths of 50 to 200 meters are divided according to the coverage range of the acoustic device.

[0058] For example, if a sidescan sonar covers 150 meters, square sampling grids with a side length of 50 meters are correspondingly divided.

[0059] For example, the coverage range of a multibeam sonar can reach 200 to 300 meters, and square grids with a side length of 200 meters can be correspondingly divided.

[0060] Optionally, in a feasible embodiment, a hydrological model (such as a propagation path with a flow velocity > 0.3 m / s) and human activity data (such as ports and waterways) can also be combined to generate a heat map of invasion risk. In high-risk areas, the size of the sampling grid unit is reduced to 30 - 50 meters to improve the monitoring accuracy. This partition is used for acoustic data acquisition.

[0061] According to the coverage range of the optical device, a plurality of optical sampling grids are divided in each sampling grid; Since the coverage range of the optical device is small and usually requires a close and clear field of view, in this embodiment, based on the sampling grid unit determined by the acoustic device, optical sampling grids are further divided for each sampling grid unit according to the coverage range of the optical device.

[0062] For example, in the case where the sampling grid cell is a square grid with a side length of 150 meters, it is divided into 16 equal-sized small cells.

[0063] Optionally, in deep waters and turbid waters, the cell size should be appropriately reduced.

[0064] In each sampling grid, an acoustic device is used for sampling, and an optical device is used for sampling in its corresponding optical sampling grid. The sampled data is used as the acoustic data and image data of each sampling grid.

[0065] Acoustic and optical sampling are sequentially performed in each sampling grid. Corresponding to one sampling grid, 16 optical sampling grids are divided. The sampling data of each sampling grid includes one piece of acoustic data and 16 pieces of image data.

[0066] During data acquisition, a high-precision GPS clock is used to synchronize the acquisition times of the acoustic and optical devices to ensure that the data is recorded at the same time point; and reference points with known coordinates are set in the target water area, and these reference points are used to perform spatial calibration on the acoustic and optical devices.

[0067] In the multi-modal underwater biological invasion detection method of the present invention, the step of respectively extracting and fusing the multi-modal features of each sampling grid based on the acoustic data and image data of each sampling grid specifically includes: After preprocessing the acoustic data and image data of each sampling grid, a pre-constructed local biological feature database is used for filtering to filter the acoustic data and image data with a similarity of no less than 90% to the biological features stored in the local biological database; Through preprocessing, data cleaning, data augmentation, etc. are performed on the acoustic data and image data of each sampling grid, and then a pre-constructed local biological feature database is used to filter the acoustic data and image data of each sampling grid after preprocessing to reduce the interference of local biological features during recognition.

[0068] Optionally, in this embodiment, for the acoustic data and image data of each sampling grid, a corresponding feature extraction network is first constructed, acoustic features and image features are extracted therefrom, and then based on the feature comparison method, they are compared with the acoustic features and image features of the local organisms stored in the local biological database to achieve filtering.

[0069] In a specific embodiment, first, acoustic information such as the species swimming frequency and body length reflection intensity of each type of local organism is obtained to construct a voiceprint library of each type of local organism; then, visual features such as the morphology, texture, and color of each type of local organism are obtained to construct an optical feature library of each type of local organism. The voiceprint library and the optical feature library are integrated to obtain a local biological feature database.

[0070] Optionally, by combining historical monitoring and expert annotation, 10%-20% of the newly discovered species data is updated annually through incremental learning to filter the detected native species in real time.

[0071] Furthermore, an acoustic data feature extraction network is constructed.

[0072] Since acoustic data usually appears as one-dimensional echo signals, a convolutional neural network can be used as an effective tool for processing such signals to capture the local features and temporal dependencies of sonar echo signals.

[0073] Specifically, the convolutional neural network constructed in this embodiment includes: An input layer for receiving the preprocessed acoustic data; A convolutional layer for extracting local features of the acoustic data through one-dimensional convolution: ; In the formula, X is the input signal, W is the convolutional kernel, with a shape of where k is the convolutional kernel size, is the number of output channels. In this embodiment, a smaller convolutional kernel (such as k =3 or k =5) is selected to capture local features. represents the one-dimensional convolution operation, b i is the bias term, which can be initialized to 0.01 and updated by the Adam optimization algorithm. RELU is the activation function, defined as: ; A pooling layer for gradually reducing the spatial dimension of the feature map through max pooling operation: ; In the formula, Xi is the element in the pooling window, and a pooling window of 2×1 or 3×1 is usually selected; A fully connected layer for flattening the output after convolutional kernel pooling and obtaining the final feature representation through the fully connected layer: ; In the formula, is the flattened signal, Q is the weight matrix of the fully connected layer, with a shape of where is the number of input neurons, is the number of output neurons.

[0074] He initialization can be used: ; represents a normal distribution.

[0075] Based on the above - constructed convolutional neural network, feature extraction is performed on the acoustic data of each sampling grid to obtain the acoustic features of each sampling grid.

[0076] The pre - constructed local acoustic fingerprint library is used to perform feature matching on the acoustic features of each sampling network. The acoustic features with a matching degree not less than the preset threshold, which is 90% in this embodiment, are considered as the acoustic features of local organisms and are directly removed to filter local species information.

[0077] Then, the filtered acoustic features are output: , with a shape of .

[0078] On the other hand, for image data, a pre - trained convolutional neural network (such as ResNet, VGG, etc.) is used as a feature extractor to extract multi - level features from the image.

[0079] At the end of the feature extractor, a global average pooling layer is used to reduce the output of the feature map to a one - dimensional vector: ; In the formula, is the eigenvalue of the image at position ( i , j ), H and W are the height and width of the image respectively, is the output one - dimensional vector.

[0080] Through the above - mentioned feature extraction network, feature extraction can be performed on the optical data of each sampling grid to obtain the image features of each sampling network.

[0081] The pre - constructed local optical feature library is used to perform feature matching on the image features of each sampling grid. The optical features with a matching degree greater than the preset threshold, which is 90% in this embodiment, are considered as the optical features of local organisms and are directly removed to filter local species information.

[0082] According to the filtering result, is updated, and finally the updated is output as the filtered optical features in each sampling grid.

[0083] A cross - attention mechanism is used to perform feature fusion on the filtered acoustic data and optical data in each sampling grid to obtain the multi - modal features of each sampling grid.

[0084] In the above - mentioned manner, the filtered acoustic data in each sampling grid is represented as the filtered acoustic features , the filtered optical data in each sampling grid is represented as filtered optical features , and and are both one-dimensional vectors.

[0085] Furthermore, based on the cross-attention mechanism, and are subjected to feature fusion to obtain multi-modal features characterizing the acousto-optic features of each sampling grid.

[0086] Specifically, queries, keys, and values are first generated.

[0087] Taking as the query: ; ; ; Taking as the query: ; ; ; wherein, etc. are learnable parameter matrices.

[0088] Then, the attention scores with acoustic features as queries and optical features as keys are calculated S A1 and the attention scores with optical features as queries and acoustic features as keys S A2 : ; ; The softmax function is used to map the network output to a categorical probability distribution, ensuring that the sum of probabilities of all categories is 1.

[0089] Next, weighted fusion is performed to obtain the feature vector as the multi-modal feature: ; ; ; wherein, ⊕ represents the concatenation operation, and FC is a fully connected layer used to map the concatenated features to an appropriate dimension.

[0090] In the multi-modal underwater biological invasion detection method of the present invention, the step of preprocessing the acoustic data and image data of each sampling grid specifically includes: Construct a measurement matrix to extract the observed data reflecting the physical characteristics of the target from the acoustic data of each sampling grid, and process it into the form of a standard normal distribution to obtain the preprocessed acoustic data of each sampling grid; In the divided sampling grids, the sonar emits sound waves and receives their echo signals as the original acoustic data. The sonar echo signal is reflected by underwater objects, and the receiving device records information such as the intensity, frequency, and phase of the echo. After a series of processes, the physical characteristics of the object are obtained.

[0091] Among them, the formula is used to calculate the depth d of the underwater target generating the sonar echo. t is the sound wave propagation time detected by the sonar, and v is the sound speed. The sound speed in water is usually taken as 1500 m / s.

[0092] On this basis, the intensity of the sonar echo signal is represented by the attenuation model : ; In the formula, is the sonar echo intensity, A is the initial intensity, is the attenuation coefficient, and t is the time.

[0093] Then, denoise the signal based on the Kalman filter. The Kalman filter is a recursive filter that reduces the influence of noise by estimating the target state and updating the optimal estimate: ; In the formula, is the state estimate at time k, representing the estimated object state or signal. Here, it should include the physical characteristics of the underwater target, and may be expressed as: ; From top to bottom, they correspond to the depth position, movement speed, size, shape, and material (reflectivity) of the underwater target respectively.

[0094] is the observed value, which is the actual measurement value obtained from the system, including the echo intensity , the echo time delay and the echo frequency f : .

[0095] Since the echo intensity is the attenuation amount of the sonar echo signal and is usually related to factors such as the depth, size, shape, and material of the underwater target, therefore, according to the relationship between the echo intensity and these variables, design the measurement matrix , which is used to transform the system state vector into the observation vector .

[0096] H K The first row of ; In the formula, 1 represents the influence of depth, size, shape, and material on the echo intensity, indicating its correlation.

[0097] Since the echo time delay is directly related to the depth of the underwater target, it is affected by the target depth and movement speed. Therefore, the second row of the observation matrix is constructed to represent the relationship between the echo time delay and depth: ; Since the echo frequency is mainly affected by the shape, material of the underwater target, and the complexity of acoustic wave reflection, therefore, the third row of the measurement matrix represents the relationship between the echo frequency and shape material: ; Finally, the constructed measurement matrix is as follows: ; To represent the relationship between the various observed quantities (intensity, time delay, frequency) of the wave signal and the physical characteristics (depth, speed, size, shape, material) of the target.

[0098] K k is the Kalman gain, and the calculation method is: ; In the formula, is the transpose of the measurement matrix; R K is the observation noise covariance matrix, P k-1 represents the noise level of the observed value, which can be obtained from the sensor, represents the estimated error covariance matrix of the previous moment, and can be initially set to the identity matrix or estimated from the data. At each update, the Kalman filter updates the error covariance of the system state: ; In the formula, E is the identity matrix.

[0099] Finally, the sonar echo signal values are converted into a standard normal distribution form with a mean of 0 and a standard deviation of 1 as the preprocessed acoustic data: ; In the formula, is the smoothed sonar echo signal, is the mean of the signal, is the standard deviation of the signal, is the standardized signal.

[0100] Filter and enhance the image data of each sampling grid to obtain the preprocessed image data of each sampling grid.

[0101] Within the optical sampling grid of each sampling grid, use devices such as underwater cameras and remotely operated vehicles (ROVs) to collect images to obtain the original image data.

[0102] Optionally, for turbid waters, multi-spectral or infrared cameras can be used to enhance the clarity of the collected images, and finally merged into an RGB three-channel image.

[0103] In a specific embodiment, each channel is normalized to 0 to 255 using the following formula: ; where I is the original pixel value, I min and I max are the minimum and maximum pixel values in the image respectively, I nml is the normalized pixel value, with a range of [0, 255].

[0104] After normalization, bilateral filtering denoising is performed according to the following steps: ; ; ; ; where is the denoised image, and are the standard deviations of space and intensity respectively. represents the current pixel and the neighboring pixel is the relative offset between them. is the spatial weight, representing the similarity between neighboring pixels and the current pixel in space; is the intensity weight, representing the influence of the intensity difference between pixel values. is called the normalization factor, which is used to normalize the sum of weighted values to ensure that the filtered pixel values do not deviate from the range of the original values.

[0105] Furthermore, underwater images usually have poor image contrast and blurred details due to factors such as water turbidity, light attenuation, and halo effects. The MSRCP (Multi-Scale Retinex Color Balance) algorithm is used to dehaze and enhance the contrast to improve the image: ; ; ; In the formula, , and are respectively the pixel values of the red, green, and blue channels of the denoised image, is the intensity image of the image. Y is a mapping factor to ensure that the color distribution of the enhanced image is consistent with that of the original image, represents the chromaticity image, represents the single-channel intensity image for normalization.

[0106] To avoid color distortion caused by over-enhancement, the finally enhanced image can be fused with the original image through weighted averaging: ; ; In the formula, is the enhanced image, is the original image without enhancement after denoising; is a coefficient to control the enhancement intensity, usually taking values between [0, 1].

[0107] Through the above method, the preprocessing of the collected image data is realized to obtain clearer underwater image data by denoising and enhancing the image.

[0108] In the multi-modal underwater biological invasion detection method of the present invention, before the step of using the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network and obtaining the discrimination result output by the discriminator that there is an invasive organism or a non-invasive organism in the sampling grid, it further includes: Constructing a local biological data set based on the acoustic features and image features of the local organisms in the target water area, and training a generative adversarial network on the local biological data set to enable the generator of the generative adversarial network to generate adversarial samples similar to the local biological features that can deceive the discriminator.

[0109] In this embodiment, a local biological data set is constructed based on the acoustic features and image features of the local organisms in the target water area for training a generative adversarial network, so as to complete the invasive organism recognition task based on the discriminator of the trained generative adversarial network.

[0110] Specifically, (G) in the generative adversarial network attempts to generate adversarial samples similar to the local biological features that can deceive the discriminator, and the discriminator (D) compares the real invasion signal and the generated samples to improve the discrimination ability. The loss function is as follows: Generator loss: ; In the formula, $\mathbb{E}$ represents the expected value, $z$ is the random noise sampled from the latent space (usually a standard normal distribution), $G(z)$ is the sample that the generator attempts to forge, and $D(G(z))$ represents the predicted probability of the discriminator for the generated sample, ranging from $[0, 1]$. The closer the value is to 1, the more real the discriminator considers the generated sample to be.

[0111] Discriminator loss: ; wherein, $\mathbb{E}$ represents the expected value, x $x$ is the real sample from the training data set, that is, the real optical image or acoustic signal; D $(D(x))$ x represents the predicted probability of the discriminator for the real sample, ranging from $[0, 1]$. The closer the value is to 1, the more real the discriminator considers the real sample to be.

[0112] By pre-training the generative adversarial network on the local biological data set, the discriminator can be used to identify local organisms and invasive organisms. Specifically, the adversarial samples generated by the GAN can approximate the local environmental characteristics, making it more difficult for the model to be misled by local similar species and improving the discrimination ability of invasive species.

[0113] In the multi-modal underwater biological invasion detection method of the present invention, before the step of training the generative adversarial network on the local biological data set, it further includes: Training a basic model based on an invasive species database, wherein the basic model takes the multi-modal features of invasive species as input and the class labels of invasive species as output; Transferring the feature extraction network of the basic model to the discriminator as the feature extraction network of the discriminator.

[0114] In order to further improve the recognition ability of the discriminator of the GAN, in this embodiment, a transfer adversarial network is constructed based on transfer learning.

[0115] Specifically, based on the invasive species database, in this embodiment, a global invasive species database (such as GISD) is selected to train the basic model to learn the common features of invasive organisms across regions.

[0116] That is, acoustic and optical data of invasive species are obtained from public databases such as GISD, and a deep convolutional neural network CNN / Transformer is used as the basic model. The training process takes the multi-modal data (acoustic signals, optical images) of global invasive species as input and the class labels of invasive species as output; the cross-entropy loss is used as the loss function, and the Adam optimizer is used. The target pre-trained model can extract the common features of invasive organisms (such as specific acoustic patterns, optical morphologies).

[0117] On this basis, the feature extraction network of the base model is used as the feature extraction network of the discriminator to construct the discriminator. Then, through adversarial training, the discriminator is made to learn to identify native organisms and invasive organisms, taking into account the possible data distribution bias between the global database and the local environment in the adversarial training.

[0118] On this basis, a complete underwater invasive organism recognition process is as Figure 2 shown.

[0119] Since transfer learning can extract the common features of invasive species and improve the generalization ability of the model. Through the way of transfer adversarial learning, it can reduce domain bias, enhance robustness, and improve the stability of detection results. In the case of small samples or complex environments, transfer adversarial classification is more robust than traditional methods, which can effectively reduce false detections and improve the reliability and stability of detection.

[0120] Next, the multi-modal underwater biological invasion detection device provided by the present invention will be described. The multi-modal underwater biological invasion detection device described below can be mutually referred to the multi-modal underwater biological invasion detection method described above.

[0121] As Figure 3 shown, the multi-modal underwater biological invasion detection device includes a collection module 301, a screening module 302, and a determination module 303; The collection module 301 is used to sample respectively through an acoustic device and an image device in each sampling grid of the target water area to obtain the acoustic data and image data of each sampling grid; Sampling grids are pre-divided in the target water area.

[0122] Optionally, the sampling grids can be divided according to factors such as research requirements and device detection ranges, as long as the divided sampling grids are convenient for collecting acoustic data and image data.

[0123] For each sampling grid, an acoustic device such as a sonar is used to obtain the sonar echo of the sampling grid as the acoustic data. An image device such as a camera is used to collect several underwater images of the area as the image data of the sampling grid.

[0124] The acoustic data and several pieces of image data of each sampling grid are sorted into a group of data according to a preset rule as the acoustic data and image data of each sampling grid.

[0125] It can be understood that the preset rule is only to sort and organize the data obtained in different modalities and at different times according to a certain rule, and it can be preset based on the type and quantity of the collected data.

[0126] A screening module 302, configured to respectively extract and fuse acoustic data and image data of each sampling grid to obtain multimodal features of each sampling grid, and use the multimodal features of each sampling grid as the input of a discriminator in a generative adversarial network, so as to obtain a discrimination result output by the discriminator that there is an invasive organism or no invasive organism in the sampling grid. During the training process of the generative adversarial network, its generator generates images for simulating native organisms, and its discriminator is used to discriminate whether the generated images are native organism images or invasive organism images; Feature extraction is performed based on the acoustic data of each sampling grid to obtain acoustic features characterizing the underwater biological acoustic characteristics in the sampling grid.

[0127] Feature extraction is performed based on the image data of each sampling grid to obtain image features characterizing the underwater biological optical characteristics in the sampling grid.

[0128] After fusing the acoustic features and the image features, multimodal features characterizing the underwater biological acoustic-optical characteristics in the sampling grid are obtained.

[0129] Usually, in order to implement an identification task based on multimodal features, the extracted multimodal features are usually selected to be input into a pre-trained softmax classifier, and the probability of each task is output based on a preset classification task.

[0130] However, during the process of pre-training the softmax classifier, since the number of samples of invasive organisms is much smaller than the number of samples of native organisms, such imbalance easily leads to poor generalization ability of the trained classifier, thus affecting the identification result of the classifier; at the same time, the softmax classifier only classifies based on training data, which further leads to its identification result being easily affected by the local native data distribution bias.

[0131] Therefore, in this embodiment, a pre-trained generative adversarial network is introduced. During the training process of the generative adversarial network, its generator generates images simulating native organisms to simulate native organism interference and enhance the model's ability to distinguish real invasive signals, and the discriminator discriminates the images generated by the generator to adapt to different environments and improve the discriminator's ability to distinguish real invasive signals.

[0132] After training is completed, the discriminator in the generative adversarial network is fine-tuned and used to classify the multimodal features of each sampling grid, and identify whether the multimodal features of each sampling grid are features of invasive organisms or non-invasive organisms, that is, there is / there is no invasive organism in the sampling grid.

[0133] A determination module 303 is configured to, when the discrimination result of the discriminator indicates that there is an invasive organism in the sampling grid, sample and detect the eDNA environmental sample of the sampling grid, and determine whether there is an invasive organism in the sampling grid based on the detection result.

[0134] It can be understood that if the eDNA sampling and detection are performed on the entire target water area, the time required for sampling and detection is relatively long, the overall workload is large, and after all the sampling data is processed, the deviation of the invasive organism location information obtained is also relatively large.

[0135] Therefore, in this embodiment, by fusing the acoustic and optical data to obtain multi-modal features for recognition, the preliminary warning result of each sampling grid can be obtained to preliminarily identify whether there is an invasive organism in each sampling grid.

[0136] It can be understood that since the preliminary recognition is implemented by using a pre-trained model, the recognition speed is relatively fast, and the preliminary recognition result of whether there is an invasive organism in the sampling grid can be obtained in a short time. On this basis, for the sampling grid with the recognition result of having an invasive organism, the eDNA environmental sample is sampled and detected. The biological species contained in the sampling grid are directly identified through the detection result of the eDNA sampling, and it is determined whether there is an invasive organism in the sampling grid according to the biological species, so as to further obtain a more accurate recognition result.

[0137] Optionally, it is also possible to expand a certain range based on the sampling grid of the invasive organism, and sample and detect the eDNA environmental samples of the sampling grids within the range to simulate the movement of the invasive organism and obtain a more accurate recognition result.

[0138] The present invention samples the acoustic and optical data for each sampling grid of the target water area, preliminarily discriminates whether the target water area contains invasive organisms based on the multi-modal features characterizing the acoustic and optical characteristics and the pre-trained generative adversarial network, and performs eDNA sampling on the sampling grid with the discrimination result of having an invasive organism, so as to make a joint decision by combining eDNA to obtain a more accurate recognition result of underwater invasive organisms. By using multi-modal information, quickly recognizing by using acoustic and optical features, and improving the recognition accuracy by using eDNA, the deficiencies of a single method are made up, and the efficiency of invasive species monitoring is improved.

[0139] Figure 4 Illustrates a schematic physical structure diagram of an electronic device, such as Figure 4As shown in the figure, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communications interface 420, and the memory 430 complete communication with each other through the communication bus 440. The processor 410 may call the logical instructions in the memory 430 to execute the multi-modal underwater biological intrusion detection method, which includes: sampling in each sampling grid of the target water area through an acoustic device and an image device respectively to obtain the acoustic data and image data of each sampling grid; extracting and fusing the multi-modal features of each sampling grid based on the acoustic data and image data of each sampling grid respectively, taking the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network, and obtaining the discrimination result output by the discriminator that there is an invasive organism or no invasive organism in the sampling grid. Among them, during the training process of the generative adversarial network, its generator generates images for simulating local organisms, and its discriminator is used to discriminate whether the generated images are local organism images or invasive organism images; in the case where the discrimination result of the discriminator is that there is an invasive organism in the sampling grid, sampling and detecting the eDNA environmental sample of the sampling grid, and determining whether there is an invasive organism in the sampling grid based on the detection result.

[0140] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software function units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0141] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-modal underwater biological invasion detection method provided by each of the above methods. The method includes: sampling respectively through an acoustic device and an image device in each sampling grid of the target water area to obtain the acoustic data and image data of each sampling grid; extracting and fusing respectively based on the acoustic data and image data of each sampling grid to obtain the multi-modal features of each sampling grid, taking the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network, and obtaining the discriminant result output by the discriminator that there is an invasive organism or no invasive organism in the sampling grid. Among them, during the training process of the generative adversarial network, its generator generates images for simulating native organisms, and its discriminator is used to discriminate whether the generated images are native organism images or invasive organism images; when the discriminant result of the discriminator is that there is an invasive organism in the sampling grid, sampling and detecting the eDNA environmental sample of the sampling grid, and determining whether there is an invasive organism in the sampling grid based on the detection result.

[0142] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the multi-modal underwater biological invasion detection method provided by each of the above methods. The method includes: sampling respectively through an acoustic device and an image device in each sampling grid of the target water area to obtain the acoustic data and image data of each sampling grid; extracting and fusing respectively based on the acoustic data and image data of each sampling grid to obtain the multi-modal features of each sampling grid, taking the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network, and obtaining the discriminant result output by the discriminator that there is an invasive organism or no invasive organism in the sampling grid. Among them, during the training process of the generative adversarial network, its generator generates images for simulating native organisms, and its discriminator is used to discriminate whether the generated images are native organism images or invasive organism images; when the discriminant result of the discriminator is that there is an invasive organism in the sampling grid, sampling and detecting the eDNA environmental sample of the sampling grid, and determining whether there is an invasive organism in the sampling grid based on the detection result.

[0143] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0144] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal underwater biological invasion detection method, characterized in that, Including: Sampling is respectively carried out on each sampling grid in the target water area through an acoustic device and an image device to obtain the acoustic data and image data of each sampling grid; Based on the acoustic data and image data of each sampling grid, multi-modal features of each sampling grid are respectively extracted and fused, and the multi-modal features of each sampling grid are used as the input of the discriminator in the generative adversarial network, and the discriminant result output by the discriminator is that there is an invasive organism or no invasive organism in the sampling grid. Among them, during the training process of the generative adversarial network, its generator generates images used to simulate native organisms, and its discriminator is used to discriminate whether the generated images are native organism images or invasive organism images; When the discriminant result of the discriminator is that there is an invasive organism in the sampling grid, eDNA environmental samples of the sampling grid are sampled and detected, and based on the detection result, it is determined whether there is an invasive organism in the sampling grid.

2. The multimodal underwater biological invasion detection method according to claim 1, wherein, The step of respectively sampling through an acoustic device and an image device on each sampling grid in the target water area to obtain the acoustic data and image data of each sampling grid specifically includes: Determine the size of the sampling grid according to the coverage range of the acoustic device, and divide the target water area into a plurality of the sampling grids as acoustic sampling grids; According to the coverage range of the optical device, a plurality of optical sampling grids are divided in each sampling grid; In each sampling grid, an acoustic device is used for sampling, and an optical device is used for sampling in its corresponding optical sampling grid, and the sampled data is used as the acoustic data and image data of each sampling grid.

3. The multimodal underwater biological invasion detection method according to claim 1, characterized in that The step of respectively extracting and fusing the multi-modal features of each sampling grid based on the acoustic data and image data of each sampling grid specifically includes: After preprocessing the acoustic data and image data of each sampling grid, a pre-constructed native organism feature database is used for filtering to filter the acoustic data and image data whose similarity with the organism features stored in the native organism database is not less than 90%; A cross-attention mechanism is adopted to perform feature fusion on the filtered acoustic data and optical data in each sampling grid to obtain the multi-modal features of each sampling grid.

4. The multimodal underwater biological invasion detection method according to claim 3, wherein, The step of preprocessing the acoustic data and image data of each sampling grid specifically includes: Construct a measurement matrix to extract the observation data reflecting the target physical features in the acoustic data of each sampling grid and process it into the form of a standard normal distribution to obtain the preprocessed acoustic data of each sampling grid; Filter and enhance the image data of each sampling grid to obtain the preprocessed image data of each sampling grid.

5. The multimodal underwater biological invasion detection method according to any one of claims 1-3, characterized in that, Before the step of using the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network to obtain the discriminant result output by the discriminator that there is an invasive organism or a non-invasive organism in the sampling grid, it further includes: Construct a native organism data set based on the acoustic features and image features of the native organisms in the target water area, and train the generative adversarial network on the native organism data set so that the generator of the generative adversarial network generates adversarial samples similar to the native organism features that can deceive the discriminator.

6. The multimodal underwater biological invasion detection method according to claim 5, wherein, Before the step of training the generative adversarial network on the local biological dataset, the following steps are further included: Training a basic model based on an invasive species database, where the basic model takes the multi-modal features of invasive species as input and the class labels of invasive species as output; Transferring the feature extraction network of the basic model to the discriminator as the feature extraction network of the discriminator.

7. A multi-modal underwater biological invasion detection device, characterized in that, It includes: A collection module, configured to sample through an acoustic device and an image device respectively at each sampling grid in the target water area to obtain the acoustic data and image data of each sampling grid; A screening module, configured to respectively extract and fuse the multi-modal features of each sampling grid based on the acoustic data and image data of each sampling grid, use the multi-modal features of each sampling grid as the input of the discriminator in the generative adversarial network, and obtain the discriminant result output by the discriminator that there are invasive organisms or non-invasive organisms in the sampling grid. During the training process of the generative adversarial network, its generator generates images for simulating local organisms, and its discriminator is used to discriminate whether the generated images are local organism images or invasive organism images; A determination module, configured to, when the discriminant result of the discriminator is that there are invasive organisms in the sampling grid, sample and detect the eDNA environmental samples of the sampling grid, and determine whether there are invasive organisms in the sampling grid based on the detection results.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-modal underwater biological invasion detection method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-modal underwater biological invasion detection method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-modal underwater biological invasion detection method according to any one of claims 1 to 6.

Images