Deep-sea mining vehicle acoustic image target detection method and system
By optimizing the deep learning network structure and combining local-dynamic feature fusion and adaptive adjustment modules, the high complexity and real-time problems of deep-sea mining sonar image target detection were solved, and accurate target detection and safe operation support for deep-sea mining vehicles were achieved.
Patent Information
- Application Number
- CN202510875724.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-27
- Publication Date
- 2025-10-17
AI Technical Summary
Existing deep-sea mining sonar image target detection technology is difficult to adapt to complex marine environments. It has problems such as high computational complexity, poor target recognition effect, high risk of missed detection and false detection, and it is difficult to meet real-time processing requirements.
The local-dynamic feature fusion module, multi-level feature fusion module, adaptive adjustment module and dynamic adjustment module are adopted, combined with D-Conv and P-Conv branch structures, and the hybrid attention mechanism and adaptive difficulty loss function are used to optimize the deep learning network structure to achieve real-time target detection in the acoustic images of deep-sea mining vehicles.
It improves the ability to accurately perceive complex targets at multiple scales, reduces missed detections and false detections, reduces computing power and power consumption, extends the operating life of equipment, enhances operational safety and efficiency, adapts to different deep-sea environments, and possesses stable real-time processing capabilities.
Smart Images
Figure CN120804573A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of visual detection, and in particular to a deep-sea mining vehicle acoustic image target detection method and system. BACKGROUND
[0002] The deep-sea mining area has a complex terrain with obstacles such as rocks, trenches and pits of various sizes, which pose a threat to the safe driving and operation of the mining vehicle. Large rocks or terrain protrusions can cause the mining vehicle to collide and be damaged, and small rocks can easily get stuck in the tracks, resulting in increased equipment wear and maintenance costs. Therefore, accurately identifying and detecting the seabed terrain obstacles are crucial for deep-sea mining.
[0003] Current methods for underwater target detection mainly include optical image detection and acoustic image detection. Optical detection has high resolution and rich details, but it faces serious problems such as light attenuation, scattering and sediment obstruction in deep-sea environments, which severely affect its reliability. Acoustic detection, on the other hand, has strong penetration and low environmental sensitivity, and has gradually become the mainstream of deep-sea exploration.
[0004] However, the existing deep-sea mining sonar image target detection technology still has obvious deficiencies, including:
[0005] Most algorithms rely on idealized environments and artificial data, making it difficult to adapt to real complex marine environments.
[0006] There is a lack of specialized design for the imaging characteristics of forward-looking sonar, especially for the identification of targets with large size differences (large rocks and small gravel).
[0007] The complex seabed terrain leads to multi-path propagation of sonar signals and increased scattering noise, making it difficult to distinguish between targets and backgrounds and increasing the risk of missed and false detections.
[0008] There is also a high computational complexity, making it difficult to meet the real-time processing requirements in the deep-sea mining scenario.
[0009] In view of this, the present application proposes a deep-sea mining vehicle acoustic image target detection method and system. SUMMARY
[0010] The purpose of the present application is to provide a deep-sea mining vehicle acoustic image target detection method and system to assist in the construction of the perception front end of the deep-sea mining vehicle and ensure the safety of the mining vehicle's travel and operation.
[0011] In a first aspect, the present application provides a deep-sea mining vehicle acoustic image target detection method, comprising a local-dynamic feature fusion module, a multi-level feature fusion module, an adaptive adjustment module and a dynamic adjustment module:
[0012] The local-dynamic feature fusion module adopts a double-branch structure, the double-branch structure includes a D-Conv branch structure and a P-Conv branch structure: a fused global dynamic feature map is obtained based on the D-Conv branch structure, and an integrated channel local channel fusion feature map is obtained based on the P-Conv branch structure; the local-dynamic feature fusion module is used to replace a C2f module in the YOLOv8 network;
[0013] The multi-level feature fusion module uses a hybrid attention mechanism, processes the input image from three parallel branches of global, local and multi-scale respectively, and generates a unified spatial-channel attention feature map by weighting fusion of the features extracted from the three branches;
[0014] The adaptive adjustment module increases a high-resolution auxiliary detection layer, adopts an adaptive multi-scale fusion strategy, dynamically selects the feature layer that is most effective for final prediction according to spatial context information, performs nonlinear weight fusion, and forms a final target detection feature map;
[0015] The dynamic adjustment module adopts an adaptive difficulty loss function, based on the adaptive difficulty loss function, adaptively adjusts the loss weight of each target prediction box according to the intersection over union between the target prediction box and the real target box in the target detection feature map, performs model training, updates the model parameters, deploys the updated model to the host computer of the deep-sea mining vehicle, receives acoustic image data collected by the forward-looking sonar in real time, and outputs real-time target detection results.
[0016] As a preferred technical solution of the first aspect of the application, it further includes a data acquisition module, which acquires deep-sea environment acoustic image data through a forward-looking sonar carried by a deep-sea mining vehicle, and forms a deep-sea terrain data set through data cleaning and labeling of the image to be detected collected by the forward-looking sonar deployed at the front end of the deep-sea mining vehicle, the deep-sea terrain data set includes a training set, a validation set and a test set.
[0017] As a preferred technical solution of the first aspect of the application, the analysis logic of the D-Conv branch structure is:
[0018] The acoustic image data obtains global feature information through global average pooling;
[0019] A dynamic routing network is used to extract dynamic information from the global feature information, and a set of adaptive weights is generated based on the dynamic information;
[0020] According to the dynamic information, a convolution kernel of variable size is selected to perform convolution operation on the acoustic image data to obtain a fused global dynamic feature map.
[0021] As a preferred technical solution of the first aspect of the application, the analysis logic of the P-Conv branch structure is:
[0022] For the channel of acoustic image data, selective convolution calculation is performed to obtain local feature information,
[0023] The local features after selection processing and the information of the untreated channel are integrated by using 1*1 convolution to obtain a complete channel fusion feature map.
[0024] As a preferred technical solution of the first aspect of the application, the three parallel branches in the multi-level feature fusion module extract the following contents:
[0025] The global branch extracts the global feature information of the acoustic image data in the D-Conv branch structure; and the global weight vector is calculated by global average pooling;
[0026] The local branch extracts the local feature information of the local region of the acoustic image data in the P-Conv branch structure; and the local weight vector is calculated by local average pooling;
[0027] The multi-scale branch uses different scale pooling operations to extract multi-scale feature information of the acoustic image data; and the multi-scale weight vector capable of covering multi-scale target features is finally generated by up-sampling to align different scale features;
[0028] The above three weight vectors are nonlinearly weighted and fused to form a spatial-channel attention feature map.
[0029] As a preferred technical solution of the first aspect of the application, the nonlinear weighted fusion includes:
[0030] One-dimensional convolution operations are respectively performed on the global weight vector, the local weight vector and the multi-scale weight vector to generate respective attention weights;
[0031] The attention weights are weighted and calculated by a nonlinear function to adaptively integrate global, local and multi-scale information, and finally a spatial-channel attention feature map is obtained.
[0032] As a preferred technical solution of the first aspect of the application, the application logic of the adaptive spatial feature fusion detection head includes:
[0033] Global feature information of the spatial-channel attention feature map is obtained by global average pooling, and an adaptive convolution kernel is generated by using a dynamic routing network;
[0034] The generated adaptive convolution kernel is used to perform convolution operation on the feature map, and at the same time, selective convolution is performed on part of the channel features of the spatial-channel attention feature map;
[0035] The global feature and the feature after local selective convolution are nonlinearly fused to obtain a local-dynamic feature map.
[0036] As a preferred technical solution of the first aspect of the application, the nonlinear spatial fusion processing of the adaptive spatial feature fusion detection head comprises:
[0037] Spatial alignment is performed by upsampling and downsampling of feature maps of different scales;
[0038] According to the spatial context information, the feature layer most effective for final prediction is dynamically selected for nonlinear weight fusion to form the final target detection feature map.
[0039] As a preferred technical solution of the first aspect of the application, the adaptive difficulty loss function module extracts the following content:
[0040] The intersection over union value between the target prediction frame and the real target frame is calculated, and the target samples are divided into easy-to-detect samples, boundary samples and difficult-to-detect samples according to the size of the intersection over union;
[0041] The easy-to-detect sample is higher than the average intersection over union value;
[0042] The boundary sample is within the average intersection over union value range;
[0043] The difficult-to-detect sample is lower than the average intersection over union value;
[0044] Different weights are adaptively assigned according to the difficulty.
[0045] Secondly, the application provides a deep-sea mining vehicle acoustic image target detection method, based on the implementation of the first aspect, comprising the following steps:
[0046] A double-branch structure is adopted, which includes a D-Conv branch structure and a P-Conv branch structure: a fused global dynamic feature map is obtained based on the D-Conv branch structure, and a local channel fusion feature map is obtained after integrating the channels based on the P-Conv branch structure; a local-dynamic feature fusion module is used to replace the C2f module in the YOLOv8 network;
[0047] A hybrid attention mechanism is used to process the input image from three parallel branches of global, local and multi-scale, and the features extracted from the three branches are fused by weighting to generate a unified spatial-channel attention feature map;
[0048] A high-resolution auxiliary detection layer is added, an adaptive multi-scale fusion strategy is adopted, and according to the spatial context information, the feature layer most effective for final prediction is dynamically selected for nonlinear weight fusion to form the final target detection feature map;
[0049] Adopting an adaptive difficulty loss function, based on the adaptive difficulty loss function, according to the intersection over union between each target prediction box in the target detection feature map and the real target box, the loss weight of each target prediction box is adaptively adjusted, the model is trained, the model parameters are updated, the updated model is deployed to the host computer of the deep-sea mining vehicle, the acoustic image data collected by the forward-looking sonar is received in real time, and the real-time target detection result is output.
[0050] In the above technical solution, the technical effects and advantages provided by the present application are as follows:
[0051] The present application can provide more detailed operation environment information through accurate perception of the seabed environment and obstacles, which helps to realize intelligent monitoring, decision support and risk warning of the deep-sea mining operation process, and promotes the development of deep-sea mining towards intelligence and high safety; enhances the accurate perception ability of the deep-sea mining vehicle to multi-scale complex targets, especially improves the detection accuracy of subtle and easily ignored targets, reduces the missed detection and false detection phenomenon; optimizes the deep learning network structure, significantly reduces the calculation amount and power consumption while ensuring the accuracy, which is beneficial to prolong the service life of the host computer equipment of the deep-sea mining vehicle and reduce the maintenance cost; the modular network structure is convenient for quick adaptation and flexible adjustment according to different deep-sea environment and task requirements, effectively improves the universality and adaptability of the technical scheme, and expands the application range of the system. It still has stable real-time processing capability under complex marine topographic conditions, which can significantly reduce the operation delay caused by insufficient real-time performance, and improve the overall mining operation efficiency and economic benefit. BRIEF DESCRIPTION OF DRAWINGS
[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.
[0053] Figure 1 The flow chart of the acoustic image target detection system of the deep-sea mining vehicle of the present application;
[0054] Figure 2 The structural schematic diagram of the local-dynamic feature fusion module of the present application;
[0055] Figure 3 The structural schematic diagram of the multi-level feature extraction and noise suppression module of the present application;
[0056] Figure 4 The structural schematic diagram of the adaptive spatial feature fusion detection head of the present application;
[0057] Figure 5 The schematic diagram of the adaptive loss function of the present application;
[0058] Figure 6 Figure 1 is a detection result diagram of the present application. DETAILED DESCRIPTION
[0059] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The described embodiments are part of the embodiments of the present application, rather than all of the embodiments of the present application. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0060] Embodiment 1
[0061] Please refer to Figure 1 As shown in the figure, the present embodiment provides an acoustic image target detection system for a deep-sea mining vehicle, which comprises a data acquisition module, a local-dynamic feature fusion module, a multi-level feature fusion module, an adaptive adjustment module and a dynamic adjustment module; each module is connected through wired and / or wireless connection;
[0062] The data acquisition module is deployed in front of the front-looking sonar of the deep-sea mining vehicle to collect images to be detected after data cleaning and labeling, forming a deep-sea terrain data set;
[0063] Specifically, the deep-sea terrain data set comprises a training set, a verification set and a test set; the deep-sea terrain data set is trained according to the deep-sea terrain data set, the improved YOLOv8 deep neural network is trained to optimize the hyperparameter configuration and obtain the training weight, and the trained deep neural network model is deployed to the host computer of the deep-sea mining vehicle, the terrain pictures around the vehicle body that need to be detected are photographed in real time by the front-looking sonar, and the inference result is obtained by the model.
[0064] The local-dynamic feature fusion module adopts a double-branch structure, and the double-branch structure comprises a D-Conv branch structure and a P-Conv branch structure: the D-Conv branch structure is used to obtain a fused global dynamic feature map, and the P-Conv branch structure is used to obtain a local channel fusion feature map after integrating channels; the local-dynamic feature fusion module is used to replace the C2f module in the YOLOv8 network.
[0065] Specifically, as Figure 2As shown, the local-dynamic feature fusion module is used to replace the C2f module in the YOLOv8 network, integrating the feature extraction and representation capabilities of the target detection network, and solving the problem of rapid increase in network parameter quantity and redundant calculation when stacking deep neural networks, leading to waste of computing resources and risk of overfitting.
[0066] It should be noted that the two feature maps are connected to the feature fusion part together with the residual branch to enhance the overall feature expression capability of the network. The input feature map of the deep neural network is dynamically processed, and an adaptive convolution kernel is generated using a dynamic routing network to obtain global features. The efficiency is improved by selectively processing part of the channel features, and finally the efficient extraction and fusion of network features are realized.
[0067] The local-dynamic feature fusion module uses two parallel branches D-Conv and P-Conv to process the input feature map simultaneously, and the processing result of each branch is connected to the feature fusion part together with the residual branch at each layer of convolution. In the D-Conv branch, the acoustic image data X is first subjected to global average pooling, and the extracted feature information is subjected to a dynamic routing network to generate a set of adaptive weights. D-Conv is configured with 4 variable convolution kernels Wi, which can be dynamically adjusted to sizes of 1x1, 2x2 and 3x3 according to the input features. They are respectively convolved with the corresponding weights and weighted to obtain the output feature map Y. In the P-Conv branch, only part of the channels of the acoustic image data is processed, and the other channels are ignored to achieve efficiency improvement. A convolution kernel of size kxk is applied to a feature map of size hxwxc. In order to ensure the integrity of the feature information, after convolution of part of the channels, a 1x1 size convolution is used to integrate the information of all channels, enhancing the flow and exchange of information between channels.
[0068] The multi-level feature fusion module uses a hybrid attention mechanism to extract features from global, local, and multi-scale dimensions, respectively. The features extracted from the three branches are fused through weighted fusion to generate a unified spatial-channel attention feature map, effectively suppressing noise in deep-sea sonar images and enhancing the ability to accurately capture complex size targets.
[0069] Specifically, as shown in Figure 3 The multi-level feature fusion module uses a hybrid attention mechanism to integrate spatial and channel information, and designs and adds a multi-scale target branch to enhance the ability to capture complex size terrain or rock in deep sea, solving the problem of a large number of invalid information regions in forward-looking sonar images, and the existing spatial-channel attention cannot balance accuracy and computational cost, etc. Specifically, the following steps are included:
[0070] Firstly, the input image is processed through three parallel branches of global, local and multi-scale. In the global branch, global average pooling is used to extract the global information of the image, and a one-dimensional convolution is used to generate a global weight vector representing the background features of the overall scene. The global average pooling captures the statistical information of the entire image by averaging the feature maps of each channel and calculating the importance of each channel. This process not only smooths the scattering, absorption and background noise in the sonar image, but also enhances the expression of the overall image information. In the local branch, local average pooling is used to extract the features of high information density areas in the image, and a one-dimensional convolution is also used to generate a local weight vector to capture the fine features of the target area. The local average pooling preserves the details of the local area by averaging the local feature maps and calculating the importance of each spatial position, which is particularly helpful for distinguishing and detecting small rocks and other target features. In the multi-scale branch, different scale pooling operations (1x1, 2x2, 4x4 pooling) are used to extract multi-scale image features, and up-sampling operations are used to align the feature maps of different scales, and then generate a multi-scale weight vector. Subsequently, the features of the three branches are fused through weighted fusion to obtain a unified attention map, which is used to adaptively adjust the feature representation of each scale and each area. This module is inserted after the SPPF layer in the entire model, further enhancing the feature expression. In the calculation process, FTSA represents the output features of the TSA module, Fglobal, Flocal and Fmscale represent the features extracted by the GAP, LAP and Multi-Scale branches respectively, and Wglobal, Wlocal and Wmscale represent the attention weights of the corresponding branches, which are obtained through one-dimensional convolution and used to weight the features of each branch.
[0071] F TSA =W global ·F global +W local ·F local +W mscale ·F mscale ;
[0072] The adaptive adjustment module increases the high-resolution auxiliary detection layer and adopts an adaptive multi-scale fusion strategy to dynamically select the most effective feature layer for final prediction based on spatial context information, and performs nonlinear weight fusion to form the final target detection feature map.
[0073] Specifically, as Figure 4As shown, the adaptive adjustment module increases a detection layer P2 with a resolution of 160x160 for auxiliary feature fusion, and introduces an adaptive multi-level fusion idea to form an adaptive spatial feature fusion detection head, which solves the problem of false detection or missed detection of traditional FPN methods when processing different scale targets in the same image due to the significant difference in feature scales of various targets in deep-sea mining.
[0074] In the adaptive spatial feature fusion detection head, each level respectively processes four feature map information from the neck network, which have different spatial resolutions, and aligns the spatial dimensions through upsampling and downsampling. In the calculation, ASFF*-n represents the detection head corresponding to a certain layer feature map, n-th represents the feature map of different levels, and x1-n represents adjusting the feature map of level-1 to the size of level-n, and so on. α, β, γ, δ represent the weight parameters of each layer. This detection head can dynamically select the feature layer most valuable to the final prediction result according to the spatial context information. By flexibly assigning the weights of different scale feature maps, the most discriminative features are extracted, and these multi-scale information is effectively fused in the spatial dimension, thereby improving the detection accuracy. It is especially suitable for deep-sea mining which requires high target detection accuracy and limited computing resources.
[0075]
[0076] The dynamic adjustment module adopts an adaptive difficulty loss function, based on the adaptive difficulty loss function, adjusts the loss weight of each target prediction box in the target detection feature map according to the intersection over union between the target prediction box and the real target box, performs model training, updates the model parameters, deploys the updated model to the host computer of the deep-sea mining vehicle, receives the acoustic image data collected by the forward-looking sonar in real time, and outputs the real-time target detection result.
[0077] Specifically, as shown in FIG. 6, the dynamic adjustment module includes an adaptive difficulty loss function, a model training module, and a model updating module. Figure 5As shown, the difficulty of the detection sample is dynamically adjusted to assign the weight, thereby improving the detection accuracy and enhancing the stability of the model, solving the imbalance problem of seabed target categories in the sonar image, and improving the accuracy of detecting difficult targets with complex shapes. Specifically, an adaptive difficulty loss function is provided to improve the sample weight assignment mechanism, thereby improving the detection accuracy and enhancing the stability of the model. Higher weight is assigned to difficult-to-detect samples (such as large rocks, complex terrain, and difficult-to-identify boundary samples), while the loss contribution of easy-to-detect samples (such as numerous small-volume rocks) is significantly reduced to avoid model overfitting on easy-to-detect samples. The loss function evaluates the difficulty of the sample by the IoU value between the predicted bounding box and the true bounding box. In design, the average IoU of all samples is used as a threshold to divide the samples into easy-to-detect samples, boundary samples and difficult-to-detect samples. Since the misclassification risk of boundary samples is high, a piecewise function is used to weight them, smooth the error curve, and reduce the gradient discontinuity, thereby improving the stability and convergence speed of the model. Figure 5
[0078] As shown in Figure 6 , the embodiment uses a local-dynamic feature fusion module with a double-branch structure to optimize the calculation efficiency, a multi-level feature fusion module, an adaptive spatial feature fusion detection head to enhance the detection efficiency of small ore, an adaptive adjustment module, and a dynamic adjustment module corresponding to the adaptive loss function to solve the imbalance problem of difficult-to-detect target categories in real marine data. Through the above algorithm improvement, the invention realizes the rapid and accurate detection of seabed terrain targets in the forward-looking sonar image, effectively improves the operation efficiency and safety of the deep-sea mining vehicle, and promotes the development of deep-sea mining technology.
[0079] Embodiment 2
[0080] Based on embodiment 1, the embodiment provides a deep-sea mining vehicle acoustic image target detection method, including the following steps:
[0081] A double-branch structure is adopted, which includes a D-Conv branch structure and a P-Conv branch structure: based on the D-Conv branch structure, a fused global dynamic feature map is obtained, and based on the P-Conv branch structure, a local channel fusion feature map after integrating channels is obtained; a local-dynamic feature fusion module is used to replace the C2f module in the YOLOv8 network;
[0082] A hybrid attention mechanism is used to process the input image from three parallel branches of global, local and multi-scale, and the features extracted from the three branches are fused to generate a unified spatial-channel attention feature map;
[0083] An auxiliary detection layer with high resolution is added, an adaptive multi-scale fusion strategy is adopted, and the most effective feature layer for final prediction is dynamically selected based on spatial context information to perform nonlinear weight fusion and form a final target detection feature map;
[0084] An adaptive difficulty loss function is adopted, based on the adaptive difficulty loss function, the intersection over union between each target prediction box in the target detection feature map and the real target box is used to adaptively adjust the loss weight of each target prediction box, the model is trained, the model parameters are updated, the updated model is deployed to the host computer of the deep sea mining vehicle, real-time acoustic image data collected by the forward-looking sonar is received, and real-time target detection results are output.
[0085] It also includes a data acquisition module, which acquires deep sea environment acoustic image data through the forward-looking sonar carried by the deep sea mining vehicle, and forms a deep sea terrain data set by cleaning and labeling the image to be detected collected by the forward-looking sonar deployed at the front end of the deep sea mining vehicle. The deep sea terrain data set includes a training set, a validation set and a test set.
[0086] The analysis logic of the D-Conv branch structure is:
[0087] The acoustic image data is subjected to global average pooling to obtain global feature information;
[0088] A set of adaptive weights is generated based on the dynamic information extracted from the global feature information using a dynamic routing network;
[0089] According to the dynamic information, a convolution kernel with variable size is selected to perform convolution operation on the acoustic image data to obtain a fused global dynamic feature map.
[0090] The analysis logic of the P-Conv branch structure is:
[0091] The channels of the acoustic image data are selectively subjected to convolution calculation to obtain local feature information,
[0092] A 1x1 convolution is used to integrate the processed local feature and the information of the unprocessed channel to obtain a complete channel fusion feature map.
[0093] The three parallel branches in the multi-level feature fusion module extract the following contents:
[0094] The global branch extracts the global feature information of the acoustic image data in the D-Conv branch structure, and calculates a global weight vector through global average pooling;
[0095] The local branch extracts the local feature information of the local area of the acoustic image data in the P-Conv branch structure, and calculates a local weight vector through local average pooling;
[0096] Multi-scale branch: use different scale of pooling operation to extract multi-scale feature information of acoustic image data; align features of different scales through upsampling, and finally generate a multi-scale weight vector that can cover multi-scale target features;
[0097] The above three weight vectors are nonlinearly weighted and fused to form a spatial-channel attention feature map.
[0098] The nonlinear weighted fusion includes:
[0099] Respectively, one-dimensional convolution operation is performed on the global weight vector, the local weight vector and the multi-scale weight vector to generate respective attention weights;
[0100] The attention weights are calculated by a nonlinear function to adaptively integrate global, local and multi-scale information, and finally obtain a spatial-channel attention feature map.
[0101] The application logic of the adaptive spatial feature fusion detection head:
[0102] The global feature information of the spatial-channel attention feature map is obtained by global average pooling, and the adaptive convolution kernel is generated by using a dynamic routing network;
[0103] The generated adaptive convolution kernel is used for convolution operation on the feature map, and at the same time, selective convolution is performed on part of the channel features of the spatial-channel attention feature map;
[0104] The global feature and the feature after local selective convolution are nonlinearly fused to obtain a local-dynamic feature map.
[0105] The nonlinear spatial fusion processing of the adaptive spatial feature fusion detection head includes:
[0106] The features of different scales are spatially aligned by upsampling and downsampling;
[0107] According to the spatial context information, the feature layer most effective for final prediction is dynamically selected for nonlinear weight fusion to form the final target detection feature map.
[0108] The adaptive difficulty loss function module extracts the following content:
[0109] The intersection over union value between the target prediction box and the real target box is calculated, and the target samples are divided into easy detection samples, boundary samples and difficult detection samples according to the size of the intersection over union;
[0110] The easy detection sample is the intersection over union value higher than the average intersection over union value;
[0111] The boundary sample is the intersection over union value within the average intersection over union value range;
[0112] The difficult-to-detect sample is a sample with an average intersection-over-union ratio;
[0113] Different weights are adaptively assigned according to the difficulty.
[0114] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.
Claims
1. A deep-sea mining vehicle acoustic image target detection system, characterized in that: It includes local-dynamic feature fusion module, multi-level feature fusion module, adaptive adjustment module and dynamic adjustment module: The local-dynamic feature fusion module adopts a dual-branch structure, which includes a D-Conv branch structure and a P-Conv branch structure: the D-Conv branch structure obtains the fused global dynamic feature map, and the P-Conv branch structure obtains the local channel fusion feature map after channel integration; Use the local-dynamic feature fusion module to replace the C2f module in the YOLOv8 network; The multi-level feature fusion module uses a hybrid attention mechanism to process the input image from three parallel branches: global, local, and multi-scale. The features extracted by the three branches are weightedly fused to generate a unified spatial-channel attention feature map. Adaptive adjustment module, adds a high-resolution auxiliary detection layer, adopts an adaptive multi-scale fusion strategy, dynamically selects the feature layer that is most effective for the final prediction based on spatial context information, performs nonlinear weight fusion, and forms the final target detection feature map; The dynamic adjustment module adopts an adaptive difficulty loss function. Based on the adaptive difficulty loss function, it adaptively adjusts the loss weight of each target prediction box according to the intersection-over-union ratio between each target prediction box and the true target box in the target detection feature map, conducts model training, updates model parameters, and deploys the updated model to the host computer of the deep-sea mining vehicle. It receives acoustic image data collected by the forward-looking sonar in real time and outputs real-time target detection results.
2. The deep-sea mining vehicle acoustic image target detection system according to claim 1, characterized in that: It also includes a data acquisition module, which collects deep-sea environment acoustic image data through the forward-looking sonar installed on the deep-sea mining vehicle. The forward-looking sonar deployed at the front end of the deep-sea mining vehicle collects the images to be detected, and after data cleaning and annotation, forms a deep-sea terrain dataset. The deep-sea terrain dataset includes a training set, a validation set, and a test set.
3. The deep-sea mining vehicle acoustic image target detection system according to claim 1, characterized in that: The analysis logic of the D-Conv branch structure is: The acoustic image data is pooled globally to obtain global feature information; Using a dynamic routing network, dynamic information is extracted from global feature information, and a set of adaptive weights is generated based on the dynamic information; According to the dynamic information, a convolution kernel of variable size is selected to perform convolution operation on the acoustic image data to obtain the fused global dynamic feature map.
4. The deep-sea mining vehicle acoustic image target detection system according to claim 1, characterized in that: The analysis logic of the P-Conv branch structure is: Selectively perform convolution calculations on the channels of acoustic image data to obtain local feature information. A 1×1 convolution is used to integrate the selected local features and the information of the unprocessed channels to obtain a complete channel fusion feature map.
5. The deep-sea mining vehicle acoustic image target detection system according to claim 1, characterized in that: The three parallel branches in the multi-level feature fusion module extract the following contents respectively: The global branch extracts the global feature information of the acoustic image data in the D-Conv branch structure; the global weight vector is calculated through global average pooling; Local branch: extracts local feature information of the local area of the acoustic image data in the P-Conv branch structure; calculates the local weight vector through local average pooling; Multi-scale branch: Uses pooling operations at different scales to extract multi-scale feature information from acoustic image data. Upsampling aligns features at different scales, ultimately generating a multi-scale weight vector that covers multi-scale target features. The above three weight vectors are nonlinearly weighted fused to form a spatial-channel attention feature map.
6. The deep-sea mining vehicle acoustic image target detection system according to claim 5, characterized in that: The nonlinear weighted fusion includes: Perform one-dimensional convolution operations on the global weight vector, local weight vector, and multi-scale weight vector respectively to generate their respective attention weights; The attention weights are weighted by nonlinear functions to adaptively integrate global, local and multi-scale information, and finally a spatial-channel attention feature map is obtained.
7. The deep-sea mining vehicle acoustic image target detection system according to claim 6, characterized in that: The application logic of the adaptive spatial feature fusion detection head: The global feature information of the spatial-channel attention feature map is obtained through global average pooling, and the adaptive convolution kernel is generated using a dynamic routing network; The generated adaptive convolution kernel is used to perform convolution operations on the feature map, and at the same time, selective convolution is performed on some channel features of the spatial-channel attention feature map; The global features and the local selective convolution features are nonlinearly fused to obtain the local-dynamic feature map.
8. The deep-sea mining vehicle acoustic image target detection system according to claim 7, characterized in that: The nonlinear spatial fusion processing of the adaptive spatial feature fusion detection head includes: Utilize feature maps of different scales for spatial alignment through upsampling and downsampling; According to the spatial context information, the feature layer that is most effective for the final prediction is dynamically selected for nonlinear weight fusion to form the final target detection feature map.
9. The deep-sea mining vehicle acoustic image target detection system according to claim 1, characterized in that: The adaptive difficulty loss function module extracts the following: Calculate the intersection-over-union ratio between the target prediction box and the true target box, and classify the target samples into easy-to-detect samples, boundary samples, and difficult-to-detect samples based on the size of the intersection-over-union ratio; The samples that are easy to detect have an intersection-to-union ratio higher than the average intersection-to-union ratio; The boundary samples have an intersection-union ratio that is within the range of the average intersection-union ratio; The difficult-to-detect samples have an intersection-to-union ratio lower than the average intersection-to-union ratio; Different weights are adaptively assigned according to the difficulty level.
10. A method for detecting targets in acoustic images of a deep-sea mining vehicle, based on the implementation of a deep-sea mining vehicle acoustic image target detection system according to any one of claims 1 to 9, characterized in that: The following steps are involved: A dual-branch structure is adopted, which includes a D-Conv branch structure and a P-Conv branch structure: the fused global dynamic feature map is obtained based on the D-Conv branch structure, and the local channel fusion feature map after the integrated channel is obtained based on the P-Conv branch structure; Use the local-dynamic feature fusion module to replace the C2f module in the YOLOv8 network; Using the hybrid attention mechanism, the input image is processed from three parallel branches: global, local, and multi-scale. The features extracted by the three branches are weighted fused to generate a unified spatial-channel attention feature map. Add a high-resolution auxiliary detection layer, adopt an adaptive multi-scale fusion strategy, dynamically select the feature layer that is most effective for the final prediction based on spatial context information, perform nonlinear weight fusion, and form the final target detection feature map; An adaptive difficulty loss function is adopted. Based on the adaptive difficulty loss function, the loss weight of each target prediction box is adaptively adjusted according to the intersection-over-union ratio between each target prediction box and the true target box in the target detection feature map. Model training is performed, and model parameters are updated. The updated model is deployed to the host computer of the deep-sea mining vehicle, which receives acoustic image data collected by the forward-looking sonar in real time and outputs real-time target detection results.