River water surface target intelligent visual detection and identification method

Abstract

The application discloses a river water surface target intelligent visual detection and identification method, relates to the technical field of deep learning, and first enhances the expression capacity of a model through a structural reparameterization feature extraction module; then, the feature information is caused to flow across layers through a feature cross-layer fusion and reconstruction module, the feature expression of a target is enhanced, redundant information is removed through spatial channel reconstruction, and the detection capacity of different scale targets in a complex background is enhanced; finally, the recall rate of the model is enhanced through an improved loss function; after comparing the target detection algorithm with mainstream target detection algorithms, the applicability of the algorithm to the river water surface target detection task is proved, and the generalization capacity of the algorithm in different environments is proved through experiments; experiments show that the average recognition rate of 16 classes of objects on an actual data set is higher than that of existing methods, and an effective solution is provided for intelligent supervision of a river.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to an intelligent visual detection and recognition method for river surface targets. Background Technology

[0002] With rapid urbanization, river environments have been severely damaged, necessitating effective monitoring and management methods. Target detection on the river surface plays a crucial role in river monitoring systems and forms the foundation for detecting illegal activities in rivers. Traditional river pollution monitoring methods typically rely on manual patrols or real-time observation from fixed monitoring points, which suffer from limited coverage, low efficiency, and high costs. However, utilizing drone aerial photography combined with target detection technology to monitor rivers significantly improves monitoring efficiency and accuracy while reducing costs.

[0003] In drone aerial videos, water surface targets often face problems such as low pixel count and unclear feature outlines, and are also affected by complex environmental factors in the river channel (such as weather and light intensity). Different types of water surface targets have huge differences in scale. All of these problems can lead to unsatisfactory model performance. Therefore, it is extremely important to study an effective detection method that can adapt to the complex river environment and targets of different scales on the water surface.

[0004] Existing deep learning-based object detection algorithms are mainly divided into two-stage object detection models and single-stage object detection models. Classic two-stage object detection models, such as R-CNN, achieve object detection through candidate region generation and feature extraction. Fast R-CNN further improves the detection accuracy of deformable objects based on R-CNN by using region-of-interest pooling. Faster R-CNN introduces a region proposal network to improve detection speed. An improved Faster R-CNN model proposed in existing technologies uses feature fusion and anchor point optimization to improve detection accuracy.

[0005] In recent years, the performance of single-stage object detection models has gradually improved. These models treat object detection as a regression problem, eliminating the need to generate candidate regions. They use the entire image as input for computation, offering advantages such as fast detection speed and wide adaptability.

[0006] However, existing detection methods struggle with targets of varying sizes due to insufficient feature information fusion and limited receptive field size. Furthermore, the complex background environment of river channels, including overcast skies and rainy / foggy weather, obscures target features, making detection difficult, particularly for small targets. While multi-branch structures can reduce information loss and improve detection accuracy during feature extraction, they also decrease model detection efficiency. Summary of the Invention

[0007] To address the above technical problems, this invention provides an intelligent visual detection and recognition method for river surface targets, comprising the following steps:

[0008] S1. Acquire images of the river surface and perform image preprocessing operations;

[0009] S2. Construct a river surface target detection network, which includes a backbone network, a feature cross-layer fusion and reconstruction module, and a prediction head. The backbone network includes a structure reparameterization feature extraction module.

[0010] S3. Extract multi-level feature maps from the input image using a backbone network;

[0011] S4. Enhance feature maps through feature cross-layer fusion and reconstruction modules;

[0012] S5. The prediction head generates outputs for object localization and classification based on enhanced feature maps;

[0013] S6. Post-process the output using nonmaximum suppression to remove low-scoring predictions.

[0014] The technical solution further defined in this invention is:

[0015] Furthermore, in step S2, the structure reparameterization feature extraction module adopts a multi-branch structure during training; during inference, the multi-branch structure is reparameterized into a single-path structure; the structure reparameterization feature extraction module downsamples the input features through convolutions with a kernel size of 3×3 and a stride of 2, reducing the length and width of the feature map to half of the input.

[0016] As described above, in the intelligent visual detection and recognition method for river surface targets, step S3 involves splitting the feature map into multiple parts for processing. Feature information is extracted through multiple SR modules. During training, the SR module adopts a multi-branch structure, including 3×3 convolutional layer branches, 1×1 convolutional layer branches, and direct mapping of the previous layer output. During the structural reparameterization process, the 3×3 and 1×1 convolutional layers in the multi-branch structure are first batch normalized. Then, the normalized 1×1 convolutional kernel is transformed into a 3×3 convolutional kernel with only a valid weight at the center and zero weights elsewhere. The direct mapping of the previous layer output is regarded as a 3×3 convolutional kernel with a weight of 1 at the center and zero weights elsewhere. Finally, the 1×1 convolutional layer branches and the previous layer output are added to the weights of the normalized 3×3 convolutional layer kernels, respectively. During inference, this is equivalent to extracting features using a 3×3 convolutional kernel.

[0017] As described above, in the intelligent visual detection and recognition method for river surface targets, in step S4, after extracting feature maps of different levels from the backbone network, the feature cross-layer fusion and reconstruction module performs weighted fusion of the original input at the same level and the feature maps at different levels, and then reconstructs the spatial information and channel information through the spatial channel reconstruction unit.

[0018] As described above, in the intelligent visual detection and recognition method for river surface targets, in step S4, the feature cross-layer fusion and reconstruction module further deepens the feature information of the feature map extracted by the backbone network through a convolutional layer with a kernel size of 1×1 and a stride of 1. Then, the information after upsampling of the high-level feature map, the information after downsampling of the low-level feature map, and the deepened information are fused with different weights, as shown in the following formula:

[0019]

[0020] Among them, F i-1 F i and F i+1 w represents the features after downsampling from the low-level feature map, the features at this level, and the features after upsampling from the high-level feature map, respectively. i-1 w i and w i+1 Representing the corresponding weights at each trainable level; Concat represents the operation of concatenating features at different scales; the fused feature F c After processing by the spatial channel reconstruction unit, the result F is output through weight refinement and separation reconstruction. out .

[0021] As described above, the intelligent visual detection and recognition method for river surface targets includes a spatial channel reconstruction unit comprising a Weight unit, a convolutional layer, a Concat layer, a Trans1 module, a Trans2 module, and a separation reconstruction layer.

[0022] As described above, the intelligent visual detection and recognition method for river surface targets includes a Weight unit algorithm as shown in the following formula:

[0023]

[0024] Where W represents weight information; γ i ∈R N γ j ∈R N Represents the input feature F in The parameters learned through group normalization, R N Represents an N-dimensional real vector; i and j are both positive integers from 1 to N; μ and σ represent F. in The mean and standard deviation; ε represents the normal value; γ denotes element-wise multiplication; γ and β denote trainable affine transformations.

[0025] As described above, the intelligent visual detection and recognition method for river surface targets includes the following steps in the spatial channel reconstruction unit process:

[0026] S4.1 The weight information W is gated with a threshold of 0.5 to obtain the informative feature weight W1 and the non-informative weight W2;

[0027] S4.2, Input feature Fi n The feature map after point convolution is multiplied element-wise with the informative feature weight W1 and the non-informative feature weight W2 to obtain the information-rich feature F1 and the less informative feature F2.

[0028] S4.3. Segment features F1 and F2 along the channels respectively to obtain four features F1 and F2, each with only half the original number of channels. 11 F 12 F 21 and F 22 The features are then cross-summed to obtain features F3 and F4. Features F3 and F4 are then concatenated along the channel dimension to restore the input feature F1. n Same number of channels;

[0029] S4.4. Re-segment the feature map along the channel into half the number of channels, compress the number of channels of the feature map through point convolution, and obtain features F5 and F6 respectively;

[0030] S4.5. After processing the features using grouped convolution and pointwise convolution respectively through the Trans1 module, the features are summed one by one to obtain feature F7, as shown in the following formula: Where GWC and PWC represent learnable weight matrices, This indicates element-wise summation;

[0031] S4.6. The pointwise convolutional processed features are fused with the unprocessed features using the Trans2 module to obtain feature F8, as shown in the following formula: F8 = PWC(F6)UF6;

[0032] S4.7 Connect features F7 and F8 along the channel dimension, and output the result F by refining the weights and separating and reconstructing. out .

[0033] As described above, in the intelligent visual detection and recognition method for river surface targets, step S6 uses the minimum point distance-based bounding box similarity comparison metric MPDIoU as the loss function for model bounding box regression.

[0034] As described above, the intelligent visual detection and recognition method for river surface targets uses MPDIoU, which is calculated based on the minimum point distance between the predicted bounding box and the ground truth bounding box, as shown in the following formula:

[0035]

[0036] Where w and h represent the width and height of the input image, respectively. and These represent the coordinates of the top-left and bottom-right corners of the predicted bounding box, respectively. and These represent the coordinates of the top-left and bottom-right corners of the ground truth bounding box, respectively, and IoU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box.

[0037] The beneficial effects of this invention are:

[0038] (1) In this invention, a structural reparameterization feature extraction module SRFE is proposed. During the training process, a multi-branch structure is used to obtain sufficient information. During the inference process, it is reparameterized into a single-path structure, thereby maintaining high efficiency and low memory consumption and effectively improving the backbone feature extraction capability.

[0039] (2) In this invention, a feature cross-layer fusion and reconstruction module FCFR is proposed. FCFR expands the receptive field by fusing the original input and features from different layers, and then improves the performance of recognizing targets of different sizes and eliminates the influence of the river background environment through feature reconstruction. It can strengthen the correlation of features at different levels and improve the recognition ability of targets of different sizes. It can also reduce the dynamic background redundant feature calculation of the river surface environment through spatial channel reconstruction and improve information utilization.

[0040] (3) In this invention, the ability to detect dense targets and the convergence speed are improved by improving the loss function; a large-scale UAV aerial photography target detection dataset (UARODD) is also established. This dataset collects 16 types of marine targets and labels 9,500 images, which has a certain promoting effect on the research of multi-target detection of river surface. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0042] Figure 2 This is a schematic diagram of the structure of the river surface target detection network in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the structure reparameterization feature extraction module in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the spatial channel reconstruction unit in an embodiment of the present invention;

[0045] Figure 5 The above are schematic diagrams of bounding box regression in an embodiment of the present invention when the predicted bounding box and the true bounding box have the same intersection-union ratio but different heights and widths. (a) is a schematic diagram when the predicted bounding box is twice the size of the true bounding box, and (b) is a schematic diagram when the predicted bounding box is half the size of the true bounding box. Detailed Implementation

[0046] This embodiment provides an intelligent visual detection and recognition method for river surface targets, such as... Figure 1 As shown, it includes the following steps:

[0047] S1. Acquire images of the river surface and perform image preprocessing operations such as image enhancement and noise reduction.

[0048] S2. Construct a target detection network for river surface, such as Figure 2 As shown, the network includes a backbone network, a Feature Cross-Layer Fusion and Reconstruction (FCFR) module, and a prediction head. The backbone network includes a Structured Reparameterized Feature Extraction (SRFE) module.

[0049] Structural reparameterization feature extraction is a feature extraction method that can improve the accuracy of backbone feature extraction while ensuring the efficiency of backbone feature extraction. During the training process, SRFE adopts a multi-branch structure, which can extract more image feature information and enhance the expressive power of the model. During the inference process, the multi-branch structure is reparameterized into a single-path structure, which achieves the advantages of high efficiency and low memory consumption of the single-backbone model.

[0050] S3. For an input RGB image of 640×640×3, a backbone network is used to extract multi-level feature maps to fully preserve the spatial and semantic information of the target.

[0051] like Figure 3 As shown, SRFE downsamples the input features using convolutions with a kernel size of 3×3 and a stride of 2, reducing the width and height of the feature map to half that of the input. Point convolutions can increase the network depth. The feature map is split into multiple parts for processing, and feature information is extracted through multiple Structured Reparameterized (SR) modules. During training, the SR module adopts a multi-branch structure, including 3×3 convolutional layer branches, 1×1 convolutional layer branches, and direct mapping of the previous layer output.

[0052] In the structural reparameterization process, the 3×3 and 1×1 convolutional layers in the multi-branch structure are first batch normalized (BN). Then, the normalized 1×1 convolutional kernel is transformed into a 3×3 convolutional kernel with only a valid weight at the center and zero weights elsewhere. The direct mapping of the previous layer output is regarded as a 3×3 convolutional kernel with a weight of 1 at the center and zero weights elsewhere. Finally, the branches of the 1×1 convolutional layer and the output of the previous layer are added to the weights of the normalized 3×3 convolutional kernel. During inference, this is equivalent to extracting features using a 3×3 convolutional kernel. Compared to using a normally trained 3×3 convolutional kernel, SR has both excellent feature extraction capabilities of multi-branch structures and efficient detection speed of single-path structures.

[0053] S4. Enhance feature maps through cross-layer feature fusion and reconstruction modules, integrate feature maps of different scales to improve the perception of multi-scale targets, and then reconstruct the fused feature maps through spatial channels to further strengthen the features of key areas.

[0054] After extracting feature maps from different levels in the backbone network, the feature cross-layer fusion and reconstruction module performs weighted fusion of the original input at the same level and feature maps from different levels, promoting contextual propagation of feature information, obtaining a larger receptive field and richer semantic information, which is beneficial for detecting aerial targets in river channels at different scales. Then, the Spatial-Channel Reconstruction Unit (SCRU) reconstructs spatial information such as shape and structure, as well as channel information such as color and depth, reducing redundant information generated after fusion. It performs well in the complex background of constantly changing river channel lighting and weather.

[0055] like Figure 2 As shown, FCFR further deepens the feature maps extracted from the backbone network by passing them through convolutional layers with a kernel size of 1×1 and a stride of 1. Then, it fuses the upsampled information from the high-level feature maps, the downsampled information from the low-level feature maps, and the deepened information according to different weights, as shown in the following formula:

[0056]

[0057] Among them, F i-1 F i and F i+1 w represents the features after downsampling from the low-level feature map, the features at this level, and the features after upsampling from the high-level feature map, respectively. i-1 w i and w i+1 Representing the corresponding weights at each trainable level; Concat represents the operation of concatenating features at different scales; the fused feature F cAfter SCRU processing to reduce background redundancy, the result F is output by refining weights and performing separation and reconstruction. out .

[0058] like Figure 4 As shown, the spatial channel reconstruction unit includes a Weight unit, a convolutional layer, a Concat layer, a Trans1 module, a Trans2 module, and a separation reconstruction layer; the Weight unit algorithm is shown in the following equation:

[0059]

[0060] Where W represents weight information; γ i ∈R N γ j ∈R N R represents the parameters learned by group normalization of the input feature Fin. N Represents an N-dimensional real vector; i and j are both positive integers from 1 to N; μ and σ represent F. in The mean and standard deviation; ε represents a small positive constant to ensure the stability of division; γ denotes element-wise multiplication; γ and β denote trainable affine transformations.

[0061] The SCRU processing flow reduces redundant feature information and improves information utilization during detection. Specifically, it includes the following steps:

[0062] S4.1 The weight information W is gated with a threshold of 0.5 to obtain the informative feature weight W1 and the non-informative weight W2;

[0063] S4.2, Input features F in The feature map after point convolution is multiplied element-wise with the informative feature weight W1 and the non-informative feature weight W2 to obtain the information-rich feature F1 and the less informative feature F2.

[0064] S4.3. Segment features F1 and F2 along the channels respectively to obtain four features F1 and F2, each with only half the original number of channels. 11 F 12 F 21 and F 22 The features are then cross-summed to obtain features F3 and F4. Features F3 and F4 are then concatenated along the channel dimension to reconstruct the input feature F. in Same number of channels;

[0065] S4.4. Re-segment the feature map along the channel into half the number of channels, compress the number of channels of the feature map through point convolution, and obtain features F5 and F6 respectively;

[0066] S4.5. Using the Transl module, the features are processed separately using Group-Wise Convolution (GWC) and Point-Wise Convolution (PWC), and then summed item by item to obtain feature F7, as shown in the following formula: Where GWC and PWC represent learnable weight matrices, This indicates element-wise summation;

[0067] S4.6. The pointwise convolutional processed features are fused with the unprocessed features using the Trans2 module to obtain feature F8, as shown in the following formula: F8 = PWC(F6)UF6;

[0068] S4.7 Connect features F7 and F8 along the channel dimension, and output the result F by refining the weights and separating and reconstructing. out .

[0069] S5. Using the detection head, the system identifies and locates targets on the water surface, outputs the type of target and its specific location in the image, thereby achieving accurate detection and intelligent perception of targets on the river surface.

[0070] S6. Post-process the output using nonmaximum suppression to remove low-scoring predictions.

[0071] In aerial images of river surfaces captured by drones, there are numerous dense samples with multiple targets occluding each other, resulting in reduced target feature information and decreased model detection accuracy. To effectively alleviate these problems and improve detection accuracy, this embodiment will use the Minimum Point Distance-IoULoss Function (MPDIoU) as the loss function for the model's bounding box regression.

[0072] When the predicted bounding box and the ground truth bounding box have the same intersection-union ratio, but different heights and widths, such as... Figure 5 As shown, the green box represents the true bounding box, and the red box represents the predicted bounding box. The traditional bounding box regression loss function will calculate the same value in both cases, which will be regarded as equivalent during the optimization process, hindering the model from adjusting the parameters correctly and reducing the convergence speed and accuracy.

[0073] MPDIoU is calculated based on the minimum point distance between the predicted bounding box and the ground truth bounding box, as shown in the following formula:

[0074]

[0075] Where w and h represent the width and height of the input image, respectively. and These represent the coordinates of the top-left and bottom-right corners of the predicted bounding box, respectively. and These represent the coordinates of the top-left and bottom-right corners of the ground truth bounding box, respectively, and IoU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box.

[0076] MPDIoU comprehensively addresses various factors related to bounding box regression, such as overlapping regions, variations in height and width, and distances between center points. It can accurately distinguish these situations and effectively avoid gradient vanishing during dense target recognition, thereby improving model robustness.

[0077] Handling multi-scale targets in complex backgrounds remains a key challenge in river target detection. This embodiment first enhances the model's expressive power through a structural reparameterization feature extraction module. Then, it promotes cross-layer feature fusion and reconstruction modules to enhance the feature representation of scale targets. Further spatial channel reconstruction removes redundant information, enhancing the detection capability of targets at different scales in complex backgrounds. Finally, it improves the model's recall by refining the loss function. Comparison of this invention's target detection algorithm with mainstream target detection algorithms demonstrates its applicability in river surface target detection tasks. Experiments prove its generalization ability in different environments. Experiments show that this algorithm achieves an average recognition rate of 79.6% for 16 object classes on a real dataset (map@0.5), outperforming existing methods and providing an effective solution for intelligent river monitoring.

[0078] In addition to the embodiments described above, the present invention may have other implementations. All technical solutions formed by equivalent substitution or equivalent transformation fall within the protection scope claimed by the present invention.

Claims

1. A method for intelligent visual detection and recognition of targets on the surface of a river, characterized in that: Includes the following steps: S1. Acquire images of the river surface and perform image preprocessing operations; S2. Construct a river surface target detection network, which includes a backbone network, a feature cross-layer fusion and reconstruction module, and a prediction head. The backbone network includes a structure reparameterization feature extraction module. S3. Extract multi-level feature maps from the input image using a backbone network; S4. Enhance feature maps through cross-layer feature fusion and reconstruction module; after extracting feature maps of different levels from the backbone network, the cross-layer feature fusion and reconstruction module performs weighted fusion of the original input at the same level and feature maps at different levels, and then reconstructs spatial and channel information through spatial channel reconstruction unit; the spatial channel reconstruction unit includes Weight unit, convolutional layer, Concat layer, Trans1 module, Trans2 module and separation reconstruction layer; the Weight unit algorithm is shown in the following formula: ； Where W represents weight information; Representing input features Parameters learned through group normalization Represents an N-dimensional real vector; i and j are both positive integers from 1 to N; and express The mean and standard deviation; Represents positive numbers; This represents element-wise multiplication; and Represents a trainable affine transformation; S5. The prediction head generates outputs for object localization and classification based on enhanced feature maps; S6. Post-process the output using nonmaximum suppression to remove low-scoring predictions.

2. The intelligent visual detection and recognition method for river surface targets according to claim 1, characterized in that: In step S2, the structure reparameterization feature extraction module adopts a multi-branch structure during training; during inference, the multi-branch structure is reparameterized into a single-path structure; the structure reparameterization feature extraction module downsamples the input features through a convolution with a kernel size of 3×3 and a stride of 2, reducing the length and width of the feature map to half of the input.

3. The intelligent visual detection and recognition method for river surface targets according to claim 1, characterized in that: In step S3, the feature map is split into multiple parts for processing, and feature information is extracted through multiple SR modules. During training, the SR module adopts a multi-branch structure, including 3×3 convolutional layer branches, 1×1 convolutional layer branches, and direct mapping of the previous layer output. During the structural reparameterization process, the 3×3 convolutional layers and 1×1 convolutional layers in the multi-branch structure are first batch normalized, and then the normalized 1×1 convolutional kernel is transformed into a 3×3 convolutional kernel with only a valid weight at the center and the rest weights being 0. The direct mapping of the previous layer output is regarded as a 3×3 convolutional kernel with a weight of 1 at the center and the rest weights being 0. Finally, the 1×1 convolutional layer branches and the previous layer output are added to the convolutional kernel weights of the normalized 3×3 convolutional layer, respectively. During inference, this is equivalent to extracting features with a 3×3 convolutional kernel.

4. The intelligent visual detection and recognition method for river surface targets according to claim 1, characterized in that: In step S4, the feature cross-layer fusion and reconstruction module further deepens the feature information of the feature map extracted by the backbone network through a convolutional layer with a kernel size of 1×1 and a stride of 1. Then, the information after upsampling of the high-level feature map, the information after downsampling of the low-level feature map, and the deepened information are fused with different weights, as shown in the following formula: ； in, , as well as These represent the features after downsampling from the low-level feature map, the features at this level, and the features after upsampling from the high-level feature map, respectively. , as well as Representing the corresponding weights at each trainable level; Concat represents the operation of concatenating features at different scales; the fused features After processing by the spatial channel reconstruction unit, the results are output through weight refinement and separation reconstruction. .

5. The intelligent visual detection and recognition method for river surface targets according to claim 1, characterized in that: The process of the spatial channel reconstruction unit specifically includes the following steps: S4.1 The weight information W is obtained by gating with a threshold of 0.5 to obtain the informative feature weights. Non-informative weights ; S4.2, Input Features The feature maps, after being enhanced by dot convolution, are compared with informative feature weights. Non-informative weights Element-wise multiplication yields information-rich features. and less information features ; S4.3, Features and Segmenting along each channel yields four channels with features that are only half the original size. , , as well as And by cross-adding them one by one, we can obtain the features. and Then the features and Connect along the channel dimension to recover the input features Same number of channels; S4.

4. Re-segment the feature map along the channels into half the number of channels, compress the number of channels of the feature map through point convolution, and obtain the feature maps respectively. and ; S4.

5. The features are processed by grouped convolution and pointwise convolution respectively through the Trans1 module, and then summed item by item to obtain the features. As shown in the following formula: Where GWC and PWC represent learnable weight matrices, This indicates element-wise summation; S4.

6. The Trans2 module is used to fuse the pointwise convolutional processed features with the unprocessed features to obtain the final features. As shown in the following formula: ; S4.7, Features and The results are output by connecting along the channel dimension, refining the weights, and separating and reconstructing. .

6. The intelligent visual detection and recognition method for river surface targets according to claim 1, characterized in that: In step S6, the bounding box similarity comparison metric MPDIoU based on minimum point distance is used as the loss function for the model bounding box regression.

7. The intelligent visual detection and recognition method for river surface targets according to claim 6, characterized in that: The MPDIoU is calculated based on the minimum point distance between the predicted bounding box and the ground truth bounding box, as shown in the following formula: ； Where w and h represent the width and height of the input image, respectively. and These represent the coordinates of the top-left and bottom-right corners of the predicted bounding box, respectively. and These represent the coordinates of the top-left and bottom-right corners of the ground truth bounding box, respectively, and IoU represents the intersection-union ratio of the predicted bounding box and the ground truth bounding box.

Images