Adaptive spatial transformation network for cross-medium optical distortion

Abstract

The invention discloses a self-adaptive spatial transformation network for cross-medium optical distortion, and solves the problem that the existing spatial transformation network is difficult to consider both global transformation and local nonlinear optical distortion. The network is integrated in a target detection model, and the method comprises the steps that firstly, a parameter prediction network synchronously outputs affine transformation parameters and a water surface fluctuation physical parameter diagram; then, the local distortion grid generation network generates a distortion sampling grid based on the parameter diagram and a predefined water surface fluctuation physical model; meanwhile, the grid generator generates an affine sampling grid according to the affine transformation parameters; then, a grid fusion device fuses the two grids to generate a final sampling grid; and finally, the sampler samples the input image or the feature map according to the final sampling grid to obtain an output image or a feature map. Through a space transformation mechanism guided by physical prior, optical distortion caused by water surface fluctuation can be effectively relieved, and the accuracy and robustness of a target detection model in a cross-medium underwater scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and more specifically to an adaptive spatial transformation network for cross-medium optical distortion. Background Technology

[0002] Detecting underwater targets using aerial platforms such as drones has significant applications in marine monitoring, ecological research, and military security. However, when photographing underwater targets from the air, the random fluctuations of the water surface and the presence of water turbulence make the refraction path of light across the air-water interface unpredictable, resulting in severe geometric distortion of underwater targets in the generated images. This distortion differs significantly from global linear transformations (such as rotation and scaling), manifesting as local, nonlinear, and non-rigid deformation that evolves dynamically over time. This undoubtedly poses a significant challenge to the accurate identification and location of underwater targets. Adding to the complexity, underwater targets may also move passively due to water surface fluctuations or actively based on their own characteristics (such as the swimming of live fish), causing their orientation and position to change at different times, further complicating their accurate detection.

[0003] Existing deep learning-based object detection methods (such as classic algorithms like YOLO and Faster R-CNN) exhibit significant performance degradation when faced with such complex scenarios. The fundamental reason lies in the inherent lack of invariance to complex spatial transformations within convolutional neural networks. While Spatial Transformer Networks (STNs), as embeddable modular components, can enhance spatial invariance by transforming input images or feature maps to "more standard" and "more manageable" forms, they still have significant limitations when dealing with cross-medium underwater object detection scenarios. Standard STNs typically employ affine transformations or thin-plate spline transformations as their core transformation mechanisms. Affine transformations can only model global linear transformations (such as rotation, translation, scaling, and shearing), failing to effectively handle local nonlinear optical distortions caused by water surface ripples. While thin-plate spline transformations offer greater flexibility, their high parameter prediction difficulty can lead to unstable or even divergent training processes, and their lack of explicit incorporation of problem-related physical prior knowledge limits their generalization ability.

[0004] In summary, there is an urgent need for an adaptive spatial transformation network that can simultaneously handle global transformations and local nonlinear optical distortions, in order to improve the robustness and detection accuracy of target detection models in complex cross-medium underwater environments. Summary of the Invention

[0005] To address the challenges of varying underwater target morphologies caused by random surface fluctuations and turbulent water flow, this invention provides an Optical-Distortion Adaptive STN (ODA-STN) for cross-medium optical distortion. This network not only inherits the standard STN's approach of transforming input images or feature maps to a more "standard" and "processable" form to enhance spatial invariance, but also expands its capabilities to simultaneously handle global rotation, translation, scaling, shearing, and local nonlinear optical distortions. This significantly improves the accuracy and environmental adaptability of target detection models in complex cross-medium underwater environments.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An adaptive spatial transformation network for cross-medium optical distortion, the network being integrated into a target detection model, is used to perform adaptive spatial transformation on an input image or feature map, and includes:

[0008] Step S1, Construct the parameter prediction network: The parameter prediction network includes a backbone network and (In this invention) The backbone network has 10 parallel prediction branches; it is used to process the input image or feature map. Feature extraction is performed, and the generated feature maps are simultaneously output to the two branches; the first prediction branch is a global parameter regression branch, used for output. Parameters of 1D affine transformation The second prediction branch is the spatial parameter map prediction branch, used to output a graph containing... Physical parameters of water surface fluctuations in each channel ;

[0009] Step S2, constructing a locally distorted mesh generation network: The locally distorted mesh generation network is connected to the spatial parameter map prediction branch described in step S1, and is used to generate the network based on the physical parameter map of the water surface fluctuations. A wavefront feature map is constructed using a predefined physical model of water surface ripples. This wavefront feature map is then nonlinearly mapped using a lightweight fully convolutional network to generate a distorted sampling mesh that represents the sampling offset. ;

[0010] Step S3, Build a mesh generator: The mesh generator is connected to the global parameter regression branch described in step S1, and is used to generate a mesh based on the affine transformation parameters. Generate affine sampling mesh ;

[0011] Step S4, Build a mesh fusion device: The mesh fusion device is connected to the local twisted mesh generation network described in step S2 and the mesh generator described in step S3, and is used to merge the affine sampled mesh. With the twisted sampling grid Weighted fusion is performed to obtain the final sampling grid. ;

[0012] Step S5, Set up the sampler: The sampler is connected to The mesh fusion unit described in step S4 is used to adjust the final sampled mesh. Determine the output image or feature map. Each pixel in The sampling coordinates are used to calculate the corresponding pixel values ​​based on these coordinates to complete the process. The construction of.

[0013] The beneficial effects of this invention are:

[0014] (1) Morphological standardization guidance: Through the synergistic effect of parameter prediction network, local twisted mesh generation network and mesh generator, this invention can actively learn how to transform underwater targets with varied morphologies in input images or feature maps into a more standard and easier-to-process morphology for the target detection model, thereby significantly improving the rotation invariance and twist invariance of the target detection model at the feature level.

[0015] (2) Physical Prior Enhancement for Generalization: By introducing physical parameter maps of water surface waves and predefined physical models of water surface waves, reasonable physical constraints of optical refraction are embedded into the spatial transformation process. This not only improves the interpretability of the learned transformation rules, but also more effectively constrains the understanding space, avoids overfitting to meaningless deformation patterns under limited data, and significantly enhances the generalization ability of the model.

[0016] (3) End-to-end joint optimization: The adaptive spatial transformation network for cross-medium optical distortion and the target detection model together constitute an end-to-end trainable system. The learning of transformation rules is optimized directly with the goal of improving the final detection performance, ensuring the effectiveness and relevance of the preprocessing transformation.

[0017] (4) Modularity and universality: The adaptive spatial transformation network for cross-medium optical distortion proposed in this invention is designed as an independent module, which can be flexibly integrated with various mainstream target detection models such as YOLO, Faster RCNN, and RetinaNet, and has good universality and application flexibility. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the structure of an adaptive spatial transformation network for cross-medium optical distortion in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the system architecture for integrating an adaptive spatial transformation network for cross-medium optical distortion with a target detection model in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] See Figure 1 and Figure 2 This invention provides an adaptive spatial transformation network for cross-medium optical distortion. This network is integrated as a preprocessing module into the front end of the YOLO11 target detection model, together forming an end-to-end cross-medium underwater target detection system. The specific implementation process of this system is as follows:

[0022] Step 1: Build the parameter prediction network.

[0023] The parameter prediction network comprises a backbone network and two parallel prediction branches. The backbone network employs a lightweight convolutional neural network structure, consisting of a sequence of convolutional layers, ReLU activation functions, and max-pooling layers, as well as two cascaded convolutional layers, used to process the input image. Feature extraction is performed, and the generated feature maps are simultaneously output to the two branches.

[0024] The first prediction branch is a global parameter regression branch, which consists of a global average pooling layer and a fully connected layer, and is used to output 6-dimensional affine transformation parameters. The second prediction branch is the spatial parameter map prediction branch, used to output the image compared to the input image. Physical parameter diagrams of water surface ripples of the same size and containing 4 channels This branch will execute the following processing steps in sequence:

[0025] (1) Upsample the feature map output by the backbone network to the size of the input image and stitch it with the input image in the channel dimension;

[0026] (2) Input the spliced ​​feature map into an encoder consisting of two convolutional layers for feature encoding;

[0027] (3) Input the encoded feature map into four parallel independent convolutional layers to generate initial amplitude maps respectively. Initial wavenumber plot Initial orientation angle diagram Compared with the initial phase diagram ;

[0028] (4) Initial amplitude diagram Compared with the initial wavenumber plot Apply the Softplus activation function to keep its value positive; the specific formula is as follows:

[0029] ,

[0030] in, Amplitude plot, Wavenumber plot;

[0031] (5) Initial orientation angle diagram Compared with the initial phase diagram Perform linear scaling to map it to a preset range; the calculation formula is as follows:

[0032] ,

[0033] in, This is a direction angle diagram. For phase diagram, and This is a preset scaling factor. In this embodiment, it is set... , This maps the direction angle and phase to... interval;

[0034] (6) The above amplitude diagram Wavenumber plot Direction Angle Diagram Phase diagram By stitching the data along the channel dimension, a physical parameter map of water surface fluctuations representing spatial changes is generated. .

[0035] Step 2: Build a local twisted mesh generation network.

[0036] The twisted mesh generation network is connected to the spatial parameter map prediction branch described in step 1, and is used to generate the network based on the water surface wave physical parameter map. Generate a warped offset mesh representing the sampling offset. Its generation process includes:

[0037] (1) Create a new image with the same output image Normalization rule reference coordinate grid of the same size ,in For the first on the grid The coordinates of one pixel;

[0038] (2) For the rule reference coordinate grid Each coordinate point on Based on pixel index From the physical parameters diagram of water surface ripples Extract the corresponding local physical parameter amplitude from the corresponding channel , wave number Direction angle With phase value The wavefront displacement was calculated using a sinusoidal superposition model. and gradient , The specific calculation formula is as follows:

[0039] ;

[0040] As a specific example, for the rule reference coordinate grid An example point on Assuming it's based on pixel index From the physical parameters diagram of water surface ripples The local physical parameters extracted from the corresponding channel are: amplitude , wave number Direction angle With phase value Then its wavefront displacement With gradient , The specific values ​​are:

[0041] .

[0042] (3) Traverse the reference coordinate grid of the rule. For all coordinate points, generate complete wavefront displacement and gradient fields; concatenate the obtained wavefront displacement and gradient fields along the channel dimension to form a wavefront feature map; input the wavefront feature map into a lightweight fully convolutional network for nonlinear mapping, and output a distorted offset mesh. This grid is used to characterize the output image. Each position Input image The sampling offset in the diagram. In this embodiment, the lightweight fully convolutional network sequentially includes two sets of "convolutional layer-ReLU activation function" structures, one convolutional layer, and one Tanh activation function.

[0043] Step 3: Build the mesh generator.

[0044] The mesh generator is connected to the global parameter regression branch described in step 1, for use based on the affine transformation parameters. Generate an affine sampling mesh Its generation process includes:

[0045] (1) Create a new image with the same output image Normalized target coordinate grid of the same size ,in For the first on the grid The coordinates of one pixel;

[0046] (2) For the regular target coordinate grid Each coordinate point on Based on the affine transformation parameters The generated affine transformation matrix is ​​used to calculate the affine sampling position of the point. The calculation formula is:

[0047] ;

[0048] As a concrete example, let's assume it's based on An example of the generated affine transformation matrix is:

[0049] ,

[0050] For the target coordinate grid An example point on Its corresponding affine sampling position for:

[0051] .

[0052] (3) Traverse the target coordinate grid Generate a complete affine sampling mesh from all coordinate points. .

[0053] Step 4: Build the mesh merger.

[0054] The mesh fusion unit connects the local distorted mesh generation network described in step 2 and the mesh generator described in step 3, and is used to merge the affine sampled mesh. With the twisted sampling grid Weighted fusion is performed to obtain the final sampling grid. The fusion formula is ,in It is a learnable weighting coefficient.

[0055] As a concrete example, suppose the following The value is 0.3. For the output image... An example point on In the twisted sampling grid The corresponding sampling offset is In the affine sampling grid The corresponding affine sampling position is According to the weighted fusion formula, the example point in the input image Final sampling coordinates on for:

[0056] .

[0057] Step 5: Set up the sampler.

[0058] The sampler is connected to the input image. The mesh fusion unit described in step 4 is used to adjust the final sampled mesh. right Sampling is performed, and the transformed image is output. The sampling process of the sampler includes: for For each target pixel on the grid, according to the final sampling grid Find it in The corresponding sampling coordinates And denormalize it to the input image If the pixel coordinates ;like If it is a floating-point number, then it is obtained through bilinear interpolation. Calculate pixel values ​​in the middle; otherwise, The pixel value at that coordinate is directly assigned to The target pixel in the image.

[0059] As a specific embodiment, Example points on The final sampling coordinates are Input image The size is The specific calculation process for bilinear interpolation is as follows:

[0060] (1) Transform the normalized coordinate system back to the pixel coordinate system of the input image. In this embodiment, pixel center alignment mapping is used. The specific calculation formula is as follows:

[0061] ;

[0062] (2) Determine its position based on pixel coordinates. The coordinates and pixel values ​​of the four nearest neighbors can be obtained. , , and .

[0063] (3) Calculate the weight of each neighboring pixel, where the horizontal weight parameter is: The vertical weighting parameter is ;

[0064] (4) The final pixel value calculated by bilinear interpolation is

[0065]

[0066] (5) Extend the above calculation process to For each coordinate point in the image, the output image is obtained. .

[0067] To illustrate how to apply the adaptive spatial transformation network for cross-medium optical distortion to target detection tasks, this embodiment provides the following scheme: 6000 original images containing underwater targets (using the 26 letters of the English alphabet as an example) are collected via an unmanned aerial vehicle (UAV) platform. After annotation, these images are divided into training, validation, and test sets in a 7:2:1 ratio. The ODA-STN network is integrated as a preprocessing module into the front end of the YOLO11 target detection model to construct an end-to-end target detection system. The training of the end-to-end target detection system is supervised by the real labels of the dataset, with target detection loss as the optimization objective, jointly optimizing all parameters of the ODA-STN network and YOLO11. The performance of the trained end-to-end target detection system is evaluated using a reserved test set.

Claims

1. An adaptive spatial transformation network for cross-medium optical distortion, characterized in that, The network is configured to be integrated into an object detection model and includes: Step S1, Construct the parameter prediction network: The parameter prediction network includes a backbone network and (In this invention) The backbone network has 10 parallel prediction branches; it is used to process the input image or feature map. Feature extraction is performed, and the generated feature maps are simultaneously output to the two branches; the first prediction branch is a global parameter regression branch, used for output. Parameters of 1D affine transformation The second prediction branch is the spatial parameter map prediction branch, used to output a graph containing... Physical parameters of water surface fluctuations in each channel ; Step S2, constructing a locally distorted mesh generation network: The locally distorted mesh generation network is connected to the spatial parameter map prediction branch described in step S1, and is used to generate the network based on the physical parameter map of the water surface fluctuations. A wavefront feature map is constructed using a predefined physical model of water surface ripples. This wavefront feature map is then nonlinearly mapped using a lightweight fully convolutional network to generate a distorted sampling mesh that represents the sampling offset. ; Step S3, Build a mesh generator: The mesh generator is connected to the global parameter regression branch described in step S1, and is used to generate a mesh based on the affine transformation parameters. Generate affine sampling mesh ; Step S4, Build a mesh fusion device: The mesh fusion device is connected to the local twisted mesh generation network described in step S2 and the mesh generator described in step S3, and is used to merge the affine sampled mesh. With the twisted sampling grid Weighted fusion is performed to obtain the final sampling grid. ; Step S5, Set up the sampler: The sampler is connected to The mesh fusion unit described in step S4 is used to adjust the final sampled mesh. Determine the output image or feature map. Each pixel in The sampling coordinates are used to calculate the corresponding pixel values ​​based on these coordinates to complete the process. The construction of.

2. The network according to claim 1, characterized in that, The backbone network described in step S1 adopts a lightweight convolutional neural network structure, which includes multiple convolutional layers, activation functions, and pooling layers; the global parameter regression branch has a fully connected layer at the end to output 6-dimensional affine transformation parameters. The spatial parameter map prediction branch output includes a water surface wave physical parameter map with four channels. This branch includes the following steps: upsampling the feature map output by the backbone network to the input image size and concatenating it with the input image in the channel dimension; inputting the concatenated feature map into an encoder composed of convolutional layers for feature encoding; passing the encoded feature map through four parallel independent convolutional layers and subsequent constraint processing to generate amplitude map, wavenumber map, orientation angle map, and phase map respectively; and concatenating the four parameter maps in the channel dimension to generate the output image or feature map. Physical parameter diagrams of water surface ripples of the same size.

3. The network according to claim 1, characterized in that, The wavefront feature map construction process in step S2 includes: creating and outputting an image or feature map. Normalization rule reference coordinate grid of the same size Based on the physical parameter diagram of the water surface fluctuation And predefined physical models of water surface waves The wavefront displacement field and gradient field are calculated and generated, and the wavefront displacement field and gradient field are concatenated along the channel dimension to form the wavefront feature map; the lightweight fully convolutional network contains multiple convolutional layers and activation functions.

4. The network according to claim 3, characterized in that, The physical model for water surface waves adopts a sinusoidal wave superposition model, and the wave surface displacement field and gradient field are calculated by traversing the reference coordinate grid. All coordinate points, based on the pixel index of each coordinate point, are obtained from the physical parameter map of water surface ripples. Extracting local physical parameter amplitude from the corresponding channel , wave number Direction angle With phase The wavefront displacement was calculated using a sinusoidal superposition model. and gradient , The specific calculation formula is as follows: ， in, Reference coordinate grid Upper The coordinates of each pixel.

5. The network according to claim 1, characterized in that, Step S3 describes the generation process of the mesh generator, which includes: creating and... Normalized target coordinate grid of the same size Traverse the target coordinate grid of the rule. All coordinate points, based on the affine transformation parameters The generated affine transformation matrix is ​​used to calculate the affine sampling position of each coordinate point, thereby generating the affine sampling mesh. The specific calculation formula is as follows: ， in, Let be the affine transformation matrix. Target coordinate grid Upper The coordinates of a pixel. Affine sampling grid Upper The coordinates of each pixel.

6. The network according to claim 1, characterized in that, The final sampled mesh in the mesh fusion machine The fusion formula is: ， in, It is a learnable weighting coefficient.

7. The network according to claim 1, characterized in that, The sampling process of the sampler includes: for the output image or feature map For each target pixel on the grid, according to the final sampling grid Find it in The corresponding sampling coordinates And denormalize it to the input image If the pixel coordinates ;like If it is a floating-point number, then an interpolation algorithm is used to... Calculate pixel values ​​in the middle, otherwise... The pixel value at that coordinate is directly assigned to The target pixel in the image.

8. The network according to claim 7, characterized in that, The interpolation algorithm is bilinear interpolation, and its calculation formula is as follows: ， in, Is it the output image or feature map? The Middle On the first channel The value of each pixel. Is it an input image or feature map? The Middle Location on each channel pixel values, and It is the weight function for bilinear interpolation.

Images