An autonomous obstacle avoidance navigation method for water area unmanned vehicles based on wavelet attention visual enhancement and rule pre-training SAC

Abstract

The application provides a water area unmanned vehicle autonomous obstacle avoidance navigation method based on wavelet attention visual enhancement and rule pre-training SAC. The method first enhances the underwater environment image through the U-Net with wavelet decomposition and double attention modules, and constructs a reinforcement learning state vector; generates offline expert samples based on a rule strategy, and completes Actor pre-training through behavior cloning; then generates new samples online to expand the experience replay pool, and iteratively updates the double Critic network, the target value network and the Actor strategy network in turn; after training, the SAC network is used to output control actions in real time to realize navigation and obstacle avoidance. The scheme improves the underwater image perception effect, reduces the reinforcement learning training difficulty with the help of behavior cloning, and improves the vehicle target tracking, obstacle avoidance and path smoothing control ability.

Description

Technical Field

[0001] This invention belongs to the field of obstacle avoidance technology for unmanned aerial vehicles (UAVs), specifically relating to an autonomous obstacle avoidance and navigation method for waterborne UAVs based on wavelet attention visual enhancement and rule pre-training SAC. Background Technology

[0002] Unmanned surface vessels and autonomous underwater robots can undertake tasks such as water area inspection, water quality monitoring, water area security patrol, underwater target search, water emergency rescue, deep-sea pipeline and cable inspection, and seabed facility operation and maintenance. During the operation, they rely on cameras, binocular vision sensors, sonar, positioning sensors, inertial navigation units and motion control systems to collect environmental information and autonomously make motion decisions.

[0003] However, the complex working conditions of actual aquatic environments severely restrict the perception and navigation performance of equipment. In surface environments, interference such as water reflection, wave disturbance, fog, strong light, and shadows significantly reduce image quality, resulting in blurred obstacle edges and insufficient distinction between targets and backgrounds. Underwater environments experience severe light attenuation and significant color shifts. Suspended particles in the water cause light scattering, which, combined with partial occlusion by objects, ultimately leads to low contrast, poor clarity, and insufficient recognition of target semantic features in underwater images. Although traditional image enhancement algorithms can slightly improve image brightness and contrast, they are prone to defects such as color distortion, over-enhancement, noise amplification, and loss of detail, making them unsuitable for imaging needs in complex aquatic environments.

[0004] In terms of autonomous obstacle avoidance decision-making, existing solutions also have significant limitations. Traditional rule-based obstacle avoidance algorithms rely on manually pre-setting distance thresholds, heading deviations, and obstacle avoidance turning logic, resulting in conservative path planning, a tendency to get trapped in local optima, and poor adaptability when facing dynamic obstacles. Ordinary reinforcement learning obstacle avoidance strategies are trained from scratch through random exploration, generating a large number of collision samples in the early stages, leading to low training efficiency, poor training process safety, and a tendency for policy degradation under complex scenarios such as strong water flow disturbances and abrupt changes in obstacle distribution, resulting in insufficient obstacle avoidance stability.

[0005] Therefore, the industry urgently needs an autonomous navigation method for unmanned underwater vehicles that combines enhanced visual perception with optimized intelligent obstacle avoidance decision-making, so as to ensure that the equipment can achieve stable, real-time, safe and highly robust autonomous obstacle avoidance control in complex water surface and underwater environments. Summary of the Invention

[0006] This invention proposes an autonomous obstacle avoidance navigation method for unmanned aerial vehicles (UAVs) in waterways based on wavelet attention visual enhancement and rule-based pre-training (SAC). By enhancing key visual regions and important feature channels through wavelet attention, the method improves image quality and obstacle perception stability in complex water environments. Behavioral cloning pre-training is performed through a policy network. Then, the policy is further fine-tuned in a simulated or real-world environment using the SAC algorithm, enabling the UAV to achieve safe, smooth, and efficient autonomous obstacle avoidance control in a continuous action space.

[0007] To achieve the above objectives, the present invention proposes the following technical content:

[0008] An autonomous obstacle avoidance and navigation method for unmanned aerial vehicles in waterways based on wavelet attention visual enhancement and rule-based pre-trained SAC includes the following steps:

[0009] S1: The unmanned aerial vehicle collects the current time. Environmental images and navigation status;

[0010] S2: View the environment image The input is fed into a pre-built, trained U-Net visual enhancement network, which outputs an enhanced image. ;

[0011] S3: Based on the enhanced image Construct reinforcement learning state vectors;

[0012] S4: Based on expert experience, offline samples are used to optimize and pre-train the Actor policy network of SAC;

[0013] S5: After the Actor policy network is pre-trained, the initial parameters of the network obtained from pre-training are fixed, the behavior cloning loss is discarded, and only the SAC policy loss is relied upon. By combining the samples in the overall experience replay pool, reinforcement learning fine-tuning is carried out, and the value network and Actor network optimization are completed step by step;

[0014] S6: During the online prediction process, the reinforcement learning state vector at the current time step from S3 above will be used. The data is input into the SAC network trained in S5 to obtain the real-time control actions of the unmanned aerial vehicle. Unmanned aerial vehicles rely on Perform navigation and obstacle avoidance.

[0015] Further, step S2 includes the following steps:

[0016] S2.1: Use convolutional layers to process the environment image Mapped to an initial feature map; the formula is:

[0017]

[0018] In the formula, Represents the initial feature map; Indicates the basic convolutional layer deal with

[0019] S2.2: The Layer encoder previous layer features As input, we get the first Layer coding features;

[0020]

[0021] In the formula, Indicates the first Layer coding features; Indicates the first Layer encoder processing; Indicates the number of layers in the encoder;

[0022] S2.3: First define the first The features are convolutional in the spatial domain, and then wavelet decomposition is performed on these features; the formula is:

[0023]

[0024]

[0025] In the formula, Indicates the first Spatial domain convolutional features of layers; Indicates the first Convolution processing of layer encoders; Indicates the first Layer coding features; Represents a two-dimensional discrete wavelet transform; Indicates the first Spatial domain convolutional features of layers; Indicates the first The horizontal low-frequency component in the spatial domain convolutional features of a layer; Indicates the first Horizontal high-frequency components in the spatial domain convolutional features of a layer; Indicates the first Vertical high-frequency components in the spatial domain convolutional features of a layer; Indicates the first Diagonal high-frequency components in the spatial domain convolutional features of a layer;

[0026] S2.4: Dimension alignment is performed through 1×1 convolution, so that the wavelet components and convolutional features have the same spatial size and channel dimension;

[0027]

[0028] In the formula, Indicates the first Wavelet frequency domain characteristics of the layer; This indicates that the process has undergone 1×1 convolution. Indicates concatenation of channel dimensions;

[0029] S2.5: The first The spatial domain convolutional features of the layer are concatenated with the wavelet frequency domain features, and then fused features are obtained through convolution and nonlinear activation; the formula is:

[0030]

[0031]

[0032] In the formula, Indicates the first Layer splicing characteristics; Represents a nonlinear activation function; Indicates the parameters of the fusion convolution kernel; Indicates the bias term; Indicates the first Layer fusion characteristics;

[0033] S2.6: Channel attention; with the first Layer fusion features As input, global average pooling and global max pooling are performed respectively. The pooling results are processed by a shared multilayer perceptron and then summed. An activation function is used to obtain the channel attention weights. Finally, the weights are multiplied element-wise with the input features to obtain the channel enhancement features.

[0034]

[0035]

[0036] In the formula, Indicates channel attention weights; This represents the Sigmoid activation function; Indicates a shared multilayer perceptron; Indicates global average pooling; Indicates global max pooling; This indicates element-wise multiplication; Indicates channel enhancement features;

[0037] S2.7: Spatial Attention; Spatial attention is used to emphasize key regions in an image. Spatial attention enhances features using the aforementioned channels. As input, spatial augmentation features are obtained; the formula is:

[0038]

[0039]

[0040] In the formula, Indicates spatial attention weights; This represents the Sigmoid activation function; express Large and small convolution kernels; Indicates concatenation of channel dimensions; This represents average pooling along the channel direction; This represents maximum pooling along the channel direction; This indicates element-wise multiplication; Indicates channel enhancement features; Represents spatially enhanced features;

[0041] S2.8: The U-Net decoder upsamples the fused attention features stepwise and fuses the shallow detail features in the encoder by skip connections, generating the enhanced image through the output convolutional layer;

[0042]

[0043]

[0044] In the formula, Indicates the first Decoding characteristics of the layer decoder; Represents spatially enhanced features; Indicates an upsampling operation; Indicates concatenation of channel dimensions; Indicates the first Layer decoder convolution processing; Indicates the first Decoding characteristics of the layer decoder; This represents the surface features after decoding by the decoder; This indicates that the output layer has undergone convolution processing; This represents the enhanced image.

[0045] Further, step S3 includes the following steps:

[0046] S3.1: Based on the enhanced image Construct the set of obstacles detected at the current moment;

[0047]

[0048] In the formula, Represents the set of obstacles at the current moment; Indicates the current time. The location of the obstacle; This represents the enhanced image at the current time. The Middle The equivalent radius of each obstacle;

[0049] S3.2: Select the obstacle closest to the unmanned aerial vehicle;

[0050]

[0051]

[0052] In the formula, This represents the enhanced image at the current time. The Middle The distance between each obstacle and the unmanned vehicle at the current moment; Indicates the obstacle closest to the unmanned aerial vehicle; Indicates the current position of the unmanned vehicle; Indicates the current time. The location of the obstacle; This represents the enhanced image at the current time. The Middle The equivalent radius of each obstacle;

[0053] S3.3: Calculate the relevant parameters of the nearest obstacle and construct the reinforcement learning state vector;

[0054] At the current moment, the distance between the nearest obstacle and the unmanned aerial vehicle:

[0055]

[0056] In the formula, Indicates the location of the nearest obstacle; Indicates the equivalent radius of the nearest obstacle;

[0057] At the current moment, the relative azimuth angle between the nearest obstacle and the unmanned aerial vehicle:

[0058]

[0059] In the formula, This represents the angle-around normalization function; Represent the arctangent function in the second quadrant; This indicates the heading angle of the unmanned aerial vehicle at the current moment;

[0060] The target heading angle at the current moment is:

[0061]

[0062] In the formula, Indicates the target heading angle; Represent the arctangent function in the second quadrant; Indicates the location of the target point; Indicates the current position of the unmanned aerial vehicle;

[0063] The relative distance to the target is:

[0064]

[0065] In the formula, Indicates the relative distance to the target; This indicates the preset distance normalization scale;

[0066] Constructing reinforcement learning state vectors:

[0067]

[0068] In the formula, Represents the reinforcement learning state vector; These represent the velocity and angular velocity of the unmanned vehicle at the current moment, respectively, which are obtained in step S1; Indicates normalized self-heading. , This indicates the heading angle of the unmanned aerial vehicle at the current moment.

[0069] Furthermore, in step S4, the previously defined action space takes the following form:

[0070]

[0071] In the formula, This represents the heading control action for the m-th execution step; This represents the speed control action of the m-th execution step; This indicates the depth or vertical control action of the m-th execution step;

[0072] The reward function for the reinforcement learning interactive environment is then defined as follows:

[0073]

[0074] In the formula, This represents the immediate reward value returned by the environment after the m-th execution step. This indicates that the current objective is close to the reward; This represents the obstacle avoidance reward for the m-th execution step; This represents the path smoothing reward for the m-th execution step; This represents the event reward for the m-th execution step;

[0075] in,

[0076]

[0077] In the formula, Indicates the target distance weight; Indicates the heading error weight; This represents the normalized distance between the unmanned aerial vehicle and the target position at the m-th execution step; This represents the normalized self-heading of the m-th execution step; This represents the target heading angle for the m-th execution step;

[0078]

[0079] In the formula, This indicates the set obstacle detection distance threshold; Indicates the weight of the avoidance penalty; This represents the distance to the nearest obstacle at the m-th execution step; This represents the relative azimuth angle between the nearest obstacle and the unmanned aerial vehicle at the m-th execution step;

[0080] The formula is:

[0081]

[0082]

[0083] In the formula, Indicates the adjustment factor; This indicates a danger proximity marker for the m-th execution step; This indicates a preset safety threshold;

[0084]

[0085] In the formula, This represents the reward value for reaching the finish line; Indicates the penalty value for close-range danger; Indicates the collision penalty value; This represents the penalty value for exceeding the boundary. This indicates a preset distance threshold; This indicates a preset safety threshold; Indicates a collision symbol; Indicates an out-of-bounds marker; Indicates other scenarios;

[0086]

[0087] In the formula, This represents the smoothing weight.

[0088] Furthermore, the SAC agent includes an Actor policy network, a dual Critic value network, a target Critic network, a temperature coefficient adaptive adjustment module, and an experience replay pool.

[0089] Step S4 specifically includes the following steps:

[0090] S4.1: Generate a sample set of expert experience;

[0091]

[0092]

[0093] In the formula, This represents a sample set of expert experience. This indicates the total number of transferred samples collected offline; Indicates the first The sample corresponding to the next state transition; Indicates the first The state vector of the next state transition; Indicates based on Normalized actions output by rule-based obstacle avoidance strategies; This indicates the immediate reward or penalty value returned by the environment after the action is performed; This represents the state vector of the next state transition obtained after the environment jumps following the execution of an action; This indicates whether the voyage round has ended after this transfer. Indicates the end of the voyage, when This indicates that the voyage continues;

[0094] The calculation formula is as follows:

[0095]

[0096]

[0097] In the formula, This represents the action clipping function; Indicates the target heading control weight; Indicates obstacle avoidance weights; Indicates the preset action space range; Indicates obstacle avoidance correction items; This indicates the set obstacle detection distance threshold; Indicates an indicator function;

[0098] S4.2: Behavioral cloning pre-training loss; the formula is:

[0099]

[0100] In the formula, Expressing expectations; This represents the result after processing by the Actor policy network; Represent the L2 norm; update using gradient descent. Continuously minimize Complete pre-training.

[0101] Furthermore, the current state vector is obtained by calling S3. ;

[0102] Will The input is fed into the pre-trained Actor network, which outputs control actions. Reward Value Generate round termination indication ;

[0103] Assemble a brand-new autonomous interactive sample;

[0104]

[0105] Will Add it to the total experience replay pool;

[0106] After completion, perform the following network optimization steps:

[0107] S5.1: From the total experience replay pool In the middle, the first one is extracted The sample corresponding to the next state transition The action is ;

[0108] S5.2: Calculate the dual Critic target value;

[0109]

[0110] In the formula, Indicates execution of the first Reward and penalty values ​​after the next state transition; Indicates the discount factor; Indicates the first A target Critic network; Indicates the temperature coefficient; Indicates that the Actor policy network is in Under the condition, output action The probability value of the action, which includes yaw angular velocity, linear velocity, or thruster thrust; Indicates to and Enter the number The numerical values ​​output in the target Critic network;

[0111] The Critic loss function is:

[0112]

[0113] In the formula, This represents the total loss of the dual-critic network; Indicates to and Enter the number The numerical values ​​output in the target Critic network; Indicates in and Below, dual Critic target values;

[0114] The target Critic network uses soft updates:

[0115]

[0116] In the formula, Indicates the first The parameters of the target Critic network before the update; Indicates the first The updated parameters of the target Critic network; For the first Parameters of the main Critic network; Indicates the soft update coefficient;

[0117] S5.3: Update the Actor network;

[0118]

[0119] In the formula, This represents the loss function of the Actor network.

[0120] The beneficial effects that can be achieved by adopting the above technologies are:

[0121] 1. Combining wavelet decomposition with deep vision enhancement networks can simultaneously preserve the global structure of an image and local high-frequency details, thereby improving the visual perception quality in complex water surface and underwater environments.

[0122] 2. By adaptively focusing on key channels, key spatial regions, important frequency domain components, and cross-view complementary information through an attention mechanism, the feature representation ability of obstacle regions and occluded regions is improved.

[0123] 3. By using a rule-based pre-training mechanism, the SAC agent acquires basic obstacle avoidance capabilities in the early stages of training, significantly reducing the risk of collisions caused by random exploration.

[0124] 4. The SAC strategy can be further optimized based on the rule-based strategy, learning a smoother and more efficient continuous control strategy in dynamic obstacle scenarios. Attached Figure Description

[0125] Figure 1 This is the flowchart of this solution. Detailed Implementation

[0126] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0127] An autonomous obstacle avoidance and navigation method for unmanned aerial vehicles in waterways based on wavelet attention visual enhancement and rule-based pre-trained SAC includes the following steps:

[0128] S1: The unmanned aerial vehicle collects the current time. Environmental images and navigation status.

[0129] Navigation status includes the current speed. and angular velocity .

[0130] S2: View the environment image The image is input into a pre-built and trained U-Net visual enhancement network, which then performs wavelet decomposition, frequency domain fusion, and attention weighting layer by layer, ultimately outputting an enhanced image. .

[0131] U-Net is used as the backbone network for visual enhancement, including convolutional layers, encoders, bottleneck layers, decoders, and skip connection structures. The specific processing sub-steps are as follows:

[0132] S2.1: The convolutional layer will convert the environmental image... Mapped to an initial feature map; the formula is:

[0133]

[0134] In the formula, Represents the initial feature map; Indicates the basic convolutional layer deal with.

[0135] S2.2: The Layer encoder previous layer features As input, the result is processed through convolution, normalization, activation function, and downsampling to obtain the... Layer coding features; the formula is:

[0136]

[0137] In the formula, Indicates the first Layer coding features; Indicates the first Layer encoder processing; Indicates the number of layers in the encoder.

[0138] S2.3: In order to fuse spatial domain convolutional features with frequency domain wavelet features, first define the... The features are convolutional in the spatial domain, and then wavelet decomposition is performed on these features. The formula is:

[0139]

[0140] In the formula, Indicates the first Spatial domain convolutional features of layers; Indicates the first Convolution processing of layer encoders; Indicates the first Layer coding features;

[0141] Perform wavelet decomposition on this feature:

[0142]

[0143] In the formula, Represents a two-dimensional discrete wavelet transform; Indicates the first Spatial domain convolutional features of layers; Indicates the first The horizontal low-frequency component in the spatial domain convolutional features of a layer; Indicates the first Horizontal high-frequency components in the spatial domain convolutional features of a layer; Indicates the first Vertical high-frequency components in the spatial domain convolutional features of a layer; Indicates the first Diagonal high-frequency components in the spatial domain convolutional features of a layer.

[0144] S2.4: To ensure that wavelet components and convolutional features have the same spatial dimensions and channel dimensions, dimension alignment can be achieved through 1×1 convolution. The formula is:

[0145]

[0146] In the formula, Indicates the first Wavelet frequency domain characteristics of the layer; This indicates that the process has undergone 1×1 convolution. This indicates concatenation of channel dimensions.

[0147] S2.5: The first The spatial domain convolutional features of the layer are concatenated with the wavelet frequency domain features, and the fused features are obtained through convolution and nonlinear activation. The formula is:

[0148]

[0149]

[0150] In the formula, Indicates the first Layer splicing characteristics; Represents a non-linear activation function; Indicates the parameters of the fusion convolution kernel; Indicates the bias term; Indicates the first Layer fusion characteristics.

[0151] S2.6: Channel attention; with the first Layer fusion features As input, global average pooling and global max pooling are performed respectively. The pooling results are processed by a shared multilayer perceptron and then summed. An activation function is used to obtain the channel attention weights. Finally, the weights are multiplied element-wise with the input features to obtain the channel enhancement features.

[0152]

[0153]

[0154] In the formula, Indicates channel attention weights; This represents the Sigmoid activation function; Indicates a shared multilayer perceptron; Indicates global average pooling; Indicates global max pooling; This indicates element-wise multiplication; This indicates channel enhancement features.

[0155] S2.7: Spatial Attention; Spatial attention is used to emphasize key regions in an image. Spatial attention enhances features using the aforementioned channels. As input, spatial enhancement features are obtained.

[0156] The formula is:

[0157]

[0158]

[0159] In the formula, Indicates spatial attention weights; This represents the Sigmoid activation function; express Large and small convolution kernels; Indicates concatenation of channel dimensions; This represents average pooling along the channel direction; This represents maximum pooling along the channel direction; This indicates element-wise multiplication; Indicates channel enhancement features; This represents spatially enhanced features.

[0160] S2.8: The U-Net decoder upsamples the fused attention features stepwise and fuses shallow detail features from the encoder via skip connections, generating the enhanced image through the output convolutional layer. The formula is:

[0161]

[0162] In the formula, Indicates the first Decoding characteristics of the layer decoder; Represents spatially enhanced features; Indicates an upsampling operation; Indicates concatenation of channel dimensions; Indicates the first Layer decoder convolution processing; Indicates the first Decoding characteristics of layer decoders.

[0163] The enhanced image is calculated using the following formula:

[0164]

[0165] In the formula, This represents the surface features after decoding by the decoder; This indicates that the output layer has undergone convolution processing; This represents the enhanced image.

[0166] S3: Based on the enhanced image Construct reinforcement learning state vectors. This includes the following steps:

[0167] S3.1: Based on the enhanced image Construct the set of obstacles detected at the current moment.

[0168] The set of obstacles is as follows:

[0169]

[0170] In the formula, Represents the set of obstacles at the current moment; Indicates the current time. The location of the obstacle; This represents the enhanced image at the current time. The Middle The equivalent radius of an obstacle.

[0171] S3.2: Select the obstacle closest to the unmanned vehicle.

[0172]

[0173]

[0174] In the formula, This represents the enhanced image at the current time. The Middle The distance between each obstacle and the unmanned vehicle at the current moment; Indicates the obstacle closest to the unmanned aerial vehicle; Indicates the current position of the unmanned vehicle; Indicates the current time. The location of the obstacle; This represents the enhanced image at the current time. The Middle The equivalent radius of an obstacle.

[0175] S3.3: Calculate the relevant parameters of the nearest obstacle and construct the reinforcement learning state vector.

[0176] At the current moment, the distance between the nearest obstacle and the unmanned aerial vehicle:

[0177]

[0178] In the formula, Indicates the location of the nearest obstacle; This represents the equivalent radius of the nearest obstacle.

[0179] At the current moment, the relative azimuth angle between the nearest obstacle and the unmanned aerial vehicle:

[0180]

[0181] In the formula, This represents the angle-around normalization function; Represent the arctangent function in the second quadrant; This indicates the heading angle of the unmanned aerial vehicle at the current moment.

[0182] The target heading angle at the current moment is:

[0183]

[0184] In the formula, Indicates the target heading angle; Represent the arctangent function in the second quadrant; Indicates the location of the target point; This indicates the current location of the unmanned aerial vehicle.

[0185] The relative distance to the target is:

[0186]

[0187] In the formula, Indicates the relative distance to the target; This indicates the preset distance normalization scale.

[0188] Constructing reinforcement learning state vectors:

[0189]

[0190] In the formula, Represents the reinforcement learning state vector; These represent the velocity and angular velocity of the unmanned vehicle at the current moment, respectively, which are obtained in step S1; Indicates normalized self-heading. , This indicates the heading angle of the unmanned aerial vehicle at the current moment.

[0191] S4: Based on expert experience, offline samples are used to optimize and pre-train the SAC Actor policy network.

[0192] The action space of the unmanned aerial vehicle (UAV) corresponding to the reinforcement learning state vector is defined to facilitate subsequent training of the SAC policy network. The form of the action space is as follows:

[0193]

[0194] In the formula, This represents the heading control action for the m-th execution step; This represents the speed control action of the m-th execution step; This indicates the depth or vertical control action of the m-th execution step.

[0195] To evaluate the quality of the SAC network's output action and guide the parameter updates of the policy network, the reward function for the reinforcement learning interactive environment is further defined as follows:

[0196]

[0197] In the formula, This represents the immediate reward value returned by the environment after the m-th execution step. This indicates that the current objective is close to the reward; This represents the obstacle avoidance reward for the m-th execution step; This represents the path smoothing reward for the m-th execution step; This represents the event reward for the m-th execution step;

[0198] in,

[0199]

[0200] In the formula, Indicates the target distance weight; Indicates the heading error weight; This represents the normalized distance between the unmanned aerial vehicle and the target position at the m-th execution step; This represents the normalized self-heading of the m-th execution step; This represents the target heading angle for the m-th execution step.

[0201]

[0202] In the formula, This indicates the set obstacle detection distance threshold; Indicates the weight of the avoidance penalty; This represents the distance to the nearest obstacle at the m-th execution step; This represents the relative azimuth angle between the nearest obstacle and the unmanned aerial vehicle at the m-th execution step; The formula is:

[0203]

[0204]

[0205] In the formula, Indicates the adjustment factor; This indicates a danger proximity marker for the m-th execution step; This indicates a preset safety threshold;

[0206]

[0207] In the formula, This represents the reward value for reaching the finish line; Indicates the penalty value for close-range danger; Indicates the collision penalty value; This represents the penalty value for exceeding the boundary. This indicates a preset distance threshold; This indicates a preset safety threshold; Indicates a collision symbol; Indicates an out-of-bounds marker; Indicates other scenarios.

[0208]

[0209] In the formula, Indicates smoothing weights;

[0210] The SAC agent includes an Actor policy network, a dual Critic value network, a target Critic network, a temperature coefficient adaptive adjustment module, and an experience replay pool.

[0211] Specifically, the following steps are included:

[0212] S4.1: Generate a sample set of expert experience.

[0213]

[0214]

[0215] In the formula, This represents a sample set of expert experience. This indicates the total number of transferred samples collected offline; Indicates the first The sample corresponding to the next state transition; Indicates the first The state vector for the next state transition is generated by the logic in step S3. Indicates based on Normalized actions output by rule-based obstacle avoidance strategies; This indicates the immediate reward or penalty value returned by the environment after the action is performed; This represents the state vector of the next state transition obtained after the environment jumps following the execution of an action; This indicates whether the voyage round has ended after this transfer. The value can be 0 or 1, when This indicates the end of the voyage, meaning: the vehicle collided with an obstacle, the vehicle reached its destination, or the maximum number of simulation rounds was reached. This indicates that the voyage continues.

[0216] In the above formula, The calculation formula is as follows:

[0217]

[0218]

[0219] In the formula, This represents the action clipping function; Indicates the target heading control weight; Indicates obstacle avoidance weights; Indicates the preset action space range; Indicates obstacle avoidance correction items; This indicates the set obstacle detection distance threshold; Indicates an indicator function;

[0220] S4.2: Behavioral Cloning Pre-training Loss. The formula is:

[0221]

[0222] In the formula, Expressing expectations; This represents the result after processing by the Actor policy network; Represents the L2 norm. Updated using gradient descent. Continuously minimize After completing pre-training, an initial strategy with basic obstacle avoidance capabilities is obtained.

[0223] S5: After completing the pre-training of the Actor policy network, fix the initial parameters of the pre-trained network, discard the behavior cloning loss, and rely only on the SAC policy loss. By combining the samples in the overall experience replay pool, reinforcement learning fine-tuning is carried out, and the parameter optimization of the value network and the Actor policy network is completed step by step.

[0224] Initialize the total experience replay pool ,at this time ;

[0225] Call S3 to obtain the current state vector ;

[0226] Will The input is fed into the pre-trained Actor network, which outputs control actions. Reward Value Generate round termination indication ;

[0227] Assemble entirely new autonomous interactive samples;

[0228]

[0229] Will Add it to the total experience replay pool.

[0230] After completion, perform the following network optimization steps:

[0231] S5.1: From the total experience replay pool In the middle, the first one is extracted The sample corresponding to the next state transition The action is ;

[0232] S5.2: Calculate the dual Critic target value;

[0233] The formula for the dual Critic target value is:

[0234]

[0235] In the formula, This indicates the immediate reward or penalty value returned by the environment after the action is performed; Indicates the discount factor; Indicates the first A target Critic network; Indicates the temperature coefficient; Indicates that the Actor policy network is in Under the condition, output action The probability value of the action, which includes yaw angular velocity, linear velocity, or thruster thrust; Indicates to and Enter the number The numerical values ​​output in a target Critic network.

[0236] The Critic loss function is:

[0237]

[0238] In the formula, This represents the total loss of the dual-critic network; Indicates to and Enter the number The numerical values ​​output in the target Critic network; Indicates in and Below, the dual Critic target values ​​are defined. Based on the Critic loss function above, the gradient backpropagation algorithm is used to update the parameters, updating the two main Critic networks.

[0239] The target Critic network uses a soft update method to slowly synchronize the parameters of the main Critic network, avoiding drastic fluctuations in target value and improving training stability. The update formula is as follows:

[0240]

[0241] In the formula, Indicates the first The parameters of the target Critic network before the update; Indicates the first The updated parameters of the target Critic network; For the first The parameters of the main Critic network are trained in real time and continuously updated. This represents the soft update coefficient.

[0242] S5.3: Update the Actor network.

[0243]

[0244] In the formula, This represents the loss function of the Actor network.

[0245] S6: During the online prediction process, the reinforcement learning state vector at the current time step from S3 above will be used. The data is input into the SAC network trained in S5 to obtain the real-time control actions of the unmanned aerial vehicle. Unmanned aerial vehicles rely on Perform navigation and obstacle avoidance.

[0246] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for autonomous obstacle avoidance and navigation of unmanned underwater vehicles (UUVs) in waterways based on wavelet attention visual enhancement and rule-based pre-trained SAC, characterized in that, Includes the following steps: S1: The unmanned aerial vehicle collects the current time. Environmental images and navigation status; S2: View the environment image The input is fed into a pre-built, trained U-Net visual enhancement network, which outputs an enhanced image. ; S3: Based on the enhanced image Construct reinforcement learning state vectors; S4: Based on expert experience, offline samples are used to optimize and pre-train the Actor policy network of SAC; S5: After the Actor policy network is pre-trained, the initial parameters of the network obtained from pre-training are fixed, the behavior cloning loss is discarded, and only the SAC policy loss is relied upon. By combining the samples in the overall experience replay pool, reinforcement learning fine-tuning is carried out, and the value network and Actor network optimization are completed step by step; S6: During the online prediction process, the reinforcement learning state vector at the current time step from S3 above will be used. The data is input into the SAC network trained in S5 to obtain the real-time control actions of the unmanned aerial vehicle. Unmanned aerial vehicles rely on Perform navigation and obstacle avoidance.

2. The autonomous obstacle avoidance and navigation method for unmanned underwater vehicles based on wavelet attention visual enhancement and rule pre-training SAC as described in claim 1, characterized in that, Step S2 includes the following steps: S2.1: Use convolutional layers to process the environment image Mapped to an initial feature map; the formula is: In the formula, Represents the initial feature map; Indicates the basic convolutional layer deal with S2.2: The Layer encoder previous layer features As input, we get the first Layer coding features; In the formula, Indicates the first Layer coding features; Indicates the first Layer encoder processing; Indicates the number of layers in the encoder; S2.3: First define the first The features are convolutional in the spatial domain, and then wavelet decomposition is performed on these features; the formula is: In the formula, Indicates the first Spatial domain convolutional features of layers; Indicates the first Convolution processing of layer encoders; Indicates the first Layer coding features; Represents a two-dimensional discrete wavelet transform; Indicates the first Spatial domain convolutional features of layers; Indicates the first The horizontal low-frequency component in the spatial domain convolutional features of a layer; Indicates the first Horizontal high-frequency components in the spatial domain convolutional features of a layer; Indicates the first Vertical high-frequency components in the spatial domain convolutional features of a layer; Indicates the first Diagonal high-frequency components in the spatial domain convolutional features of a layer; S2.4: Dimension alignment is performed through 1×1 convolution, so that the wavelet components and convolutional features have the same spatial size and channel dimension; In the formula, Indicates the first Wavelet frequency domain characteristics of the layer; This indicates that the process has undergone 1×1 convolution. Indicates concatenation of channel dimensions; S2.5: The first The spatial domain convolutional features of the layer are concatenated with the wavelet frequency domain features, and then fused features are obtained through convolution and nonlinear activation; the formula is: In the formula, Indicates the first Layer splicing characteristics; Represents a non-linear activation function; Indicates the parameters of the fusion convolution kernel; Indicates the bias term; Indicates the first Layer fusion characteristics; S2.6: Channel attention; with the first Layer fusion features As input, global average pooling and global max pooling are performed respectively. The pooling results are processed by a shared multilayer perceptron and then summed. An activation function is used to obtain the channel attention weights. Finally, the weights are multiplied element-wise with the input features to obtain the channel enhancement features. In the formula, Indicates channel attention weights; This represents the Sigmoid activation function; Indicates a shared multilayer perceptron; Indicates global average pooling; Indicates global max pooling; This indicates element-wise multiplication; Indicates channel enhancement features; S2.7: Spatial Attention; Spatial attention is used to emphasize key regions in an image. Spatial attention enhances features using the aforementioned channels. As input, spatial augmentation features are obtained; the formula is: In the formula, Indicates spatial attention weights; This represents the Sigmoid activation function; express Large and small convolution kernels; Indicates concatenation of channel dimensions; This represents average pooling along the channel direction; This represents maximum pooling along the channel direction; This indicates element-wise multiplication; Indicates channel enhancement features; Represents spatially enhanced features; S2.8: The U-Net decoder upsamples the fused attention features stepwise and fuses the shallow detail features in the encoder by skip connections, generating the enhanced image through the output convolutional layer; In the formula, Indicates the first Decoding characteristics of the layer decoder; Represents spatially enhanced features; Indicates an upsampling operation; Indicates concatenation of channel dimensions; Indicates the first Layer decoder convolution processing; Indicates the first Decoding characteristics of the layer decoder; This represents the surface features after decoding by the decoder; This indicates that the output layer has undergone convolution processing; This represents the enhanced image.

3. The autonomous obstacle avoidance and navigation method for unmanned underwater vehicles based on wavelet attention visual enhancement and rule pre-training SAC as described in claim 1, characterized in that, Step S3 includes the following steps: S3.1: Based on the enhanced image Construct the set of obstacles detected at the current moment; In the formula, Represents the set of obstacles at the current moment; Indicates the current time. The location of the obstacle; This represents the enhanced image at the current time. The Middle The equivalent radius of each obstacle; S3.2: Select the obstacle closest to the unmanned aerial vehicle; In the formula, This represents the enhanced image at the current time. The Middle The distance between each obstacle and the unmanned vehicle at the current moment; Indicates the obstacle closest to the unmanned aerial vehicle; Indicates the current position of the unmanned vehicle; Indicates the current time. The location of the obstacle; This represents the enhanced image at the current time. The Middle The equivalent radius of each obstacle; S3.3: Calculate the relevant parameters of the nearest obstacle and construct the reinforcement learning state vector; At the current moment, the distance between the nearest obstacle and the unmanned aerial vehicle: In the formula, Indicates the location of the nearest obstacle; Indicates the equivalent radius of the nearest obstacle; At the current moment, the relative azimuth angle between the nearest obstacle and the unmanned aerial vehicle: In the formula, This represents the angle-around normalization function; Represent the arctangent function in the second quadrant; This indicates the heading angle of the unmanned aerial vehicle at the current moment; The target heading angle at the current moment is: In the formula, Indicates the target heading angle; Represent the arctangent function in the second quadrant; Indicates the location of the target point; Indicates the current position of the unmanned aerial vehicle; The relative distance to the target is: In the formula, Indicates the relative distance to the target; This indicates the preset distance normalization scale; Constructing reinforcement learning state vectors: In the formula, Represents the reinforcement learning state vector; These represent the velocity and angular velocity of the unmanned vehicle at the current moment, respectively, which are obtained in step S1; Indicates normalized self-heading. , This indicates the heading angle of the unmanned aerial vehicle at the current moment.

4. The autonomous obstacle avoidance and navigation method for unmanned underwater vehicles based on wavelet attention visual enhancement and rule pre-training SAC as described in claim 1, characterized in that, In step S4, the previously defined action space takes the following form: In the formula, This represents the heading control action for the m-th execution step; This represents the speed control action of the m-th execution step; This indicates the depth or vertical control action of the m-th execution step; The reward function for the reinforcement learning interactive environment is then defined as follows: In the formula, This represents the immediate reward value returned by the environment after the m-th execution step. This indicates that the current objective is close to the reward; This represents the obstacle avoidance reward for the m-th execution step; This represents the path smoothing reward for the m-th execution step; This represents the event reward for the m-th execution step; in, In the formula, Indicates the target distance weight; Indicates the heading error weight; This represents the normalized distance between the unmanned aerial vehicle and the target position at the m-th execution step; This represents the normalized self-heading of the m-th execution step; This represents the target heading angle for the m-th execution step; In the formula, This indicates the set obstacle detection distance threshold; Indicates the weight of the avoidance penalty; This represents the distance to the nearest obstacle at the m-th execution step; This represents the relative azimuth angle between the nearest obstacle and the unmanned aerial vehicle at the m-th execution step; The formula is: In the formula, Indicates the adjustment factor; This indicates a danger proximity marker for the m-th execution step; This indicates a preset safety threshold; In the formula, This represents the reward value for reaching the finish line; Indicates the penalty value for close-range danger; Indicates the collision penalty value; This represents the penalty value for exceeding the boundary. This indicates a preset distance threshold; This indicates a preset safety threshold; Indicates a collision symbol; Indicates an out-of-bounds marker; Indicates other scenarios; In the formula, This represents the smoothing weight.

5. The autonomous obstacle avoidance and navigation method for unmanned underwater vehicles based on wavelet attention visual enhancement and rule pre-training SAC as described in claim 4, characterized in that, The SAC agent includes an Actor policy network, a dual Critic value network, a target Critic network, a temperature coefficient adaptive adjustment module, and an experience replay pool. Step S4 specifically includes the following steps: S4.1: Generate a sample set of expert experience; In the formula, This represents a sample set of expert experience. This indicates the total number of transferred samples collected offline; Indicates the first The sample corresponding to the next state transition; Indicates the first The state vector of the next state transition; Indicates based on Normalized actions output by rule-based obstacle avoidance strategies; This indicates the immediate reward or penalty value returned by the environment after the action is performed; This represents the state vector of the next state transition obtained after the environment jumps following the execution of an action; This indicates whether the voyage round has ended after this transfer. Indicates the end of the voyage, when This indicates that the voyage continues; The calculation formula is as follows: In the formula, This represents the action clipping function; Indicates the target heading control weight; Indicates obstacle avoidance weights; Indicates the preset action space range; Indicates obstacle avoidance correction items; This indicates the set obstacle detection distance threshold; Indicates an indicator function; S4.2: Behavioral cloning pre-training loss; the formula is: In the formula, Expressing expectations; This represents the result after processing by the Actor policy network; Represent the L2 norm; update using gradient descent. Continuously minimize Complete pre-training.

6. The autonomous obstacle avoidance and navigation method for unmanned underwater vehicles based on wavelet attention visual enhancement and rule pre-training SAC as described in claim 1, characterized in that, Call S3 to obtain the current state vector ; Will The input is fed into the pre-trained Actor network, which outputs control actions. Reward Value Generate round termination indication ; Assemble a brand-new autonomous interactive sample; Will Add it to the total experience replay pool; After completion, perform the following network optimization steps: S5.1: From the total experience replay pool In the middle, the first one is extracted The sample corresponding to the next state transition The action is ; S5.2: Calculate the dual Critic target value; In the formula, Indicates execution of the first Reward and penalty values ​​after the next state transition; Indicates the discount factor; Indicates the first A target Critic network; Indicates the temperature coefficient; Indicates that the Actor policy network is in Under the condition, output action The probability value of the action, which includes yaw angular velocity, linear velocity, or thruster thrust; Indicates to and Enter the number The numerical values ​​output in the target Critic network; The Critic loss function is: In the formula, This represents the total loss of the dual-critic network; Indicates to and Enter the number The numerical values ​​output in the target Critic network; Indicates in and Below, dual Critic target values; The target Critic network uses soft updates: In the formula, Indicates the first The parameters of the target Critic network before the update; Indicates the first The updated parameters of the target Critic network; For the first Parameters of the main Critic network; Indicates the soft update coefficient; S5.3: Update the Actor network; In the formula, This represents the loss function of the Actor network.

Images