An aerial robot hierarchical reinforcement learning control method

Abstract

The present application relates to the technical field of image recognition, in particular to a layered reinforcement learning control method for an aerial robot, comprising establishing an environment state model, decomposing a road marking detection task into three control levels according to environment parameters, setting a region-level reward function, a flight path-level reward function and a pixel-level reward function respectively, wherein the region-level reward function evaluates target road section coverage integrity, the flight path-level reward function optimizes flight trajectory efficiency, and the pixel-level reward function guarantees image quality; firstly, training an environment adaptation layer to determine optimal detection region division; then training a path planning layer to generate a flight trajectory; finally training a posture control layer to realize position control; the layers interact through state information and reward signals; during flight, image quality is evaluated, and when image blurriness and / or contrast do not meet preset threshold values, a re-shooting mechanism and / or flight parameter adjustment are triggered, and a layered reinforcement learning model is updated simultaneously.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a layered reinforcement learning control method for an aerial robot. BACKGROUND

[0002] With the rapid development of intelligent transportation systems, the demand for automated detection and maintenance of road infrastructure is increasingly urgent. Traditional pavement marking detection mainly relies on manual inspection or ground vehicle-mounted equipment, which has problems such as low detection efficiency, limited coverage, high personnel safety risk, etc.

[0003] Especially when performing pavement marking detection tasks, the robot needs to accurately adjust the attitude and position while maintaining stable flight to obtain clear marking images. Traditional control methods are difficult to achieve coordinated optimization of multiple tasks. Although reinforcement learning technology has shown good adaptive ability in the field of robot control, traditional reinforcement learning methods have limitations such as high state space dimension, slow convergence speed, and difficulty in handling multi-level decision-making problems. The existing technology lacks a mechanism to effectively decompose complex control tasks, and cannot fully utilize the hierarchical characteristics of the task to improve learning efficiency and control performance.

[0004] Therefore, a layered reinforcement learning control method for an aerial robot is proposed. SUMMARY

[0005] The purpose of the present application is to provide a layered reinforcement learning control method for an aerial robot, which establishes an environment state model, decomposes the pavement marking detection task into three control levels according to environmental parameters, sets a regional level reward function, a flight path level reward function and a pixel level reward function, wherein the regional level reward function evaluates the completeness of the target road section coverage, the flight path level reward function optimizes the efficiency of the flight trajectory, and the pixel level reward function ensures the image quality; first, train the environment adaptation layer to determine the optimal detection area division; then train the path planning layer to generate the flight trajectory; finally, train the attitude control layer to achieve position control; the layers interact through state information and reward signals; during flight, evaluate the image quality, and when the image blur and / or contrast do not meet the preset threshold, trigger the re-shooting mechanism and / or adjust the flight parameters, while updating the layered reinforcement learning model.

[0006] To achieve the above purpose, the present application provides the following technical scheme:

[0007] A layered reinforcement learning control method for an aerial robot for pavement marking fuzzy identification, comprising:

[0008] Collecting environmental parameters and establishing an environment state model, dynamically decomposing the pavement marking detection task into three control levels according to the environmental parameters, namely the environment adaptation layer, the path planning layer and the attitude control layer;

[0009] The region-level reward function, the track-level reward function and the pixel-level reward function are respectively arranged for the three control levels, wherein the region-level reward function is used for evaluating target road section coverage integrity, the track-level reward function is used for optimizing flight trajectory efficiency, and the pixel-level reward function is used for ensuring image quality, and a layered reward mechanism is formed through weighted fusion;

[0010] A top-down training strategy is adopted, the environment adaptation layer is first trained to determine optimal detection region division, then the path planning layer is trained to generate a flight trajectory, and finally the attitude control layer is trained to realize position control; the layers interact through state information and reward signals;

[0011] Image quality is evaluated during flight, when image blurriness and / or contrast do not meet preset thresholds, a re-shooting mechanism and / or flight parameter adjustment are triggered, and the detection result is fed back to the layered reinforcement learning model for optimization and update.

[0012] Preferably, the environmental parameters include light intensity, wind speed, wind direction, temperature, humidity and shadow coverage rate; the environmental state model fuses and processes the environmental parameters through a Kalman filtering algorithm to generate an environmental adaptability index and a flight condition evaluation parameter; the environmental adaptability index is used to guide decision making of the environment adaptation layer, and the flight condition evaluation parameter is used for trajectory optimization of the path planning layer.

[0013] Preferably, the dynamic decomposition process includes: the environment adaptation layer calculates an environmental complexity index according to real-time environmental parameters, when the environmental complexity index exceeds an environmental preset threshold, the detection task is decomposed into multiple sub-tasks and priorities are assigned; the path planning layer receives sub-task information and generates a flight path set in combination with aircraft performance constraints; the attitude control layer calculates optimal flight attitude parameters according to the flight path set and image quality requirements.

[0014] Preferably, the region-level reward function determines a coverage integrity reward value by calculating a ratio of a detected region area to a total target region area; the track-level reward function determines a trajectory efficiency reward value by calculating a ratio of a flight path length to a theoretical shortest path length in combination with an energy consumption efficiency parameter; the pixel-level reward function determines an image quality reward value by calculating an image gradient variance and a contrast mean value, and the three reward functions are linearly weighted and fused through weight coefficients.

[0015] Preferably, the top-down training strategy specifically includes: a first stage of training an environment adaptation layer, a fixed path planning layer and a posture control layer parameter, training an environment perception network through environment simulation data; a second stage of training a path planning layer, training a path generation network using the environment state information output by the environment adaptation layer; a third stage of training a posture control layer, training a posture control network based on the output information of the previous two layers, and performing joint fine-tuning after training each layer.

[0016] Preferably, the image quality is evaluated by an image evaluation model; the image evaluation model includes:

[0017] An environment perception layer obtains illumination intensity values, shadow coverage percentages obtained through image analysis, wind speed values and aerial robot acceleration data, calculates an environment complexity index through an environment parameter fusion algorithm, and outputs an evaluation strategy signal;

[0018] A feature extraction layer extracts edge gradient amplitude calculated by a Sobel operator and image saturation pixel ratio under strong light conditions, extracts local binary pattern texture features and block contrast mean under shadow conditions, extracts second derivative variance calculated by a Laplacian operator and global standard deviation under normal illumination, and simultaneously extracts frequency domain features of image sharpness as auxiliary indicators according to the evaluation strategy signal of the environment perception layer, and outputs a multi-dimensional feature vector;

[0019] A quality evaluation layer calculates a comprehensive image quality score and a confidence index based on the multi-dimensional feature vector, dynamically allocates weights to different features by using an environment adaptive weighting function, wherein the weight coefficients are adjusted in real time according to the current illumination intensity and the shadow ratio, and outputs a quality evaluation result;

[0020] A decision fusion layer obtains a comprehensive score by using a fuzzy logic controller to perform decision fusion on the quality evaluation result of the quality evaluation layer, the current aerial robot power percentage, the task completion progress and a preset image quality lower threshold; when the comprehensive score is lower than the dynamically adjusted quality threshold and the remaining power is sufficient, a retake instruction and flight parameter adjustment suggestion are output.

[0021] Preferably, the state information interaction between each layer includes: the environment adaptation layer transmits the calculated environment complexity index, risk assessment matrix, recommended flight window time period and priority detection area coordinates to the path planning layer, while feeding back the current environment stability prediction and abnormal event warning information; the path planning layer transmits the target position sequence containing a time stamp, the expected flight speed vector, the attitude angle constraint range and the energy consumption budget allocation to the attitude control layer, while receiving the execution state confirmation and trajectory tracking deviation data fed back by the attitude control layer; the attitude control layer provides the real-time flight height, attitude angle, vibration amplitude and camera stability parameters to the image quality evaluation environment perception layer, and receives the flight parameter adjustment suggestions output by the decision fusion layer, including height increment, angle correction value and speed adjustment coefficient.

[0022] Preferably, the optimization update process of the layered reinforcement learning model comprises:

[0023] Collecting state-action-reward data during flight to build an experience replay buffer;

[0024] Using a priority sampling method to select training samples from the buffer;

[0025] Using time difference error to update network parameters of each layer, and preventing gradient explosion through gradient clipping, and updating model parameters once a complete detection task is completed.

[0026] Compared with the prior art, the present application has the following advantages:

[0027] 1、The present application proposes to divide the whole control task into three layers of environment adaptation layer, path planning layer and attitude control layer in view of the complex environmental changes of aerial robots in actual execution of road marking detection tasks. Through modeling and layered response to the environment state, the detection strategy is flexibly adjusted when the environment parameters fluctuate, avoiding problems such as image blur and path deviation. Especially under the influence of strong shadows, sudden wind speed changes and other interference factors, the environment adaptation layer can preferentially identify high-risk areas and guide the path planning strategy to avoid processing, enhancing the anti-interference ability and adaptability of the system in non-ideal environments, solving the problem that traditional single control strategy cannot be adjusted in real time, and improving the reliability and stability of the overall system in complex road scenes.

[0028] 2、The application introduces three types of reward functions, namely regional level, track level and pixel level, in the hierarchical structure, and constructs a weighted fusion hierarchical reward mechanism. During the task execution of the aerial robot, the value of the detection behavior is comprehensively evaluated from different levels: the regional level reward function is used to evaluate the integrity of the coverage range to avoid detection blind area; the track level reward function combines path length and energy efficiency to optimize flight path design; the pixel level reward function focuses on image sharpness and contrast to improve image recognition accuracy. According to the task requirements, the reward focus is flexibly adjusted to realize multi-objective collaborative optimization, improve the dual performance of flight path efficiency and image acquisition quality, and meet the composite demand for image accuracy and coverage efficiency in the road marking recognition task.

[0029] 3、The application proposes an image quality evaluation model and decision fusion mechanism to evaluate multi-dimensional image quality characteristics such as image blurring, contrast, frequency domain sharpness, etc. in real time. During flight, when the image quality does not meet the preset threshold, according to the power, task progress and environmental conditions, dynamically decide whether to re-shoot or adjust the flight parameters. The mechanism integrates fuzzy logic controller to realize the linkage adjustment of image quality score and flight control strategy, not only improves the effectiveness of image data, but also effectively avoids data distortion caused by abnormal light and vibration interference, improves the real-time and intelligent level of data acquisition, and ensures the system to continuously output high-quality image data in complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 A flowchart of the hierarchical reinforcement learning control method of the aerial robot provided by the application is provided.

[0031] Figure 2 A flowchart of task decomposition provided by the application is provided.

[0032] Figure 3 An image evaluation model structure diagram provided by the application is provided. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the application will be described in detail below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0034] Embodiment one:

[0035] The application provides a hierarchical reinforcement learning control method of an aerial robot for road marking fuzzy recognition, and the technical scheme is referred to Figure 1, the specific content is: collecting environmental parameters, and establishing an environmental state model, according to the environmental parameters, the road marking detection task is dynamically decomposed into three control levels, which are environment adaptation layer, path planning layer and attitude control layer; Aiming at the three control levels, the regional level reward function, the track level reward function and the pixel level reward function are set respectively, wherein the regional level reward function is used to evaluate the target road section coverage integrity, the track level reward function is used to optimize the flight trajectory efficiency, and the pixel level reward function is used to ensure the image quality, and a layered reward mechanism is formed by weighted fusion; A top-down training strategy is adopted, the environment adaptation layer is trained first to determine the optimal detection area division; Then train the path planning layer to generate the flight trajectory; Finally, the attitude control layer is trained to realize position control; The state information and reward signal are interacted between the layers; In the flight process, the image quality is evaluated, when the image blur and / or contrast do not meet the preset threshold, the re-shooting mechanism and / or the flight parameter adjustment are triggered, and the detection result is fed back to the layered reinforcement learning model for optimization and update.

[0036] The environmental parameters include light intensity, wind speed, wind direction, temperature, humidity and shadow coverage rate; The environmental state model fuses the environmental parameters by Kalman filtering algorithm to generate environmental adaptability index and flight condition evaluation parameters; The environmental adaptability index is used to guide the decision making of the environment adaptation layer, and the flight condition evaluation parameters are used for trajectory optimization of the path planning layer.

[0037] Specifically, the collection of environmental parameters covers the key factors affecting the flight of unmanned aerial vehicle and the image quality, and a comprehensive environmental perception system is established by simultaneously monitoring parameters such as light intensity, wind speed, wind direction, temperature, humidity and shadow coverage rate through multiple sensors. Further, Kalman filtering algorithm can effectively process the uncertainty and noise interference of multi-sensor data, and fuse the dispersed environmental data into unified environmental adaptability index and flight condition evaluation parameters, providing reliable environmental state description for upper layer decision making, and ensuring that the control system can make appropriate response according to environmental changes.

[0038] In this embodiment, the key parameters such as light, wind speed and temperature are cooperatively monitored to perceive the environmental factors affecting flight safety and image quality. Kalman filtering algorithm effectively eliminates sensor noise and fuses multi-dimensional environmental data into a unified adaptability index to provide accurate and reliable environmental state description for layered control. The aerial robot can adaptively adjust the flight strategy and shooting parameters according to real-time environmental changes, improve the execution efficiency and detection accuracy of road marking detection task in complex environmental conditions, and enhance the robustness and practicality of the system.

[0039] The dynamic decomposition process refers to Figure 2, specifically comprising: the environment adaptation layer calculates an environment complexity index according to real-time environment parameters, and when the environment complexity index exceeds a preset threshold, the detection task is decomposed into multiple subtasks and priority is assigned; the path planning layer receives the subtask information and generates a flight path set in combination with the performance constraints of the aircraft; the attitude control layer calculates optimal flight attitude parameters according to path requirements and image quality requirements.

[0040] Specifically, the dynamic decomposition mechanism realizes intelligent division from a complex task to a simple subtask, and the environment adaptation layer judges the challenge level of the current environment by calculating the environment complexity index, and automatically starts the task decomposition process when the complexity exceeds the system processing capability. Further, the priority assignment of the subtask considers factors such as the importance of the detection area, environmental stability, and resource constraints, the path planning layer generates an optimized flight path according to the endurance capability, flight speed limit, and other performance constraints of the aircraft, and the attitude control layer ensures that the aircraft can fly stably according to the planned path and obtain high-quality images.

[0041] In this embodiment, the task difficulty is evaluated in real time by the environment complexity index, the complex detection task is decomposed into manageable subtasks, the system load and decision complexity are reduced. The priority assignment mechanism ensures that key detection areas are processed first, improving task execution efficiency; the path planning layer generates an optimal path in combination with the performance constraints of the aircraft, avoiding the planning of infeasible paths and improving flight safety. The accurate parameter calculation of the attitude control layer ensures the image quality requirements and ensures the reliability of the detection data. This hierarchical dynamic decomposition method enables the system to maintain efficient and stable detection performance in complex environments, enhancing the adaptability and robustness of the system.

[0042] The process of decomposing the detection task into multiple subtasks and assigning priorities includes: dividing the target area into several detection units according to the continuity and integrity of the road markings, and determining the priority weight of each detection unit according to the traffic importance, marking wear degree, and detection difficulty coefficient; high-priority subtasks include main road markings, areas near traffic signs, and areas where problems have been found in historical detection; medium-priority subtasks include secondary roads and general road segments; low-priority subtasks include auxiliary roads and well-maintained areas; when the system detects insufficient power or deteriorating environmental conditions, the priority assignment strategy is automatically adjusted to prioritize the detection of high-importance areas.

[0043] The priority of each detection unit is determined by multi-dimensional weight calculation based on traffic importance, marking wear degree and detection difficulty, ensuring that key areas are detected first. The three-level priority classification strategy makes resource allocation more accurate, and the priority processing of main roads and problem areas improves the traffic safety guarantee effect. The dynamic priority adjustment mechanism automatically reallocates tasks when the power is insufficient or the environment deteriorates, ensuring that the most important areas are still detected under limited resources. Task decomposition and priority management significantly improve detection efficiency and resource utilization, enhancing the system's adaptability under various constraints.

[0044] The area-level reward function determines the coverage integrity reward value by calculating the ratio of the detected area to the total target area. The track-level reward function determines the trajectory efficiency reward value by calculating the ratio of the flight path length to the theoretical shortest path length, combined with the energy efficiency parameter. The pixel-level reward function determines the image quality reward value by calculating the image gradient variance and contrast mean. The three reward functions are linearly weighted and fused by weight coefficients.

[0045] Specifically, the hierarchical reward mechanism designs a special evaluation function for different control targets. The area-level reward function focuses on the comprehensiveness of task completion, ensuring complete coverage of the detection area. The track-level reward function optimizes flight efficiency, minimizing flight distance and energy consumption while ensuring task completion. Further, the pixel-level reward function ensures image acquisition quality, ensuring the accuracy of marking identification through gradient variance and contrast evaluation. The three levels of reward functions are linearly combined by weight coefficients to form a unified optimization target, realizing multi-objective collaborative optimization.

[0046] In this embodiment, by designing special evaluation functions for different control levels, multi-objective collaborative optimization of coverage integrity, flight efficiency and image quality is achieved. The area-level reward ensures the comprehensiveness of the detection task, the track-level reward minimizes flight distance and energy consumption while ensuring task completion, and the pixel-level reward ensures the accuracy of marking identification through gradient variance and contrast evaluation. Linear weighted fusion of the three reward functions forms a unified optimization target, avoiding performance deviation that may be caused by single target optimization, and improving overall detection performance and system efficiency.

[0047] The top-down training strategy specifically includes: a first stage training environment adaptation layer, fixing path planning layer and attitude control layer parameters, training environment perception network through environment simulation data; a second stage training path planning layer, training path generation network using environment state information output by the environment adaptation layer; a third stage training attitude control layer, training attitude control network based on the output information of the previous two layers, and joint fine-tuning after each layer training is completed.

[0048] Specifically, the top-down training strategy adopts a phased training method, which avoids parameter interference and convergence difficulties caused by simultaneous training of multiple network layers. Further, the first phase focuses on cultivating environmental perception ability, ensuring that the environment adaptation layer can accurately identify and analyze the environment state by fixing the parameters of other layers; the second phase trains path planning ability based on environmental perception; the third phase trains precise attitude control ability, and the final joint fine-tuning phase optimizes the coordination between layers to form a unified and efficient control system.

[0049] In the present embodiment, the phased training method avoids the problems of parameter interference and convergence difficulties caused by simultaneous training of multiple network layers, improving training stability and efficiency. By training layer by layer with fixed parameters of other layers, better feature learning effect is obtained. The progressive training from environmental perception to path planning to attitude control conforms to the natural hierarchical structure of the task, enabling the network to gradually master complex control skills. The final joint fine-tuning phase optimizes the coordination between layers to form a unified and efficient control system.

[0050] The image quality is evaluated by an image evaluation model, which is based on Figure 3 ; the image evaluation model comprises:

[0051] The environmental perception layer obtains the illumination intensity value, the shadow coverage percentage obtained by image analysis, the wind speed value, and the aerial robot acceleration data, calculates the environmental complexity index through an environmental parameter fusion algorithm, and outputs an evaluation strategy signal;

[0052] The feature extraction layer extracts the edge gradient amplitude calculated by the Sobel operator and the proportion of saturated pixels in the image under strong light conditions, extracts the local binary pattern texture feature and the block contrast mean under shadow conditions, extracts the second derivative variance calculated by the Laplacian operator and the global standard deviation under normal illumination, and simultaneously extracts the frequency domain feature of image sharpness as an auxiliary index, and outputs a multi-dimensional feature vector according to the evaluation strategy signal of the environmental perception layer;

[0053] The quality evaluation layer, based on the multi-dimensional feature vector, adopts an environment-adaptive weighting function to dynamically allocate weights to different features, wherein the weight coefficients are adjusted in real time according to the current illumination intensity and shadow proportion, calculates the comprehensive image quality score and confidence index, and outputs the quality evaluation result;

[0054] The decision fusion layer integrates the quality evaluation result of the quality evaluation layer, the current aerial robot power percentage, the task completion progress, and the preset image quality lower threshold, uses a fuzzy logic controller for decision fusion to obtain a comprehensive score; when the comprehensive score is lower than the dynamically adjusted quality threshold and the remaining power is sufficient, outputs a retake instruction and flight parameter adjustment suggestion.

[0055] The evaluation strategy signals specifically include: First, a lighting condition classification signal, which classifies the environment into three typical conditions—strong light, normal light, and shadow—based on the current light intensity value; Second, an environmental complexity level identifier, which quantifies and converts the environmental complexity index into a corresponding complexity level by fusing data from light intensity, shadow coverage percentage, wind speed, and aerial robot acceleration; Third, a feature extraction strategy instruction, which selects the optimal feature extraction scheme for different environmental conditions; Fourth, weight allocation reference information, which provides an environmental basis for the dynamic weight adjustment of the subsequent quality assessment layer. This achieves environmental adaptive optimization for image quality assessment, improving the accuracy of image quality judgment and the overall environmental adaptability of the system by establishing a mapping relationship between environmental parameters and feature effectiveness at the environmental perception layer.

[0056] Specifically, the image quality assessment model employs a four-layer architecture to implement a complete assessment process from environmental perception to decision execution. The environmental perception layer establishes the correlation between environmental parameters and image quality, while the feature extraction layer selects the most suitable image feature extraction method based on different environmental conditions. Furthermore, the quality assessment layer dynamically adjusts the importance of different features through an adaptive weight allocation mechanism, and the decision fusion layer comprehensively considers multiple factors such as image quality, power constraints, and task progress, employing a fuzzy logic controller for intelligent decision-making to ensure optimal detection results under resource-limited conditions.

[0057] The process of dynamically assigning weights to different features using the environment-adaptive weighting function includes: establishing a mapping relationship between light intensity and feature effectiveness; increasing the weight of edge gradient features while decreasing the weight of global contrast features under strong light conditions; increasing the weight of texture features and local contrast under shadow conditions; dynamically adjusting the feature fusion strategy based on shadow coverage; and using a regional feature extraction and evaluation method when shadow coverage is high. The weight adjustment process employs a smooth transition mechanism to avoid abrupt changes, and establishes an empirical model for weight adjustment using historical environmental data to achieve intelligent prediction and adaptive adjustment of weight coefficients.

[0058] By establishing a mapping relationship between illumination intensity and feature effectiveness, the system automatically selects the most effective feature combinations under different environmental conditions, improving the accuracy of image quality assessment. Dynamic adjustment strategies for shadow coverage and region-based processing effectively address the problem of uneven illumination. A smooth transition mechanism avoids the impact of abrupt weight changes on system stability, and an empirical model based on historical data enables intelligent prediction of weight coefficients, enhancing the system's self-learning ability. This adaptive weight allocation mechanism allows the system to maintain stable assessment performance in complex and changing environments, significantly improving the reliability of image quality judgment and the overall environmental adaptability of the system.

[0059] The state information interaction between each layer includes: the environmental adaptation layer transmits the calculated environmental complexity index, risk assessment matrix, recommended flight window time period and priority detection area coordinates to the path planning layer, while feeding back the current environmental stability prediction and abnormal event warning information; the path planning layer transmits the target position sequence containing a time stamp, the expected flight speed vector, the attitude angle constraint range and the energy consumption budget allocation to the attitude control layer, while receiving the execution state confirmation and trajectory tracking deviation data fed back by the attitude control layer; the attitude control layer provides the real-time flight height, attitude angle, vibration amplitude and camera stability parameters to the image quality evaluation environmental perception layer, and receives the flight parameter adjustment suggestions output by the decision fusion layer, including the height increment, angle correction value and speed adjustment coefficient; a direct communication channel is established between the environmental adaptation layer and the attitude control layer, and when a sudden environmental change is detected, the environmental adaptation layer directly sends an emergency adjustment instruction to the attitude control layer, bypassing the path planning layer to reduce the response delay, forming a multi-level linkage closed-loop control architecture.

[0060] Specifically, a comprehensive information interaction mechanism is established between each control layer, the environmental adaptation layer not only provides environmental analysis results, but also includes risk warning and flight suggestions, the path planning layer provides detailed flight instructions and constraint conditions, while receiving execution feedback for dynamic adjustment. Further, the direct interaction between the attitude control layer and the image quality evaluation module ensures that the flight parameters can be adjusted in real time according to the image quality requirements, forming a complete closed-loop control system from environmental perception to flight execution, improving the response speed and control accuracy of the system.

[0061] In this embodiment, by establishing a state information transmission system, each control layer can share key data in real time, ensuring the coordination and consistency of the decision. The risk assessment and flight suggestions provided by the environmental adaptation layer provide reliable basis for the upper layer planning, and the detailed instructions and constraint conditions of the path planning layer ensure the accurate execution of the flight mission. The direct interaction between the attitude control layer and the image quality evaluation module realizes the real-time optimization adjustment of the flight parameters, forming a complete closed-loop control. The multi-dimensional information interaction mechanism improves the response speed, control accuracy and environmental adaptation ability, ensures the collaborative work of each layer, and realizes efficient coordinated control from environmental perception to flight execution.

[0062] The optimization and update process of the hierarchical reinforcement learning model includes:

[0063] Collect state-action-reward data during flight to build an experience replay buffer;

[0064] Use priority sampling method to select training samples from the buffer;

[0065] Update the network parameters of each layer using time difference error, and prevent gradient explosion through gradient clipping, and update the model parameters once after completing a complete detection task.

[0066] Specifically, the optimization update of the hierarchical reinforcement learning model adopts an online learning method, an experience buffer is constructed by collecting actual flight data to ensure the authenticity and effectiveness of the learning samples. Further, the priority sampling method ensures that important experiences are learned preferentially, the time difference error update mechanism realizes the progressive optimization of parameters, the gradient clipping technique prevents numerical instability in the training process, and regular model updates ensure that the control strategy can continuously improve and adapt to new flight environments.

[0067] In this embodiment, an experience replay buffer is constructed by collecting actual flight data to ensure the authenticity and representativeness of the learning samples and improve the practicality of the model. The priority sampling method allows important experiences to be learned preferentially, accelerating the model convergence process and the mastery of key skills. The time difference error update realizes the progressive optimization of parameters, avoiding the impact of large parameter changes on system stability. The gradient clipping technique effectively prevents numerical instability problems in the training process, ensuring the reliability of the learning process. The regular model update mechanism allows the control strategy to continuously improve and adapt to new flight environments and task requirements.

[0068] The optimization update process of the hierarchical reinforcement learning model adopts an online incremental learning mechanism, including:

[0069] Real-time data acquisition and preprocessing: state-action-reward triple data is collected at fixed time intervals during flight, including environmental parameter vectors, control instruction sequences, image quality scores, and task completion degrees. The data is denoised by a sliding window filter, and a circular buffer with a capacity limit of the last 1000 groups of data is constructed;

[0070] Hierarchical sampling and weight update: differentiated sampling strategies are used for different control levels. The environmental adaptation layer updates its parameters every 10 detection areas, the path planning layer updates its parameters every complete flight path, and the attitude control layer continuously updates its parameters using a sliding average method. The weight update of each layer uses an adaptive learning rate adjustment, and the learning rate is dynamically adjusted according to the current task completion quality and environmental complexity;

[0071] Environment-adaptive model optimization: an environment type classifier is established to classify the environment into three categories: simple, medium, and complex, based on lighting conditions, wind speed levels, and task complexity. Independent subsets of network parameters are maintained for different environment types, and the corresponding parameters are updated based on the environment recognition results to avoid parameter interference in different environments;

[0072] Gradient optimization and stability guarantee, the gradient accumulation mechanism is adopted instead of the traditional experience replay, the gradient of continuous time steps is weighted and accumulated, then updated uniformly, the parameter divergence is prevented through adaptive gradient clipping and weight regularization, the upper and lower limits of parameter update are set to ensure the stability of the control system during the learning process, and when the system performance is detected to decline, the parameters are automatically rolled back to the previous stable version.

[0073] Real-time data acquisition and sliding window filtering ensure the timeliness and quality of learning samples, the circular buffer design balances the storage efficiency and data representativeness, the differentiated sampling strategy optimizes the update frequency according to the characteristics of each control layer, the adaptive learning rate adjustment improves the learning efficiency, the environment classifier and independent parameter subset design effectively avoid parameter interference in different environments, and the generalization ability of the model is enhanced, the gradient accumulation mechanism replaces the traditional experience replay to reduce storage overhead, and the adaptive gradient clipping and weight regularization guarantee the training stability, and the parameter rollback mechanism provides safety protection when the performance declines, so that the system can continuously adapt to environmental changes, and the learning efficiency and system stability are improved.

[0074] The present application realizes all-round intelligent decision-making from macro-environment perception to micro-posture control by establishing a three-layer architecture of environment adaptation layer, path planning layer and posture control layer, and the multi-sensor cooperative environment perception system combined with Kalman filter algorithm provides accurate and reliable environment state description, and improves the adaptability in complex environment. The hierarchical reward mechanism realizes the multi-objective balance of detection coverage integrity, flight efficiency and image quality through the cooperative optimization of regional, track and pixel level reward functions. The top-down training strategy and online incremental learning mechanism ensure continuous improvement and adaptation to new environments. The four-layer image quality evaluation model ensures high-quality data acquisition through environment adaptive weighting and fuzzy logic decision-making. The comprehensive information interaction mechanism between layers forms a complete closed-loop control, which significantly improves the response speed and control accuracy. This innovative hierarchical control architecture enables the aerial robot to exhibit excellent intelligent level, environmental adaptability and detection accuracy in the road marking detection task, providing an efficient and reliable technical solution for intelligent transportation infrastructure detection.

[0075] Embodiment two:

[0076] The aerial robot carries a multi-sensor system, including an illumination sensor, a wind speed and direction instrument, a temperature and humidity sensor, and an onboard camera, and performs a road marking detection task on a city three-lane section. First, environmental parameters are collected, including illumination intensity, wind speed, wind direction, temperature, humidity, and shadow coverage rate.

[0077] The environmental state model uses a Kalman filter algorithm to fuse the above multi-dimensional environmental parameters; the state vector of the Kalman filter contains four main components: light intensity, wind speed vector, temperature and humidity composite index, and shadow distribution density; through the two steps of state prediction and measurement update, the system eliminates sensor noise and random disturbances, generates an environmental adaptability index of 0.75 (value range 0-1, the higher the value, the more suitable the environment for flight) and a flight condition evaluation parameter of good level; the environmental adaptability index guides the environmental adaptation layer to determine the priority of the detection area, and the flight condition evaluation parameter provides the trajectory optimization constraint condition for the path planning layer.

[0078] The environmental adaptation layer calculates the environmental complexity index according to real-time environmental parameters; for example, the weighted calculation results of the three dimensions of light conditions, wind speed influence and shadow distribution show that the environmental complexity index is 0.68, which exceeds the preset threshold value of 0.6, triggering the task decomposition mechanism.

[0079] The detection task divides the 12-kilometer loop section into 18 detection units according to the continuity and integrity features of the road markings, each detection unit covering about 667 meters in length. The priority weight of each detection unit is determined according to the traffic importance, marking wear degree and detection difficulty coefficient. Specifically, the 6 detection units in the intersection area of the main road are marked as high priority, including the main entrance and exit markings, the area near the traffic signs and the road section where severe marking wear is found in historical detection; the 8 detection units of the secondary road connecting section are divided into medium priority; the 4 detection units of the auxiliary lane and the well-maintained section are set as low priority.

[0080] When the system detects that the power decreases from full power to 40% during flight, the priority allocation strategy automatically adjusts, increasing the weight of the high-priority area from the original 0.5 to 0.7, ensuring that the key area detection task is completed first, while postponing the detection task of the low-priority area to the next flight task.

[0081] Special reward functions are set for the three control levels respectively:

[0082] The area-level reward function determines the coverage integrity reward value by calculating the ratio of the detected area to the total target area. When the system completes the first 6 high-priority detection units, the coverage integrity reaches 33.3%, corresponding to a reward value of 0.33;

[0083] The track-level reward function considers flight path efficiency and energy consumption control. The actual flight path length planned is 15.8 kilometers, while the theoretical shortest path length is 14.2 kilometers, with a path length ratio of 1.11; combined with the energy consumption efficiency parameter (current energy consumption rate is 8% per kilometer), the trajectory efficiency reward value is calculated as 0.82, indicating that the flight trajectory is relatively efficient;

[0084] The pixel-level reward function evaluates image quality using the image gradient variance and the mean contrast. The marking image taken under strong light conditions has a gradient variance of 1847 and a mean contrast of 0.68, with a corresponding image quality reward of 0.85, indicating that the image quality meets the requirements for subsequent marking recognition.

[0085] The three reward functions are linearly weighted and fused using weight coefficients, with a weight of 0.4 for the region level, 0.35 for the track level, and 0.25 for the pixel level, to obtain the final comprehensive reward value.

[0086] The training process was conducted in three phases. The first phase trained the environment adaptation layer, fixed the parameters of the path planning layer and the attitude control layer, and used 1000 sets of simulated environmental data containing different lighting, wind speed, and shadow conditions to train the environment perception network. During the training process, the network learned to establish a mapping relationship between environmental parameters and detection difficulty, which took 200 training cycles.

[0087] The second stage trains the path planning layer, utilizing the environmental state information output from the environment adaptation layer to train the path generation network. The input includes the coordinates of all detection units, their priority weights, and environmental constraints. The network learns to generate the optimal flight trajectory sequence.

[0088] The third stage trains the attitude control layer. Based on the output information of the first two layers, the attitude control network is trained. The network receives the target position sequence, the desired flight speed and attitude angle constraints, and learns to generate accurate flight attitude parameters.

[0089] The image quality assessment model adopts a four-layer architecture to realize a complete quality control process; the environment perception layer acquires data such as light intensity, shadow coverage, wind speed and robot acceleration, calculates the environment complexity index through an environmental parameter fusion algorithm, and outputs the corresponding assessment strategy signal.

[0090] Based on the strategy signal under strong light conditions, the feature extraction layer extracts the average edge gradient magnitude calculated by the Sobel operator and the proportion of saturated pixels in the image; at the same time, it extracts the frequency domain features of image sharpness, the proportion of high-frequency component energy, as an auxiliary indicator to evaluate image sharpness.

[0091] The quality assessment layer employs an environment-adaptive weighting function to dynamically assign weights to different features. Under the current strong light conditions, the edge gradient feature weight is set to 0.6, the global contrast feature weight is reduced to 0.25, and the saturation pixel ratio feature weight is 0.15. Based on the weighted multidimensional feature vector, a comprehensive image quality score is calculated.

[0092] The decision fusion layer integrates the quality evaluation result, the current percentage of power, the task completion progress and the preset image quality lower threshold, and adopts a fuzzy logic controller for decision fusion. The fuzzy logic rule base includes control rules, covering combinations of different image qualities, power states and task progress.

[0093] A detailed mapping relationship between the light intensity and the feature effectiveness is established. Under the current strong light condition, the effectiveness coefficient of the edge gradient feature is obtained, the weight of the edge gradient feature is correspondingly increased, and the weight of the global contrast feature is reduced.

[0094] According to the shadow coverage rate, the feature fusion strategy is dynamically adjusted. Since the shadow coverage rate does not reach the threshold, a unified processing method for the whole image is adopted instead of a regional processing method. An experience model for weight adjustment is established based on the environmental data of historical flight tasks, and the optimal weight combination under different environmental conditions is predicted through a machine learning algorithm.

[0095] The environment adaptation layer transmits detailed state information to the path planning layer, including the environmental complexity index, the risk assessment matrix, the recommended flight window time period and the GPS coordinates of the priority detection area; at the same time, it feeds back the current environmental stability prediction as medium stability (the environmental condition change amplitude is less than 20% in the next 2 hours) and the abnormal event alarm information (no abnormality at present).

[0096] The path planning layer transmits the target position sequence containing time stamps to the attitude control layer, including the accurate GPS coordinates and the predicted arrival time of the detection unit, the expected flight speed vector, the vertical flight height, the attitude angle constraint range and the energy budget allocation.

[0097] The attitude control layer provides real-time flight state parameters to the environment perception layer of image quality evaluation, including the current flight height, pitch angle, roll angle, yaw angle, and root mean square value of vibration amplitude in X, Y and Z directions; receives the flight parameter adjustment output by the decision fusion layer.

[0098] An online incremental learning mechanism is adopted for continuous optimization. The real-time data acquisition module collects state-action-reward triple data at fixed time intervals, including environmental parameter vectors, control instruction sequences, image quality scores and task completion percentage. A sliding window filter uses a sliding window to denoise the data, effectively filtering out high-frequency noise and outliers. A circular buffer with a capacity limit of 1000 groups of data is constructed, and the history data is managed using the first-in-first-out strategy.

[0099] Hierarchical sampling and weight updating adopt differentiated strategies; the environment adaptation layer updates the parameters once after completing each detection area, and the first update is triggered after completing the detection unit in the current task; the path planning layer updates once after completing a complete track, ensuring the overall consistency of the path optimization strategy; the attitude control layer continuously updates in a moving average manner.

[0100] The weight update of each layer adopts an adaptive learning rate adjustment mechanism. The learning rate is dynamically adjusted according to the current task completion quality and environmental complexity.

[0101] An environment type classifier is established to classify the environment into 27 combined types according to the light conditions (strong light / normal / weak light), wind speed levels (gentle breeze / light breeze / medium wind) and task complexity (simple / medium / complex). The current environment is classified as the “strong light-light breeze-medium complexity” type. Independent subsets of network parameters are maintained for different environment types. The corresponding parameter subset is selected for updating in this task to avoid interference with parameters under other environmental conditions.

[0102] A cumulative mechanism is used to replace the traditional experience regression. The continuous gradients are weighted and accumulated, and then updated uniformly. The adaptive gradient clipping sets the maximum gradient norm to 1.0. The weight regularization adopts an L2 regularization term, and the regularization coefficient is 0.0001. The parameter update is set with upper and lower limit constraints. The single update amplitude does not exceed 10% of the current value of the parameter. When the system monitors that the control performance continuously decreases for 5 times of evaluation, it automatically rolls back to the parameter configuration of the previous stable version.

[0103] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. An aerial robot hierarchical reinforcement learning control method for pavement marking blur identification, characterized in that, The method comprises the following steps: Collecting environmental parameters and establishing an environmental state model, and dynamically decomposing a road marking detection task into three control levels according to the environmental parameters, namely an environment adaptation level, a path planning level and a posture control level; A regional level reward function, a track level reward function and a pixel level reward function are set for the three control levels, wherein the regional level reward function is used to evaluate the completeness of the target road section coverage, the track level reward function is used to optimize the flight trajectory efficiency, and the pixel level reward function is used to ensure image quality, and a layered reward mechanism is formed by weighted fusion; A top-down training strategy is adopted, the environment adaptation level is trained first to determine the optimal detection area division, then the path planning level is trained to generate a flight trajectory, and finally the posture control level is trained to realize position control; the layers interact through state information and reward signals; During flight, the image quality is evaluated, when the image blur and / or contrast do not meet the preset threshold, the re-shooting mechanism and / or the flight parameter adjustment are triggered, and the detection result is fed back to the layered reinforcement learning model for optimization and update; The image quality is evaluated by an image evaluation model; The image evaluation model comprises: An environment perception layer, which obtains the illumination intensity value, the shadow coverage percentage obtained by image analysis, the wind speed value and the aerial robot acceleration data, calculates the environmental complexity index through an environmental parameter fusion algorithm, and outputs an evaluation strategy signal; A feature extraction layer, which extracts the edge gradient amplitude calculated by the Sobel operator and the image saturated pixel ratio under strong light conditions, extracts the local binary pattern texture feature and the block contrast mean under shadow conditions, extracts the second derivative variance calculated by the Laplace operator and the global standard deviation under normal illumination, and extracts the frequency domain feature of image definition as an auxiliary index, and outputs a multi-dimensional feature vector; A quality evaluation layer, which dynamically allocates weights to different features based on the multi-dimensional feature vector using an environment adaptive weighting function, adjusts the weight coefficients in real time according to the current illumination intensity and shadow ratio, calculates the comprehensive image quality score and confidence index, and outputs the quality evaluation result; A decision fusion layer, which integrates the quality evaluation result of the quality evaluation layer, the current aerial robot power percentage, the task completion progress and the preset image quality lower threshold, uses a fuzzy logic controller for decision fusion to obtain a comprehensive score; when the comprehensive score is lower than the dynamically adjusted quality threshold and the remaining power is sufficient, a re-shooting instruction and flight parameter adjustment suggestion are output.

2. The aerial robot layered reinforcement learning control method according to claim 1, wherein: The environmental parameters include illumination intensity, wind speed, wind direction, temperature, humidity and shadow coverage; the environmental state model fuses the environmental parameters through a Kalman filtering algorithm to generate an environmental adaptability index and a flight condition evaluation parameter; The environmental adaptability index is used to guide the decision making of the environment adaptation level, and the flight condition evaluation parameter is used for trajectory optimization of the path planning level.

3. The aerial robot layered reinforcement learning control method according to claim 1, wherein: The dynamic decomposition process comprises: the environment adaptation layer calculates an environment complexity index according to real-time environment parameters, and when the environment complexity index exceeds a preset threshold of the environment, the detection task is decomposed into multiple subtasks and priorities are assigned; The path planning layer receives the subtask information and generates a flight path set in combination with the performance constraints of the aircraft; The attitude control layer calculates optimal flight attitude parameters according to the flight path set and image quality requirements.

4. The hierarchical reinforcement learning control method for an aerial robot according to claim 1, characterized in that: The area-level reward function determines the coverage integrity reward value by calculating the ratio of the detected area to the total area of the target region; The track-level reward function determines the track efficiency reward value by calculating the ratio of the flight path length to the theoretical shortest path length, in combination with the energy efficiency parameter; The pixel-level reward function determines the image quality reward value by calculating the image gradient variance and contrast mean, and the three reward functions are linearly weighted and fused through weight coefficients.

5. The hierarchical reinforcement learning control method for an aerial robot according to claim 1, characterized in that: The top-down training strategy specifically comprises: In the first stage, the environment adaptation layer is trained, the parameters of the path planning layer and the attitude control layer are fixed, and the environment perception network is trained through environment simulation data; In the second stage, the path planning layer is trained, and the path generation network is trained using the environment state information output by the environment adaptation layer; In the third stage, the attitude control layer is trained based on the output information of the previous two layers, and after the training of each layer is completed, joint fine-tuning is performed.

6. The hierarchical reinforcement learning control method for an aerial robot according to claim 1, characterized in that: The state information interaction between the layers comprises: The environment adaptation layer transmits the calculated environment complexity index, risk assessment matrix, recommended flight window time period, and priority detection region coordinates to the path planning layer, and simultaneously feeds back the current environment stability prediction and abnormal event warning information; the path planning layer transmits the target position sequence containing a time stamp, the expected flight speed vector, the attitude angle constraint range, and the energy consumption budget allocation to the attitude control layer, and simultaneously receives the execution state confirmation and trajectory tracking deviation data fed back by the attitude control layer; the attitude control layer provides the real-time flight height, attitude angle, vibration amplitude, and camera stability parameters to the image quality evaluation environment perception layer, and receives the flight parameter adjustment suggestions output by the decision fusion layer, including the height increment, angle correction value, and speed adjustment coefficient.

7. The hierarchical reinforcement learning control method for an aerial robot according to claim 1, characterized in that: The optimization and update process of the hierarchical reinforcement learning model comprises: State-action-reward data in the flight process are collected to construct an experience replay buffer; A priority sampling method is used to select training samples from the buffer; The time difference error is used to update the network parameters of each layer, and gradient clipping is used to prevent gradient explosion, and the model parameter update is performed once after completing a complete detection task.

Images