A Visual Algorithm Generation Method Based on Reinforcement Learning

Abstract

This invention discloses a visual algorithm generation method based on reinforcement learning, relating to the field of UAV control technology, including: constructing a joint state vector; generating composite action commands; executing algorithm state transitions; and executing communication state transitions. This invention solves the algorithm-communication resource mismatch problem under traditional static configuration through mechanisms such as joint modeling, dynamic decision-making, and human-machine collaboration. It can dynamically adjust strategies based on real-time channel quality, improving the mission resilience and efficiency of UAV swarms in complex environments, while also supporting expert knowledge injection to enhance system adaptability.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and more specifically to a visual algorithm generation method based on reinforcement learning. Background Technology

[0002] In UAV inspection missions, the onboard vision algorithms are typically configured statically before the mission begins. This fixed strategy cannot be dynamically adjusted according to the real-time changes in communication bandwidth during the mission. Once the channel quality drops sharply, the large amount of data generated by the highly complex algorithms will far exceed the link capacity, leading to the loss of critical information packets, loss of decision-making, and ultimately causing a decline in mission efficiency or even systemic risks.

[0003] Therefore, the new challenge facing current technology is: how to endow the system with true intelligence on an advanced hardware platform, enabling it to perceive its own and environmental status in real time, dynamically adjust its internal algorithms and communication strategies, and solve the problem of real-time matching of algorithms and communication resources; this requires a completely new dynamic decision-making mechanism to replace the traditional static configuration scheme, thereby systematically improving the task resilience and execution efficiency of intelligent equipment in changing and harsh environments. Summary of the Invention

[0004] The purpose of this invention is to provide a visual algorithm generation method based on reinforcement learning, which solves the problems existing in the background technology.

[0005] To address the aforementioned technical problems, this invention provides a visual algorithm generation method based on reinforcement learning, comprising: Step 1, constructing a joint state vector, wherein the joint state vector is configured to include: a visual algorithm complexity level for quantifying the resource consumption of the airborne visual algorithm, a communication link quality level for quantifying the communication channel carrying capacity between UAV nodes, and a communication protocol mode level for quantifying the data transmission strategy;

[0006] Step 2: Generate composite action instructions. The composite action instructions are generated by inputting the joint state vector into a policy generation model trained by reinforcement learning, and outputting the policy generation model. The composite action instructions contain algorithm complexity adjustment instructions and communication protocol mode adjustment instructions.

[0007] Step 3: Execute the algorithm state transition and set the visual algorithm complexity level of the next moment to be: the result of adjusting the visual algorithm complexity level of the current moment according to the algorithm complexity adjustment instruction;

[0008] Step four: Perform a communication state transition and set the communication protocol mode level for the next moment to be the result of adjusting the current communication protocol mode level according to the communication protocol mode adjustment instruction.

[0009] Preferably, in step one, the method for constructing the visual algorithm complexity level includes: pre-setting a visual algorithm library, which contains visual algorithms sorted from low to high resource consumption; setting the visual algorithm complexity level as a discrete scalar, which serves as an index to uniquely determine the visual algorithm implementation from the visual algorithm library.

[0010] Preferably, in step one, the method for constructing the communication link quality level includes: real-time monitoring of the effective communication bandwidth between UAV nodes; and converting the continuous values ​​of the effective communication bandwidth through a preset quantization mapping function to generate the communication link quality level.

[0011] Preferably, the joint state vector is further configured to include a task progress vector; the task progress vector is used to characterize the execution status of the inspection task, and the task progress vector includes the inspection area coverage and the number of targets discovered.

[0012] Preferably, in step two, the training process of the strategy generation model is further defined as follows: optimizing the internal parameters of the strategy generation model by maximizing the scenario-based reward function; the scenario-based reward function is constructed as a weighted combination, the weighted combination including: an inspection benefit item associated with the newly added coverage area, a discovery benefit item associated with the number of newly added effective targets detected, and a resource consumption cost item associated with the total energy consumption of the system.

[0013] Preferably, the calculation method of the resource consumption cost item in the scenario-based reward function includes: obtaining the computational resource consumption cost corresponding to the visual algorithm complexity level at the current time and the communication resource consumption cost corresponding to the communication protocol mode level at the current time; and weighted summing the computational resource consumption cost and the communication resource consumption cost to obtain the total system energy consumption.

[0014] Preferably, in step three, the execution method of the algorithm state transition includes: adding the algorithm complexity adjustment instruction to the visual algorithm complexity level at the current moment to obtain a temporary complexity level; comparing the temporary complexity level with a preset highest algorithm complexity level and a preset lowest algorithm complexity level, ensuring through amplitude limiting processing that the temporary complexity level does not exceed the boundary between the highest algorithm complexity level and the lowest algorithm complexity level, and using the processed result as the visual algorithm complexity level at the next moment.

[0015] Preferably, in step four, the execution method of the communication state transition includes: adding the communication protocol mode adjustment instruction to the current communication protocol mode level to obtain a temporary mode level; comparing the temporary mode level with a preset highest performance communication protocol level and a preset lowest robust communication protocol level, ensuring that it does not exceed the boundary between the highest performance communication protocol level and the lowest robust communication protocol level through amplitude limiting processing, and using the processed result as the communication protocol mode level for the next moment.

[0016] Preferably, the method further includes: after the strategy generation model is trained centrally in the cloud, it is solidified and distributed to each drone agent in the drone swarm; during task execution, each drone agent takes its own independent local observation data as input and generates its own composite action command locally through the solidified strategy generation model to achieve decentralized decision-making.

[0017] Beneficial effects

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. By constructing a joint state space for algorithm and communication, the complexity of the visual algorithm and multi-dimensional parameters such as the quality of the communication link are jointly modeled, providing complete state input for reinforcement learning. This solves the problem of independent modeling of algorithm and communication resources under static configuration, and realizes the upgrade from static optimization to dynamic adaptation.

[0020] 2. By utilizing multi-agent reinforcement learning to construct a dynamic policy generation engine, a joint control policy is generated based on the real-time environmental state. The algorithm complexity and communication protocol are dynamically adjusted to avoid data packet loss and decision-making asynchrony caused by channel fluctuations, thereby improving the system's adaptability under extreme conditions.

[0021] 3. Through the development of an interactive platform that supports digital twin monitoring and human-machine collaborative calibration, domain experts can adjust the reward function weights and inject prior knowledge to guide the strategy to converge to a target that meets the needs of agricultural production, thereby enhancing the system's practicality and flexibility. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0025] Example 1:

[0026] Please see Figure 1 This invention provides a reinforcement learning-based visual algorithm generation method, comprising: Step 1, constructing a joint state vector, wherein the joint state vector is configured to include: a visual algorithm complexity level for quantifying the resource consumption of the airborne visual algorithm, a communication link quality level for quantifying the communication channel carrying capacity between UAV nodes, and a communication protocol mode level for quantifying the data transmission strategy; Step 2, generating a composite action command, wherein the composite action command is generated by inputting the joint state vector into a policy generation model trained based on reinforcement learning, and outputting by the policy generation model, wherein the composite action command contains an algorithm complexity adjustment command and a communication protocol mode adjustment command; Step 3, executing an algorithm state transition, setting the visual algorithm complexity level at the next time step to: the result of adjusting the visual algorithm complexity level at the current time step according to the algorithm complexity adjustment command; Step 4, executing a communication state transition, setting the communication protocol mode level at the next time step to: the result of adjusting the communication protocol mode level at the current time step according to the communication protocol mode adjustment command;

[0027] This embodiment provides a visual algorithm generation method based on reinforcement learning. The construction of the joint state vector realizes a unified mathematical representation of the UAV's own computing power, external communication environment, and task execution strategy. This provides a complete state perception foundation for subsequent intelligent decision-making. The introduction of the strategy generation model maps the joint state vector into composite action commands. This model can autonomously generate the optimal joint control strategy based on the current state. By executing algorithm state transitions and communication state transitions, the system accurately transforms decision commands into actual adjustments to the airborne visual algorithm and communication protocol. This closed-loop control process solves the problem of mismatch between algorithm and communication resource requirements under static configuration, upgrading from static optimization to dynamic adaptation, and systematically improving the operational resilience and efficiency of UAV swarms in changing environments.

[0028] Example 2:

[0029] In step one, the construction method of the visual algorithm complexity level includes: pre-setting a visual algorithm library, which contains at least two visual algorithm implementations sorted by resource consumption from low to high; setting the visual algorithm complexity level as a discrete scalar, which serves as an index to uniquely determine a visual algorithm implementation from the visual algorithm library;

[0030] In step one, the method for constructing the communication link quality level includes: real-time monitoring of the effective communication bandwidth between UAV nodes; and converting the continuous values ​​of the effective communication bandwidth through a preset quantization mapping function to generate the communication link quality level.

[0031] This embodiment refines the construction method of the joint state vector based on Embodiment 1; the preset visual algorithm library provides a solid foundation for quantifying the complexity level of visual algorithms; this algorithm library was established through preliminary testing and includes visual algorithm implementations of various complexities, from basic color segmentation to deep neural networks; the visual algorithm complexity level is defined as a discrete scalar, which serves as an index to directly select a specific algorithm from the visual algorithm library; this design ensures a one-to-one correspondence between the abstract complexity level and the specific algorithm implementation, thus ensuring the executability of decision instructions;

[0032] Simultaneously, the effective communication bandwidth between UAV nodes is monitored in real time to obtain continuous physical quantities reflecting the current channel conditions. A preset quantization mapping function is used to process this continuous value. The design principle of this quantization mapping function is to discretize the continuously changing bandwidth value into a finite number of levels, each level representing a typical communication carrying capacity. Its conversion process is defined by the following formula:

[0033] ;

[0034] It represents the link quality level, and converts continuous bandwidth values ​​into discrete levels through quantization mapping, reflecting the communication carrying capacity;

[0035] This represents the quantization mapping function, which is used to discretize continuous bandwidth values ​​into a finite number of levels, making it easier for reinforcement learning models to process.

[0036] This represents the effective communication bandwidth between drone nodes, a physical quantity monitored in real time.

[0037] This approach transforms raw monitoring data from the physical world into standardized communication link quality levels that are understandable to reinforcement learning models. This enables agents to clearly perceive macroscopic changes in the communication environment and make adaptive decisions.

[0038] Example 3:

[0039] The joint state vector is further configured to include a task progress vector; the task progress vector is used to characterize the execution status of the inspection task, and the task progress vector includes an inspection area coverage rate and a number of targets discovered.

[0040] This embodiment expands upon Embodiment 1 by adding a task progress vector. The task progress vector includes two key indicators: inspection area coverage and the number of targets detected. This aims to provide task-level contextual information for decision-making. Through this design, the strategy generation model can consider not only resource matching status but also task completion when making decisions. For example, when coverage is low, the model might choose a strategy with lower energy consumption but higher coverage efficiency; when entering a dense target area, the model might switch to a high-precision recognition algorithm. The joint state vector after introducing the task progress vector... The composition is as follows:

[0041] ;

[0042] Represents the joint state vector at time t, integrating the algorithm, communication, and task states;

[0043] This represents the complexity level of a visual algorithm; it is a discrete scalar, with higher levels indicating higher algorithm accuracy but also greater computational load.

[0044] This indicates the link quality level, obtained by bandwidth quantization.

[0045] This represents the task progress vector, which includes task context information such as inspection coverage and number of targets discovered.

[0046] This design enables drones to make decisions with a more holistic perspective, better balance mission objectives and resource consumption, and achieve a higher level of intelligence.

[0047] Example 4:

[0048] In step two, the training process of the strategy generation model is further defined as follows: the internal parameters of the strategy generation model are optimized by maximizing a scenario-based reward function; the scenario-based reward function is constructed as a weighted combination, which includes: an inspection benefit item associated with the newly added coverage area, a discovery benefit item associated with the number of newly detected effective targets, and a resource consumption cost item associated with the total energy consumption of the system.

[0049] The calculation method for the resource consumption cost item in the scenario-based reward function includes: obtaining the computational resource consumption cost corresponding to the complexity level of the visual algorithm at the current time and the communication resource consumption cost corresponding to the communication protocol mode level at the current time; and weighted summing the computational resource consumption cost and the communication resource consumption cost to obtain the total energy consumption of the system.

[0050] This embodiment, based on Embodiments 1 and 3, defines in detail the training logic of the policy generation model; the model parameters are optimized by maximizing a scenario-based reward function; the design of this reward function directly reflects the core objective of the inspection task; it weights and combines inspection gains with resource consumption to guide the agent's learning process; time... Instant rewards Defined as:

[0051] ;

[0052] The immediate reward at time t is used to guide the optimization of the reinforcement learning strategy;

[0053] , , This represents the weighting coefficient, which adjusts the importance of each reward item.

[0054] This indicates the newly added coverage area and reflects the efficiency of the inspection.

[0055] This indicates the number of newly detected valid targets, reflecting the recognition effect;

[0056] This represents the total system energy consumption, including computing and communication energy consumption.

[0057] Indicates a time index;

[0058] The total system energy consumption in the reward function It is explicitly defined, rather than a black-box parameter; it is determined by the complexity level of the current vision algorithm. Corresponding computing resource consumption costs and communication protocol mode levels The corresponding communication resource consumption cost composition; these two cost functions Based on prior benchmark testing and calibration, the total system energy consumption is calculated as follows:

[0059] ;

[0060] This represents the total energy consumption of the system.

[0061] , The conversion factor is indicated, and the subscripts c and m represent computation and communication, respectively.

[0062] Represents the cost function;

[0063] Indicates the level of algorithm complexity;

[0064] Indicates the communication mode level;

[0065] This design links abstract energy consumption costs with each specific decision (choice) made by the agent. and This directly correlates the reward function with the actual energy consumption parameters, which solves the problem of the reward function being isolated from the actual energy consumption parameters in the original scheme. It ensures the effectiveness and closed-loop nature of the reward signal, thereby guiding the model to learn truly energy-efficient strategies.

[0066] Example 5:

[0067] In step three, the execution method of the algorithm state transition includes: adding the algorithm complexity adjustment instruction to the visual algorithm complexity level at the current moment to obtain a temporary complexity level; comparing the temporary complexity level with the preset highest algorithm complexity level and the preset lowest algorithm complexity level, ensuring that the temporary complexity level does not exceed the boundary between the highest algorithm complexity level and the lowest algorithm complexity level through amplitude limiting processing, and using the processed result as the visual algorithm complexity level at the next moment;

[0068] In step four, the execution method of the communication state transition includes: adding the communication protocol mode adjustment instruction to the current communication protocol mode level to obtain a temporary mode level; comparing the temporary mode level with a preset highest performance communication protocol level and a preset lowest robust communication protocol level, and ensuring that it does not exceed the boundary between the highest performance communication protocol level and the lowest robust communication protocol level through amplitude limiting processing, and using the processed result as the communication protocol mode level at the next moment;

[0069] This embodiment, based on embodiment 1, clarifies the execution mechanism of state transition; the algorithm complexity adjustment instructions output by the agent. Directly affects the complexity level of visual algorithms The state transition function is explicitly defined to ensure that decision instructions can accurately and predictably change the system state; the complexity level of the visual algorithm at the next time step. It is calculated using the following deterministic function:

[0070] ;

[0071] Indicates the level of algorithm complexity;

[0072] Indicates an algorithm adjustment instruction;

[0073] Indicates the highest level;

[0074] Indicates a time index;

[0075] Similarly, communication protocol mode adjustment instructions The execution method has also been clarified; the communication protocol mode level for the next moment. Determined by the following function:

[0076] ;

[0077] Indicates the communication mode level;

[0078] Display mode adjustment instructions;

[0079] Indicates the highest command;

[0080] Indicates a time index;

[0081] The introduction of these two state transition functions, through amplitude limiting, ensures that the algorithm complexity and communication mode are always within the preset effective range, preventing system crashes caused by improper instructions. This design provides a clear physical execution path and measurable results for previously isolated decision-making actions, constructing a complete, black-box-free control loop from decision to execution, ensuring the determinism and closed-loop nature of the entire control loop.

[0082] Example 6:

[0083] Further, it includes: after the strategy generation model is centrally trained in the cloud, it is solidified and distributed to each drone agent in the drone swarm; during the task execution, each drone agent takes its own independent local observation data as input and generates its own composite action command locally through the solidified strategy generation model to achieve decentralized decision-making.

[0084] This embodiment, based on Embodiment 1, introduces a decentralized decision-making execution framework; the strategy generation model First, centralized training is conducted in the cloud; after training, the solidified policy model is distributed to every drone agent in the swarm; during the task execution phase, each drone agent... As an independent decision-making unit, it relies solely on its own complete local observation data. Make decisions; local observation data contains the current visual algorithm complexity level of the agent. Communication link quality level Communication protocol mode level and task progress vector The process of generating decision actions is as follows:

[0085] ;

[0086] Indicates a compound action instruction;

[0087] Representational strategy model;

[0088] This represents local observation data;

[0089] Indicates the agent index;

[0090] Indicates a time index;

[0091] This compound action instruction Includes specific adjustment instructions This instruction is immediately transmitted to the execution layer of the airborne control system, where the state update is completed independently locally. This centralized training and decentralized execution architecture realizes a complete autonomous process from complete local observation to independent decision-making and then to closed-loop state update. As a result, the system has high scalability and robustness, and the mission execution does not depend on continuous central communication, which greatly improves the success rate and efficiency of UAV swarms in performing missions in complex and harsh environments.

[0092] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for generating a visual algorithm based on reinforcement learning, characterized in that, The method comprises the following steps: Step 1: constructing a joint state vector, which is configured to include: a visual algorithm complexity level for quantifying onboard visual algorithm resource consumption, a communication link quality level for quantifying unmanned aerial vehicle inter-node communication channel carrying capacity, and a communication protocol mode level for quantifying data transmission strategy; Step 2: generating a composite action instruction, which is generated by inputting the joint state vector into a policy generation model trained based on reinforcement learning, and outputting the composite action instruction from the policy generation model, wherein the composite action instruction contains an algorithm complexity adjustment instruction and a communication protocol mode adjustment instruction; Step 3: performing algorithm state transition, and setting the visual algorithm complexity level at the next time point as the result of adjusting the visual algorithm complexity level at the current time point according to the algorithm complexity adjustment instruction; Step 4: performing communication state transition, and setting the communication protocol mode level at the next time point as the result of adjusting the communication protocol mode level at the current time point according to the communication protocol mode adjustment instruction; In the step 1, the visual algorithm complexity level is constructed in the following manner: a preset visual algorithm library is included, the visual algorithm library contains visual algorithms sorted from low to high according to resource consumption; the visual algorithm complexity level is set as a discrete scalar, which is used as an index to uniquely determine a visual algorithm implementation from the visual algorithm library; In the step 1, the communication link quality level is constructed in the following manner: the effective communication bandwidth between unmanned aerial vehicle nodes is monitored in real time; the continuous value of the effective communication bandwidth is converted by a preset quantization mapping function to generate the communication link quality level; In the step 4, the communication state transition is performed in the following manner: the communication protocol mode adjustment instruction is added to the communication protocol mode level at the current time point to obtain a temporary mode level; the temporary mode level is compared with a preset highest performance communication protocol level and a preset lowest robustness communication protocol level, and the result after amplitude limiting processing is ensured not to exceed the boundary of the highest performance communication protocol level and the lowest robustness communication protocol level, and the processed result is used as the communication protocol mode level at the next time point.

2. The method of claim 1, wherein, The joint state vector is further configured to include a task progress vector; the task progress vector is used to represent the execution state of the inspection task, and the task progress vector includes an inspection area coverage rate and a target discovery number.

3. The method of claim 1, wherein, In the step 2, the training process of the policy generation model is further limited as follows: the internal parameters of the policy generation model are optimized by maximizing a scenario reward function; The scenario reward function is constructed as a weighted combination, which includes: an inspection income item associated with newly added coverage area, a discovery income item associated with newly added effective target detection number, and a resource consumption cost item associated with total system energy consumption.

4. The method of claim 3, wherein, The calculation manner of the resource consumption cost term in the scenario reward function comprises: acquiring a calculation resource consumption cost corresponding to the visual algorithm complexity level at the current moment and a communication resource consumption cost corresponding to the communication protocol mode level at the current moment respectively; and performing weighted summation on the calculation resource consumption cost and the communication resource consumption cost to obtain the system total energy consumption.

5. The method of claim 1, wherein, In the step three, the execution manner of the algorithm state transition comprises: adding the algorithm complexity adjustment instruction and the visual algorithm complexity level at the current moment to obtain a temporary complexity level; comparing the temporary complexity level with a preset highest algorithm complexity level and a preset lowest algorithm complexity level, performing limiting processing to ensure that the temporary complexity level does not cross the boundary of the highest algorithm complexity level and the lowest algorithm complexity level, and taking the processed result as the visual algorithm complexity level at the next moment.

6. The method of claim 1, wherein, Further comprising: After the cloud centralized training of the strategy generation model is completed, the strategy generation model is solidified and distributed to each unmanned aerial vehicle intelligent agent in the unmanned aerial vehicle swarm; During task execution, each unmanned aerial vehicle intelligent agent takes its own independent local observation data as input, independently generates its own composite action instruction locally through the solidified strategy generation model, and realizes decentralized decision-making.

Images