Adaptive control system of mobile intelligent sampling equipment based on reinforcement learning

Abstract

The present invention relates to the field of adaptive control technology, and in particular to an adaptive control system for a mobile intelligent sampling device based on reinforcement learning. The system comprises: a state and action construction unit, a reinforcement learning construction unit and an adaptive control unit; the state and action construction unit is used to perform superposition modeling on multiple concentration sources of a target substance to obtain a concentration field of the target substance, and calculate the concentration field gradient, obtain the real-time state of the device, and use the real-time state, concentration field and concentration field gradient of the device as elements in the state space to construct the state; define the action; the reinforcement learning construction unit is used to construct a reinforcement learning reward function according to the state and action to generate a reinforcement learning model; the adaptive control unit is used to perform strategy optimization, generate a control trajectory and evaluate the sampling quality according to the reinforcement learning model. The present invention realizes autonomous decision-making and efficient sampling of the device in a dynamic concentration field.

Description

Technical Field

[0001] The present invention belongs to the technical field of adaptive control, and in particular relates to an adaptive control system of a movable intelligent sampling device based on reinforcement learning. Background Art

[0002] In the fields of industrial testing, environmental monitoring and medicine, accurate sampling of target substances is crucial for applications such as data analysis, environmental governance and disease diagnosis. With the development of Internet of Things technology, sensor technology and intelligent control systems, mobile intelligent sampling equipment has received widespread attention. By combining sensor networks with automatic control technology, this type of equipment can achieve dynamic sampling of target substances in complex environments. However, the existing technology still has many shortcomings in terms of equipment adaptability, intelligence level and sampling quality, which is particularly evident in concentration fields with dynamic changes or complex distributions.

[0003] At present, the research on intelligent sampling equipment mainly focuses on path planning, environmental perception and control optimization. Typical technical solutions include rule-based path planning, model predictive control, and sampling strategy optimization based on sensor fusion. Rule-based path planning technology guides the movement of equipment through preset paths or rules. For example, the equipment evenly distributes samples in the target area according to a fixed grid scanning mode. This method is simple and easy to implement, and is suitable for scenes where the target material is relatively uniformly distributed in a static environment. However, when the concentration field distribution has large spatial differences or dynamic changes, fixed path sampling may cause some high-concentration areas to be not effectively covered, while increasing invalid sampling in low-concentration areas, thereby reducing sampling efficiency and quality. The optimization method based on model predictive control uses a mathematical model to predict the future motion state of the equipment and generates real-time control instructions through an optimization algorithm. This method can dynamically adjust the motion path of the equipment to adapt to a certain degree of environmental changes. However, MPC relies heavily on the accuracy of the environmental model. When the distribution of target materials in the environment has strong randomness or nonlinear changes, the accuracy of the model prediction decreases, resulting in the inability of the equipment to cope with complex scenes in real time. In addition, the computational complexity of MPC is high, and it may be limited by hardware performance in scenes with high real-time requirements. With the advancement of sensor technology, multi-sensor fusion has gradually become an important research direction for intelligent sampling equipment. By integrating concentration sensors, visual sensors, and inertial sensors, the equipment can achieve more accurate environmental perception in the concentration field. However, this approach still relies on preset rules or heuristic algorithms for motion control and lacks the ability to actively optimize. In addition, sensor errors and data delays may cause deviations in the device sampling path, affecting the sampling quality. Summary of the invention

[0004] The main purpose of the present invention is to provide an adaptive control system for a mobile intelligent sampling device based on reinforcement learning. Through state and action modeling, reward function design, reinforcement learning strategy optimization and value function evaluation, the device can make autonomous decisions and perform efficient sampling in a dynamic concentration field. The system combines real-time environmental perception with multi-objective optimization strategies, and can dynamically adjust the motion path, give priority to covering high-concentration areas, and effectively control energy consumption and task completion time.

[0005] The technical solution of the present invention is achieved in this way:

[0006] A reinforcement learning-based adaptive control system for a mobile intelligent sampling device, the system comprising: a state and action construction unit, a reinforcement learning construction unit and an adaptive control unit; the state and action construction unit is used to perform superposition modeling on multiple concentration sources of a target substance to obtain a concentration field of the target substance, calculate the concentration field gradient, obtain the real-time state of the device, and use the real-time state, concentration field and concentration field gradient of the device as elements in the state space to construct the state; define an action, whose elements include an instruction linear velocity vector and an instruction angular velocity vector; the reinforcement learning construction unit is used to construct a reinforcement learning reward function according to the state and action, and design a reinforcement learning policy network and value function to generate a reinforcement learning model; the adaptive control unit is used to perform policy optimization according to the reinforcement learning model, generate a control trajectory, and evaluate the sampling quality.

[0007] Furthermore, the state and action construction unit obtains the intensity of each concentration source through the sensor, and sets the diffusion coefficient of each concentration source based on the intensity of each concentration source; assuming that the target substance distribution is formed by the superposition of multiple concentration sources, and its concentration field The distribution in space is represented by the Gaussian distribution function:

[0008] ;

[0009] in, For equipment The location of time, For the The location of the concentration source; For the The intensity of a concentration source; For the The diffusion coefficient of a concentration source is a set value. The higher the intensity of the concentration source, the larger the corresponding diffusion coefficient value. It is the modulus length operation of the vector; is the number of concentration sources; is an integer subscript index; the gradient of the concentration field Calculated using the following formula:

[0010] .

[0011] Furthermore, the real-time state of the device includes: position, attitude rotation matrix and speed; the state is expressed using the following formula:

[0012] ;

[0013] in, For equipment location in time; For equipment The speed of time; motion is defined as ;in, is the command linear velocity vector; is the command angular velocity vector.

[0014] Furthermore, the reward function Use the following formula to express it:

[0015] ;

[0016] in, , which indicates the degree of alignment between the speed of the device and the concentration field gradient. The closer the value is to 1, the more efficient the direction of movement of the device is; , is the acceleration of the device; , is the angular acceleration of the device; is the maximum allowed task time.

[0017] Furthermore, the policy network of reinforcement learning is expressed using the following formula:

[0018] ;

[0019] in, For the status Next select action The probability of is the action mean, indicating the center of the optimal action; is the action covariance matrix, which represents the uncertainty of the action; is the determinant of the action covariance matrix, used for normalization; is the action dimension; action mean Generated by a 3-layer neural network, the formula is as follows:

[0020] ;

[0021] in, , and are the weight matrices of the first, second, and third layers of the neural network respectively; , and The bias matrices of the first, second, and third layers of the neural network respectively; is the activation function; is the hyperbolic tangent function; the action covariance matrix The diagonal elements of are output by the network, and the positive definiteness is guaranteed by the exponential function, as follows:

[0022] ;

[0023] in, is the logarithm of the covariance diagonal elements generated by the neural network; is an operation that generates a diagonal matrix.

[0024] Furthermore, the value function Use the following formula to express it:

[0025] ;

[0026] in, Indicates time Status at the time; is the standard deviation of the Gaussian distribution; is the time-integrated variable; is the characteristic matrix of the value function; is the first weight parameter, is the set value; is the second weight parameter, is the set value; is the square of the F norm; For time The reward function when ; For time The gradient of the value function corresponding to ;

[0027] ;

[0028] is the gradient operator.

[0029] Furthermore, the following formula is used for strategy optimization:

[0030] ;

[0031] in, Optimize values ​​for the strategy; To adopt the The action corresponding to each track; To adopt the The state corresponding to the trajectory; To adopt the The reward corresponding to each track; To adopt the When the track is Next select action The probability of To adopt the The value function of the strategy when the strategy is optimized is used; by continuously calculating the strategy optimization value corresponding to each trajectory, the action corresponding to each time when the strategy optimization value is the largest is found, and the actions corresponding to each time when the strategy optimization value is the largest are arranged in chronological order as the elements in the optimal action sequence.

[0032] Furthermore, generating a control trajectory according to the optimal action sequence specifically includes: at each time, executing the corresponding action in the optimal action sequence to generate the control trajectory.

[0033] Furthermore, the sampling quality is evaluated by the following formula:

[0034] ;

[0035] in, It is the quality assessment value. The larger the value is, the higher the sampling quality is.

[0036] The adaptive control system of the mobile intelligent sampling device based on reinforcement learning of the present invention has the following beneficial effects:

[0037] The present invention combines concentration field modeling with gradient perception to achieve dynamic perception of the distribution of target substances by intelligent devices. Using the Gaussian distribution model and real-time gradient calculation, the device can identify high-concentration areas in the concentration field and give priority to sampling, thereby improving the acquisition rate of high-value data. Compared with the traditional fixed path planning method, the reinforcement learning-based policy network of the present invention can dynamically adjust the movement path of the device, making the sampling behavior more targeted and avoiding invalid sampling in low-concentration areas. This optimization strategy significantly improves the sampling efficiency while ensuring that the sampling data is highly representative of environmental characteristics. The concentration field distribution in complex environments is usually affected by dynamic changes, such as diffusion effects, flow disturbances, or time-varying concentration source intensity. The existing technology shows low adaptability when dealing with these complex scenarios, while the present invention significantly improves the environmental adaptability of the device through real-time state perception and the adaptability of the reinforcement learning model.

[0038] Specifically, the device can evaluate the current state and future benefits through reward functions and value functions, and adjust the motion trajectory according to the real-time changes of the concentration field. Regardless of whether the concentration field distribution is static or dynamic, the present invention can optimize the sampling strategy so that the device is always in an efficient operating state. Energy consumption is an important constraint condition when smart devices perform sampling tasks. The present invention incorporates sampling quality and energy consumption into the same optimization framework through reward function design, effectively reducing device energy consumption while ensuring sampling quality. Specifically, the reward function combines multiple trade-offs of concentration value, gradient direction alignment, and action energy consumption. By constraining the acceleration and angular acceleration of the device, invalid speed or direction changes are reduced, thereby reducing energy consumption. In addition, the policy network can also plan the shortest path according to the distribution characteristics of the concentration field, and further optimize the ratio of sampling efficiency to energy consumption. This multi-objective optimization design enables the device to complete more efficient sampling tasks under limited energy resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the system structure of a reinforcement learning-based adaptive control system for a mobile intelligent sampling device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In the following description, specific details such as specific system structures, interfaces, and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the embodiments of the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the embodiments of the present invention.

[0041] Example 1, reference Figure 1 : An adaptive control system for a mobile intelligent sampling device based on reinforcement learning, the system comprising: a state and action construction unit, a reinforcement learning construction unit and an adaptive control unit; the state and action construction unit is used to superimpose modeling on multiple concentration sources of a target substance to obtain the concentration field of the target substance, and calculate the concentration field gradient, obtain the real-time state of the device, and use the real-time state of the device, the concentration field and the concentration field gradient as elements in the state space to construct the state; define the action, whose elements include the command linear velocity vector and the command angular velocity vector; the reinforcement learning construction unit is used to construct a reward function of reinforcement learning according to the state and action, and design a policy network and value function of reinforcement learning to generate a reinforcement learning model; the adaptive control unit is used to optimize the policy according to the reinforcement learning model, generate a control trajectory, and evaluate the sampling quality.

[0042] Specifically, in the state construction of the present invention, the real-time state of the device, the concentration field and the concentration gradient together constitute a highly integrated state space, in which each element plays an indispensable role in the decision-making mechanism of the system. The real-time state of the device includes its position information and current motion parameters, which provide the system with information on the relative relationship between the device and the environment. The concentration field and the concentration gradient respectively describe the concentration value of the target substance at the location of the device and the directional information of the concentration change. The absolute value of the concentration field guides the device to approach the high concentration area, while the concentration gradient provides the device with a basis for dynamically adjusting the direction. In the design of this state space, not only the environmental characteristics of the device are fully reflected, but also the dynamics are introduced into the state modeling process, so that the control and decision of the device can be updated in real time as the environment changes. Different from the state modeling of the traditional system, the present invention constructs a state space with more environmental adaptability and complexity processing capabilities by combining the concentration gradient and the dynamic parameters of the device. The advantage of this design is that it not only focuses on the real-time data that the device can obtain, but also embeds the structural characteristics of the environmental information in the state modeling. Through the introduction of gradient information, the device can more efficiently select the moving direction and path in the concentration field, avoiding the local optimal trap that may appear in traditional rules or heuristic methods. In addition, the focus on the real-time motion parameters of the device in the state space also provides the possibility for subsequent reinforcement learning algorithms to construct optimization strategies in multi-dimensional space. In terms of action construction, the present invention innovatively defines the behavior of the device in the concentration field, and defines the action as a combination of the command linear velocity vector and the command angular velocity vector. Such a design conforms to the physical characteristics of the device moving in three-dimensional space, while providing a more free decision-making space for the flexible movement of the device. The linear velocity vector not only controls the speed of the device's movement, but also combines with the concentration gradient to determine the direction of movement of the device, thereby achieving efficient movement of the device in the concentration change field. The angular velocity vector provides the device with the ability to rotate and adjust, enabling it to quickly correct its direction to cope with sudden changes in the environment. Such an action design not only ensures the flexibility of the device, but also enhances the system's ability to adapt to the environment.

[0043] In the previous state and action construction unit, the concentration field, concentration gradient and device state have been efficiently modeled and formed input information. On this basis, the reinforcement learning unit directly links the sampling behavior of the device with the goal of the sampling task by designing a special reward mechanism, thereby giving the device the ability to perceive the task goal. This reward mechanism is the driving core of the entire reinforcement learning unit. Its role is not only to encourage the device to choose behaviors that can improve the sampling quality, but also to guide the device to gradually learn the optimal movement strategy in multiple trials through dynamic feedback of the reward value. For example, the reward mechanism can evaluate the device based on comprehensive indicators such as the concentration field coverage, sampling point uniformity and device energy consumption obtained during the sampling process, thereby giving positive or negative feedback to the device's behavior. This design enables the device to autonomously weigh the relationship between sampling efficiency and energy consumption in a complex environment, and ultimately achieve global optimization. In order to achieve the goal of reinforcement learning, the policy network and value function are designed as the core structure of the reinforcement learning unit. The role of the policy network is to generate specific action decisions based on the input state. It maps the input of a complex environment to the optimal behavior output of the device in that environment by modeling the concentration field characteristics and the device's motion ability. The design of the policy network fully considers the continuity and dynamic adjustment ability of the device action, so that the device can achieve flexible and efficient movement under different concentration field conditions. At the same time, the output of the policy network not only depends on the current state, but also uses historical experience to correct its strategy through deep learning, so as to gradually approach the optimal solution. The design of the value function is to evaluate the long-term benefits of the current state and its corresponding action, and assist the policy network to make a comprehensive judgment on the pros and cons of the current action by predicting the cumulative rewards that the device may obtain in the future sampling process. This prediction-based mechanism enables the device to not only pay attention to the local benefits of the current sampling point, but also optimize its behavior from a global perspective to avoid falling into the local optimal solution. In the training process of the reinforcement learning model, the present invention adopts a cyclic iteration method to continuously correct the parameters of the policy network and the value function through multiple test behaviors of the device in the concentration field. This process relies on the interactive feedback mechanism in reinforcement learning, that is, after the device performs a certain action, it obtains a reward signal based on the impact of its behavior on the task goal, and transmits this signal to the reinforcement learning model for parameter adjustment. Through this reinforcement learning method, the device gradually accumulates experience in multiple experiments and can generate the optimal sampling strategy under different concentration field conditions. It is worth mentioning that the present invention introduces the consideration of environmental randomness in the model training of reinforcement learning. This design enables the device to cope with the environmental uncertainties that may exist in practical applications, thereby improving the robustness and practicality of the system.

[0044] The adaptive control unit first obtains the policy output from the reinforcement learning model. These outputs are usually expressed in the form of actions, describing the optimal movement mode and direction of the device in the current state. However, the actual device control does not directly execute these outputs, but needs to convert them into specific physical control signals to drive the sampling device to complete the adjustment of linear velocity and angular velocity. Therefore, the adaptive control unit needs a policy conversion mechanism to interpret the abstract strategy in reinforcement learning into operable control parameters. This conversion process fully considers the kinematic constraints and dynamic characteristics of the device, such as the maximum acceleration, turning radius and motion smoothness of the device, to ensure that the generated control instructions are consistent with the theoretical optimization results and can be reliably executed in practice. This conversion mechanism enhances the operability of the system, so that the abstract results of reinforcement learning can be seamlessly connected to the actual needs of physical devices. In the generation of control trajectories, the adaptive control unit generates the movement path of the device in three-dimensional space by real-time calculation of the policy output. These paths are not simple straight lines or fixed trajectories, but the results of dynamic optimization, which can continuously adjust the direction and speed according to the gradient distribution of the concentration field and the real-time state of the device. During the movement of the device, the trajectory generator will dynamically calculate the optimal movement direction for the next step by combining the concentration value at the target position and the concentration gradient of the environment. Through this dynamic planning method, the device can quickly approach the target concentration area while avoiding redundant paths and unnecessary energy consumption. The trajectory generation capability of the adaptive control unit makes the device flexible and efficient in complex environments, and can quickly adjust the direction of movement according to real-time environmental feedback to ensure the accuracy and coverage of sampling. In addition, the adaptive control unit is also embedded with a sampling quality evaluation mechanism to monitor the sampling effect of the device in real time and dynamically correct the control strategy. This evaluation mechanism analyzes the concentration data collected by the device to calculate indicators such as the uniformity of the sampling points, the concentration coverage, and the representativeness of the sampling area. These evaluation results not only serve as feedback for the current control strategy of the device, but also directly affect the planning of subsequent control trajectories. For example, when the device finds that the sampling quality of a certain area is lower than expected, the adaptive control unit will recalculate the optimal path through the reinforcement learning model and guide the device to the insufficiently sampled area. This dynamic adjustment mechanism based on real-time feedback enables the device to flexibly respond to emergencies during the sampling process, such as changes in the concentration field, abnormal device status, or environmental interference. Through this mechanism, the adaptive control unit not only achieves efficient control of the device, but also makes up for the decision-making deviation that may occur in the reinforcement learning model under extreme circumstances. Another core function of the adaptive control unit is the ability to optimize policies. When the reinforcement learning model provides the basic policies, the adaptive control unit further optimizes these policies to meet the requirements of the dynamic environment by analyzing the real-time sampling data and environmental feedback. This optimization is not limited to fine-tuning the current state, but also includes re-planning the global sampling strategy.For example, when the device moves to a certain area, the concentration gradient may change dramatically. The adaptive control unit can quickly capture these changes and adjust the direction of movement to ensure that the device always moves in the direction of the largest concentration change to improve sampling efficiency. This capability makes the device more adaptable in dynamic scenes and can effectively deal with uncertainties and sudden changes in the concentration field.

[0045] Embodiment 2: The state and action construction unit obtains the intensity of each concentration source through a sensor, and sets the diffusion coefficient of each concentration source based on the intensity of each concentration source; assuming that the target substance distribution is formed by the superposition of multiple concentration sources, and its concentration field The distribution in space is represented by the Gaussian distribution function:

[0046] ;

[0047] in, For equipment The location of time, For the The location of the concentration source; For the The intensity of a concentration source; For the The diffusion coefficient of a concentration source is a set value. The higher the intensity of the concentration source, the larger the corresponding diffusion coefficient value. It is the modulus length operation of the vector; is the number of concentration sources; is an integer subscript index; the gradient of the concentration field Calculated using the following formula:

[0048] .

[0049] Specifically, the current position of the device is The location of the concentration source Combined, through the square of the distance Achieve a mathematical expression for the distance attenuation effect. This distance attenuation effect is consistent with physical reality, that is, the concentration of the target substance will show a decreasing trend as the distance from the source increases, and the exponential form of the Gaussian function ensures that this attenuation is smooth and continuous, thus avoiding abrupt changes that may occur in modeling. Concentration field strength It is introduced into the formula as the weight factor of each concentration source's influence on the overall field. This design ensures that the influence of concentration sources of different intensities on the equipment is accurately reflected in the model. At the same time, the diffusion coefficient of the concentration source The size of its influence range is controlled. The larger the diffusion coefficient, the smoother the distribution of the concentration field in space and the wider the range, while a smaller diffusion coefficient leads to a more concentrated distribution of the concentration field. This design allows the modeling of the concentration field to not only characterize the spatial distribution of the target substance, but also be refined to reflect the diffusion characteristics of different concentration sources, so that the device can flexibly adjust its behavior based on different scene characteristics. In the gradient calculation formula, the partial derivative of the concentration field distribution is precisely solved in a form related to the spatial variation of the Gaussian function. The core idea of ​​this formula is to determine the optimal direction of movement in the current environment by calculating the sensitivity of the device's current position to the concentration field distribution. The gradient term in the formula Directly pointing to the direction of the concentration source, its length is determined by the distance and the diffusion coefficient. The contribution of a concentration source with a larger intensity to the gradient is more significant, while the influence of a concentration source with a longer distance or a larger diffusion on the local gradient is naturally suppressed. This design makes the behavior of the device dominated by the nearby concentration source, while still being able to perceive the weak gradient generated by the distant concentration source, thereby avoiding the overall sampling efficiency reduction caused by the local optimum. The dynamic mechanism of obtaining the concentration source intensity and setting the diffusion coefficient through the sensor in the formula is an innovative embodiment of the real-time and adaptability of the present invention. When the device senses that the intensity of a concentration source is higher, the system will automatically assign a larger diffusion coefficient to it, so that the concentration field has a wider distribution range for the source, thereby guiding the device to cover important areas faster. For a concentration source with a lower intensity, the diffusion coefficient is smaller and the distribution of the concentration field is more concentrated. This dynamic adjustment mechanism ensures that the difference in the strength of the concentration source is fully reflected in the model, so that the motion strategy of the device can flexibly adapt to the sampling needs in different scenarios. These formulas play a key role in the state and action modeling of reinforcement learning. The value of the concentration field is not only directly used as part of the state space, but also guides the next action of the device through the result of the gradient calculation. The device moves along the gradient direction in the current state, which means that it always tends to the direction where the concentration increases fastest. This behavior mechanism of rising along the gradient effectively improves the sampling efficiency of the device in complex scenarios. At the same time, the distribution form and gradient characteristics of the concentration field also provide a basic basis for the design of reinforcement learning reward functions. For example, a device may receive a higher reward when moving in an area with a larger gradient to encourage it to prioritize exploring areas with high concentration changes. This design, which combines mathematical modeling with reinforcement learning strategies, enables the device to not only sample efficiently in a dynamic environment, but also further optimize its behavior through multiple rounds of training.

[0050] Embodiment 3: The real-time state of the device includes: position, attitude rotation matrix and speed; the state is expressed using the following formula:

[0051] ;

[0052] in, For equipment location in time; For equipment The speed of time; motion is defined as ;in, is the command linear velocity vector; is the command angular velocity vector.

[0053] Specifically, the real-time status of the device is expressed as a multidimensional state vector Indicates that the device is The key information at the moment. The first part of the state vector Describes the spatial position of the device and is the core coordinate reference point of the device in the concentration field. The position not only determines the specific concentration value of the device , and also calculate the gradient This position correlation makes the motion planning of the device closely related to the concentration field environment. is the device's attitude rotation matrix, which is used to describe the device's orientation and attitude in three-dimensional space. This matrix is ​​defined by a set of orthogonal unit vectors, which can relate the device's local coordinate system to the global coordinate system. In the concentration field, the device's attitude directly affects the execution of its actions, such as the direction of linear velocity and angular velocity. Therefore, by introducing the rotation matrix in the state vector , the device can achieve more precise motion control in three-dimensional space. is the real-time velocity vector of the device, indicating the motion trend of the device in the current state. This component not only reflects the actual motion state of the device, but also restricts the future actions of the device. For example, when the speed of the device in a certain direction is too large, its subsequent angular velocity adjustment range will be limited. Incorporating velocity information into the state vector helps the reinforcement learning model to fully consider the dynamic characteristics of the device when optimizing the strategy. The fourth and fifth parts of the state vector are the concentration values, and gradient . Concentration value Indicates the concentration of the target substance at the current location of the device. This information directly reflects the immediate effect of the device sampling. It provides information about the direction and rate of concentration change in the device's current environment, which is an important basis for device decision-making. By combining concentration value and gradient information, the device can efficiently select the next moving direction in the concentration field to ensure maximum sampling quality. The design is to achieve the specific behavior of the device in the actual scene. The action consists of two core components: the command linear velocity vector and the command angular velocity vector . Instruction Linear Speed Determines the linear movement speed of the device, and its size and direction are determined by the concentration gradient and reinforcement learning strategy. Command angular velocity Control the rotational movement of the device so that the device can flexibly adjust its direction to adapt to changes in the concentration field. The definition of the action vector fully considers the physical motion characteristics of the device, so that the system can directly output control instructions that meet the device's motion capabilities when generating strategies. Through the above-mentioned definitions of states and actions, the present invention realizes effective modeling of devices and environments in the reinforcement learning framework. The state vector not only fully reflects the real-time state of the device, but also incorporates environmental characteristics through concentration values ​​and gradient information, providing high-dimensional input for the optimization of reinforcement learning strategies. The action vector provides precise operating instructions for strategy execution by clarifying the linear velocity and angular velocity of the device. This state and action design is still innovative in traditional methods. It closely combines the physical characteristics, motion capabilities and concentration field environment of the device to form a complete state-action closed loop.

[0054] Example 4: Reward Function Use the following formula to express it:

[0055] ;

[0056] in, , which indicates the degree of alignment between the speed of the device and the concentration field gradient. The closer the value is to 1, the more efficient the direction of movement of the device is; , is the acceleration of the device; , is the angular acceleration of the device; is the maximum allowed task time.

[0057] Specifically, the first part of the reward function Used to evaluate the sampling quality of the device at its current location. Indicates the concentration value of the current sampling point of the device. The higher the concentration, the greater the reward, which reflects the system's priority attention to high-concentration areas. The square of the concentration gradient is introduced as a penalty term. The larger the gradient, the more the reward is appropriately reduced. The physical significance of this design is that in areas with large gradients, the concentration field changes quickly, and the device may need more time and energy to adjust the direction of movement to adapt to the environment. Therefore, the reward for high-gradient areas needs to be appropriately reduced, thereby guiding the device to prioritize sampling in areas with high concentrations and relatively gentle gradients. Part II The alignment of the device's current direction of motion with the concentration gradient is evaluated. This part quantifies the device velocity vector using a cosine function. and concentration gradient The closer the value is to 1, the more consistent the direction of the device's movement is with the gradient. This design directly encourages the device to move in the direction of the rising concentration gradient, so that it can quickly approach the area with higher concentration. This not only improves the efficiency of sampling, but also optimizes the path planning of the device to avoid invalid movement. Part III is a penalty term for the energy consumption and smoothness of the device motion. Here, and Represent the acceleration and angular acceleration of the device, respectively, and their squares reflect the dynamic change amplitude of the device in motion. The introduction of the penalty term limits the excessive acceleration or violent turning behavior of the device, thereby reducing energy consumption and improving the stability of motion. This design is of great significance for extending the operating time of the device and improving the reliability of actual operation. The last part It is the limiting factor of the task time. As time goes by, the value of this term gradually decreases, which has a negative impact on the reward of the device. The introduction of this linear time penalty term effectively avoids the meaningless delay of the device in the task and prompts it to complete the sampling task within a limited time. , the system can ensure that the device completes the set goals before the task times out, thereby improving operational efficiency. Overall, this reward function fully considers the complex task requirements of the device in a dynamic environment by balancing multiple factors. First, sampling quality is the core goal of the device, and the device is guided to sample efficiently through the combination of concentration value and gradient; secondly, motion efficiency is effectively controlled through the reward term of alignment degree and the penalty term of acceleration; finally, the task time constraint factor ensures the global efficiency of the system. This multi-objective optimization reward design not only reflects the comprehensiveness of the device behavior, but also makes full use of the superiority of the reinforcement learning algorithm, enabling the device to dynamically adjust its behavior in a complex environment, achieve real-time adaptation to the environment and efficient completion of task goals.

[0058] Example 5: The policy network of reinforcement learning is expressed using the following formula:

[0059] ;

[0060] in, For the status Next select action The probability of is the action mean, indicating the center of the optimal action; is the action covariance matrix, which represents the uncertainty of the action; is the determinant of the action covariance matrix, used for normalization; is the action dimension; action mean Generated by a 3-layer neural network, the formula is as follows:

[0061] ;

[0062] in, , and are the weight matrices of the first, second, and third layers of the neural network respectively; , and The bias matrices of the first, second, and third layers of the neural network respectively; is the activation function; is the hyperbolic tangent function; the action covariance matrix The diagonal elements of are output by the network, and the positive definiteness is guaranteed by the exponential function, as follows:

[0063] ;

[0064] in, is the logarithm of the covariance diagonal elements generated by the neural network; is an operation that generates a diagonal matrix.

[0065] Specifically, the core formula of the policy network is Describes the state of the device Next select action The probability density of . This formula is based on a multidimensional Gaussian distribution, and the probability of an action depends on the action mean and the covariance matrix The formal expression of Gaussian distribution can capture the uncertainty of actions, focusing on the vicinity of the optimal action while allowing a certain range of exploration. This design is particularly important in reinforcement learning because it can balance the exploration and exploitation of strategies, ensuring that the device can learn existing good behaviors while trying better strategies in new environments. In the expression of Gaussian distribution, Represents the mean of the action distribution, that is, the center of the optimal action. This mean is generated by a three-layer neural network, and the input of the network is the current state of the device . Through the network weights and bias The neural network is able to extract complex features from the state space and map them to the optimal action mean. Specifically, the first layer of the network is activated by the hyperbolic tangent function. The nonlinear features of the state are extracted, the second layer further digs into the implicit patterns in the high-dimensional state space through another set of nonlinear transformations, and the last layer generates the action mean through linear mapping. This deep structure gives the policy network a powerful expressive ability, enabling it to adapt to complex and dynamic concentration field environments. On the other hand, the action covariance matrix The design further enhances the policy network's ability to model environmental uncertainty. The diagonal elements of the covariance matrix are output by another set of neural networks, and these diagonal elements are transformed by exponential functions to ensure the positive definiteness of the matrix. This design not only ensures the stability of the mathematical model, but also allows the policy network to dynamically adjust the variance of the action distribution to adapt to the exploration needs under different states. When the device is in a state of high uncertainty, the diagonal elements of the covariance matrix will increase, thereby increasing the diffusion of the action distribution and encouraging the device to explore new behavior patterns over a larger range; in a more certain state, the diagonal elements of the covariance matrix will decrease, prompting the device to perform more precise optimal actions. The output of the policy network is the probability distribution of the action, and its mathematical expression contains several important normalization factors. For example, the denominator Through the determinant of the covariance matrix Normalization is performed to ensure that the generated probability density function meets the requirement of integral 1. The numerator is weighted by the Euclidean distance between the action and the mean and the inverse of the covariance matrix, which reflects the state of the device. Next select action The square weight of this distance reflects the degree of deviation of the action. The greater the deviation from the optimal mean, the lower the probability density. Through the design of the above policy network, the reinforcement learning model can generate the action decision of the device in the form of probability, so that the device can efficiently weigh short-term and long-term benefits in a complex concentration field environment and avoid falling into a local optimal solution. For example, when the device needs to explore the best path in a high-gradient area, the policy network can dynamically adjust the action distribution and give the device appropriate randomness to try new directions of movement; in areas with better sampling effects, the policy network will concentrate the action distribution to ensure that the device continues to sample along the optimal path.

[0066] Example 6: Value Function Use the following formula to express it:

[0067] ;

[0068] in, Indicates time Status at the time; is the standard deviation of the Gaussian distribution; is the time-integrated variable; is the characteristic matrix of the value function; is the first weight parameter, is the set value; is the second weight parameter, is the set value; is the square of the F norm; For time The reward function when ; For time The gradient of the value function corresponding to ;

[0069] ;

[0070] is the gradient operator.

[0071] Specifically, the core formula of the value function consists of three main parts. First, the integral term is calculated by the time variable The accumulation of reflects the contribution of the device's behavior from the past to the present time to the overall benefit. In this section, is a Gaussian weighting function used to Status and current status The similarity of the reward function Weighting. The physical significance of this weighting mechanism is that the device focuses on the behavior near the current state, while the influence of the distant state is exponentially attenuated. This design can effectively enhance the sensitivity of the value function to the current environment while avoiding excessive dependence on past states. On the basis of Gaussian weighting, the integral term also includes the combination of the concentration field change rate and the gradient, that is, . This term describes the dynamic characteristics of the concentration field over time. By combining the time partial derivative, the gradient, and the speed of the device, it captures the device's ability to respond to concentration changes during the sampling process. This design makes the value function not only limited to the return estimation of the static state, but also able to adjust the future strategy of the device according to the dynamic changes of the concentration field. It is especially suitable for complex scenarios where the concentration distribution changes over time. The second part is the regularization term, which is used to constrain the smoothness and stability of the value function. First, By penalizing the square of the value function gradient, the drastic changes in the value function are limited, ensuring that the model has the ability to smoothly transition between different states. This smoothness is crucial for policy optimization in reinforcement learning, because an overly steep value function may cause drastic fluctuations in the policy and affect the robustness of the device in a dynamic environment. By the feature matrix of the value function The square of the F-norm is constrained to further restrict the second-order derivative of the value function. This regularization method smoothes the surface shape of the value function from a high-order perspective and reduces the unstable behavior of the model. In addition, the feature matrix The definition of contains a combination of first-order and second-order derivatives to describe the local curvature information of the value function. This design can not only capture the temporal trend of the value function, but also This memory mechanism is particularly important in a dynamic environment, because the strategy optimization of the device needs to combine the comprehensive information of historical behavior and current status to more accurately predict future returns.

[0072] Example 7: Use the following formula to optimize the strategy:

[0073] ;

[0074] in, Optimize values ​​for the strategy; To adopt the The action corresponding to each track; To adopt the The state corresponding to the trajectory; To adopt the The reward corresponding to each track; To adopt the When the track is Next select action The probability of To adopt the The value function of the strategy when the strategy is optimized is used; by continuously calculating the strategy optimization value corresponding to each trajectory, the action corresponding to each time when the strategy optimization value is the largest is found, and the actions corresponding to each time when the strategy optimization value is the largest are arranged in chronological order as the elements in the optimal action sequence.

[0075] Specifically, the formula Represents the gradient of strategy optimization, whose physical meaning is to adjust the strategy parameters in the direction of increasing cumulative rewards Through the policy gradient method, the optimization process is directly based on the actual performance of the device behavior in the environment, avoiding the complexity of the traditional method that requires explicit modeling of the environment dynamics. During the optimization process, each trajectory of the device (including state and actions ) are used as samples to participate in gradient estimation and strategy update. The core part of the formula is Represents the logarithmic gradient of the policy probability, which quantifies the current policy network in the state Next Generate Action By calculating its gradient, the policy network can learn the pros and cons of the current action and adjust the probability distribution of the generated action, so that future action selection is more inclined to the direction of high rewards. This logarithmic gradient-based optimization method is efficient and stable because it can avoid directly operating on the probability value, thereby reducing the problem of numerical overflow and gradient disappearance. Reward function Sum value function The difference is the weight of the policy gradient, called the advantage function. Its physical meaning is to measure the actual reward of the current action The expected cumulative reward in this state If the actual reward of an action is higher than the expected cumulative reward, the advantage function is positive, and the direction of the policy gradient will strengthen the probability of the action; if it is lower than the expected cumulative reward, the advantage function is negative, and the policy gradient will reduce the probability of the action. Through this mechanism, policy optimization can dynamically balance exploration and exploitation, and further improve on the basis of existing experience. The trajectory sampling method estimates the policy gradient by averaging multiple trajectories, where represents the average operation in each batch of trajectories. The number of trajectories (64) and the time range ( ) has a direct impact on the stability and efficiency of policy optimization. More trajectory sampling can provide more accurate gradient estimation, but it also increases the computational burden; and the appropriate time range ensures that the optimization process can fully cover the behavior sequence required by the task. The ultimate goal of policy optimization is to find In the specific implementation, by continuously calculating the strategy optimization value of each trajectory, it is possible to identify the maximum action sequence at each time step. The best action . These optimal actions are arranged in chronological order to form the final optimal action sequence. This sequence represents the best execution plan for the device's sampling task in a complex concentration field environment, which can maximize the sampling quality and optimize the device's operating efficiency. The advantage of this policy optimization method is that it can not only dynamically adapt to changes in the environment, but also effectively balance exploration and utilization through a combination of trajectory sampling and advantage functions. This method is particularly applicable in concentration field environments, because devices need to quickly adjust their behavior in dynamic and uncertain environments to maximize the cumulative benefits of sampling tasks. Compared with traditional deterministic optimization methods, the policy optimization formula of this embodiment can more flexibly respond to complex task requirements, and provides strong theoretical support and practical value for reinforcement learning control of intelligent devices.

[0076] Embodiment 8: Generating a control trajectory according to an optimal action sequence specifically includes: at each time, executing the corresponding action in the optimal action sequence to generate a control trajectory.

[0077] Specifically, the generation of the control trajectory is based on the optimal action sequence. The optimal action sequence is obtained through the policy optimization process in Example 7, which includes the actions that the device should perform in each time step, that is, the corresponding linear velocity and angular velocity instructions. These instructions are not only determined by the policy network and value function of the reinforcement learning model, but also fully combine the environmental feedback and dynamic optimization results, representing the global optimal behavior mode of the device under the current concentration field. In the actual process of generating the control trajectory, the system will execute the corresponding actions in the optimal action sequence step by time. At each time , the device adjusts its motion state, including position, posture and speed, according to the instructions of the optimal action sequence. Specifically, the linear speed instruction Determines the linear movement speed and direction of the device at the current time, while the angular velocity command The rotation angle and attitude adjustment of the device are controlled. These two parts of the action act together on the power system of the device, drive it to move in three-dimensional space, and generate continuous control trajectories. The generation of control trajectories has real-time and dynamic adjustment capabilities. In each time step, the device not only performs the current optimal action, but also obtains real-time information of the environment through sensors, including concentration field distribution, gradient changes and device state feedback. This information will be input into the reinforcement learning model to update the policy optimization results and correct the subsequent parts of the optimal action sequence. This real-time closed-loop control mechanism ensures that the control trajectory can adapt to the dynamically changing environmental characteristics, so that the device always runs along the optimal path in the concentration field. In addition, the generation process of the control trajectory also focuses on the smoothness and physical feasibility of the trajectory. Each step of the optimal action sequence is not only driven by policy optimization, but also affected by the kinematic and dynamic constraints of the device. For example, the changes in the linear velocity and angular velocity of the device between time steps need to meet the smoothness requirements to avoid the negative impact of drastic speed or direction adjustments on the stability and energy consumption of the device. The system will dynamically adjust and optimize the optimal action sequence to ensure that the generated control trajectory is executable and efficient in actual operation. Through this control trajectory generation method based on the optimal action sequence, the present invention realizes the efficient integration of reinforcement learning and device control. The control trajectory not only reflects the theoretical optimality of the reinforcement learning strategy, but also combines the constraints of the actual equipment and environment to ensure its feasibility and efficiency in physical space. This design can effectively guide the device to achieve accurate sampling in the concentration field, avoid redundant paths, improve sampling efficiency, and reduce the energy consumption of the device and the task completion time.

[0078] Example 9: The sampling quality is evaluated by the following formula:

[0079] ;

[0080] in, It is the quality assessment value. The larger the value is, the higher the sampling quality is.

[0081] Specifically, the formula defines the sampling quality assessment value , its calculation consists of two main parts. The first part Indicates that during the entire task The time average of the target substance concentration values ​​sampled by the device within a period of time. This item is a direct reflection of the sampling quality and reflects the cumulative level of the target substance concentration obtained by the device during the mission. , the sensor records the concentration value at the current position , and accumulate it into the overall sampling through time integration. Finally, through the task time The normalization process is performed to obtain the average sampling concentration value of the device. The higher the concentration value, the more the device covers the area where the target substance is densely distributed, thereby improving the effectiveness of sampling. The second part of the formula is a penalty term for the energy consumption of equipment movement, where the action Including control signals for linear velocity and angular velocity. By calculating the square of the second norm of the action, the formula reflects the energy consumption of the device during movement. The physical meaning of this penalty term is that the lower the motion consumption of the device while completing the sampling task, the higher the task efficiency. By adding the energy consumption of the action into the formula in the form of a negative value, the system can comprehensively consider the balance between efficiency and cost when evaluating the sampling quality. The coefficient 0.1 is an adjustment factor used to control the weight ratio of sampling concentration and energy consumption in the evaluation formula. For example, when the target material in the concentration field is sparsely distributed, it may be necessary to increase this coefficient to reduce the motion consumption in the invalid area. The formula as a whole embodies a multi-objective optimization idea, which not only pursues the maximization of the sampling concentration, but also strives to reduce the energy cost of task execution. Value The larger the value, the more high-concentration areas the device covers during the task and completes the task with lower energy consumption. This design is applicable to a variety of complex environments, such as scenes with uneven or dynamically changing concentration fields, and can flexibly adapt to different task requirements by adjusting the weight factor. Compared with the traditional single-index evaluation method, the innovation of this formula is that it comprehensively considers the dual factors of sampling concentration and energy consumption, making the evaluation results more realistic. In actual tasks, simply pursuing high-concentration sampling may lead to a sharp increase in equipment energy consumption, while excessive energy conservation may lead to insufficient sampling results. This formula provides a clear goal orientation for the optimization of reinforcement learning strategies by balancing these two goals, thereby driving the device to find the best balance between sampling quality and efficiency. The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A mobile intelligent sampling equipment adaptive control system based on reinforcement learning, characterized in that: The system includes: a state and action construction unit, a reinforcement learning construction unit and an adaptive control unit; the state and action construction unit is used to perform superposition modeling on multiple concentration sources of the target substance to obtain the concentration field of the target substance, calculate the concentration field gradient, obtain the real-time state of the device, and use the real-time state of the device, the concentration field and the concentration field gradient as elements in the state space to construct the state; define the action, whose elements include the command linear velocity vector and the command angular velocity vector; the reinforcement learning construction unit is used to construct the reward function of reinforcement learning according to the state and action, and design the policy network and value function of reinforcement learning to generate a reinforcement learning model; the adaptive control unit is used to optimize the policy according to the reinforcement learning model, generate the control trajectory, and evaluate the sampling quality; The state and action construction unit obtains the intensity of each concentration source through the sensor, and sets the diffusion coefficient of each concentration source based on the intensity of each concentration source; assuming that the target material distribution is formed by the superposition of multiple concentration sources, its concentration field The distribution in space is represented by the Gaussian distribution function: ； in, For equipment The location of time, For the The location of the concentration source; For the The intensity of a concentration source; For the The diffusion coefficient of a concentration source is a set value. The higher the intensity of the concentration source, the larger the corresponding diffusion coefficient value. It is the modulus length operation of the vector; is the number of concentration sources; is an integer subscript index; the gradient of the concentration field Calculated using the following formula: ； Reward Function Use the following formula to express it: ； in, , which indicates the degree of alignment between the speed of the device and the concentration field gradient. The closer the value is to 1, the more efficient the direction of movement of the device is; , is the acceleration of the device; , is the angular acceleration of the device; is the maximum allowed task time.

2. The adaptive control system of a mobile intelligent sampling device based on reinforcement learning as claimed in claim 1, characterized in that: The real-time status of the device includes: position, attitude rotation matrix and speed; the status is expressed using the following formula: ； in, For equipment location in time; For equipment The speed of time; motion is defined as ;in, is the command linear velocity vector; is the command angular velocity vector.

3. The adaptive control system of a mobile intelligent sampling device based on reinforcement learning as claimed in claim 1, characterized in that: The policy network of reinforcement learning is expressed using the following formula: ； in, For the status Next select action probability; is the action mean, indicating the center of the optimal action; is the action covariance matrix, which represents the uncertainty of the action; is the determinant of the action covariance matrix, used for normalization; is the action dimension; action mean Generated by a 3-layer neural network, the formula is as follows: ； in, , and are the weight matrices of the first, second, and third layers of the neural network respectively; , and The bias matrices of the first, second, and third layers of the neural network respectively; is the activation function; is the hyperbolic tangent function; the action covariance matrix The diagonal elements of are output by the network, and the positive definiteness is guaranteed by the exponential function, as follows: ； in, is the logarithm of the covariance diagonal elements generated by the neural network; is an operation that generates a diagonal matrix.

4. The adaptive control system of a mobile intelligent sampling device based on reinforcement learning as claimed in claim 3, characterized in that: Value Function Use the following formula to express it: ； in, Indicates time Status at the time; is the standard deviation of the Gaussian distribution; is the time-integrated variable; is the characteristic matrix of the value function; is the first weight parameter, is the set value; is the second weight parameter, is the set value; is the square of the F norm; For time The reward function when ; For time The gradient of the value function corresponding to ; ； is the gradient operator.

5. The adaptive control system of a mobile intelligent sampling device based on reinforcement learning as claimed in claim 4, characterized in that: Use the following formula for strategy optimization: ； in, Optimize values ​​for the strategy; To adopt the The action corresponding to each track; To adopt the The state corresponding to the trajectory; To adopt the The reward corresponding to each track; To adopt the When the track is Next select action probability; To adopt the The value function of the strategy when the strategy is optimized is used; by continuously calculating the strategy optimization value corresponding to each trajectory, the action corresponding to each time when the strategy optimization value is the largest is found, and the actions corresponding to each time when the strategy optimization value is the largest are arranged in chronological order as the elements in the optimal action sequence.

6. The adaptive control system of a mobile intelligent sampling device based on reinforcement learning as claimed in claim 5, characterized in that: Generating a control trajectory according to the optimal action sequence specifically includes: executing the corresponding action in the optimal action sequence at each time to generate the control trajectory.

7. The adaptive control system of a mobile intelligent sampling device based on reinforcement learning as claimed in claim 6, characterized in that: The sampling quality is evaluated by the following formula: ； in, It is the quality assessment value. The larger the value is, the higher the sampling quality is.

Images