End-to-end unmanned platform autonomous reconnaissance method and system integrating safety control

Abstract

The invention relates to the technical field of unmanned platform autonomous reconnaissance, and discloses an end-to-end unmanned platform autonomous reconnaissance method and system integrating safety control. The method comprises the following steps: S1, receiving multi-modal input data, and performing visual feature extraction on the multi-modal input data based on a visual encoder of a deformable convolutional network (DCN); s2, generating a context vector according to the visual feature vector, and generating and outputting a future action sequence according to the context vector; s3, optimizing a future action sequence based on a quadratic programming method of a control barrier function-Lyapunov function to obtain an optimized unmanned platform control quantity; and S4, executing unmanned platform control according to the optimized unmanned platform control quantity, acquiring new state information in a control process in real time, outputting the new state information acquired in real time as multi-modal input data, and returning to S1. Therefore, the action sequence can be dynamically adjusted in an uncertain environment, and the system is ensured to always meet security constraints and gradually approach a task target.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned platform autonomous reconnaissance, and particularly relates to an end-to-end unmanned platform autonomous reconnaissance method and system integrated with safety control. BACKGROUND

[0002] In recent years, end-to-end autonomous driving technology has become a focus of autonomous driving research due to its unified integration of perception, planning, decision-making and control, and has stimulated extensive exploration and in-depth research in the industry. It has also been applied in the exploration and reconnaissance of unmanned platforms. This method directly generates control instructions from sensor data (such as cameras and lidar) through a deep learning model, avoiding information loss, cumulative errors and complex interfaces in traditional modular methods (such as perception-planning-control). However, existing end-to-end methods still face the following deficiencies in practical applications: (1) insufficient safety: the output of traditional end-to-end models cannot directly guarantee the safety of vehicles or unmanned platforms, especially in dynamic and uncertain environments, which may pose a risk of collision or task failure; (2) low decision-making transparency: the black box nature of end-to-end models makes it difficult to explain failures; (3) lack of robustness: in high complexity or unseen environments, model performance may significantly decrease, especially in tasks that require dynamic obstacle avoidance and target navigation switching.

[0003] Currently, safety is usually supplemented by rule-based obstacle avoidance algorithms (such as dynamic window method DWA or reinforcement learning-based strategies), which rely on heuristic rules or data-driven optimization, and have problems such as response lag, lack of theoretical guarantee, and may only obtain local optimal solutions, making it difficult to meet high safety and efficiency requirements. SUMMARY

[0004] The present application provides an end-to-end unmanned platform autonomous reconnaissance method and system integrated with safety control, which can solve the problems in the prior art.

[0005] The present application provides an end-to-end unmanned platform autonomous reconnaissance method integrated with safety control, wherein the method comprises:

[0006] S1, receiving multi-modal input data and performing visual feature extraction on the multi-modal input data based on a deformable convolution network DCN visual encoder, outputting a visual feature vector representing environmental information;

[0007] S2, generating a context vector according to the visual feature vector, and generating and outputting a future action sequence according to the context vector;

[0008] S3, optimizing the future action sequence based on a control barrier function-Lyapunov function quadratic programming method to obtain an optimized unmanned platform control quantity;

[0009] S4, performing unmanned platform control according to the optimized unmanned platform control amount, obtaining new state information in the control process in real time, and outputting the real-time obtained new state information as multi-modal input data, returning to S1.

[0010] Preferably, the multi-modal input data includes environment perception data and task target information.

[0011] Preferably, generating and outputting the future action sequence according to the context vector includes:

[0012] Modeling the action distribution according to the context vector using a diffusion model, and generating the future action sequence of the unmanned platform in the multi-modal distribution using a diffusion process;

[0013] Outputting the future action sequence, the future action sequence representing a preliminary navigation planning result of the unmanned platform.

[0014] Preferably, the future action sequence is optimized based on a control barrier function-Lyapunov function quadratic programming method, and the optimized unmanned platform control amount includes:

[0015] According to the first action in the future action sequence, calculating the task target state of the unmanned platform at the next time, the task target state including position, velocity or trajectory;

[0016] Constructing a control Lyapunov function CLF;

[0017] Constructing a control barrier function CBF;

[0018] Constructing a control input quadratic programming optimization function, and optimizing the future action sequence through CLF constraint and CBF constraint according to the control input optimization function, to obtain the optimized unmanned platform control amount.

[0019] Preferably, the control Lyapunov function CLF is:

[0020] V(x) = ||x-x goal || 2 ,

[0021] Wherein, V(x) is a typical candidate Lyapunov function, representing the squared Euclidean distance from the current position to the target position, by minimizing V(x), the system is driven to converge to the target position, x is the current position of the unmanned platform, and x goal is the target position.

[0022] System dynamics The control input quadratic programming optimization function is:

[0023]

[0024] wherein u is a control input of the platform, by which the running of the platform is controlled, u ref is a reference action;

[0025] The control barrier function CBF is:

[0026] h(x)=d(x,obs)-r safe ,

[0027] wherein h(x) is the control barrier function, when h(x) is greater than or equal to 0, the system is in a safe state, d(x,obs) is the distance from the current position of the unmanned platform to the obstacle, and r safe is a safety radius;

[0028] The CLF constraint is:

[0029]

[0030] wherein K is a convergence speed parameter, and the greater K is, the faster the convergence speed is;

[0031] The CBF constraint is:

[0032]

[0033] wherein a(h(x)) is a nonlinear term of the control barrier function, and a is a K-like function.

[0034] The application further provides an end-to-end unmanned platform autonomous reconnaissance system integrated with safety control, wherein the system comprises:

[0035] An input and perception module is configured to receive multi-modal input data, and perform visual feature extraction on the multi-modal input data based on a visual encoder of a deformable convolutional network (DCN), and output a visual feature vector representing environmental information;

[0036] An action sequence generation module is configured to generate a context vector according to the visual feature vector, and generate and output a future action sequence according to the context vector;

[0037] An optimization module is configured to optimize the future action sequence based on a quadratic programming method of a control barrier function-Lyapunov function, and obtain an optimized unmanned platform control variable;

[0038] An execution and closed loop module is configured to execute unmanned platform control according to the optimized unmanned platform control variable, acquire new state information in a real-time manner during the control process, and output the real-time acquired new state information as multi-modal input data to the input and perception module.

[0039] Preferably, the multi-modal input data comprises environmental perception data and task target information.

[0040] Preferably, the generating and outputting the future action sequence according to the context vector comprises:

[0041] modeling the action distribution according to the context vector using a diffusion model, generating the future action sequence of the unmanned platform in a multi-modal distribution by a diffusion process;

[0042] outputting the future action sequence, the future action sequence representing a preliminary navigation planning result of the unmanned platform.

[0043] Preferably, the future action sequence is optimized based on a quadratic programming method of a control barrier function-Lyapunov function, and the optimized unmanned platform control quantity comprises:

[0044] calculating a task target state of the unmanned platform at a next time according to a first action in the future action sequence, the task target state comprising a position, a velocity or a trajectory;

[0045] constructing a control Lyapunov function CLF;

[0046] constructing a control barrier function CBF;

[0047] constructing a control input quadratic programming optimization function, and optimizing the future action sequence by the CLF constraint and the CBF constraint according to the control input optimization function, to obtain the optimized unmanned platform control quantity.

[0048] Preferably,

[0049] the control Lyapunov function CLF is:

[0050] V(x) = ||x-x goal || 2 ,

[0051] wherein V(x) is a typical candidate Lyapunov function, representing the squared Euclidean distance from the current position to the target position, by minimizing V(x), the system is driven to converge to the target position, x is the current position of the unmanned platform, and x goal is the target position;

[0052] Let the system dynamics be the control input quadratic programming optimization function is:

[0053]

[0054] wherein u is the control input of the platform, by which the operation of the platform is controlled, and u ref is the reference action;

[0055] the control barrier function CBF is:

[0056] h(x) = d(x, obs) - r safe ,

[0057] where h(x) is the control barrier function, when h(x) ≥ 0, the system is in a safe state, d(x, obs) is the distance from the current position of the unmanned platform to the obstacle, r safe is the safety radius;

[0058] The CLF constraint is:

[0059]

[0060] where κ is the convergence speed parameter, the larger κ is, the faster the convergence speed is;

[0061] The CBF constraint is:

[0062]

[0063] where α(h(x)) is a nonlinear term of the control barrier function, and α is a K-like function.

[0064] Through the above technical scheme, the deformable convolution network DCN is used as a visual feature extraction module in an end-to-end architecture for visual feature extraction, and the control barrier function (CBF) and the control Lyapunov function (CLF) are combined to optimize the action sequence generated by the end-to-end architecture. Specifically, 1. The improved end-to-end architecture based on DCN improves the extraction ability of dynamic targets, obstacles and complex environmental features through adaptive receptive field. In unstructured environments, the flexibility of this improvement is significantly better than the convolutional network such as EfficientNet used by the traditional end-to-end architecture with fixed receptive field; 2. The optimization operation is combined with the control barrier function (CBF) and the control Lyapunov function (CLF). The CBF is used in the end-to-end architecture to provide strict mathematical constraints, ensuring that the unmanned platform avoids collision in a dynamic environment and maintains safe operation; the CLF uses the asymptotic convergence property of the Lyapunov function to guide the unmanned platform to reach the target stably and avoid task failure. 3. The method described in the present application provides a complete real-time closed-loop control based on end-to-end architecture. The upper end-to-end provides an action sequence, the action optimization module provides safety control, and the state information is updated through feedback to dynamically adjust the action sequence and control input, so that the end-to-end system can maintain high efficiency in dynamic and complex scenes. Unlike traditional end-to-end black box models, it has higher interpretability and reliability, and unlike traditional perception-planning-control multi-layer architecture, it avoids cumulative errors and local optimal traps, and has higher reconnaissance success rate and flexibility. BRIEF DESCRIPTION OF DRAWINGS

[0065] The accompanying drawings, which are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0066] Figure 1 A flow chart of an integrated security controlled end-to-end unmanned platform autonomous reconnaissance method according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0067] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The description of the at least one example embodiment is actually only illustrative, but not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0068] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form, unless the context clearly indicates otherwise, and it should also be understood that the terms "comprise" and / or "include" as used in the specification indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0069] Unless specifically stated otherwise, the relative arrangements of the components and steps illustrated in these embodiments and the numerical expressions and values set forth herein are not limiting. It should be understood that the various parts of the drawings are not necessarily drawn to scale, and that, for the purpose of convenience and clarity, not all components can be shown in a given figure. Techniques, methods, and devices known to those of ordinary skill can not be discussed in detail, but rather can be summarily described in order not to unnecessarily obscure aspects of the present application. In the description of the example embodiments above, any specific values should be interpreted as merely example, and not as a limitation. Other example embodiments of the example embodiments can have different values. It is to be noted that like reference numerals and letters refer to like items in the drawings and a discussion of one item in a drawing does not necessitate further discussion of that item in subsequent drawings.

[0070] Figure 1 A flow chart of an integrated safety control end-to-end unmanned platform autonomous reconnaissance method according to an embodiment of the present application is shown.

[0071] The unmanned platform may be a vehicle, for example.

[0072] As shown in the figure, the embodiment of the present application provides an integrated safety control end-to-end unmanned platform autonomous reconnaissance method, wherein the method comprises: Figure 1

[0073] S1, receiving multi-modal input data and performing visual feature extraction on the multi-modal input data based on a visual encoder of a deformable convolutional network (DCN), and outputting a visual feature vector representing environmental information;

[0074] That is, a large amount of multi-modal input can be provided for the model, and then its visual features are extracted based on the convolutional network in deep learning. Based on this, the context information extracted by the deep learning method can be obtained from the situation pictures collected by the unmanned platform, so as to directly generate the subsequent action (future action) of the unmanned platform.

[0075] The deformable convolutional network can flexibly adapt to feature changes in complex scenes, such as dynamic obstacles or environmental occlusions. Therefore, the use of the visual encoder based on the deformable convolutional network DCN to extract features can improve the perception accuracy.

[0076] S2, generating a context vector according to the visual feature vector, and generating and outputting a future action sequence according to the context vector;

[0077] That is, the subsequent action of the unmanned platform is directly generated, the environmental information obtained in S1 is first taken as input, a Transformer module is used for encoding to obtain environmental context information, then a diffusion model is used for modeling to obtain a future action sequence, and finally output is performed for subsequent optimization operation.

[0078] S3, optimizing the future action sequence based on a quadratic programming (QP) method of a control barrier function-Lyapunov function to obtain an optimized unmanned platform control variable;

[0079] S4, performing unmanned platform control according to the optimized unmanned platform control variable, acquiring new state information in real time in the control process, and outputting the real-time acquired new state information as multi-modal input data, returning to S1.

[0080] For example, control input execution can be performed: the optimized control input (control variable) u * ​The data link is sent to the bottom drive module of the unmanned platform, and the unmanned platform is controlled to complete the action execution at the next moment. Then, real-time feedback and iterative updating operations can be performed: new state information such as speed, position, RGB image of the camera, depth information and laser radar point cloud is obtained in real time through the sensor of the unmanned platform, and is fed back to the input and perception module and the optimization module (safety and stability optimization module) of the unmanned platform. After the feedback, the environment and target changes are collected by S1, and the action sequence at the subsequent moment is dynamically updated by S2, and the control input is optimized by S3, forming a closed-loop control (the state information is updated by sensor feedback, and the action sequence and control input are dynamically adjusted).

[0081] The DCN+Transformer+diffusion model is an end-to-end architecture.

[0082] Through the above technical solution, the deformable convolution network DCN is used as a visual feature extraction module in the end-to-end architecture for visual feature extraction, and the control barrier function (CBF) and the control Lyapunov function (CLF) are combined to optimize the action sequence generated by the end-to-end architecture. Specifically, 1. The improved end-to-end architecture based on DCN improves the extraction ability of dynamic targets, obstacles and complex environmental features through adaptive receptive field. In unstructured environments, the flexibility of this improvement is significantly better than the convolutional network such as EfficientNet used by the traditional end-to-end architecture with fixed receptive field; 2. Combined with the control barrier function (CBF) and the control Lyapunov function (CLF), the optimization operation is performed, the CBF is used in the end-to-end architecture to provide strict mathematical constraints, ensuring that the unmanned platform avoids collision in a dynamic environment and maintains safe operation; the CLF uses the asymptotic convergence property of the Lyapunov function to guide the unmanned platform to robustly reach the target and avoid task failure. 3. The method described in the application provides a complete real-time closed-loop control based on end-to-end architecture. The action sequence is provided by the upper end-to-end, the safety control is provided by the action optimization module, the state information is updated by feedback, the action sequence and control input are dynamically adjusted, so that the end-to-end system can maintain high efficiency in dynamic and complex scenes. Unlike traditional end-to-end black box models, it has higher interpretability and reliability, and unlike traditional perception-planning-control multi-layer architecture, it avoids cumulative errors and local optimal traps, and has higher reconnaissance success rate and flexibility.

[0083] Specifically, by introducing a Deformable Convolutional Network (DCN), a Controlled Barrier Function (CBF), and a Controlled Lyapunov Function (CLF), this method provides flexible adaptability, mathematically rigorous safety, and stability assurance for the entire system in the field of autonomous reconnaissance for unmanned platforms, balancing mission efficiency and operational safety. This method can dynamically adjust action sequences in uncertain environments, ensuring that the system (unmanned platform) always meets safety constraints and gradually approaches the mission objectives.

[0084] It can be seen that compared with the existing technology, the present invention has significant improvements in perception performance, strategy generation, security assurance, task efficiency and system robustness, especially in complex dynamic scenes.

[0085] According to one embodiment of the present invention, the multimodal input data includes environmental perception data and task target information.

[0086] The environmental perception data may be, for example, at least one of the following: speed, position, camera RGB image, depth information, and lidar point cloud; the mission target information may be, for example, target image, GPS coordinates, or language description. Preferably, the environmental perception data may also include the time series in which the data is observed. t , so that the visual encoder can extract features.

[0087] According to an embodiment of the present invention, generating a context vector based on a visual feature vector includes:

[0088] The visual features extracted by DCN are passed into the Transformer module to encode the current environment state and mission target state of the unmanned platform and generate a context vector.

[0089] According to one embodiment of the present invention, generating and outputting a future action sequence according to a context vector includes:

[0090] Use the diffusion policy to model the action distribution based on the context vector and use the diffusion process to generate the future action sequence of the unmanned platform with multimodal distribution. t ={a t+1 ,…,a t+H}; In this process, the generated action distribution takes into account complex behaviors such as collision avoidance and path reconnaissance and exploration.

[0091] Output future action sequence a t , future action sequence a t Represents the preliminary navigation planning results of the unmanned platform.

[0092] According to one embodiment of the present application, a quadratic programming method based on control barrier function-Lyapunov function (CBF-CLF) is used to optimize the future action sequence to obtain the optimized unmanned platform control quantity. The CLF constraint is used to meet the stability constraint of the end-to-end unmanned autonomous reconnaissance platform, that is, under the control of the output optimized control quantity, the platform always stably advances towards the target. The CBF constraint is used to meet the safety constraint of the end-to-end unmanned autonomous reconnaissance platform, that is, under the control of the output optimized control quantity, the motion of the platform always remains in the safety set, which specifically includes:

[0093] According to the first action in the future action sequence, the task target state of the unmanned platform at the next time is calculated, and the task target state includes position, velocity or trajectory.

[0094] For example, the first action a t+1 output can be used to calculate the task target state of the unmanned platform at the next time, such as position, velocity or trajectory x goal . Thus, the state and target information acquisition is realized.

[0095] A control Lyapunov function CLF is constructed.

[0096] A control barrier function CBF is constructed.

[0097] A control input quadratic programming optimization function is constructed, and the future action sequence is optimized by the CLF constraint and the CBF constraint according to the control input optimization function to obtain the optimized unmanned platform control quantity.

[0098] The CLF constraint can ensure that the action of the unmanned platform gradually approaches the target. The CBF constraint can ensure that the action of the unmanned platform meets the safety requirement.

[0099] In the actual autonomous reconnaissance process of the unmanned platform, factors such as terrain undulation, unknown road conditions and unknown dynamic obstacles may cause the end-to-end autonomous reconnaissance system to give unreasonable action behavior planning, so that the tracking error exceeds the error boundary, and then violates the safety hard constraint, thereby bringing potential unsafe hidden dangers such as collision. The present application introduces a quadratic programming (QP) method based on control barrier function-Lyapunov function (CBF-CLF) to optimize the unmanned platform control quantity and ensure safe control.

[0100] According to one embodiment of the present application, the control Lyapunov function CLF is:

[0101] V(x)=||x-x goal || 2 ,

[0102] where V(x) is a typical candidate Lyapunov function, representing the squared Euclidean distance from the current position to the target position, by minimizing V(x), the system (autonomous reconnaissance system) is driven to converge to the target position, x is the current position of the unmanned platform, x goal is the target position;

[0103] Let the system dynamics be The control input quadratic programming optimization function is:

[0104]

[0105] where u is the control input of the platform, through which the operation of the platform is controlled, u ref is the reference action;

[0106] The control barrier function CBF is:

[0107] h(x) = d(x, obs) - r safe ,

[0108] where h(x) is the control barrier function, when h(x) ≥ 0, the system is in a safe state, d(x, obs) is the distance from the current position of the unmanned platform to the obstacle, r safe is the safety radius;

[0109] The CLF constraint is:

[0110]

[0111] where κ is the convergence speed parameter, the larger κ is, the faster the convergence speed is, but it may lead to too large control input;

[0112] The CBF constraint is:

[0113]

[0114] where α(h(x)) is a nonlinear term of the control barrier function, α is a K-like function, which is strictly increasing and α(0) = 0, used to adjust the strictness of safety, in the design, it is necessary to ensure sufficient safety, while ensuring the freedom of control input.

[0115] Specifically, the end-to-end architecture generates a multi-modal future action sequence, the CBF-CLF prunes and optimizes the action distribution; in the CBF-CLF, the barrier function h(x) provides strict mathematical guarantee to ensure that the unmanned platform always maintains a safe area in a complex dynamic environment; the Lyapunov function V(x) realizes the asymptotic stability of the target state, guiding the unmanned platform to stably and autonomously drive to the target point, and then through QP (Quadratic Programming) optimization, the high-level strategy is seamlessly connected with the low-level control, and efficient and safe execution capability is provided in a multi-task scenario.

[0116] Therefore, based on the above formula for solving the quadratic programming (QP) problem, the optimized unmanned platform control input u * .

[0117] The embodiment of the application further provides an end-to-end unmanned platform autonomous reconnaissance system integrated with safety control, wherein the system comprises:

[0118] An input and perception module is configured to receive multi-modal input data, perform visual feature extraction on the multi-modal input data based on a visual encoder of a deformable convolutional network (DCN), and output a visual feature vector representing environmental information;

[0119] An action sequence generation module is configured to generate a context vector based on the visual feature vector, and generate and output a future action sequence based on the context vector;

[0120] An optimization module (safety and stability optimization module) is configured to optimize the future action sequence based on a quadratic programming method of a control barrier function-Lyapunov function, to obtain an optimized unmanned platform control quantity;

[0121] An execution and closed loop module is configured to execute unmanned platform control based on the optimized unmanned platform control quantity, acquire new state information in a real-time manner during the control process, and output the real-time acquired new state information as multi-modal input data to the input and perception module.

[0122] By the technical scheme, the deformable convolution network DCN is used as a visual feature extraction module in an end-to-end architecture to perform visual feature extraction, and a control barrier function (CBF) and a control Lyapunov function (CLF) are combined to optimize the action sequence generated by the end-to-end architecture. Specifically, 1. The improved end-to-end architecture based on the DCN improves the extraction capability of dynamic targets, obstacles and complex environmental features by using an adaptive receptive field. In an unstructured environment, the flexibility of the improved end-to-end architecture is significantly better than that of the traditional end-to-end architecture using the EfficientNet convolution network with a fixed receptive field; 2. The optimization operation is combined with the control barrier function (CBF) and the control Lyapunov function (CLF). The CBF is used in the end-to-end architecture to provide strict mathematical constraints, ensuring that the unmanned platform avoids collision in a dynamic environment and maintains safe operation. The CLF uses the asymptotic convergence property of the Lyapunov function to guide the unmanned platform to reach the target stably and avoid task failure. 3. The method provides a complete real-time closed-loop control based on the end-to-end architecture. The upper end-to-end architecture provides an action sequence, the action optimization module provides safety control, and the state information is updated through feedback to dynamically adjust the action sequence and the control input, so that the end-to-end system can maintain high efficiency in a dynamic and complex scene. Unlike the traditional end-to-end black box model, the method has higher interpretability and reliability, and unlike the traditional perception-planning-control multi-layer architecture, the method avoids cumulative errors and local optimal traps, and has higher reconnaissance success rate and flexibility.

[0123] According to an embodiment of the present application, the multi-modal input data includes environmental perception data and task target information.

[0124] According to an embodiment of the present application, generating and outputting the future action sequence according to the context vector includes:

[0125] Using the diffusion model to model the action distribution according to the context vector, the diffusion process is used to generate the future action sequence of the unmanned platform in the multi-modal distribution;

[0126] Outputting the future action sequence, the future action sequence represents a preliminary navigation planning result of the unmanned platform.

[0127] According to an embodiment of the present application, the future action sequence is optimized based on the control barrier function-Lyapunov function quadratic programming method, and the optimized unmanned platform control quantity includes:

[0128] According to the first action in the future action sequence, the task target state of the unmanned platform at the next time is calculated, and the task target state includes position, velocity or trajectory;

[0129] Constructing a control Lyapunov function CLF;

[0130] constructing a control barrier function CBF;

[0131] constructing a control input quadratic programming optimization function, optimizing the future action sequence according to the control input optimization function through the CLF constraint and the CBF constraint, and obtaining an optimized unmanned platform control quantity.

[0132] According to an embodiment of the present application, the control Lyapunov function CLF is:

[0133] The control Lyapunov function CLF is:

[0134] V(x) = ||x-x goal || 2 ,

[0135] wherein V(x) is a typical candidate Lyapunov function, representing the square of the Euclidean distance from the current position to the target position, by minimizing V(x), the system is driven to converge to the target position, x is the current position of the unmanned platform, and x goal is the target position;

[0136] Let the system dynamics be The control input quadratic programming optimization function is:

[0137]

[0138] wherein u is the control input of the platform, by which the operation of the platform is controlled, and u ref is the reference action;

[0139] The control barrier function CBF is:

[0140] h(x) = d(x, obs) - r safe ,

[0141] wherein h(x) is the control barrier function, when h(x) ≥ 0, the system is in a safe state, d(x, obs) is the distance from the current position of the unmanned platform to the obstacle, and r safe is the safety radius;

[0142] The CLF constraint is:

[0143]

[0144] wherein κ is a convergence speed parameter, the larger κ is, the faster the convergence speed is, but it may cause the control input to be too large;

[0145] The CBF constraint is:

[0146]

[0147] Wherein, a(h(x)) is a nonlinear term of control barrier function, a is a K-like function, which is strictly increasing and a(0)=0, used to adjust the strictness of safety, and in the design, sufficient safety needs to be ensured while ensuring the freedom of control input.

[0148] An integrated safety control end-to-end unmanned platform autonomous reconnaissance system is described below in conjunction with examples.

[0149] The embodiment provides an integrated safety control end-to-end unmanned platform autonomous reconnaissance system, which is used for end-to-end autonomous reconnaissance with safe real-time closed-loop control in the process of autonomous reconnaissance task performed by an unmanned platform. That is, an end-to-end unmanned platform autonomous reconnaissance method can be performed by using an end-to-end unmanned platform autonomous reconnaissance system.

[0150] 1. System composition

[0151] The system of the present application is composed of the following modules, each module cooperates with the other modules through a pre-defined interface, and together completes perception, policy generation, control optimization and finally closed-loop task execution. (1) Input and perception module (2) Action sequence generation module (3) Safety and stability optimization module (4) Execution and closed-loop module.

[0152] (1) System input and perception module:

[0153] The input and perception module is used to collect real-time information of the environment around the unmanned platform, and provides input data for subsequent feature extraction and policy generation. This module can be composed of a depth camera or an RGB camera plus a laser radar, which is used to capture two-dimensional visual information and depth information of the environment. The depth camera can capture two-dimensional visual information and depth information at the same time; the RGB camera collects high-resolution environment images (such as four surrounding images, obstacle positions) in real time. Two-dimensional visual information is used to capture the color, texture and shape features of obstacles, which play a key role in subsequent policy generation. Laser radar generates point cloud data of the environment by ranging scanning, which is used to construct a three-dimensional depth map of the environment. Depth information provides support for dynamic obstacle detection and barrier function construction. The collected data is standardized, denoised and frame-synchronized by the preprocessing module to ensure the stability and accuracy of subsequent processing.

[0154] After that, a deformable convolution network (DCN) is used to extract visual features from the environment data provided by the perception module. Compared with traditional convolution networks, DCN has the feature of adaptive receptive field, which can more effectively process complex features in dynamic environments, such as occlusion, small targets and multi-scale objects. In this invention, a multi-layer convolution architecture DCN backbone network is designed, the first few layers are used to capture low-level features of the environment (such as edges and textures), and the last few layers are used to extract higher-level semantic features (such as obstacle shape and dynamic object position). These features are encoded into feature vectors in the subsequent steps and input into the strategy generation module. By using the DCN encoding method, the adaptive receptive field improves the adaptability to complex scenes, such as occlusion or dynamic targets.

[0155] (2) Action sequence generation module:

[0156] The action sequence generation module is based on an end-to-end architecture, specifically based on Transformer and diffusion model, which generates future action sequences by combining visual features and target conditions. This module first inputs the feature vectors obtained from the input and perception module into a Transformer-based context encoder to generate a context vector c t that describes the current environment state. The action sequence generation module generates a future action sequence a t = {a t+1 ,…,a t+H} through a diffusion model. These action sequences can cover a variety of possible paths and provide a multi-modal decision basis for autonomous reconnaissance tasks.

[0157] (3) Safety and stability optimization module:

[0158] The safety and stability optimization module optimizes the action sequence output by the action sequence generation module through control barrier function (CBF) and control Lyapunov function (CLF) to ensure the safety and stability of task execution. In this invention, CBF is responsible for constructing safety constraints, calculating the minimum safety distance between the unmanned platform and obstacles through barrier function h(x), and ensuring that the unmanned platform always stays within the safety area in a dynamic environment. While CLF ensures that the unmanned platform gradually stabilizes and approaches the target state through Lyapunov function V(x). This module uses quadratic programming (QP) optimization method, combined with the constraint conditions of CBF and CLF, to dynamically adjust the reference action u ref to the actual control input u * , so that the unmanned platform meets the safety and target orientation during execution.

[0159] (4) Execution and closed-loop module:

[0160] The execution and closed-loop module receives the actual control input u* and convert them into specific driving signals for controlling the motion of the platform. During the execution of the task, real-time sensor feedback mechanisms monitor the current state (e.g., position, velocity, RGBD information around the platform) and send this information back to the safety and stability optimization module and the input and perception module, forming a closed-loop control structure. This design ensures that the platform can dynamically adapt to environmental changes, improving the reliability and robustness of the task.

[0161] 2. Specific working process

[0162] (1) System input and perception

[0163] When the platform enters a dynamic and complex environment (e.g., an indoor scene containing multiple obstacles and moving pedestrians), the perception module begins to work.

[0164] Data collection: The RGB camera collects environmental images at a rate of 30 frames per second, generating a set of color images with a resolution of 1920x1080. The laser radar performs 10 scans per second, generating point cloud data for measuring the distance and position of obstacles. For example, in the current frame, obstacles are detected at distances of 1.5 meters, 2.8 meters, and 4.0 meters.

[0165] Data preprocessing: The collected data is preprocessed by the preprocessing module, which performs the following operations: image data is cropped to a resolution of 640x480 to fit the DCN input. The laser radar point cloud is denoised, and after removing noise points, a sparse point cloud image is formed.

[0166] Subsequently, the preprocessed image data and point cloud data are passed to the visual feature extraction module. The first two layers of convolution operations of the DCN network extract low-level features of the image (such as edges and contours), and then generate feature maps through its deformable convolution layer.

[0167] In the detected obstacle area (such as a moving pedestrian), the DCN generates a set of dynamic feature vectors, for example, [v obs ,(x obs ,y obs ), θ obs ], which represent the feature information of the obstacle's speed, position, and direction, respectively. These features will be used for action sequence generation.

[0168] (2) Action sequence generation

[0169] The feature vectors generated by the input and perception module are input into the action sequence generation module, which is completed by an end-to-end architecture. Suppose the platform needs to autonomously drive to a certain target point, which is represented by a target image o gProvide (e.g., a photo of a target desk). At the same time, the current visual observation image of the unmanned platform is o t (For example, the image around the unmanned platform). Through the Transformer-based context encoder, the module converts o t and the target image o g into a context vector c t . The diffusion model uses the context vector c t to generate a future action sequence, such as [a t+1 , a t+2 , …, a t+H ]. Each action includes the speed and direction angle of the unmanned platform, and the action sequence is passed to the safety and stability optimization module after generation.

[0170] (3) Safety and stability optimization:

[0171] The safety and stability optimization module generates the final unmanned platform control input u * through constraint optimization of CBF and CLF.

[0172] First, obtain the current real-time state information of the unmanned platform from sensor feedback, including the current speed v t , the current position (x t , y t ), and the distance to the target d.

[0173] Second, construct the obstacle function, define the obstacle function h(x), and ensure that the unmanned platform maintains a safe distance from the nearest obstacle. For example, the current obstacle position is (x obs , y obs ), and the safety distance is optionally set to 0.5m. The obstacle function is defined as: h(x) = d(x, obs) - 0.5, if h(x) > 0, indicating that the unmanned platform is in a safe area.

[0174] Next, construct the Lyapunov function, define the Lyapunov function V(x) to represent the asymptotic stability of task completion: V(x) = ||x-x goal || 2 , where x represents the state information of the unmanned platform, which can usually be its geometric position in the environment (i.e., the current position of the unmanned platform), and x goal represents the target position, which can be obtained from task input parameters, system input, and target position obtained by the perception module.

[0175] Finally, perform QP optimization, taking the first action a t+1 generated by the second action sequence generation module as the reference input u ref , and construct the quadratic programming problem:

[0176]

[0177] The constraint condition is:

[0178]

[0179] wherein the value of kappa can be selected as 0.1, and the value of alpha can be selected as 0.2, so that the optimized constraint condition is After optimization, the control input u * can be calculated, wherein u * includes the speed v and the direction angle theta of the unmanned platform.

[0180] In addition, the value of kappa can be adjusted to control the speed and characteristics of the unmanned platform converging to the target state (Lyapunov stable point), and the value of alpha can be adjusted to control the dynamic characteristics (such as acceleration or deceleration) of the unmanned platform approaching the obstacle.

[0181] (4) Perform closed-loop control

[0182] The closed-loop control module receives the optimized control input u * , and drives the unmanned platform to complete the current action. The unmanned platform advances at a speed v and a direction angle theta. The sensor monitors the current position of the unmanned platform and the state of the obstacle in real time, and feeds back the updated current position (x t , y t ) and speed v t to the input and perception and action sequence generation module. The future action sequence is regenerated according to the latest observation, and the safety and stability optimization module adjusts the constraint condition according to the new state to form a dynamic closed-loop control.

[0183] As can be seen from the above embodiments, by introducing the variable convolutional network (DCN), the control barrier function (CBF) and the control Lyapunov function (CLF), the present application provides flexible adaptability, mathematically rigorous safety and stability guarantee for the whole system in the field of autonomous reconnaissance of unmanned platforms, and takes into account the task completion efficiency and operation safety. The method and system can dynamically adjust the action sequence in an uncertain environment, ensure that the unmanned platform always meets the safety constraints, and gradually approach the task target.

[0184] Compared with the prior art, the present application has at least the following advantages:

[0185] 1. Improved adaptability: Traditional end-to-end methods mostly use fixed architectures (such as EfficientNet). The present invention uses a deformable convolutional network (DCN) to enhance the adaptability of the perception module, especially for dynamic changes in complex environments such as occlusion and small targets. It has better flexible adaptability, which is significantly different from traditional perception models that use fixed receptive fields. In other words, by enhancing perception performance, the present invention can extract more accurate dynamic features, making subsequent action strategy generation more accurate. The dynamic receptive field optimization design of DCN helps to process diverse visual inputs in end-to-end autonomous reconnaissance scenarios of unmanned platforms.

[0186] 2. Improved safety: Through the obstacle constraints provided by CBF, strict protection of obstacles and safe areas is achieved, avoiding the collision risks caused by rule failure or insufficient optimization in traditional methods.

[0187] 3. Enhanced stability: CLF is used to ensure the asymptotic stability of mission objectives, taking into account both global mission completion efficiency and local safety optimization, so that the unmanned platform can obtain safe and stable autonomous driving control input.

[0188] 4. Higher interpretability: By extracting environmental features through a variable convolutional network and modeling future action sequences using a diffusion model, a high-level action sequence is generated. Combined with the constraints and optimization process of CBF-CLF, a closed-loop optimization framework from global planning to local control is formed, which provides a mathematical explanation for the system's decision-making process and facilitates analysis and improvement.

[0189] In summary, the present invention provides a more flexible, safer, more robust and more interpretable end-to-end unmanned platform autonomous reconnaissance method and system, providing an innovative technical path for intelligent navigation tasks in complex environments.

[0190] In the description of the present invention, it should be understood that the directions or positional relationships indicated by directional words such as "front, back, up, down, left, right", "horizontal, vertical, perpendicular, horizontal" and "top, bottom" are usually based on the directions or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description. Unless otherwise specified, these directional words do not indicate or imply that the device or element referred to must have a specific direction or be constructed and operated in a specific direction. Therefore, they cannot be understood as limiting the scope of protection of the present invention; the directional words "inside and outside" refer to the inside and outside relative to the outline of each component itself.

[0191] For purposes of the description hereinafter, the terms "upper", "lower", "right", "left", "rear", "front", "vertical" and "horizontal" as can be perceived herein relative to the accompanying drawings refer to the orientation of the components being described. However, it is to be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device described herein relative to the other device or structure is inverted, then a spatially relative term such as "above" can be interpreted as meaning "below" or "below" can be interpreted as meaning "above". The device can also be oriented in other ways (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. The devices can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0192] In addition, it should be pointed out that the use of the terms "first", "second" and the like in connection with various elements is used for convenience in distinguishing one element from another. It does not necessarily imply that the elements so designated are to be understood as being preferred over or superior to other elements. Moreover, no additional meaning is to be inferred from use of such terms.

[0193] The preferred embodiments herein disclosed are not intended to limit or restrict the scope of the invention. The scope of the invention is defined solely by the appended claims.

Claims

1. An end-to-end unmanned platform autonomous reconnaissance method with integrated safety control, characterized in that: The method includes: S1 receives multimodal input data and extracts visual features from the multimodal input data based on the visual encoder of the deformable convolutional network (DCN), outputting a visual feature vector representing the environmental information; S2, generates a context vector based on the visual feature vector, and generates and outputs the future action sequence based on the context vector; S3, based on the quadratic programming method of the control obstacle function-Lyapunov function, optimizes the future action sequence to obtain the optimized unmanned platform control quantity; S4, executes unmanned platform control according to the optimized unmanned platform control quantity, obtains new status information in the control process in real time, and outputs the new status information obtained in real time as multimodal input data, and returns to S1.

2. The method according to claim 1, characterized in that Multimodal input data includes environmental perception data and task target information.

3. The method according to claim 2, characterized in that Generating and outputting future action sequences based on the context vector includes: Use the diffusion model to model the action distribution based on the context vector, and use the diffusion process to generate the future action sequence of the unmanned platform with multimodal distribution; Output the future action sequence, which represents the preliminary navigation planning results of the unmanned platform.

4. The method according to claim 3, characterized in that The quadratic programming method based on the control obstacle function - Lyapunov function is used to optimize the future action sequence. The optimized control variables of the unmanned platform include: Based on the first action in the future action sequence, calculate the mission target state of the unmanned platform at the next moment. The mission target state includes position, speed or trajectory; Construct control Lyapunov function CLF; Construct control barrier function CBF; A control input quadratic programming optimization function is constructed. Based on the control input optimization function, the future action sequence is optimized through CLF constraints and CBF constraints to obtain the optimized control quantity of the unmanned platform.

5. The method according to claim 4, characterized in that The control Lyapunov function CLF is: V(x)=‖x-x goal ‖ 2 , Among them, V(x) is a typical candidate Lyapunov function, which represents the square of the Euclidean distance from the current position to the target position. By minimizing V(x), the system is driven to converge to the target position. x is the current position of the unmanned platform, x goal is the target location; System Dynamics The control input quadratic programming optimization function is: Among them, u is the control input of the platform, and the operation of the platform is controlled by this parameter. ref is the reference action; The control barrier function CBF is: h(x)=d(x,obs)-r safe , Among them, h(x) is the control obstacle function. When h(x)≥0, the system is in a safe state. d(x,obs) is the distance from the current position of the unmanned platform to the obstacle. r safe is the safety radius; The CLF constraints are: Among them, κ is the convergence speed parameter, the larger the κ is, the faster the convergence speed is; The CBF constraints are: Where α(h(x)) is the nonlinear term that controls the barrier function, and α is a K-like function.

6. An end-to-end unmanned platform autonomous reconnaissance system with integrated safety control, characterized in that: The system includes: The input and perception module is used to receive multimodal input data, extract visual features from the multimodal input data based on the visual encoder of the deformable convolutional network (DCN), and output a visual feature vector representing environmental information; An action sequence generation module is used to generate a context vector based on the visual feature vector, and generate and output a future action sequence based on the context vector; The optimization module is used to optimize the future action sequence based on the quadratic programming method of the control obstacle function-Lyapunov function to obtain the optimized control quantity of the unmanned platform; The execution and closed-loop module is used to execute the unmanned platform control according to the optimized unmanned platform control quantity, obtain the new status information in the control process in real time, and output the new status information obtained in real time as multimodal input data to the input and perception module.

7. The system according to claim 6, characterized in that Multimodal input data includes environmental perception data and task target information.

8. The system according to claim 7, characterized in that Generating and outputting future action sequences based on the context vector includes: Use the diffusion model to model the action distribution based on the context vector, and use the diffusion process to generate the future action sequence of the unmanned platform with multimodal distribution; Output the future action sequence, which represents the preliminary navigation planning results of the unmanned platform.

9. The system according to claim 8, characterized in that The quadratic programming method based on the control obstacle function - Lyapunov function is used to optimize the future action sequence. The optimized control variables of the unmanned platform include: Based on the first action in the future action sequence, calculate the mission target state of the unmanned platform at the next moment. The mission target state includes position, speed or trajectory; Construct control Lyapunov function CLF; Construct control barrier function CBF; A control input quadratic programming optimization function is constructed. Based on the control input optimization function, the future action sequence is optimized through CLF constraints and CBF constraints to obtain the optimized control quantity of the unmanned platform.

10. The system according to claim 9, characterized in that The control Lyapunov function CLF is: V(x)=‖x-x goal ‖ 2 , Among them, V(x) is a typical candidate Lyapunov function, which represents the square of the Euclidean distance from the current position to the target position. By minimizing V(x), the system is driven to converge to the target position. x is the current position of the unmanned platform, x goal is the target location; System Dynamics The control input quadratic programming optimization function is: Among them, u is the control input of the platform, and the operation of the platform is controlled by this parameter. ref is the reference action; The control barrier function CBF is: h(x)=d(x,obs)-r safe , Among them, h(x) is the control obstacle function. When h(x)≥0, the system is in a safe state. d(x,obs) is the distance from the current position of the unmanned platform to the obstacle. r safe is the safety radius; The CLF constraints are: Among them, κ is the convergence speed parameter, the larger the κ is, the faster the convergence speed is; The CBF constraints are: Where α(h(x)) is the nonlinear term that controls the barrier function, and α is a K-like function.

Images