Robust neural network control method based on embedded physical knowledge

Abstract

The invention discloses a robust neural network control method based on embedded physical knowledge, and belongs to the technical field of robust control and safety. The method comprises the following steps: 1, in a constructed embedded physical module, extracting angle features and line features of an environment area subset divided by a point cloud in a rectangular plane coordinate system, and identifying a task mode; 2, constructing and utilizing SAC deep reinforcement learning training to parallelly access an embedded physical module to a robust neural network of embedded physical knowledge of a penultimate layer of a hidden layer of the mixed mode neural network of combined reinforcement learning; and 3, inputting radar data into the trained embedded physical neural network, performing feature fusion through full connection to form a steering angle, and performing corresponding actions in real time by an intelligent agent along a task trajectory in a manner of avoiding obstacles according to the steering angle. Compared with the prior art, the robustness of the controller of the hybrid mode neural network for combined reinforcement learning is improved by using embedded physical knowledge.

Description

Technical Field

[0001] This invention relates to a robust neural network control method based on embedded physical knowledge, belonging to the field of robust control and safety technology, and applied to hybrid mode neural network control using combined reinforcement learning. Background Technology

[0002] In recent years, the performance of deep learning models has been continuously improving, and they are increasingly being used in safety-critical systems to train controllers. The principle behind this is learning "experienced knowledge" from a large amount of sensory data and then outputting control signals to control system changes. In real-world safety-critical systems, environmental uncertainties and variability can disturb the sensory data. However, models based on physical knowledge explicitly constrain system behavior through mathematical equations. Input noise is naturally filtered out by the smoothness and conservation properties of these mathematical equations, resulting in control signals that conform to physical laws within their applicable range. Therefore, training a more robust neural network controller using embedded physical knowledge is crucial.

[0003] Currently, robust controllers can be categorized into four types based on their methods for improving system robustness: model-optimized robust controllers, validation-optimized robust controllers, test-optimized robust controllers, and adversarial-optimized robust controllers. Existing robust controllers suffer from the following drawbacks: model-optimized robust controllers can guarantee robustness when training simple tasks, but are not suitable for mode switching; adversarial-optimized robust controllers only improve robustness during subtask switching in ensemble learning, without considering robustness within subtasks; test-optimized robust controllers cannot perform comprehensive system testing, failing to ensure complete security and robustness; and validation-optimized robust controllers have poor scalability, hampered by factors such as neural network architecture, activation function selection, parameter quantity, and dimensionality.

[0004] Therefore, how to leverage embedded physical knowledge to improve the robustness of controllers in hybrid neural networks that combine reinforcement learning has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the technical problem of improving the robustness of controllers for hybrid-mode neural networks using embedded physical knowledge, and to propose a robust neural network control method based on embedded physical knowledge.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] This invention discloses a robust neural network control method based on embedded physical knowledge, comprising the following steps:

[0008] Step 1: Construct an embedded physical module for combining hybrid-mode neural networks for reinforcement learning; In the embedded physical module, extract angular and line features from the environmental region subset divided by the point cloud in a Cartesian coordinate system with the agent as the origin and the agent's orientation as the y-axis, and use the angular and line features to identify the task pattern.

[0009] Step 1.1: Construct a Cartesian coordinate system for the embedded physical module with the agent as the origin and the agent's orientation as the y-axis; form a dynamic threshold by the distance between adjacent radar ranging points and the distance between adjacent radar rays in the quadrant of the Cartesian coordinate system; segment the point cloud by dividing the endpoints of radar rays that meet the dynamic threshold and the preset threshold conditions.

[0010] Step 1.1.1: Establish a Cartesian coordinate system with the agent as the origin and the agent's orientation as the y-axis;

[0011] Step 1.1.2: Convert the radar ray to the quadrant of the Cartesian coordinate system as shown in Equation (1);

[0012]

[0013] In the formula, (x n ,y n ) represents the endpoint coordinates detected by the nth radar ray, L represents the length measured by each radar, and x represents the angle difference between radars;

[0014] Step 1.1.3: Convert the radar ray to the coordinates of a Cartesian coordinate system and use them as the coordinates of the obstacle; form the obstacle from the set of coordinates of the radar ray's endpoint;

[0015] Step 1.1.4: Using the calculation of the distance between adjacent radar ranging points and the distance between adjacent radar rays, the endpoints of radar rays that meet the dynamic threshold and preset threshold conditions are used to segment the point cloud boundary.

[0016] Step 1.1.4.1: Obtain the projected distance between adjacent radar rays in the Cartesian coordinate system using the method shown in equation (2);

[0017] ac=2·min{oa,ob}·sin(θ / 2) (2)

[0018] In the formula, ac represents the distance between adjacent radar rays, θ represents the angular increment of the laser beam, and oa and ob represent two adjacent radar rays projected onto the wall;

[0019] Step 1.1.4.2: Obtain the dynamic segmentation threshold as shown in equation (3) based on the projection distance between adjacent radar rays;

[0020]

[0021] In the formula, t threshold This represents the dynamic segmentation threshold, ab represents the distance between adjacent radar ranging points, and ac represents the distance between adjacent radar rays;

[0022] Step 1.1.4.3: Filter the distances between adjacent radar ranging points and the distances between adjacent radar rays to obtain the range of dynamic threshold values;

[0023] Step 1.1.4.3.1: When the distance between adjacent radar ranging points tends to +∞ and the distance between adjacent radar rays tends to 0, the radar ray is parallel to the straight line where the obstacle is located, and the dynamic segmentation threshold tends to +∞.

[0024] Step 1.1.4.3.2: When any two radar rays symmetrically oriented towards the agent are equal, the dynamic segmentation threshold tends to 1;

[0025] Step 1.1.4.4: Set a predefined threshold and segment the point cloud by the endpoint of the radar ray that meets the dynamic threshold shown in Equation (4);

[0026] t threshold >t set (4)

[0027] In the formula, t threshold Indicates the dynamic threshold, t set This represents a predefined threshold.

[0028] Step 1.2: Based on the boundary segmentation, the environmental region subsets of the point cloud are divided, and the polygon fitting method is used to extract the diagonal features and the slope and intercept of the boundary line formed by the initial and final point cloud coordinates of the linear point in the environmental region subset are used to extract the line features.

[0029] Step 1.2.1: Divide the point cloud into environmental regions using boundary segmentation;

[0030] Step 1.2.2: Extract angular features from a subset of the environmental region using a polygon fitting method;

[0031] Step 1.2.2.1: Construct the boundary line formed by the initial and final point cloud coordinates of the environmental region subset as shown in equation (5);

[0032]

[0033] In the formula, (x1, y1) represents the first coordinate of the environmental region subset, (x n ,y n () represents the last coordinate of the subset of the environment region;

[0034] Step 1.2.2.2: Obtain the perpendicular distance from each point in the environmental region subset to the boundary line using the method shown in equation (6);

[0035]

[0036] In the formula, L i Let x represent the vertical distance from each point in the region, and n represent the total number of points in the environmental region subset; where (x i ,y i The condition Ax + By + C = 0 is satisfied; furthermore, A = y1 - y n B = x n -x1, C=x1y n -x n y1;

[0037] Step 1.2.2.3: Set a predefined angle threshold, compare the maximum value of the vertical distance with the predefined angle threshold, and take the point that satisfies the expression (7) as the first turning point; take the point that does not satisfy the expression (7) as the linear point;

[0038] max{L i C threshold (7)

[0039] In the formula, max{L i} represents the maximum value of all calculated vertical distances, C threshold Indicates a predefined angle threshold;

[0040] Step 1.2.2.4: Use the first corner point as the boundary segment, and use Step 1.2.2.3 to judge the second corner point; if the second corner point exists, mark the second corner point; if the second corner point does not exist, retain the first corner point;

[0041] Step 1.2.2.5: Represent the corner point as an angle feature using the method shown in equation (8);

[0042] (x1,y1,x n ,y n ,x j ,y j ,L j ,j) (8)

[0043] In the formula, j is the index of the radar ray; (x j ,y j ) represents the corner point; L j This is the perpendicular distance from the corner point to the boundary line;

[0044] Step 1.2.3: Express the slope k and intercept b of the boundary line formed by the coordinates of the beginning and end points of the subset of the environmental region where the linear points are located as the line features shown in Equation (9);

[0045] (x1,y1,x n ,y n ,k,b) (9)

[0046] Step 1.3: Divide the task mode into straight sub-mode and turning sub-mode. Recognize the straight sub-mode based on line features of the environmental region subset, and recognize the turning sub-mode based on the angular features in the continuous and discrete types of the environmental region subset.

[0047] Step 1.3.1: Construct a task pattern consisting of a straight-line sub-pattern and a turning sub-pattern for a subset of the environment region; divide the turning sub-pattern into left turn mode, right turn mode, sharp left turn mode and sharp right turn mode according to the left and right directions;

[0048] Step 1.3.2: Identify straight sub-patterns using line features;

[0049] Step 1.3.3: Identify turning patterns using angular features;

[0050] Step 1.3.3.1: Divide the environmental region subset into continuous environmental region subsets and discrete environmental region subsets;

[0051] Step 1.3.3.2: For a continuous environmental region subset, identify the corner feature with only the first corner point as the turning mode, and identify the corner feature with the second corner point as the sharp turn mode; in the left and right direction division, divide the turning mode and sharp turn mode into left turn mode, right turn mode, left sharp turn mode and right sharp turn mode according to the environmental region subset index number.

[0052] Step 1.3.3.3: Divide the discrete environment region subset into three sub-regions, compare the distances between adjacent sub-regions to divide the left and right directions, identify the sub-region located in the middle position with angular features as a sharp turn pattern, and identify the sub-region located in the middle position without angular features as a turn pattern;

[0053] Step 2: Construct and utilize SAC deep reinforcement learning to train a robust neural network with embedded physics knowledge in the penultimate hidden layer of a hybrid neural network that integrates embedded physics modules into the network in parallel with the reinforcement learning.

[0054] Step 2.1: Connect the embedded physics module in parallel to the penultimate hidden layer of the hybrid mode neural network of combined reinforcement learning to form a robust neural network with embedded physics knowledge.

[0055] Step 2.2: Train the robust neural network with embedded physical knowledge using SAC deep reinforcement learning;

[0056] Step 3: Input the radar data into the trained robust neural network with embedded physical knowledge, and perform feature fusion through full connection to form the steering angle. The agent then performs the corresponding action in real time along the task trajectory to avoid obstacles based on the steering angle.

[0057] Step 3.1: Input radar data into a robust neural network with embedded physical knowledge, and obtain perception data formed by angular features and line features and recognition data for task pattern recognition through the embedded physical module;

[0058] Step 3.2: Input the perceived data and recognition data into the robust neural network with embedded physical knowledge, and fully connect the penultimate hidden layer and the last hidden layer of the robust neural network to form feature fusion;

[0059] Step 3.3: Output the steering angle by passing the fused features through a robust neural network with embedded physical knowledge;

[0060] Step 3.4: The agent generates actions based on the steering angle and interacts with the task trajectory formed by the permutation and combination of task patterns;

[0061] Step 3.4.1: Arrange the task patterns in random combinations to form a task trajectory;

[0062] Step 3.4.2: The agent performs corresponding actions in real time along the task trajectory according to the steering angle to avoid obstacles;

[0063] Compared with existing technologies, it has the following beneficial effects:

[0064] This invention addresses the problem of controller perturbations caused by complex and unpredictable environmental changes by proposing a robust neural network controller based on embedded physical knowledge. The core idea is to propose a combinatorial reinforcement learning framework that integrates multi-physics coupling knowledge. By introducing multi-physics coupling knowledge and a hybrid neural network structure during the combinatorial reinforcement learning training process, the safety and stability of the system under uncertain environments are improved. A hybrid neural network structure is designed to train the neural network controller in combinatorial reinforcement learning. By embedding multi-physics coupling knowledge into the intermediate layers of the neural network, real-time guidance is provided to the network, effectively responding to real-time environmental conditions. A sub-task pattern recognition method based on environmental geometric feature calculation is proposed. Key environmental geometric features are calculated in real time using radar sensing data, and sub-task pattern recognition is effectively performed to enhance the controller's perception capability in complex scenes. This invention enables the training of a more robust neural network controller using embedded physical knowledge. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the control method of the present invention;

[0066] Figure 2 This is a schematic diagram of the radar ray model of the present invention;

[0067] Figure 3 This is a visualization of LiDAR data during normal driving in the mission mode of this invention;

[0068] Figure 4 This is a schematic diagram of the neural network structure of the present invention;

[0069] Figure 5 This is a schematic diagram of the combined reinforcement learning framework of the present invention;

[0070] Figure 6 This is a schematic diagram of the F1 / 10th environment training of the present invention;

[0071] Figure 7 This is a graph showing the collision count results after adding noise to the sensing data according to the present invention;

[0072] Figure 8 This is a graph showing the number of collisions after adding noise to the control data according to the present invention. Detailed Implementation

[0073] To better illustrate the purpose and advantages of this invention, the invention will be further described below with reference to the accompanying drawings and examples. It should be noted that the implementation of this invention is not limited to the following embodiments, and any modifications or alterations made to this invention will fall within the scope of protection of this invention.

[0074] Example

[0075] like Figure 1 As shown in the figure, the specific implementation steps of a robust neural network control method based on embedded physical knowledge in this embodiment are as follows:

[0076] Step 1: Construct an embedded physical module for combining hybrid-mode neural networks for reinforcement learning; In the embedded physical module, extract angular and line features from the environmental region subset divided by the point cloud in a Cartesian coordinate system with the agent as the origin and the agent's orientation as the y-axis, and use the angular and line features to identify the task pattern.

[0077] Step 1.1: Construct a Cartesian coordinate system for the embedded physical module with the agent as the origin and the agent's orientation as the y-axis; form a dynamic threshold by the distance between adjacent radar ranging points and the distance between adjacent radar rays in the quadrant of the Cartesian coordinate system; segment the point cloud by dividing the endpoints of radar rays that meet the dynamic threshold and the preset threshold conditions.

[0078] Step 1.1.1: Establish a Cartesian coordinate system with the agent as the origin and the agent's orientation as the y-axis;

[0079] Step 1.1.2: Convert the radar ray to the quadrant of the Cartesian coordinate system as shown in Equation (1);

[0080]

[0081] In the formula, (x n ,y n ) represents the endpoint coordinates detected by the nth radar ray, L represents the length measured by each radar, and x represents the angle difference between radars;

[0082] In this embodiment, a Cartesian coordinate system is established with the agent as the origin and the agent's orientation as the y-axis. The agent has a total of 1081 LiDAR rays, which are evenly distributed counterclockwise in the range of [-120°, 120°]. The 120° position is the first LiDAR ray, and the -120° position is the 1081st LiDAR ray. Based on their distribution and length characteristics, a total of four quadrant transformations are involved. Taking the n=100th LiDAR ray with a ray length L=10 meters as an example.

[0083]

[0084] Step 1.1.3: Convert the radar ray to the coordinates of a Cartesian coordinate system and use them as the coordinates of the obstacle; form the obstacle from the set of coordinates of the radar ray's endpoint;

[0085] Step 1.1.4: Using the calculation of the distance between adjacent radar ranging points and the distance between adjacent radar rays, the endpoints of radar rays that meet the dynamic threshold and preset threshold conditions are used to segment the point cloud boundary.

[0086] Step 1.1.4.1: Obtain the projected distance between adjacent radar rays in the Cartesian coordinate system using the method shown in equation (2);

[0087] ac=2·min{oa,ob}·sin(θ / 2) (2)

[0088] In the formula, ac represents the distance between adjacent radar rays, θ represents the angular increment of the laser beam, and oa and ob represent two adjacent radar rays projected onto the wall;

[0089] In the embodiments, such as Figure 2As shown, this is a model of radar ray projection, where θ represents the angular increment of the radar ray. Let oa and ob represent two adjacent radar rays projected onto the wall; calculate the distance ac between adjacent radar rays, assuming oa = oc = 8 meters, ob = 8.5 meters, and the angular increment θ = 2° / 9, the calculation yields...

[0090]

[0091] Step 1.1.4.2: Obtain the dynamic segmentation threshold as shown in equation (3) based on the projection distance between adjacent radar rays;

[0092]

[0093] In the formula, t threshold This represents the dynamic segmentation threshold, ab represents the distance between adjacent radar ranging points, and ac represents the distance between adjacent radar rays;

[0094] Step 1.1.4.3: Filter the distances between adjacent radar ranging points and the distances between adjacent radar rays to obtain the range of dynamic threshold values;

[0095] Step 1.1.4.3.1: When the distance between adjacent radar ranging points tends to +∞ and the distance between adjacent radar rays tends to 0, the radar ray is parallel to the straight line where the obstacle is located, and the dynamic segmentation threshold tends to +∞.

[0096] Step 1.1.4.3.2: When any two radar rays symmetrically oriented towards the agent are equal, the dynamic segmentation threshold tends to 1;

[0097] Step 1.1.4.4: Set a predefined threshold and segment the point cloud by the endpoint of the radar ray that meets the dynamic threshold shown in Equation (4);

[0098] t threshold >t set (4)

[0099] In the formula, t threshold Indicates the dynamic threshold, t set This represents a predefined threshold.

[0100] In the embodiments, such as Figure 2 As shown, the dynamic threshold is calculated based on the values ​​from oa to ob, and the dynamic threshold range is obtained for different cases, assuming ab = 0.1 meters. The dynamic threshold is obtained through calculation. If ab→+∞ and ac→0, the radar ray is parallel to the line containing the obstacle, thus making the threshold t... threshold →+∞;When od=oe, t threshold ≡1; therefore, t threshold∈[1,+∞]. Introduce a predefined threshold t. set Assuming threshold t set =2. Since 3.226 > 2, the radar ray endpoints are used to segment the point cloud boundary;

[0101] Step 1.2: Based on the boundary segmentation, the environmental region subsets of the point cloud are divided, and the polygon fitting method is used to extract the diagonal features and the slope and intercept of the boundary line formed by the initial and final point cloud coordinates of the linear point in the environmental region subset are used to extract the line features.

[0102] Step 1.2.1: Divide the point cloud into environmental regions using boundary segmentation;

[0103] Step 1.2.2: Extract angular features from a subset of the environmental region using a polygon fitting method;

[0104] Step 1.2.2.1: Construct the boundary line formed by the initial and final point cloud coordinates of the environmental region subset as shown in equation (5);

[0105]

[0106] In the formula, (x1, y1) represents the first coordinate of the environmental region subset, (x n ,y n () represents the last coordinate of the subset of the environment region;

[0107] In this embodiment, the boundary lines are defined within each segmented region using the coordinates of the first and last subsets of the environment. Assume the first subset of the environment has coordinates (x1, y1) = (2, 3), and the last subset has coordinates (x...). n ,y n The calculation of (6,7) yields the following results:

[0108]

[0109] Step 1.2.2.2: Obtain the perpendicular distance from each point in the environmental region subset to the boundary line using the method shown in equation (6);

[0110]

[0111] In the formula, L i Let x represent the vertical distance from each point in the region, and n represent the total number of points in the environmental region subset; where (x i ,y i The condition Ax + By + C = 0 is satisfied; furthermore, A = y1 - y n B = x n -x1, C=x1y n -x ny1;

[0112] In the embodiment, it is assumed that the coordinates (x, y) of a point cloud of a subset of the environmental region are... i ,y i )=(4,5); then (x i ,y i The following condition must be met: Ax + By + C = -x + y - 1 = 0; where A = y1 - y n =3-7=-4, B=x n -x1=6-2=4,C=x1y n -x n y1 = 2 × 7 - 6 × 3 = -4; therefore, the vertical distance is:

[0113] Step 1.2.2.3: Set a predefined angle threshold, compare the maximum value of the vertical distance with the predefined angle threshold, and take the point that satisfies the expression (7) as the first turning point; take the point that does not satisfy the expression (7) as the linear point;

[0114] max{L i C threshold (7)

[0115] In the formula, max{L i} represents the maximum value of all calculated vertical distances, C threshold Indicates a predefined angle threshold;

[0116] In this embodiment, the maximum value of all calculated vertical distances is compared with a predefined angle threshold. It is assumed that the maximum value of the calculated vertical distance is max{L}. i} = 0.5, predefined angle threshold C threshold =0.3. Since 0.5 > 0.3, the recorded coordinates belong to the 100th LiDAR ray (angle index) and coordinates (4,5) (angular coordinates). The detected angle is used as the new segmentation boundary; there is no second corner point.

[0117] Step 1.2.2.4: Use the first corner point as the boundary segment, and use Step 1.2.2.3 to judge the second corner point; if the second corner point exists, mark the second corner point; if the second corner point does not exist, retain the first corner point;

[0118] Step 1.2.2.5: Represent the corner point as an angle feature using the method shown in equation (8);

[0119] (x1,y1,x n ,y n ,x j ,y j ,L j ,j) (8)

[0120] In the formula, j is the index of the radar ray; (x j ,y j ) represents the corner point; L j This is the perpendicular distance from the corner point to the boundary line;

[0121] In this embodiment, the extracted angular features are represented as (x1, y1, x...). n ,y n ,x i ,y i ,L i ,i)=(2,3,6,7,4,5,0.5,100);

[0122] Step 1.2.3: Express the slope k and intercept b of the boundary line formed by the coordinates of the beginning and end points of the subset of the environmental region where the linear points are located as the line features shown in Equation (9);

[0123] (x1,y1,x n ,y n In the embodiment (9), the slope of the line is calculated using the starting and ending points corresponding to the region. The line is fitted to a straight line using the two-point method and the least squares method to obtain the line intercept property b = y - kx = 3 - 1 × 2 = 1. The extracted line features can be represented as (x1, y1, x n ,y n (k,b)=(2,3,6,7,1,1).

[0124] Step 1.3: Divide the task mode into straight sub-mode and turning sub-mode. Recognize the straight sub-mode based on line features of the environmental region subset, and recognize the turning sub-mode based on the angular features in the continuous and discrete types of the environmental region subset.

[0125] Step 1.3.1: Construct a task pattern consisting of a straight-line sub-pattern and a turning sub-pattern for a subset of the environment region; divide the turning sub-pattern into left turn mode, right turn mode, sharp left turn mode and sharp right turn mode according to the left and right directions;

[0126] Step 1.3.2: Identify straight sub-patterns using line features;

[0127] Step 1.3.3: Identify turning patterns using angular features;

[0128] Step 1.3.3.1: Divide the environmental region subset into continuous environmental region subsets and discrete environmental region subsets;

[0129] Step 1.3.3.2: For a continuous environmental region subset, identify the corner feature with only the first corner point as the turning mode, and identify the corner feature with the second corner point as the sharp turn mode; in the left and right direction division, divide the turning mode and sharp turn mode into left turn mode, right turn mode, left sharp turn mode and right sharp turn mode according to the environmental region subset index number.

[0130] Step 1.3.3.3: Divide the discrete environment region subset into three sub-regions, compare the distances between adjacent sub-regions to divide the left and right directions, identify the sub-region located in the middle position with angular features as a sharp turn pattern, and identify the sub-region located in the middle position without angular features as a turn pattern;

[0131] In this embodiment, the turning mode is determined based on different scenarios during subtask mode identification; for example... Figure 3 As shown in the three images below, in general, the number of angular features is used to distinguish between sharp turns and right-angle turns. The order of the area segments is obtained, and based on this, it can be directly determined whether it is a left turn or a right turn. If area 1 contains angular features, it is a left turn; if area 2 contains angular features, it is a right turn. For example... Figure 3 As shown in the Square_right diagram, for a turning angle identified as two line segments (i.e., the number of region segments becomes 3), parameters δ1 and δ2 are introduced to represent the distances between each region segment. This helps determine whether the segment division is due to the radar's discrete nature or the width of the trajectory itself. δ1 is the distance between the end coordinates of the first region segment and the starting coordinates of the second region segment, and δ2 is the distance between the end coordinates of the second region segment and the starting coordinates of the third region segment. The magnitudes of these two values ​​distinguish between left and right turns. Furthermore, based on the visualization, we can determine whether it's a sharp turn or a right-angle turn by checking if the second region segment contains a corner.

[0132] Step 2: Construct and utilize SAC deep reinforcement learning to train a robust neural network with embedded physics knowledge in the penultimate hidden layer of a hybrid neural network that integrates embedded physics modules into the network in parallel with the reinforcement learning.

[0133] Step 2.1: Connect the embedded physics module in parallel to the penultimate hidden layer of the hybrid mode neural network of combined reinforcement learning to form a robust neural network with embedded physics knowledge.

[0134] Step 2.2: Train the robust neural network with embedded physical knowledge using SAC deep reinforcement learning;

[0135] In the embodiments, such as Figure 4As shown, the physical module with embedded physical knowledge processes the perceived data, extracts valuable perceived features as perceived data parameters containing physical knowledge, adds them to the penultimate hidden layer of the neural network as the input of the last hidden layer, and performs deep information fusion through a fully connected structure to finally output control signals, thereby embedding physical knowledge into the neural network and training a hybrid mode neural network controller with embedded physical knowledge.

[0136] like Figure 5 As shown, during training, the trajectory is divided into five basic sub-trajectories, and a series of trajectories formed by arbitrary permutations and combinations of these sub-trajectories are generated through adversarial generation. For each sub-trajectory in this permutation and combination, the SAC deep reinforcement learning algorithm is employed. Perceptual and recognition data are added to the experience buffer, enabling the trained controller to effectively perceive key information about the surrounding environment. Guided training is then conducted using the multi-physics coupling knowledge present within this buffer, allowing the policy function based on this knowledge to make safer actions. For the actor and q-networks used in deep reinforcement learning, all networks are modified to hybrid-mode neural networks. During training, the gradient update method of the update function is modified to accommodate more input parameters, the initial entropy value is increased to enhance the model's exploratory nature, and the learning rate of the model itself is reduced to balance parameter updates.

[0137] Step 3: Input the radar data into the trained robust neural network with embedded physical knowledge, and perform feature fusion through full connection to form the steering angle. The agent then performs the corresponding action in real time along the task trajectory to avoid obstacles based on the steering angle.

[0138] Step 3.1: Input radar data into a robust neural network with embedded physical knowledge, and obtain perception data formed by angular features and line features and recognition data for task pattern recognition through the embedded physical module;

[0139] Step 3.2: Input the perceived data and recognition data into the robust neural network with embedded physical knowledge, and fully connect the penultimate hidden layer and the last hidden layer of the robust neural network to form feature fusion;

[0140] Step 3.3: Output the steering angle by passing the fused features through a robust neural network with embedded physical knowledge;

[0141] Step 3.4: The agent generates actions based on the steering angle and interacts with the task trajectory formed by the permutation and combination of task patterns;

[0142] Step 3.4.1: Arrange the task patterns in random combinations to form a task trajectory;

[0143] Step 3.4.2: The agent performs corresponding actions in real time along the task trajectory according to the steering angle to avoid obstacles;

[0144] In this embodiment, the extracted environmental geometric features and sub-task patterns are embedded into a hybrid-mode neural network, and the LiDAR data (dimension 1) is combined with these features. As the initial input data, the perceived data dimension is... LiDAR data and extracted sensory data are fed together into the subtask pattern recognition module to obtain recognition data, with dimensions of [missing information]. The perceived data and the recognition data were added together to the penultimate hidden layer of the neural network, serving as the input to the final hidden layer. The penultimate hidden layer of the hybrid neural network has a dimension of... The last dimension is It also performs deep information fusion through a fully connected structure, and finally outputs a control signal, namely the steering angle ω.

[0145] To evaluate the performance of this invention, a comparative experiment was conducted between this invention and existing robust neural network controllers. The chosen comparison method is the robust neural network controllers NAIVE and ROSAC in combinatorial reinforcement learning, which are most closely related to this invention.

[0146] This invention is trained and tested in a multi-task environment (F1 / 10th racing environment). The invention uses a publicly available F1 / 10th simulator that takes LiDAR measurements as input as sensing signals and steering angles as output as control signals. As shown in previous work, individual neural networks can be trained to predict the completion of each sub-task; therefore, this invention assumes that the controller can detect when each sub-task is completed.

[0147] This invention evaluates the performance of these algorithms in two modes. The first mode is a randomized mode, which randomly selects the next subtask from all subtask sets. The second mode utilizes the Monte Carlo Tree Search (MCTS) algorithm to estimate the worst-case sequence of subtasks for a given set of options. In MCTS mode, a reward of 1 is assigned when a subtask that cannot be completed within a fixed time limit is selected, and a reward of 0 is assigned otherwise. For the F1 / 10th environment, this invention considers subtask sequences with a maximum length of 25. This invention evaluates the average number of completed subtasks and the probability of completing the maximum set of subtasks.

[0148] like Figure 6The diagram shows the training process for two validation modes in the F1 / 10th environment. The x-axis represents the number of sample steps, and the y-axis represents the average number of subtasks that can be safely completed or the probability of completing a combination of 25 consecutive subtasks. This invention performed 5 training iterations for each method and averaged the results. The error bar represents ± standard deviation. It can be observed that the NAIVE method cannot complete the training of long sequence tasks, and the ROSAC method requires 2.5 * 10^5 training steps to complete training. Compared with other methods, the method of this invention completes training with fewer training steps in both random and MCTS validation modes.

[0149] To illustrate the robustness of the controller trained by the method of this invention, different levels of Gaussian noise were applied to the sensing data (radar data) and control data (azimuth angle), with a constant mean of 0. The changes in the adversarial frequency were then compared. Adversarial frequency is an important indicator of robustness, referring to the number of inputs in which the model fails to maintain robustness for a given disturbance magnitude. Since the NAI VE method cannot complete the training, this invention only compares the ROSAC algorithm.

[0150] The results are as follows Figure 7 , Figure 8 As shown, Figure 7 To counteract frequency variations when different levels of noise are added to the sensed data, Figure 8 To combat frequency variations under different levels of noise introduced into the control data, the ROSAC controller began to collide when Gaussian noise with a standard deviation of 0.4 was introduced into the sensed data, while the controller of this invention began to collide when Gaussian noise with a standard deviation of 0.7 was introduced. Similarly, the ROSAC controller began to collide when Gaussian noise with a standard deviation of 0.3 was introduced into the control data, while the controller of this invention began to collide when Gaussian noise with a standard deviation of 0.7 was introduced. This demonstrates that the controller trained using the method of this invention has better robustness.

[0151] In summary, the robust neural network controller based on embedded physical knowledge disclosed in this invention can train a more robust neural network controller using embedded physical knowledge.

Claims

1. A robust neural network control method based on embedded physical knowledge, characterized in that: Includes the following steps, Step 1: Construct an embedded physical module for combining hybrid-mode neural networks for reinforcement learning; In the embedded physical module, extract angular and line features from the environmental region subset divided by the point cloud in a Cartesian coordinate system with the agent as the origin and the agent's orientation as the y-axis, and use the angular and line features to identify the task pattern. Step 1.1: Construct a Cartesian coordinate system for the embedded physical module with the agent as the origin and the agent's orientation as the y-axis; form a dynamic threshold by the distance between adjacent radar ranging points and the distance between adjacent radar rays in the quadrant of the Cartesian coordinate system; segment the point cloud by dividing the endpoints of radar rays that meet the dynamic threshold and the preset threshold conditions. Step 1.2: Based on the boundary segmentation, the environmental region subsets of the point cloud are divided, and the polygon fitting method is used to extract the diagonal features and the slope and intercept of the boundary line formed by the initial and final point cloud coordinates of the linear point in the environmental region subset are used to extract the line features. Step 1.3: Divide the task mode into straight sub-mode and turning sub-mode. Recognize the straight sub-mode based on line features of the environmental region subset, and recognize the turning sub-mode based on the angular features in the continuous and discrete types of the environmental region subset. Step 1.3.1: Construct a task pattern consisting of a straight-line sub-pattern and a turning sub-pattern for a subset of the environment region; divide the turning sub-pattern into left turn mode, right turn mode, sharp left turn mode and sharp right turn mode according to the left and right directions; Step 1.3.2: Identify straight sub-patterns using line features; Step 1.3.3: Identify turning patterns using angular features; Step 2: Construct and utilize SAC deep reinforcement learning to train a robust neural network with embedded physics knowledge in the penultimate hidden layer of a hybrid neural network that integrates embedded physics modules into the network in parallel with the reinforcement learning. Step 3: Input the radar data into the trained robust neural network with embedded physical knowledge, and perform feature fusion through full connection to form the steering angle. The agent then performs the corresponding action in real time along the task trajectory to avoid obstacles based on the steering angle.

2. The robust neural network control method based on embedded physical knowledge as described in claim 1, characterized in that: Step 1.1 is implemented as follows: Step 1.1.1: Establish a Cartesian coordinate system with the agent as the origin and the agent's orientation as the y-axis; Step 1.1.2: Convert the radar ray to the quadrant of the Cartesian coordinate system as shown in Equation (1); In the formula, (x n ,y n ) represents the endpoint coordinates detected by the nth radar ray, L represents the length measured by each radar, and x represents the angle difference between radars; Step 1.1.3: Convert the radar ray to the coordinates of a Cartesian coordinate system and use them as the coordinates of the obstacle; form the obstacle from the set of coordinates of the radar ray's endpoint; Step 1.1.4: Using the calculation of the distance between adjacent radar ranging points and the distance between adjacent radar rays, the endpoints of radar rays that meet the dynamic threshold and preset threshold conditions are used to segment the point cloud boundary.

3. The robust neural network control method based on embedded physical knowledge as described in claim 2, characterized in that: The implementation method for step 1.1.4 is as follows: Step 1.1.4.1: Obtain the projected distance between adjacent radar rays in the Cartesian coordinate system using the method shown in equation (2); ac=2·min{oa,ob}·sin(θ / 2) (2) In the formula, ac represents the distance between adjacent radar rays, θ represents the angular increment of the laser beam, and oa and ob represent two adjacent radar rays projected onto the wall; Step 1.1.4.2: Obtain the dynamic segmentation threshold as shown in equation (3) based on the projection distance between adjacent radar rays; In the formula, t threshold This represents the dynamic segmentation threshold, ab represents the distance between adjacent radar ranging points, and ac represents the distance between adjacent radar rays; Step 1.1.4.3: Filter the distances between adjacent radar ranging points and the distances between adjacent radar rays to obtain the range of dynamic threshold values; Step 1.1.4.4: Set a predefined threshold and segment the point cloud by the endpoint of the radar ray that meets the dynamic threshold shown in Equation (4); t threshold >t set (4) In the formula, t threshold Indicates the dynamic threshold, t set This represents a predefined threshold.

4. The robust neural network control method based on embedded physical knowledge as described in claim 3, characterized in that: The implementation method for step 1.1.4.3 is as follows: Step 1.1.4.3.1: When the distance between adjacent radar ranging points tends to +∞ and the distance between adjacent radar rays tends to 0, the radar ray is parallel to the straight line where the obstacle is located, and the dynamic segmentation threshold tends to +∞. Step 1.1.4.3.2: When any two radar rays symmetrical to the direction of the agent are equal, the dynamic segmentation threshold tends to 1.

5. The robust neural network control method based on embedded physical knowledge as described in claim 1, characterized in that: Step 1.2 is implemented as follows: Step 1.2.1: Divide the point cloud into environmental regions using boundary segmentation; Step 1.2.2: Extract angular features from a subset of the environmental region using a polygon fitting method; Step 1.2.3: Express the slope k and intercept b of the boundary line formed by the coordinates of the beginning and end points of the subset of the environmental region where the obtained linear points are located as the line feature shown in Equation (9). (x1,y1,x n ,y n ,k,b) (9) 6. The robust neural network control method based on embedded physical knowledge as described in claim 5, characterized in that: The implementation method for step 1.2.2 is as follows: Step 1.2.2.1: Construct the boundary line formed by the initial and final point cloud coordinates of the environmental region subset as shown in equation (5); In the formula, (x1, y1) represents the first coordinate of the environmental region subset, (x n ,y n () represents the last coordinate of the subset of the environment region; Step 1.2.2.2: Obtain the perpendicular distance from each point in the environmental region subset to the boundary line using the method shown in equation (6); In the formula, L i This represents the vertical distance from each point in the region, and n represents the total number of points in the subset of the environmental region. Among them, (x i ,y i The condition Ax + By + C = 0 is satisfied; furthermore, A = y1 - y n B = x n -x1, C=x1y n -x n y1; Step 1.2.2.3: Set a predefined angle threshold, compare the maximum value of the vertical distance with the predefined angle threshold, and take the point that satisfies the expression (7) as the first turning point; take the point that does not satisfy the expression (7) as the linear point; max{L i }>C threshold (7) In the formula, max{L i } represents the maximum value of all calculated vertical distances, C threshold Indicates a predefined angle threshold; Step 1.2.2.4: Use the first corner point as the boundary segment, and use Step 1.2.2.3 to judge the second corner point; if the second corner point exists, mark the second corner point; if the second corner point does not exist, retain the first corner point; Step 1.2.2.5: Represent the corner point as an angle feature using the method shown in equation (8); (x1,y1,x n ,y n ,x j ,y j ,L j ,j) (8) In the formula, j is the index of the radar ray; (x j ,y j ) represents the corner point; L j This is the perpendicular distance from the corner point to the boundary line.

7. The robust neural network control method based on embedded physical knowledge as described in claim 1, characterized in that: The implementation method for step 1.3.3 is as follows: Step 1.3.3.1: Divide the environmental region subset into continuous environmental region subsets and discrete environmental region subsets; Step 1.3.3.2: For a continuous environmental region subset, identify the corner feature with only the first corner point as the turning mode, and identify the corner feature with the second corner point as the sharp turn mode; in the left and right direction division, divide the turning mode and sharp turn mode into left turn mode, right turn mode, left sharp turn mode and right sharp turn mode according to the environmental region subset index number. Step 1.3.3.3: Divide the discrete environment region subset into three sub-regions, compare the distance between adjacent sub-regions to divide the left and right directions, identify the sub-region located in the middle position with angular features as a sharp turn pattern, and identify the sub-region located in the middle position without angular features as a turn pattern.

8. The robust neural network control method based on embedded physical knowledge as described in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Connect the embedded physics module in parallel to the penultimate hidden layer of the hybrid mode neural network of combined reinforcement learning to form a robust neural network with embedded physics knowledge. Step 2.2: Use SAC deep reinforcement learning to train a robust neural network with embedded physical knowledge.

9. The robust neural network control method based on embedded physical knowledge as described in claim 1, characterized in that: Step 3 is implemented as follows: Step 3.1: Input radar data into a robust neural network with embedded physical knowledge, and obtain perception data formed by angular features and line features and recognition data for task pattern recognition through the embedded physical module; Step 3.2: Input the perceived data and recognition data into the robust neural network with embedded physical knowledge, and fully connect the penultimate hidden layer and the last hidden layer of the robust neural network to form feature fusion; Step 3.3: Output the steering angle by passing the fused features through a robust neural network with embedded physical knowledge; Step 3.4: The agent generates actions based on the steering angle and interacts with the task trajectory formed by the permutation and combination of task patterns.

10. The robust neural network control method based on embedded physical knowledge as described in claim 9, characterized in that: Step 3.4 is implemented as follows: Step 3.4.1: Arrange the task patterns in random combinations to form a task trajectory; Step 3.4.2: The agent performs the corresponding action in real time along the task trajectory according to the steering angle to avoid obstacles.

Images