High-speed rail vehicle body plane spraying full-coverage path planning method

Abstract

The invention discloses a high-speed rail vehicle body plane spraying full-coverage path planning method which comprises the following steps: selecting a high-speed rail plane vehicle body to be planned, and determining a spraying area; building a grid map for path planning; different gray values represent the states, the repetition degrees and the real-time positions of the Agents of different grid units, and a gray state matrix is formed; defining the action direction of the Agent; constructing a multi-level feature aggregation network model as an Agent structure; gray values of movable grid units around the current position of the Agent are converted into a direction probability matrix through calculation, and the direction probability matrix acts on network output of the Agent; formulating a reward function and a path quality evaluation index; and training a multi-level feature aggregation network model in the constructed spraying environment, and selecting an optimal path according to a path quality evaluation index. The spraying device can act on any planar vehicle body with different sizes and different door and window configurations, so that the spraying coverage rate reaches 100%, and meanwhile, the extremely low repetition rate is guaranteed.

Description

Technical Field

[0001] The invention relates to the field of intelligent industry, and in particular to a method for planning a path for full coverage of a flat surface spraying of a high-speed railway body. Background Art

[0002] Full coverage path planning technology is an important research topic in many fields, such as industrial automation robots, unmanned driving and signal communication. At present, the high-speed rail body spraying process mainly relies on manual spraying, and the experience and physical condition of workers significantly affect the spraying quality, making it difficult to ensure the uniformity and thickness of the layer. Automated spraying robots are considered to be able to replace manual spraying due to their high efficiency and stability, but the automatic putty spraying process is still in the exploration and trial stage, and there are few application examples.

[0003] In automated spraying, the spray path has a significant impact on the thickness and uniformity of the final coating. Repeated paths and incorrect paths will produce "wavy" coatings, which will affect coating performance. The paths generated by the commonly used coverage path planning methods, on the one hand, have insufficient coverage, making it impossible to carry out spraying tasks. On the other hand, too high a repetition rate will lead to paint loss, and in order to ensure the flatness of the vehicle surface, the spray gun needs to be temporarily closed on the repeated path. If the spray gun is closed for too long, the paint may solidify, which may cause the spray gun to become clogged. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a high-speed railway body flat surface spray full coverage path planning method in response to the above-mentioned deficiencies in the prior art. The high-speed railway body flat surface spray full coverage path planning method can be applied to flat bodies of any different sizes and different door and window configurations, so that the spray coverage rate reaches 100% while ensuring an extremely low repetition rate.

[0005] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0006] A method for planning a full coverage path for spraying a flat surface of a high-speed railway body, comprising:

[0007] Step 1: Select the high-speed rail flat body to be planned and determine the spraying area; select a suitable spray gun and build a grid map for path planning;

[0008] Step 2: Different grayscale values ​​represent the status, repetition degree and real-time position of different grid cells, and form a grayscale state matrix; define the direction of the agent's action;

[0009] Step 3: Build a multi-level feature aggregation network model as the Agent structure in the Rainbow reinforcement learning framework;

[0010] Step 4: The grayscale values ​​of the movable grid cells around the current position of the agent are converted into a direction probability matrix through calculation and acted on the network output of the agent;

[0011] Step 5: Considering the environmental coverage and grid unit repetition, formulate the reward function and path quality evaluation index;

[0012] Step 6: Train a multi-level feature aggregation network model in the constructed spraying environment and select the optimal path based on the path quality evaluation index.

[0013] As a further improved technical solution of the present invention, the step 1 is specifically as follows:

[0014] Step 1.1, select the high-speed rail plane body to be planned, and determine the spraying position on the body surface;

[0015] Step 1.2, select a spray gun with a fan size of d, and establish a path planning grid map according to the plane size H×C of the high-speed rail body and the position of the obstacle. The edge of the grid map is filled with a circle of obstacles. The grid unit in the grid map is a square with a side length of one unit. The size of the grid map is:

[0016]

[0017] Among them, H is the vertical length of the vehicle body, C is the horizontal length of the vehicle body, Grid H Grid is the number of grid cells in the vertical direction of the grid map. C It is the number of grid cells in the horizontal direction of the grid map.

[0018] As a further improved technical solution of the present invention, the step 2 is specifically as follows:

[0019] Step 2.1, use the gray value g to represent the state of different grid cells, the number of repetitions and the real-time position of the Agent. Each grid cell in the grid map will be assigned a gray value, thus forming a gray state matrix:

[0020]

[0021] Among them, q is the gray value, N is the number of grid unit repetitions, and σ is the gray value attenuation amplitude;

[0022] Step 2.2: The Agent moves in four directions: up, down, left, and right, and the step length is 1 grid unit. When moving, the Agent cannot move to the obstacle area.

[0023] As a further improved technical solution of the present invention, the step 3 is specifically:

[0024] Step 3.1, construct a multi-level feature aggregation network model as the agent structure in the Rainbow reinforcement learning framework, including an input processing module, two basic modules, two tracking modules, an aggregation module, a flattening layer, an advantage value fully connected layer, and an action value fully connected layer;

[0025] Step 3.2, the input processing module includes a 3×3 convolutional layer with a stride of 1, a layer normalization, and a ReLU activation function layer;

[0026] Step 3.2.1: The gray state matrix calculated in step 2.1 is input into the processing module, and the output feature is A. The feature map size remains unchanged, and the feature information is extracted while retaining the spatial position information.

[0027] Step 3.3, the basic module includes two convolution blocks consisting of a 3×3 convolution layer, layer normalization, a ReLU activation function layer, a 3×3 convolution layer and layer normalization, and a channel adjustment layer consisting of a 3×3 convolution layer and layer normalization;

[0028] The output feature A of step 3.3.1 and step 3.2.1 passes through the first convolution block in the first basic module, the feature map size is halved, and the output feature B1; feature A passes through the channel adjustment layer in the first basic module, the feature map size is halved, and the output feature B2; the outputs of the two are merged B1+B2=B3, feature B3 passes through the Relu activation function layer and is processed by the second convolution block in the first basic module, the feature map size remains unchanged, and the output feature B4; the residual connection merges B3+B4=B5, and feature B5 passes through the ReLU activation function layer to obtain the activated feature B6;

[0029] Step 3.3.2, feature B6 passes through the first convolution block in the second basic module, the feature map size remains unchanged, the number of channels is doubled, and the output feature C1; feature B6 passes through the channel adjustment layer in the second basic module, the feature map size remains unchanged, the number of channels is doubled, and the output feature C2; ​​the outputs of the two are merged C1+C2=C3, feature C3 passes through the ReLU activation function layer, and is processed by the second convolution block in the second basic module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature C4; the residual connection merges C3+C4=C5, and feature C5 passes through the ReLU activation function layer to obtain the activated feature C;

[0030] Step 3.4, the first tracking module includes two convolution blocks consisting of 3×3 convolution layers, layer normalization and ReLU activation function layers, a channel adjustment layer consisting of 3×3 convolution layers and layer normalization, and an extraction layer consisting of 3×3 convolution layers and ReLU activation function layers; the second tracking module includes two convolution blocks consisting of 3×3 convolution layers, layer normalization and ReLU activation function layers, and an extraction layer consisting of 3×3 convolution layers and ReLU activation function layers;

[0031] Step 3.4.1, pass the output feature C of step 3.3.2 through the first convolution block in the first tracking module, the feature map size remains unchanged, and the output feature D1; the residual connection merges C+D1=D2, and the feature D2 passes through the second convolution block in the first tracking module again, the feature map size remains unchanged, and the output feature D3; the residual connection merges D2+D3=D4, and the feature D4 passes through the extraction layer in the first tracking module, the feature map size is halved, and the number of channels is doubled, and the output feature D5; the feature C passes through the channel adjustment layer in the first tracking module, the feature map size is halved, and the number of channels is doubled, and the output feature D6; the outputs of the two are merged D5+D6=D7 to obtain feature D7;

[0032] Step 3.4.2, feature D7 passes through the first convolution block in the second tracking module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E1; residual connection merges D7+E1=E2, feature E2 passes through the second convolution block in the second tracking module again, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E3; residual connection merges E2+E3=E4, feature E4 passes through the extraction layer in the second tracking module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E5; residual connection merges E5+D7=E to obtain feature E;

[0033] Step 3.5, the aggregation module contains two synthesis blocks consisting of a 3×3 convolutional layer and an upsampling layer;

[0034] The output feature E of step 3.5.1 and step 3.4.2 passes through the first synthesis block, the feature map size is doubled, the number of channels is halved, and the output feature F1 is output; feature F1 is merged with the output feature C of step 3.3.2 by residual connection, that is, C+F1=F2, to obtain feature F2;

[0035] Step 3.5.2, feature F2 passes through the second synthesis block, the feature map size remains unchanged, the number of channels is halved, and feature F3 is output; feature F3 is merged with the output feature B6 of step 3.3.1 by residual connection, that is, F3+B6=F4, to obtain feature F4;

[0036] Step 3.6: After the feature F4 is flattened by the flattening layer, it is processed by the advantage value fully connected layer and the action value fully connected layer respectively to obtain the advantage value Advantage and the action value Value. After the two are processed by formula (3), the output O of the Agent is obtained:

[0037]

[0038] in, represents the average value of Advantage, and O represents the initial Q value array.

[0039] As a further improved technical solution of the present invention, the step 4 is specifically as follows:

[0040] Step 4.1. In the current grayscale state matrix, find the current position of the Agent, that is, the grid cell with a grayscale value of 170, and add the grayscale values ​​of its four adjacent grid cells above, below, left, and right to the array Pixel;

[0041] Step 4.2: The elements in the array Pixel are modified to corresponding values ​​according to the rule of formula (4):

[0042]

[0043] Among them, Pixel[i] represents the value of the i-th element in the Pixel array;

[0044] Step 4.3, the pixel array is processed by the SoftMax function to become the direction probability array Action Mask;

[0045] Step 4.4: Perform the following operations according to formula (5): Action Mask acts on the output O of the Agent in step 3.6 to obtain the corrected Q value array, and replace the elements with a value of 0 in the corrected Q value array with -∞:

[0046]

[0047] As a further improved technical solution of the present invention, the step 5 is specifically as follows:

[0048] Step 5.1: Create a reward function, give different reward values ​​to the agent according to its different behaviors, and quantify the agent's actions into specific values:

[0049]

[0050] Among them, s i represents the reward generated by the i-th action, i represents the i-th step, V frepresents the effective action step reward base, α represents the discount factor of the effective action step, V n Indicates the number of valid action steps, i is Valid Step means the i-th step is a valid step, I f represents the base of the invalid action step reward, β represents the discount factor of the number of grid unit repetitions, Gr represents the number of grid repetitions, i is Invalid Step represents that the i-th step is an invalid step, EPR represents the full coverage completion reward, and Cover Ratio represents the coverage ratio;

[0051] Step 5.2: Formulate path evaluation indicators. Path evaluation indicators include environment coverage rate c, path repetition rate r and path total return R. Meeting the environment coverage rate c is the prerequisite for completing the task. Path repetition rate r and path total return R are the basis for judging the optimal solution of the coverage path:

[0052]

[0053] R=∑S n (9);

[0054] Among them, N c is the number of covered grid cells, N is the total number of grid cells to be covered in the environment, S is the number of steps consumed by the coverage path, and S n is the reward generated by the nth action.

[0055] As a further improved technical solution of the present invention, the step 6 is specifically as follows:

[0056] Step 6.1: Construct the spraying environment of step 1.2 and define the training parameters of reinforcement learning;

[0057] Step 6.2: Reset the spraying environment, Agent starting point, path reward and Reply Buffer. Reply Buffer is used to store past action information.

[0058] Step 6.3: record the gray state matrix of step 2.1 as State, and use State as the input of the multi-level feature aggregation network constructed in step 3. The network output is O;

[0059] Step 6.4: Apply the Action Mask to O according to the method in step 4 to obtain the corrected Q value array, and replace the elements with a value of 0 in the corrected Q value array with -∞; the Agent selects the action a corresponding to the largest element in the corrected Q value array and executes it;

[0060] Step 6.5: After the Agent takes action, the gray state matrix is ​​updated to State′, and the environment feeds back the single-step action reward s according to the reward function. i, task completion mark d, current coverage c, repetition rate r and path total return R;

[0061] Step 6.6, (State, a, s i ,State′,d) is stored in ReplyBuffer as a complete action process;

[0062] Step 6.7: If the number of experiences in the Reply Buffer reaches the preset value, extract Batch Size experiences from it and use them to train the Agent. After the training is completed, execute step 6.8; if the number of experiences in the Reply Buffer does not reach the preset value, directly execute step 6.8;

[0063] Step 6.8: If the task completion flag d is True or the Agent action times reach the maximum, the task is terminated and the process goes to step 6.2 to continue the loop.

[0064] Otherwise, the current gray state matrix is ​​recorded as State, and State is used as the input of the latest multi-level feature aggregation network. The network outputs O and executes step 6.4;

[0065] Step 6.9: If the number of iterations of steps 6.2 to 6.8 reaches the maximum number, exit the task, select the optimal coverage path based on the path evaluation index, and draw a graph.

[0066] In order to achieve the above technical purpose, another technical solution adopted by the present invention is:

[0067] A high-speed railway body flat spray full coverage path planning system, comprising:

[0068] The spraying environment building module is used to build a grid map for path planning based on the plane size of the high-speed rail body, the spray fan size of the spray gun and the location of obstacles;

[0069] The state-action space construction module is used to construct the agent's grayscale state matrix and action space;

[0070] Multi-level feature aggregation network module, used to build a multi-level feature aggregation network model, and use the multi-level feature aggregation network model as the Agent model structure in the Rainbow reinforcement learning framework;

[0071] The action mask module is used to convert the grayscale values ​​of the movable grid cells around the current position of the Agent into a direction probability matrix through calculation, and act on the network output of the Agent;

[0072] The reward and indicator module is used to formulate reward functions and path quality evaluation indicators based on the comprehensive coverage of the environment and the degree of grid unit repetition;

[0073] The model training module is used to train the multi-level feature aggregation network model in the constructed spraying environment and select the optimal path according to the path quality evaluation index.

[0074] The beneficial effects of the present invention are:

[0075] (1) The present invention optimizes the structure of the Agent in the Rainbow reinforcement learning framework and uses a multi-level feature aggregation network model to process environmental grayscale state information. Among them, the input processing module is used to extract simple feature information while retaining spatial position information; the basic module is used to extract shallow semantic information to improve the training stability and convergence speed of the network; the tracking module is used to efficiently capture multi-scale deep features and expand the receptive field. The special structure enables it to process local details and broader contextual information at the same time, effectively balancing feature extraction capabilities, computational efficiency, and network depth; the aggregation module is specifically used to fuse multi-scale features. This module not only improves the network's ability to fuse multi-scale information, but also optimizes the information flow between different levels.

[0076] (2) The present invention proposes an observation-based Action Mask module, which converts the gray value matrix within the feasible range into an action probability matrix and acts on the output of the Agent to guide action selection, thereby eliminating dangerous actions and guiding it to complete the coverage task.

[0077] (3) The present invention proposes a specific reward function to balance the exploration reward and inform the agent of the task goal and the degree of environmental exploration; the reward function guides the agent to effectively cover or escape from the dead zone according to the path coverage degree and the number of unit repetitions. When the agent's behavior is effective, the reward function feedbacks a gradually superimposed positive reward value; when the agent's behavior is repeated, the reward function feedbacks a negative reward value that is proportional to the number of grid unit repetitions; when the agent completes the full coverage task, the reward function will give the agent a huge reward.

[0078] (4) The present invention proposes a full coverage, low repetition rate high-speed rail body flat spraying full coverage path planning method to address the problems of insufficient path coverage generated by traditional commonly used coverage path planning methods, which leads to the inability to carry out spraying tasks, too high a repetition rate leading to paint waste, and too long a spray gun shutdown time causing spray gun clogging. This method combines deep learning with reinforcement learning, integrates a variety of innovative modules, improves the coverage rate of the planned path of the automatic spraying robot, and reduces the repetition rate. At the same time, this method and process are also applicable to other vehicles or planes of similar structures, providing an efficient coverage path planning method for automatic spraying. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1This is a flow chart of the path planning method for full coverage spraying on the flat surface of a high-speed rail body.

[0080] Figure 2 This is a training flow chart of the improved deep reinforcement learning agent of the present invention.

[0081] Figure 3 To spray color environment interface.

[0082] Figure 4 is the gray state matrix and the Agent action direction.

[0083] Figure 5 Algorithm flow chart for multi-level feature aggregation network and action mask.

[0084] Figure 6 This is a diagram of the network module structure in the multi-level feature aggregation network.

[0085] Figure 6 (a) shows the structural diagram of the basic module.

[0086] Figure 6 (b) shows the structural diagram of the tracking module.

[0087] Figure 6 (c) shows the structural diagram of the aggregation module.

[0088] Figure 7 This is a graph showing the changes in coverage and repetition rate corresponding to the number of iterations.

[0089] Figure 8 This is a graph showing the total reward changes corresponding to the number of iterations.

[0090] Fig. 9 This is the optimal path diagram of the improved deep reinforcement learning algorithm in the spraying environment. DETAILED DESCRIPTION

[0091] The specific embodiments of the present invention are further described below according to the accompanying drawings:

[0092] A method for planning a full coverage path for spraying a high-speed railway body surface, such as Figure 1 As shown, the following steps are included:

[0093] Step 1: Model the spraying environment. According to the plane size of the high-speed rail body, the spray fan size of the spray gun and the location of obstacles, a grid map for path planning is built.

[0094] The step 1 is specifically as follows:

[0095] Step 1.1, select the high-speed rail plane body to be planned, determine the spraying area on the body surface (doors, windows, etc. are obstacles and should be avoided from spraying), spraying area, obstacles such as Figure 3 As shown, Figure 3 To spray color environment interface;

[0096] Step 1.2, select a spray gun with a fan size of d (mm), and establish a path planning grid map (filling a circle of obstacles on the edge) according to the plane size H×C (mm) of the high-speed rail body and the position of the obstacle. The grid unit in the grid map is a square with a side length of one unit. The size of the grid map is:

[0097]

[0098] Among them, H is the vertical length of the vehicle body, C is the horizontal length of the vehicle body, Grid H Grid is the number of grid cells in the vertical direction of the grid map. c It is the number of grid cells in the horizontal direction of the grid map.

[0099] Step 2: Define the state and action space. Different grayscale values ​​represent the state of different grid cells, the degree of repetition of the area and the real-time position of the agent, and form a grayscale state matrix. The agent's action direction constitutes the action space (in reinforcement learning, the agent processes the environment state information and acts in the environment. Its behavior and position in the environment represent the behavior and position of the spray gun in reality).

[0100] The step 2 is specifically as follows:

[0101] Step 2.1: Figure 4 As shown in the figure, the gray value q is used to represent the state of different grid cells, the number of repetitions, and the real-time position of the Agent. Each grid cell in the grid map is assigned a gray value, thus forming a gray state matrix:

[0102]

[0103] Among them, q is the gray value, N is the number of grid unit repetitions, and σ is the gray value attenuation amplitude;

[0104] Step 2.2: Figure 4 As shown in the figure, the Agent's movement directions include up, down, left, and right. The Agent can move in the four directions, and the step length is 1 grid unit. When moving, the Agent cannot move to the obstacle area.

[0105] Step 3: Construct a multi-level feature aggregation network model based on convolutional neural network and attention mechanism as the Agent structure in the Rainbow reinforcement learning framework.

[0106] The step 3 is specifically as follows:

[0107] Step 3.1: Figure 5 As shown in the figure, a multi-level feature aggregation network model is constructed as the Agent structure in the Rainbow reinforcement learning framework, including an input processing module, two basic modules, two tracking modules, an aggregation module, a flattening layer, and two fully connected layers (including a fully connected layer of advantage value and a fully connected layer of action value). Figure 6 (a) shows the structure diagram of the basic module. Figure 6 (b) shows the structural diagram of the tracking module. Figure 6 (c) shows the structural diagram of the aggregation module, where k, s, p, g, and d represent the convolution kernel size, step size, padding edge, grouped convolution, and hole convolution parameters, respectively. Figure 6 In the basic module of (a), gray represents the convolutional layer and orange represents the channel adjustment layer. Figure 6 In the tracking module of (b), gray represents the convolution layer, orange represents the channel adjustment layer, the dotted line indicates whether the channel adjustment layer is needed according to the situation, and yellow represents the extraction layer. Figure 6 In the aggregation module of (c), gray represents the convolutional layer, blue represents the upsampling layer, E is the output feature of step 3.4.2, C is the output feature of step 3.3.2, and B6 is the output feature of step 3.3.1.

[0108] Step 3.2, the input processing module includes a 3×3 convolutional layer with a stride of 1, a layer normalization, and a ReLU activation function layer;

[0109] Step 3.2.1: The gray state matrix calculated in step 2.1 is input into the processing module, and the output feature is A. The feature map size remains unchanged, and simple feature information is extracted while retaining the spatial position information.

[0110] Step 3.3, the basic module includes two convolution blocks consisting of a 3×3 convolution layer, layer normalization, a ReLU activation function layer, a 3×3 convolution layer and layer normalization, and a channel adjustment layer consisting of a 3×3 convolution layer and layer normalization;

[0111] The output feature A of step 3.3.1 and step 3.2.1 passes through the first convolution block in the first basic module, the feature map size is halved, and the output feature B1; feature A passes through the channel adjustment layer in the first basic module, the feature map size is halved, and the output feature B2; the two outputs are merged B1+B2=B3, passed through the Relu activation function layer, and processed by the second convolution block in the first basic module, the feature map size remains unchanged, and the output feature B4; the residual connection merges B3+B4=B5, and feature B5 passes through the ReLU activation function layer to obtain the activated feature B6;

[0112] Step 3.3.2, feature B6 passes through the first convolution block in the second basic module, the feature map size remains unchanged, the number of channels is doubled, and the output feature C1; feature B6 passes through the channel adjustment layer in the second basic module, the feature map size remains unchanged, the number of channels is doubled, and the output feature C2; ​​the two outputs are merged C1+C2=C3, passed through the ReLU activation function layer, and processed by the second convolution block in the second basic module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature C4; the residual connection merges C3+C4=C5, and feature C5 passes through the ReLU activation function layer to obtain the activated feature C;

[0113] Step 3.4, the first tracking module includes two convolution blocks consisting of 3×3 convolution layers, layer normalization and ReLU activation function layers, a channel adjustment layer consisting of 3×3 convolution layers and layer normalization, and an extraction layer consisting of 3×3 convolution layers and ReLU activation function layers; the second tracking module includes two convolution blocks consisting of 3×3 convolution layers, layer normalization and ReLU activation function layers, and an extraction layer consisting of 3×3 convolution layers and ReLU activation function layers;

[0114] Step 3.4.1, pass the output feature C of step 3.3.2 through the first convolution block in the first tracking module, the feature map size remains unchanged, and the output feature D1; the residual connection merges C+D1=D2, and the feature D2 passes through the second convolution block in the first tracking module again, the feature map size remains unchanged, and the output feature D3; the residual connection merges D2+D3=D4, and the feature D4 passes through the extraction layer in the first tracking module, the feature map size is halved, the number of channels is doubled, and the output feature D5; C passes through the channel adjustment layer in the first tracking module, the feature map size is halved, the number of channels is doubled, and the output feature D6; the outputs of the two are merged D5+D6=D7 to obtain feature D7;

[0115] Step 3.4.2, feature D7 passes through the first convolution block in the second tracking module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E1; residual connection merges D7+E1=E2, feature E2 passes through the second convolution block in the second tracking module again, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E3; residual connection merges E2+E3=E4, feature E4 passes through the extraction layer in the second tracking module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E5; residual connection merges E5+D7=E to obtain feature E;

[0116] Step 3.5, the aggregation module contains two synthesis blocks consisting of a 3×3 convolutional layer and an upsampling layer;

[0117] The output feature E of step 3.5.1 and step 3.4.2 passes through the first synthesis block, the feature map size is doubled, the number of channels is halved, and the output feature F1 is output; it is merged with the output feature C of step 3.3.2 by residual connection, C+F1=F2;

[0118] Step 3.5.2, feature F2 passes through the second synthesis block, the feature map size remains unchanged, the number of channels is halved, and the output feature F3 is output; it is merged with the output feature B6 of step 3.3.1 by residual connection, F3+B6=F4, and feature F4 is obtained;

[0119] Step 3.6: After feature F4 is flattened (from two dimensions to one dimension) by the flattening layer, it is processed by the fully connected layer of advantage value and the fully connected layer of action value respectively to obtain the advantage value Advantage and the action value Value. The two are processed by formula (3) to obtain the output O of the Agent.

[0120]

[0121] in, represents the average value of Advantage, and O is the initial Q value array.

[0122] Step 4: Construct an action mask. The grayscale values ​​of the movable grid cells around the current position of the agent are converted into a directional probability distribution matrix Action Mask through calculation and acted on the network output of the agent.

[0123] The step 4 is specifically as follows:

[0124] Step 4.1. In the current grayscale state matrix, find the current position of the Agent, that is, the grid cell with a grayscale value of 170, and add the grayscale values ​​of its four adjacent grid cells above, below, left, and right to the array Pixel;

[0125] Step 4.2: The elements in the array Pixel are modified to corresponding values ​​according to the rule of formula (4):

[0126]

[0127] Among them, Pixel[i] represents the value of the i-th element in the Pixel array;

[0128] Step 4.3, the Pixel array is processed by the SOftMax function to become the direction probability array Action Mask;

[0129] Step 4.4: Perform the following operations according to formula (5): Action Mask acts on the output O of the Agent in step 3.6 to obtain the corrected Q value array, and replace the elements with a value of 0 in the corrected Q value array with -∞:

[0130]

[0131] Step 5: Considering the environmental coverage and grid unit repetition, formulate the reward function and path quality evaluation index;

[0132] The step 5 is specifically as follows:

[0133] Step 5.1: Create a reward function, give different reward values ​​to the agent according to its different behaviors, and quantify the agent's actions into specific values:

[0134]

[0135] Among them, s i represents the reward generated by the i-th action, i represents the i-th step, V f represents the effective action step reward base, α represents the discount factor of the effective action step, V n Indicates the number of valid action steps, i is Valid Step means the i-th step is a valid step, I f represents the base of the invalid action step reward, β represents the discount factor of the number of grid unit repetitions, Gr represents the number of grid repetitions, i is Invalid Step represents that the i-th step is an invalid step, EPR represents the full coverage completion reward, and Cover Ratio represents the coverage ratio;

[0136] Step 5.2: Formulate path evaluation indicators. For coverage path planning, meeting the environmental coverage rate c is a prerequisite for completing the task. The path repetition rate r and the total path reward R are the basis for judging the optimal solution of the coverage path:

[0137]

[0138] R=∑S n (9);

[0139] Among them, N c is the number of covered grid cells, N is the total number of grid cells to be covered in the environment, S is the number of steps consumed by the coverage path, and S n is the reward generated by the nth action.

[0140] Step 6: Train a multi-level feature aggregation network model in the constructed spraying environment and select the optimal path based on the path quality evaluation index.

[0141] like Figure 2 As shown, the step 6 is specifically as follows:

[0142] Step 6.1: Construct the spraying environment of step 1.2 and define the training parameters of reinforcement learning (learning rate, discount rate, number of iterations, maximum number of steps per round, and experience capacity, etc.);

[0143] Step 6.2, reset the spraying environment, Agent starting point (lower left corner), path reward and Reply Buffer, etc. Reply Buffer is used to store past action information;

[0144] Step 6.3: record the gray state matrix of step 2.1 as State, and use State as the input of the multi-level feature aggregation network constructed in step 3. The network outputs the initial Q value array;

[0145] Step 6.4: Apply the Action Mask to the initial Q value array according to the method in step 4 to obtain a modified Q value array (after multiplying the initial Q value array by the Action Mask, the modified Q value array is obtained), and replace the elements with a value of 0 in the modified Q value array with -∞; the Agent selects the action a corresponding to the largest element in the Q value array and executes it;

[0146] Step 6.5: After the Agent takes action, the gray state matrix is ​​updated to State′, and the environment feeds back the single-step action reward s according to the reward function. i , task completion mark d, current coverage c, repetition rate r and path total return R;

[0147] Step 6.6, (State, a, s i ,State′,d) is stored in ReplyBuffer as a complete action process;

[0148] Step 6.7: If the number of experiences in the Reply Buffer reaches the preset value, extract Batch Size experiences from it and use them to train the Agent. After the training is completed, execute step 6.8; if the number of experiences in the Reply Buffer does not reach the preset value, directly execute step 6.8;

[0149] Step 6.8: If the task completion flag d is True or the Agent action times reach the maximum, the task is terminated and the process goes to step 6.2 to continue the loop.

[0150] Otherwise, the current gray state matrix is ​​recorded as State, and State is used as the input of the latest multi-level feature aggregation network. The network outputs the initial Q value array and executes step 6.4;

[0151] Step 6.9: If the number of iterations of steps 6.2 to 6.8 reaches the maximum number, exit the task, select the optimal coverage path based on the path evaluation index, and draw a graph. Figure 7 This is a graph showing the changes in coverage and repetition rate corresponding to the number of iterations. Figure 8 This is a graph showing the total reward changes corresponding to the number of iterations. Fig. 9 In the path diagram, the green arrow indicates the direction of movement of the current grid cell, the red arrow indicates the repeated path, and the five-pointed star indicates the end point of the path.

[0152] This embodiment also provides a high-speed rail body flat spray full coverage path planning system, including:

[0153] The spraying environment building module is used to build a grid map for path planning based on the plane size of the high-speed rail body, the spray fan size of the spray gun and the location of obstacles;

[0154] The state and action space construction module is used to construct the agent's gray state matrix and action space;

[0155] The multi-level feature aggregation network module is used to build a multi-level feature aggregation network model and use the multi-level feature aggregation network model as the agent model structure in the Rainbow reinforcement learning framework, thereby improving the agent structure and enhancing the environmental information processing and exploration capabilities.

[0156] The action mask module is used to convert the grayscale values ​​of the movable grid cells around the current position of the agent into a direction probability matrix through calculation, and act on the network output of the agent, that is, superimpose the action mask on the output end of the agent network, while avoiding dangerous actions, guiding the intelligent agent to achieve the goal of full coverage;

[0157] The reward and indicator module is used to comprehensively consider the degree of environmental coverage and the degree of grid unit repetition, and formulate reward functions and path quality evaluation indicators. The reward function quantifies the agent's actions into specific values ​​to judge the quality of the actions. The path evaluation indicator is used to determine whether the full coverage task has been completed and the quality of the coverage path.

[0158] The model training module is used to train the multi-level feature aggregation network model in the constructed spray environment and select the optimal path according to the path quality evaluation index, that is, to train the reinforcement learning agent in the spray environment, find the optimal network parameters within the specified number of iterations, and find the optimal path according to the path evaluation index.

[0159] The protection scope of the present invention includes but is not limited to the above embodiments. The protection scope of the present invention shall be based on the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by technicians in this field shall fall within the protection scope of the present invention.

Claims

1. A method for planning a full coverage path for spraying a high-speed railway body, characterized in that: include: Step 1: Select the high-speed rail flat body to be planned and determine the spraying area; select a suitable spray gun and build a grid map for path planning; Step 2: Different grayscale values ​​represent the status, repetition degree and real-time position of different grid cells, and form a grayscale state matrix; define the direction of the agent's action; Step 3: Build a multi-level feature aggregation network model as the Agent structure in the Rainbow reinforcement learning framework; Step 4: The grayscale values ​​of the movable grid cells around the current position of the agent are converted into a direction probability matrix through calculation and acted on the network output of the agent; Step 5: Considering the environmental coverage and grid unit repetition, formulate the reward function and path quality evaluation index; Step 6: Train a multi-level feature aggregation network model in the constructed spraying environment and select the optimal path based on the path quality evaluation index.

2. The high-speed railway body plane spray full coverage path planning method according to claim 1 is characterized in that: The step 1 is specifically as follows: Step 1.1, select the high-speed rail plane body to be planned, and determine the spraying position on the body surface; Step 1.2, select a spray gun with a fan size of d, and establish a path planning grid map according to the plane size H×C of the high-speed rail body and the position of the obstacle. The edge of the grid map is filled with a circle of obstacles. The grid unit in the grid map is a square with a side length of one unit. The size of the grid map is: Among them, H is the vertical length of the vehicle body, C is the horizontal length of the vehicle body, Grid H Grid is the number of grid cells in the vertical direction of the grid map. C It is the number of grid cells in the horizontal direction of the grid map.

3. The high-speed railway body plane spraying full coverage path planning method according to claim 1 is characterized in that: The step 2 is specifically as follows: Step 2.1, use the gray value g to represent the state of different grid cells, the number of repetitions and the real-time position of the Agent. Each grid cell in the grid map will be assigned a gray value, thus forming a gray state matrix: Among them, g is the gray value, N is the number of grid unit repetitions, and σ is the gray value attenuation amplitude; Step 2.2: The Agent moves in four directions: up, down, left, and right, and the step length is 1 grid unit. When moving, the Agent cannot move to the obstacle area.

4. The method for planning a path for full coverage of a high-speed railway body surface spraying according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1, construct a multi-level feature aggregation network model as the agent structure in the Rainbow reinforcement learning framework, including an input processing module, two basic modules, two tracking modules, an aggregation module, a flattening layer, an advantage value fully connected layer, and an action value fully connected layer; Step 3.2, the input processing module includes a 3×3 convolutional layer with a stride of 1, a layer normalization, and a ReLU activation function layer; Step 3.2.1: The gray state matrix calculated in step 2.1 is input into the processing module, and the output feature is A. The feature map size remains unchanged, and the feature information is extracted while retaining the spatial position information. Step 3.3, the basic module includes two convolution blocks consisting of a 3×3 convolution layer, layer normalization, a ReLU activation function layer, a 3×3 convolution layer and layer normalization, and a channel adjustment layer consisting of a 3×3 convolution layer and layer normalization; The output feature A of step 3.3.1 and step 3.2.1 passes through the first convolution block in the first basic module, the feature map size is halved, and the output feature B1; feature A passes through the channel adjustment layer in the first basic module, the feature map size is halved, and the output feature B2; the outputs of the two are merged B1+B2=B3, feature B3 passes through the Relu activation function layer and is processed by the second convolution block in the first basic module, the feature map size remains unchanged, and the output feature B4; the residual connection merges B3+B4=B5, and feature B5 passes through the ReLU activation function layer to obtain the activated feature B6; Step 3.3.2, feature B6 passes through the first convolution block in the second basic module, the feature map size remains unchanged, the number of channels is doubled, and the output feature C1; feature B6 passes through the channel adjustment layer in the second basic module, the feature map size remains unchanged, the number of channels is doubled, and the output feature C2; ​​the outputs of the two are merged C1+C2=C3, feature C3 passes through the ReLU activation function layer, and is processed by the second convolution block in the second basic module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature C4; the residual connection merges C3+c4=C5, and feature C5 passes through the ReLU activation function layer to obtain the activated feature C; Step 3.4, the first tracking module includes two convolution blocks consisting of 3×3 convolution layers, layer normalization and ReLU activation function layers, a channel adjustment layer consisting of 3×3 convolution layers and layer normalization, and an extraction layer consisting of 3×3 convolution layers and ReLU activation function layers; The second tracking module includes two convolution blocks consisting of 3×3 convolution layers, layer normalization and ReLU activation function layers, and an extraction layer consisting of 3×3 convolution layers and ReLU activation function layers; Step 3.4.1, pass the output feature C of step 3.3.2 through the first convolution block in the first tracking module, the feature map size remains unchanged, and the output feature D1; the residual connection merges C+D1=D2, and the feature D2 passes through the second convolution block in the first tracking module again, the feature map size remains unchanged, and the output feature D3; the residual connection merges D2+D3=D4, and the feature D4 passes through the extraction layer in the first tracking module, the feature map size is halved, and the number of channels is doubled, and the output feature D5; the feature C passes through the channel adjustment layer in the first tracking module, the feature map size is halved, and the number of channels is doubled, and the output feature D6; the outputs of the two are merged D5+D6=D7 to obtain feature D7; Step 3.4.2, feature D7 passes through the first convolution block in the second tracking module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E1; residual connection merges D7+E1=E2, feature E2 passes through the second convolution block in the second tracking module again, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E3; residual connection merges E2+E3=E4, feature E4 passes through the extraction layer in the second tracking module, the feature map size remains unchanged, the number of channels remains unchanged, and the output feature E5; residual connection merges E5+D7=E to obtain feature E; Step 3.5, the aggregation module contains two synthesis blocks consisting of a 3×3 convolutional layer and an upsampling layer; The output feature E of step 3.5.1 and step 3.4.2 passes through the first synthesis block, the feature map size is doubled, the number of channels is halved, and the output feature F1 is output; feature F1 is merged with the output feature C of step 3.3.2 by residual connection, that is, C+F1=F2, to obtain feature F2; Step 3.5.2, feature F2 passes through the second synthesis block, the feature map size remains unchanged, the number of channels is halved, and feature F3 is output; feature F3 is merged with the output feature B6 of step 3.3.1 by residual connection, that is, F3+B6=F4, to obtain feature F4; Step 3.6: After the feature F4 is flattened by the flattening layer, it is processed by the advantage value fully connected layer and the action value fully connected layer respectively to obtain the advantage value Advantage and the action value Value. After the two are processed by formula (3), the output O of the Agent is obtained: in, represents the average value of Advantage, and O represents the initial Q value array.

5. The high-speed railway body plane spray full coverage path planning method according to claim 1 is characterized in that: The step 4 is specifically as follows: Step 4.

1. In the current grayscale state matrix, find the current position of the Agent, that is, the grid cell with a grayscale value of 170, and add the grayscale values ​​of its four adjacent grid cells above, below, left, and right to the array Pixel; Step 4.2: The elements in the array Pixel are modified to corresponding values ​​according to the rule of formula (4): Among them, Pixel[i] represents the value of the i-th element in the Pixel array; Step 4.3, the pixel array is processed by the SoftMax function to become the direction probability array Action Mask; Step 4.4: Perform the following operations according to formula (5): Action Mask acts on the output O of the Agent in step 3.6 to obtain the corrected Q value array, and replace the elements with a value of 0 in the corrected Q value array with -∞:

6. The high-speed railway body plane spray full coverage path planning method according to claim 1 is characterized in that: The step 5 is specifically as follows: Step 5.1: Create a reward function, give different reward values ​​to the agent according to its different behaviors, and quantify the agent's actions into specific values: Among them, s i represents the reward generated by the i-th action, i represents the i-th step, V f represents the effective action step reward base, α represents the discount factor of the effective action step, V n Indicates the number of valid action steps, i is Valid Step means the i-th step is a valid step, I f represents the base number of invalid action step rewards, β represents the discount factor of the number of grid unit repetitions, G r Indicates the number of grid repetitions, iis Invalid Step indicates that the i-th step is an invalid step, EPR indicates the full coverage completion reward, and Cover Ratio indicates the coverage ratio; Step 5.2: Formulate path evaluation indicators. Path evaluation indicators include environment coverage rate c, path repetition rate r and path total return R. Meeting the environment coverage rate c is the prerequisite for completing the task. Path repetition rate r and path total return R are the basis for judging the optimal solution of the coverage path: R=∑s n (9); Among them, N c is the number of covered grid cells, N is the total number of grid cells to be covered in the environment, S is the number of steps consumed by the coverage path, and S n is the reward generated by the nth action.

7. The method for planning a path for full coverage of a high-speed railway body surface spraying according to claim 1, characterized in that: The step 6 is specifically as follows: Step 6.1: Construct the spraying environment of step 1.2 and define the training parameters of reinforcement learning; Step 6.2: Reset the spraying environment, Agent starting point, path reward and Reply Buffer. Reply Buffer is used to store past action information. Step 6.3: record the gray state matrix of step 2.1 as State, and use State as the input of the multi-level feature aggregation network constructed in step 3. The network output is O; Step 6.4: Apply the Action Mask to O according to the method in step 4 to obtain the corrected Q value array, and replace the elements with a value of 0 in the corrected Q value array with -∞; the Agent selects the action a corresponding to the largest element in the corrected Q value array and executes it; Step 6.5: After the Agent takes action, the gray state matrix is ​​updated to State′, and the environment feeds back the single-step action reward s according to the reward function. i , task completion mark d, current coverage c, repetition rate r and path total return R; Step 6.6, (State, a, s i ,State′,d) is stored in the Reply Buffer as a complete action process; Step 6.7: If the number of experiences in the Reply Buffer reaches the preset value, extract Batch Size experiences from it and use them to train the Agent. After the training is completed, execute step 6.8; if the number of experiences in the Reply Buffer does not reach the preset value, directly execute step 6.8; Step 6.8: If the task completion flag d is True or the Agent action times reach the maximum, the task is terminated and the process goes to step 6.2 to continue the loop. Otherwise, the current gray state matrix is ​​recorded as State, and State is used as the input of the latest multi-level feature aggregation network. The network outputs O and executes step 6.4; Step 6.9: If the number of iterations of steps 6.2 to 6.8 reaches the maximum number, exit the task, select the optimal coverage path based on the path evaluation index, and draw a graph.

8. A high-speed railway body flat spray full coverage path planning system, characterized in that: include: The spraying environment building module is used to build a grid map for path planning based on the plane size of the high-speed rail body, the spray fan size of the spray gun and the location of obstacles; The state-action space construction module is used to construct the agent's grayscale state matrix and action space; Multi-level feature aggregation network module, used to build a multi-level feature aggregation network model, and use the multi-level feature aggregation network model as the Agent model structure in the Rainbow reinforcement learning framework; The action mask module is used to convert the grayscale values ​​of the movable grid cells around the current position of the Agent into a direction probability matrix through calculation, and act on the network output of the Agent; The reward and indicator module is used to formulate reward functions and path quality evaluation indicators based on the comprehensive coverage of the environment and the degree of grid unit repetition; The model training module is used to train the multi-level feature aggregation network model in the constructed spraying environment and select the optimal path according to the path quality evaluation index.

Images