A method for dynamically optimizing a path of an inspection robot based on reinforcement learning
By introducing a path structure unit reconstruction mechanism and a continuous-time mapping model driven by structural residuals, the problem of refining the path planning of inspection robots in dynamic environments is solved, and stable path optimization and parameter updates are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 邯郸泓联智宇科技有限公司
- Filing Date
- 2026-02-28
- Publication Date
- 2026-06-02
Smart Images
Figure CN122131765A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent decision-making and dynamic optimization technology that combines reinforcement learning with inspection robot path planning, and particularly to a method for dynamic path optimization of inspection robots based on reinforcement learning. Background Technology
[0002] Inspection robots have been widely used in the inspection of industrial facilities, public spaces, and enclosed environments. They complete the sequential visits to inspection points through preset paths, replacing manual labor in performing repetitive and high-frequency inspection tasks. Existing path planning methods for inspection robots are mostly based on static maps or fixed rules. Typically, path planning is completed once before the task begins, and the robots are executed in a predetermined order during the inspection process.
[0003] However, in real-world inspection scenarios, the inspection environment is characterized by dynamic changes, such as variations in local traffic conditions, updates to inspection point status, and fluctuations in path execution costs. Existing path planning methods often require regenerating the entire path or rely on manual intervention when faced with these changes, resulting in coarse-grained path adjustments that struggle to address structural changes during the inspection process.
[0004] On the other hand, some studies have attempted to introduce reinforcement learning methods to optimize robot paths, but existing solutions are mostly focused on position-level or action-level decisions, simplifying the path planning problem into a continuous motion control problem. They do not clearly model the structural composition of the inspection path, and lack a unified state representation and feedback mechanism between the path adjustment process and the historical structure execution results, resulting in limited correlation between the path decision-making process and the inspection task structure.
[0005] Furthermore, existing reinforcement learning path optimization methods typically employ uniform perturbation or gradient update strategies during parameter updates, failing to distinguish the structural differences in different path adjustment behaviors. Consequently, there is a lack of correspondence between parameter updates and path structure changes, making it difficult to form a stable decision-making process in inspection tasks with frequent structural reconstruction.
[0006] Therefore, there is an urgent need for a method that can model the path structure during the inspection process and combine it with information on structural changes to achieve dynamic path optimization, so as to adapt to complex and ever-changing inspection application scenarios. Summary of the Invention
[0007] One objective of this invention is to propose a dynamic path optimization method for inspection robots based on reinforcement learning. This invention introduces a path structure unit reconstruction mechanism and a continuous-time mapping model driven by structural residuals to perform dynamic recalculation and structural-level rearrangement of the path structure state of the inspection robot, thereby constructing a continuously evolving path optimization process with the characteristics of coherent path response, fine structural adjustment, and stable parameter updates.
[0008] A method for dynamic path optimization of an inspection robot based on reinforcement learning according to an embodiment of the present invention includes the following steps:
[0009] S1. Construct a set of path structure units, each containing a start position identifier, an end position identifier, a set of covered inspection points identifiers, and structural cost features. S2. Construct a path structure state vector, composed of selected path structure unit identifiers, uncovered inspection point identifiers, and structural cost components. S3. Construct a set of structural actions, including path structure unit replacement, merging, splitting, and order adjustment. S4. Perform continuous-time mapping on the path structure state vector using a liquid time constant neural network that incorporates a structural residual correction time constant, generated from the difference results of the structural cost components. S5. Select structural actions based on the output of the liquid time constant neural network and reconstruct the path structure units, forming an updated set of path structure units. S6. Collect information on structural cost changes after path structure unit reconstruction and construct a reward vector. S7. Use a natural evolution strategy that incorporates hierarchical noise in structural actions to perform perturbation sampling and weighted update of the gating parameter subset of the liquid time constant neural network.
[0010] Optionally, S1 specifically includes:
[0011] S11. Read the description data of the passable space in the inspection area, perform connectivity parsing on the passable space, generate several passable path segments, and record the start and end position identifiers for each passable path segment; S12. Perform inspection point coverage mapping operation on each passable path segment, aggregate inspection points located on the same passable path segment to form a coverage inspection point set identifier, and calculate the structural cost features for the passable path segment; S13. Combine the start and end position identifiers, coverage inspection point set identifiers, and structural cost features corresponding to the same passable path segment to generate path structure units, and write all path structure units into the path structure unit set.
[0012] Optionally, S2 specifically includes:
[0013] S21. Read the selected path structure unit identifiers from the path structure unit set, and generate a selected path structure unit identifier sequence according to the encoding order of the selected path structure unit identifiers; S22. Read the uncovered inspection point identifiers from the inspection point set identifiers, and generate an uncovered inspection point identifier sequence according to the encoding order of the uncovered inspection point identifiers; S23. Perform vectorized concatenation on the selected path structure unit identifier sequence and the uncovered inspection point identifier sequence, and append the structural cost components corresponding to the selected path structure units according to the fixed field order to form a path structure state vector.
[0014] Optionally, S3 specifically includes:
[0015] S31. Read all path structure unit identifiers and the start and end position identifiers of each path structure unit from the path structure unit set, and generate a set of path structure unit adjacency pairs according to the matching rule that the end position identifier and the start position identifier are consistent; S32. Generate merging action entries and order adjustment action entries based on the set of path structure unit adjacency pairs. The merging action entries record the first path structure unit identifier and the second path structure unit identifier, and the order adjustment action entries record the path structure unit identifier and the target order position identifier; S33. Select path structure unit pairs whose coverage inspection point set identifiers intersect from the path structure unit set to generate replacement action entries, and select path structure units whose structural cost features meet the splitting conditions from the set of path structure unit adjacency pairs to generate splitting action entries. The replacement action entries record the path structure unit identifier to be replaced and the replacement path structure unit identifier, and the splitting action entries record the path structure unit identifier to be split and the number of splitting segments parameter; S34. Execute unique action identifier encoding on the merging action entries, splitting action entries, replacement action entries, and order adjustment action entries, and write the unique action identifiers and corresponding entry parameters into the structure action set in the encoding order.
[0016] Optionally, S4 specifically includes:
[0017] S41. Read the path structure state vector, separate the structural cost component sequence, and read the structural cost component sequence of the previous decision cycle from the buffer. S42. Perform a difference operation on the current structural cost component sequence and the structural cost component sequence of the previous decision cycle according to field position to obtain the structural cost component difference vector. S43. Input the structural cost component difference vector into the time constant correction mapping layer. The time constant correction mapping layer contains the mapping weight matrix and mapping bias vector corresponding to the gating parameter subset, and outputs the structural residual correction vector. S44. Input the path structure state vector into the time constant benchmark mapping layer. The time constant benchmark mapping layer contains the time constant benchmark weight matrix and the time constant benchmark bias vector, and outputs the time constant benchmark vector. S45. Perform an element-wise summation operation on the time constant benchmark vector and the structural residual correction vector to generate the structural residual correction time. The time constant vector for structural residual correction is truncated using a preset lower and upper limit for the time constant. S46: The liquid time constant neural network initializes the hidden state vector and reads the preset time step parameters. S47: Using the structural residual correction time constant vector as the time constant input for hidden state updates, continuous time recursive updates are performed on the hidden state vector. At each time step, the continuous time recursive update performs a hidden state increment calculation and a hidden state write-back operation. The hidden state increment calculation includes a state mapping matrix product term and a gating parameter subset control term. S48: The updated hidden state vector is input to the action output mapping layer. The action output mapping layer outputs an action output vector that corresponds one-to-one with the action identifier of the structural action set. The action output vector is used as the output of the liquid time constant neural network and written to the output buffer.
[0018] Optionally, the structural residual correction time constant vector in step S4 specifically includes:
[0019] It consists of a time constant reference component vector and a structural residual correction component vector. The time constant reference component vector is calculated from the path structure state vector through a time constant reference mapping layer, and each component corresponds one-to-one with the dimension position of the hidden state vector in the liquid time constant neural network. The structural residual correction component vector is calculated from the difference results of the structural cost component in adjacent decision cycles through a time constant correction mapping layer, and each component is consistent with the time constant reference component vector in terms of vector position. The structural residual correction time constant vector is obtained by performing a summation operation on the time constant reference component vector and the structural residual correction component vector at corresponding positions, and after generation, it is subjected to interval truncation processing according to the preset time constant lower limit and preset time constant upper limit.
[0020] Optionally, S5 specifically includes:
[0021] S51. Read the action output vector from the output buffer of the liquid time constant neural network. The position of each element in the action output vector corresponds one-to-one with the action identifier in the structure action set. S52. Perform a numerical sorting operation on the action output vector, select the action identifier corresponding to the first position of the sorting result, and read the action entry parameter corresponding to the action identifier from the structure action set. S53. Perform path structure unit replacement, merging, splitting, or order adjustment operations on the path structure unit set according to the action entry parameter. Delete the replaced or merged path structure units and write the newly generated path structure units to form an updated path structure unit set.
[0022] Optionally, S6 specifically includes:
[0023] S61. Read the structural cost component sequences from the path structure unit set and the updated path structure unit set respectively, and arrange them according to the encoding order of the path structure unit identifiers; S62. Perform a difference operation on the reconstructed structural cost component sequence and the reconstructed structural cost component sequence according to the field position to obtain the structural cost change vector. At the same time, read the uncovered inspection point identifier sequence in the path structure state vector and calculate the change value of the number of uncovered inspection point identifiers; S63. Concatenate the structural cost change vector and the change value of the number of uncovered inspection point identifiers in a fixed position order to form a reward vector and write it into the reward buffer.
[0024] Optionally, S7 specifically includes:
[0025] S71. Read the action identifier determined in S52, and read the action type label corresponding to the action identifier in the structural action set. The action type label takes a value from four types of labels: path structural unit replacement, path structural unit merging, path structural unit splitting, and path structural unit order adjustment. S72. Establish a gating parameter subset index table. The gating parameter subset index table records the correspondence between action type labels and gating parameter subset identifiers, and read the gating parameter subset identifier corresponding to the action type label from the gating parameter subset index table. S73. Read the gating parameter subset identifier from the liquid time constant neural network parameter storage area. The corresponding gating parameter subset vector is read, along with the preset sampling number, preset noise scale set, and preset update step size. The preset noise scale set contains noise scale values corresponding one-to-one with the four action type labels; S74, several sets of disturbance vectors are generated according to the preset sampling number, with the dimension of the disturbance vectors being consistent with the gating parameter subset vector. The noise scale value corresponding to the action type label is applied to each set of disturbance vectors to form a structural action layered noise disturbance vector set; S75, gating parameter replacement is performed on each set of structural action layered noise disturbance vector sets. The gating parameter replacement includes the positive perturbation of the gating parameter subset vector. The system switches between the positive and negative perturbation versions, and after each gating parameter replacement, reads the same path structure state vector and calls S46 to S48 to perform continuous time mapping to obtain the action output vector; S76, for each action output vector, selects the action identifier according to S52 and performs path structure unit reconstruction according to S53, then constructs the reward vector according to S71 to S73 and writes it to the reward buffer. The reward buffer records one reward vector for the positive perturbation version and one for the negative perturbation version; S77, reads the reward vector corresponding to each group of positive perturbation versions and the reward vector corresponding to the negative perturbation versions from the reward buffer. Perform scalarization of the reward, which includes multiplying each component of the reward vector by the reward weight coefficient and summing them to obtain the reward scalar; S78, calculate the reward difference scalar for each group of perturbation vectors, and use the reward difference scalar as a weight to perform weighted accumulation on the corresponding perturbation vector to obtain the update direction vector; S79, perform incremental update on the gating parameter subset vector along the update direction vector according to the preset update step size, and write the updated gating parameter subset vector into the liquid time constant neural network parameter storage area to overwrite the original gating parameter subset vector, while clearing the cache entry in the reward cache area corresponding to this sampling.
[0026] The beneficial effects of this invention are:
[0027] (1) This invention represents the inspection path as a structured set composed of multiple path structure units, which transforms the path planning process from position-level adjustment to structure-level reconstruction, making the granularity of path adjustment more refined, and ensuring that the path state expression is consistent with the inspection task structure.
[0028] (2) In the path decision-making process, the present invention introduces a liquid time constant neural network with structural residual correction time constant, so that the continuous time mapping process can adjust the state update rhythm synchronously with the path structure change, and a stable correspondence is formed between the path structure change and the internal state evolution.
[0029] (3) This invention constructs a set of structural actions and performs replacement, merging, splitting and order adjustment operations on the path structure units to realize the dynamic reconstruction of the path structure during the inspection process. The path update does not depend on the overall replanning, and the structural adjustment process is continuous and controllable.
[0030] (4) The present invention introduces a natural evolution strategy of hierarchical noise of structural action in the parameter update stage, so that the gating parameter update process and the path structure adjustment type are in correspondence, and the parameter disturbance and reward feedback form a closed loop, which is conducive to the continuous evolution of the path decision process. Attached Figure Description
[0031] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0032] Figure 1 This is a flowchart of a method for dynamic path optimization of an inspection robot based on reinforcement learning, as proposed in this invention.
[0033] Figure 2 This is a continuous-time mapping implementation diagram of the structural residual correction time constant of the dynamic path optimization method for inspection robots based on reinforcement learning proposed in this invention;
[0034] Figure 3 This diagram illustrates the natural evolution update of hierarchical noise in the structure-action layering of a reinforcement learning-based path dynamic optimization method for inspection robots proposed in this invention. Detailed Implementation
[0035] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0036] refer to Figure 1-3 A method for dynamic path optimization of inspection robots based on reinforcement learning includes the following steps:
[0037] S1. Construct a set of path structure units, each containing a start position identifier, an end position identifier, a set of covered inspection points identifiers, and structural cost features. S2. Construct a path structure state vector, composed of selected path structure unit identifiers, uncovered inspection point identifiers, and structural cost components. S3. Construct a set of structural actions, including path structure unit replacement, merging, splitting, and order adjustment. S4. Perform continuous-time mapping on the path structure state vector using a liquid time constant neural network that incorporates a structural residual correction time constant, generated from the difference results of the structural cost components. S5. Select structural actions based on the output of the liquid time constant neural network and reconstruct the path structure units, forming an updated set of path structure units. S6. Collect information on structural cost changes after path structure unit reconstruction and construct a reward vector. S7. Use a natural evolution strategy that incorporates hierarchical noise in structural actions to perform perturbation sampling and weighted update of the gating parameter subset of the liquid time constant neural network.
[0038] In this embodiment, S1 specifically includes:
[0039] S11. Read the traversable space description data of the inspection area, perform connectivity analysis on the traversable space, generate several traversable path segments, and record the start and end position identifiers for each traversable path segment; the traversable space description data is used to characterize the traversable space structure of the inspection robot in the inspection area, and it is stored in the form of structured data, including at least spatial node data, spatial connectivity relationship data, passage constraint data, and spatial partition and boundary data. Among them, spatial node data is used to represent discrete position points in the inspection area that can be reached or passed by the robot, and each spatial node records the corresponding node identifier and spatial coordinate information; spatial connectivity relationship data is used to describe the traversable connection relationship between different spatial nodes, and each connectivity relationship records the start and end node identifiers, which are used to indicate that the robot can perform continuous movement between the corresponding nodes; passage constraint data is used to describe the passage condition information corresponding to spatial nodes or connectivity relationships, and participates in the structural cost feature calculation of subsequent path structure units in the form of structured tags; spatial partition and boundary data is used to limit the spatial range division of the inspection area, record the boundary information of different spatial partitions, and constrain the generation range of traversable path segments during the connectivity analysis process. S12. Perform an inspection point coverage mapping operation on each passable path segment, aggregating inspection points located on the same passable path segment to form a coverage inspection point set identifier, and simultaneously calculate the structural cost features for the passable path segment; S13. Combine the start position identifier, end position identifier, coverage inspection point set identifier, and structural cost features corresponding to the same passable path segment to generate a path structure unit, and write all path structure units into the path structure unit set.
[0040] In this embodiment, the overlay mapping operation in step S12 specifically includes:
[0041] Read the set of passable path segments generated in step S11, and extract the corresponding start and end position identifiers for each passable path segment to determine the spatial range of the path segment in the passable space.
[0042] The system reads the spatial location information of all inspection points from the inspection point dataset and performs path segment attribution determination on the inspection points based on the correspondence between the spatial coordinates of the inspection points and the spatial range of the passable path segments. When the spatial coordinates of an inspection point meet the spatial range conditions of a certain passable path segment, the inspection point identifier is recorded as the associated inspection point identifier of that passable path segment.
[0043] After determining the path segment affiliation of all inspection points, the inspection point identifiers belonging to the same passable path segment are aggregated. The aggregated inspection point identifiers are then used to form a set of covered inspection point identifiers, and an association record is established between the set of covered inspection point identifiers and the corresponding passable path segments.
[0044] While generating the cover inspection point set identifier, the system calculates the structural cost characteristics of the passable path segment based on the spatial length information of the passable path segment, the turning structure information inside the path segment, and the passage constraint data. The structural cost characteristics and the cover inspection point set identifier are then passed to the subsequent path structure unit construction process.
[0045] In this embodiment, S2 specifically includes:
[0046] S21. Read the selected path structure unit identifiers from the path structure unit set, and generate a selected path structure unit identifier sequence according to the encoding order of the selected path structure unit identifiers; S22. Read the uncovered inspection point identifiers from the inspection point set identifiers, and generate an uncovered inspection point identifier sequence according to the encoding order of the uncovered inspection point identifiers; S23. Perform vectorized concatenation on the selected path structure unit identifier sequence and the uncovered inspection point identifier sequence, and append the structural cost components corresponding to the selected path structure units according to the fixed field order to form a path structure state vector.
[0047] In this embodiment, S3 specifically includes:
[0048] S31. Read all path structure unit identifiers and the start and end position identifiers of each path structure unit from the path structure unit set, and generate a set of path structure unit adjacency pairs according to the matching rule that the end position identifier and the start position identifier are consistent; S32. Generate merging action entries and order adjustment action entries based on the set of path structure unit adjacency pairs. The merging action entries record the first path structure unit identifier and the second path structure unit identifier, and the order adjustment action entries record the path structure unit identifier and the target order position identifier; S33. Select path structure unit pairs whose coverage inspection point set identifiers intersect from the path structure unit set to generate replacement action entries, and select path structure units whose structural cost features meet the splitting conditions from the set of path structure unit adjacency pairs to generate splitting action entries. The replacement action entries record the path structure unit identifier to be replaced and the replacement path structure unit identifier, and the splitting action entries record the path structure unit identifier to be split and the number of splitting segments parameter; S34. Execute unique action identifier encoding on the merging action entries, splitting action entries, replacement action entries, and order adjustment action entries, and write the unique action identifiers and corresponding entry parameters into the structure action set in the encoding order.
[0049] In this embodiment, the first path structure unit identifier and the second path structure unit identifier in step S3 specifically include:
[0050] Each adjacency pair in the path structure unit adjacency pair set consists of two path structure unit identifiers, indicating that the two path structure units have a spatially continuous passage relationship. The first path structure unit identifier represents the preceding path structure unit in the adjacency pair, whose ending position identifier matches the starting position identifier of the adjacent path structure unit. The second path structure unit identifier represents the following path structure unit in the adjacency pair, whose starting position identifier matches the ending position identifier of the preceding path structure unit. When generating a merge action entry, the system records the first and second path structure unit identifiers in the same adjacency pair as a set of merge operation objects, used to describe the action relationship for performing merge processing on two adjacent path structure units.
[0051] In this embodiment, S4 specifically includes:
[0052] S41, Read the path structure state vector, separate the structure cost component sequence, and read the structure cost component sequence of the previous decision cycle from the buffer; S42, Perform a difference operation on the current structure cost component sequence and the structure cost component sequence of the previous decision cycle according to the field position to obtain the structure cost component difference vector; S43, Input the structure cost component difference vector into the time constant correction mapping layer, the time constant correction mapping layer contains the mapping weight matrix and mapping bias vector corresponding to the gating parameter subset, and outputs the structure residual correction vector; the gating parameter subset is specifically a set of network parameters related to time constant adjustment, which are separately divided and participate in the time constant correction mapping calculation in the liquid time constant neural network; S44, Input the path structure state vector into the time constant reference mapping layer, the time constant reference mapping layer contains the time constant reference weight matrix and the time constant reference bias vector, and outputs the time constant reference vector; S45, Perform a difference operation on the time constant reference... The quasi-vector and the structural residual correction vector are summed element-wise to generate a structural residual correction time constant vector. The structural residual correction time constant vector is then truncated, using a preset lower and upper time constant limit. S46: The liquid time constant neural network initializes the hidden state vector and reads a preset time step parameter. S47: Using the structural residual correction time constant vector as the time constant input for hidden state updates, continuous time recursive updates are performed on the hidden state vector. At each time step, the continuous time recursive update performs a hidden state increment calculation and a hidden state write-back operation. The hidden state increment calculation includes a state mapping matrix product term and a gating parameter subset control term. S48: The updated hidden state vector is input to the action output mapping layer. The action output mapping layer outputs an action output vector that corresponds one-to-one with the action identifier of the structural action set. This action output vector is used as the output of the liquid time constant neural network and written to the output buffer.
[0053] In this embodiment, the structural residual correction time constant vector in step S4 specifically includes:
[0054] It consists of a time constant reference component vector and a structural residual correction component vector. The time constant reference component vector is calculated from the path structure state vector through a time constant reference mapping layer, and each component corresponds one-to-one with the dimension position of the hidden state vector in the liquid time constant neural network. The structural residual correction component vector is calculated from the difference results of the structural cost component in adjacent decision cycles through a time constant correction mapping layer, and each component is consistent with the time constant reference component vector in terms of vector position. The structural residual correction time constant vector is obtained by performing a summation operation on the time constant reference component vector and the structural residual correction component vector at corresponding positions, and after generation, it is subjected to interval truncation processing according to the preset time constant lower limit and preset time constant upper limit.
[0055] In this embodiment, the time constant correction mapping layer in step S4 specifically includes:
[0056] The time constant correction mapping layer is a pre-built parameter mapping structure within the liquid time constant neural network. It is used to map the difference results of the structural cost component in adjacent decision cycles to the structural residual correction component vector. The time constant correction mapping layer is generated during the system initialization phase and is managed uniformly as a component of the liquid time constant neural network.
[0057] When constructing the liquid time constant neural network, the output dimension of the time constant correction mapping layer is determined based on the dimension of the time constant input in the hidden state vector, and the corresponding mapping weight matrix and mapping bias vector are generated accordingly. The mapping weight matrix and mapping bias vector are obtained from the parameter set of the liquid time constant neural network and are marked as mapping parameters corresponding to the gating parameter subset.
[0058] During operation, the time constant correction mapping layer receives the difference results of the structural cost components in adjacent decision cycles as input, performs linear mapping calculations on the difference results according to the preset field order, and outputs a structural residual correction component vector. The positions of each component in the structural residual correction component vector are consistent with the positions of the components in the time constant reference component vector, so that the two types of vectors can perform subsequent summation operations at corresponding positions.
[0059] In this embodiment, S5 specifically includes:
[0060] S51. Read the action output vector from the output buffer of the liquid time constant neural network. The position of each element in the action output vector corresponds one-to-one with the action identifier in the structure action set. S52. Perform a numerical sorting operation on the action output vector, select the action identifier corresponding to the first position of the sorting result, and read the action entry parameter corresponding to the action identifier from the structure action set. S53. Perform path structure unit replacement, merging, splitting, or order adjustment operations on the path structure unit set according to the action entry parameter. Delete the replaced or merged path structure units and write the newly generated path structure units to form an updated path structure unit set.
[0061] In this embodiment, S6 specifically includes:
[0062] S61. Read the structural cost component sequences from the path structure unit set and the updated path structure unit set respectively, and arrange them according to the encoding order of the path structure unit identifiers; S62. Perform a difference operation on the reconstructed structural cost component sequence and the reconstructed structural cost component sequence according to the field position to obtain the structural cost change vector. At the same time, read the uncovered inspection point identifier sequence in the path structure state vector and calculate the change value of the number of uncovered inspection point identifiers; S63. Concatenate the structural cost change vector and the change value of the number of uncovered inspection point identifiers in a fixed position order to form a reward vector and write it into the reward buffer.
[0063] In this embodiment, S7 specifically includes:
[0064] S71. Read the action identifier determined in S52, and read the action type label corresponding to the action identifier in the structural action set. The action type label takes a value from four types of labels: path structural unit replacement, path structural unit merging, path structural unit splitting, and path structural unit order adjustment. S72. Establish a gating parameter subset index table. The gating parameter subset index table records the correspondence between action type labels and gating parameter subset identifiers, and read the gating parameter subset identifier corresponding to the action type label from the gating parameter subset index table. S73. Read the gating parameter subset identifier from the liquid time constant neural network parameter storage area. The corresponding gating parameter subset vector is read, along with the preset sampling number, preset noise scale set, and preset update step size. The preset noise scale set contains noise scale values corresponding one-to-one with the four action type labels; S74, several sets of disturbance vectors are generated according to the preset sampling number, with the dimension of the disturbance vectors being consistent with the gating parameter subset vectors. The noise scale value corresponding to the action type label is applied to each set of disturbance vectors to form a structural action layered noise disturbance vector set; S85, gating parameter replacement is performed on each set of structural action layered noise disturbance vector sets. The gating parameter replacement includes the positive perturbation of the gating parameter subset vectors. The system switches between the positive and negative perturbation versions, and after each gating parameter replacement, reads the same path structure state vector and calls S46 to S48 to perform continuous time mapping to obtain the action output vector; S76, for each action output vector, selects the action identifier according to S52 and performs path structure unit reconstruction according to S53, then constructs the reward vector according to S71 to S73 and writes it to the reward buffer. The reward buffer records one reward vector for the positive perturbation version and one for the negative perturbation version; S77, reads the reward vector corresponding to each group of positive perturbation versions and the reward vector corresponding to the negative perturbation versions from the reward buffer. Perform scalarization of the reward, which includes multiplying each component of the reward vector by the reward weight coefficient and summing them to obtain the reward scalar; S78, calculate the reward difference scalar for each group of perturbation vectors, and use the reward difference scalar as a weight to perform weighted accumulation on the corresponding perturbation vector to obtain the update direction vector; S79, perform incremental update on the gating parameter subset vector along the update direction vector according to the preset update step size, and write the updated gating parameter subset vector into the liquid time constant neural network parameter storage area to overwrite the original gating parameter subset vector, while clearing the cache entry in the reward cache area corresponding to this sampling.
[0065] In this embodiment, the positive and negative perturbations in step S85 specifically include:
[0066] When performing hierarchical noise disturbance sampling of structural motion, a set of disturbance vectors is first read from the hierarchical noise disturbance vector set of structural motion, and the disturbance vectors are used as the reference for parameter offset of the gating parameter subset vector.
[0067] A positive perturbation refers to superimposing the perturbation vector onto the original parameter values of the gated parameter subset vector according to the one-to-one correspondence of the parameter positions, forming the first type of perturbation gated parameter subset vector value. A negative perturbation refers to superimposing the sign-reversed result of the perturbation vector onto the original parameter values of the same gated parameter subset vector according to the one-to-one correspondence of the parameter positions, forming the second type of perturbation gated parameter subset vector value. Both positive and negative perturbations start with the same gated parameter subset vector, differing only in the direction of the perturbation.
[0068] During the gating parameter replacement operation, the system switches between the positive perturbation version of the gating parameter subset vector and the negative perturbation version of the gating parameter subset vector. After each parameter replacement, the same path structure state vector is read, and steps S46 to S48 are called to perform continuous-time mapping to obtain the corresponding action output vector. By using the positive and negative perturbation versions of the gating parameter subset vector under the same path structure state conditions, the system can obtain two sets of corresponding action output results, providing paired reference inputs for subsequent reward vector construction and parameter updates.
[0069] Example 1:
[0070] In a typical inspection application scenario, an inspection robot needs to cover multiple inspection points within a closed area composed of multiple traversable paths. Paths intersect, merge, and split; inspection points are unevenly distributed; and the path structure cost changes with traffic conditions and structural adjustments during the inspection process. Traditional inspection path optimization methods typically rely on fixed paths or position-level reinforcement learning decisions. When the path structure changes, a complete replanning is required, easily leading to frequent path adjustments and insufficient structural stability. This invention addresses this scenario by applying its method. The system first resolves the traversable space into several path structure units and continuously maintains the path structure state vector during the inspection process. During the path decision-making phase, a liquid time constant neural network based on the structural residual correction time constant outputs structural actions, performing replacement, merging, splitting, or sequence adjustment operations on the path structure units. After each path structure reconstruction, the system statistically analyzes the changes in structural cost and the number of uncovered inspection points, forming a reward vector for subsequent parameter updates. To verify the feasibility of this invention in implementation, its method is compared with a comparative method that does not introduce structural residual correction time constant and structural action layering noise. Both methods were run for multiple rounds of inspection tasks under the same inspection point scale and path structure complexity, and the average results of key indicators were statistically analyzed.
[0071] Table 1: Comparison of Path Dynamic Optimization Effects
[0072] Comparison Methods Average path length value Number of path structure adjustments Final number of inspection points not covered Average structural cost Structural cost fluctuation range Comparison Methods 128.6 42 7 15.3 9.1 Method of the present invention 112.4 26 2 11.7 4.3
[0073] As shown in Table 1, the statistical results of the path dynamic optimization effect comparison reveal quantifiable differences among different path optimization methods in several key indicators under the same inspection point scale and path structure complexity. The average path length generated by the comparative method during the inspection process is 128.6, while the average path length of the method of this invention is 112.4. This change in path length indicates a difference in the path structure combination method between the two methods. The method of this invention, in the process of path structure unit reconstruction, adopts more structural-level adjustments rather than overall path replacement, resulting in a more stable numerical distribution of the path combination results.
[0074] Regarding the number of path structure adjustments, the comparative method recorded 42 adjustments during multiple rounds of inspection, while the method of this invention recorded 26 adjustments. The statistics on the number of structure adjustments reflect the frequency of path structure changes during path decision-making. The method of this invention, through path structure unit modeling and structural action set constraints, limits path adjustments to the necessary structural units, avoiding excessive structural rearrangement during inspection.
[0075] Regarding the final number of uncovered inspection points, the comparative method still had 7 uncovered inspection points at the end of the inspection, while the method of this invention had 2 uncovered inspection points. This indicator reflects the difference in the completion rate of the inspection task due to the path structure adjustment strategy. The method of this invention continuously introduces uncovered inspection point information into the path structure state vector, keeping the path structure reconstruction process synchronized with the inspection point coverage status, thereby gradually reducing the number of residual uncovered inspection points during the inspection process.
[0076] Regarding structural cost-related indicators, the average structural cost of the comparative method is 15.3, with a fluctuation range of 9.1; the average structural cost of the method of this invention is 11.7, with a fluctuation range of 4.3. The numerical difference in the fluctuation range of structural costs reflects the magnitude of changes in the path structure during the inspection process. This invention adjusts the continuous time mapping rhythm by adjusting the structural residual correction time constant, ensuring the continuity of the path structure change process, and the structural cost values exhibit a relatively concentrated distribution characteristic in multiple rounds of inspection.
[0077] As can be seen from the statistical data in Table 1, in inspection scenarios with complex path structures and uneven distribution of inspection points, the method of the present invention exhibits good numerical characteristics in terms of path structure combination stability, structure adjustment frequency control, and consistency of inspection point coverage process, thus verifying the feasibility of implementing the method of the present invention in the dynamic optimization of actual inspection paths.
[0078] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for dynamic path optimization of an inspection robot based on reinforcement learning, characterized in that, Includes the following steps: S1. Construct a set of path structure units. Each path structure unit includes a start position identifier, an end position identifier, a set identifier of covered inspection points, and structural cost features. S2. Construct a path structure state vector, which consists of the selected path structure unit identifier, the uncovered inspection point identifier, and the structure cost component. S3. Construct a set of structure actions, including path structure unit replacement, merging, splitting, and order adjustment. S4. Use a liquid time constant neural network with introduced structural residual correction time constant to perform continuous-time mapping on the path structure state vector. The structural residual correction time constant is generated from the difference results of the structure cost component. S5. Select structure actions based on the output of the liquid time constant neural network and perform path structure unit reconstruction to form an updated set of path structure units. S6. Collect information on structural cost changes after path structure unit reconstruction and construct a reward vector. S7. Use a natural evolution strategy with introduced hierarchical noise in structure actions to perform perturbation sampling and weighted update of the gating parameter subset of the liquid time constant neural network.
2. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 1, characterized in that, S1 specifically includes: S11. Read the description data of the passable space in the inspection area, perform connectivity parsing on the passable space, generate several passable path segments, and record the start and end position identifiers for each passable path segment; S12. Perform inspection point coverage mapping operation on each passable path segment, aggregate inspection points located on the same passable path segment to form a coverage inspection point set identifier, and calculate the structural cost features for the passable path segment; S13. Combine the start and end position identifiers, coverage inspection point set identifiers, and structural cost features corresponding to the same passable path segment to generate path structure units, and write all path structure units into the path structure unit set.
3. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 2, characterized in that, S2 specifically includes: S21. Read the selected path structure unit identifiers from the path structure unit set, and generate a selected path structure unit identifier sequence according to the encoding order of the selected path structure unit identifiers; S22. Read the uncovered inspection point identifiers from the inspection point set identifiers, and generate an uncovered inspection point identifier sequence according to the encoding order of the uncovered inspection point identifiers; S23. Perform vectorized concatenation on the selected path structure unit identifier sequence and the uncovered inspection point identifier sequence, and append the structural cost components corresponding to the selected path structure units according to the fixed field order to form a path structure state vector.
4. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 3, characterized in that, S3 specifically includes: S31. Read all path structure unit identifiers and the start and end position identifiers of each path structure unit from the path structure unit set, and generate a set of path structure unit adjacency pairs according to the matching rule that the end position identifier and the start position identifier are consistent; S32. Generate merging action entries and order adjustment action entries based on the set of path structure unit adjacency pairs. The merging action entries record the first path structure unit identifier and the second path structure unit identifier, and the order adjustment action entries record the path structure unit identifier and the target order position identifier; S33. Select path structure unit pairs whose coverage inspection point set identifiers intersect from the path structure unit set to generate replacement action entries, and select path structure units whose structural cost features meet the splitting conditions from the set of path structure unit adjacency pairs to generate splitting action entries. The replacement action entries record the path structure unit identifier to be replaced and the replacement path structure unit identifier, and the splitting action entries record the path structure unit identifier to be split and the number of splitting segments parameter; S34. Execute unique action identifier encoding on the merging action entries, splitting action entries, replacement action entries, and order adjustment action entries, and write the unique action identifiers and corresponding entry parameters into the structure action set in the encoding order.
5. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 4, characterized in that, S4 specifically includes: S41. Read the path structure state vector, separate the structural cost component sequence, and read the structural cost component sequence of the previous decision cycle from the buffer. S42. Perform a difference operation on the current structural cost component sequence and the structural cost component sequence of the previous decision cycle according to field position to obtain the structural cost component difference vector. S43. Input the structural cost component difference vector into the time constant correction mapping layer. The time constant correction mapping layer contains the mapping weight matrix and mapping bias vector corresponding to the gating parameter subset, and outputs the structural residual correction vector. S44. Input the path structure state vector into the time constant benchmark mapping layer. The time constant benchmark mapping layer contains the time constant benchmark weight matrix and the time constant benchmark bias vector, and outputs the time constant benchmark vector. S45. Perform an element-wise summation operation on the time constant benchmark vector and the structural residual correction vector to generate the structural residual correction time. The time constant vector for structural residual correction is truncated using a preset lower and upper limit for the time constant. S46: The liquid time constant neural network initializes the hidden state vector and reads the preset time step parameters. S47: Using the structural residual correction time constant vector as the time constant input for hidden state updates, continuous time recursive updates are performed on the hidden state vector. At each time step, the continuous time recursive update performs a hidden state increment calculation and a hidden state write-back operation. The hidden state increment calculation includes a state mapping matrix product term and a gating parameter subset control term. S48: The updated hidden state vector is input to the action output mapping layer. The action output mapping layer outputs an action output vector that corresponds one-to-one with the action identifier of the structural action set. The action output vector is used as the output of the liquid time constant neural network and written to the output buffer.
6. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 5, characterized in that, The structural residual correction time constant vector in step S4 specifically includes: It consists of a time constant reference component vector and a structural residual correction component vector. The time constant reference component vector is calculated from the path structure state vector through a time constant reference mapping layer, and each component corresponds one-to-one with the dimension position of the hidden state vector in the liquid time constant neural network. The structural residual correction component vector is calculated from the difference results of the structural cost component in adjacent decision cycles through a time constant correction mapping layer, and each component is consistent with the time constant reference component vector in terms of vector position. The structural residual correction time constant vector is obtained by performing a summation operation on the time constant reference component vector and the structural residual correction component vector at corresponding positions, and after generation, it is subjected to interval truncation processing according to the preset time constant lower limit and preset time constant upper limit.
7. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 6, characterized in that, S5 specifically includes: S51. Read the action output vector from the output buffer of the liquid time constant neural network. The position of each element in the action output vector corresponds one-to-one with the action identifier in the structure action set. S52. Perform a numerical sorting operation on the action output vector, select the action identifier corresponding to the first position of the sorting result, and read the action entry parameter corresponding to the action identifier from the structure action set. S53. Perform path structure unit replacement, merging, splitting, or order adjustment operations on the path structure unit set according to the action entry parameter. Delete the replaced or merged path structure units and write the newly generated path structure units to form an updated path structure unit set.
8. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 7, characterized in that, S6 specifically includes: S61. Read the structural cost component sequences from the path structure unit set and the updated path structure unit set respectively, and arrange them according to the encoding order of the path structure unit identifiers; S62. Perform a difference operation on the reconstructed structural cost component sequence and the reconstructed structural cost component sequence according to the field position to obtain the structural cost change vector. At the same time, read the uncovered inspection point identifier sequence in the path structure state vector and calculate the change value of the number of uncovered inspection point identifiers; S63. Concatenate the structural cost change vector and the change value of the number of uncovered inspection point identifiers in a fixed position order to form a reward vector and write it into the reward buffer.
9. The method for dynamic path optimization of an inspection robot based on reinforcement learning according to claim 8, characterized in that, Specifically, S7 includes: S71. Read the action identifier determined in S52, and read the action type label corresponding to the action identifier in the structural action set. The action type label takes a value from four types of labels: path structural unit replacement, path structural unit merging, path structural unit splitting, and path structural unit order adjustment. S72. Establish a gating parameter subset index table. The gating parameter subset index table records the correspondence between action type labels and gating parameter subset identifiers, and read the gating parameter subset identifier corresponding to the action type label from the gating parameter subset index table. S73. Read the gating parameter subset identifier from the liquid time constant neural network parameter storage area. The corresponding gating parameter subset vector is read, along with the preset sampling number, preset noise scale set, and preset update step size. The preset noise scale set contains noise scale values corresponding one-to-one with the four action type labels; S74, several sets of disturbance vectors are generated according to the preset sampling number, with the dimension of the disturbance vectors being consistent with the gating parameter subset vector. The noise scale value corresponding to the action type label is applied to each set of disturbance vectors to form a structural action layered noise disturbance vector set; S75, gating parameter replacement is performed on each set of structural action layered noise disturbance vector sets. The gating parameter replacement includes the positive perturbation of the gating parameter subset vector. The system switches between the positive and negative perturbation versions, and after each gating parameter replacement, reads the same path structure state vector and calls S46 to S48 to perform continuous time mapping to obtain the action output vector; S76, for each action output vector, selects the action identifier according to S52 and performs path structure unit reconstruction according to S53, then constructs the reward vector according to S71 to S73 and writes it to the reward buffer. The reward buffer records one reward vector for the positive perturbation version and one for the negative perturbation version; S77, reads the reward vector corresponding to each group of positive perturbation versions and the reward vector corresponding to the negative perturbation versions from the reward buffer. Perform scalarization of the reward, which includes multiplying each component of the reward vector by the reward weight coefficient and summing them to obtain the reward scalar; S78, calculate the reward difference scalar for each group of perturbation vectors, and use the reward difference scalar as a weight to perform weighted accumulation on the corresponding perturbation vector to obtain the update direction vector; S79, perform incremental update on the gating parameter subset vector along the update direction vector according to the preset update step size, and write the updated gating parameter subset vector into the liquid time constant neural network parameter storage area to overwrite the original gating parameter subset vector, while clearing the cache entry in the reward cache area corresponding to this sampling.