Operation path planning method and equipment of operation device and readable storage medium

Abstract

The invention discloses a working path planning method and equipment of a working device and a readable storage medium, the working device comprises a working unit, and the method comprises the following steps: obtaining an obstacle list around the working device; based on the obstacle list, constructing a three-dimensional grid map for path planning; and on the basis of the real-time state, the target pose and the three-dimensional grid map of the operation device, a control instruction for the operation unit is generated in real time through a reinforcement learning algorithm so as to drive the operation device to move towards the target pose along the collision-free path. Thus, path planning is performed by combining the three-dimensional grid map and the reinforcement learning algorithm, the intelligence, safety and efficiency of the operation process can be remarkably improved, and meanwhile, the obstacle avoidance capability of the operation device in a complex environment is improved.

Description

Technical Field

[0001] This invention relates to the field of mechanical equipment technology, and in particular to a method for planning the operation path of an operation device, an operation device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the field of construction machinery, the precise control and safe obstacle avoidance of operating devices such as booms on fire trucks and aerial work platforms have always been core challenges. Meanwhile, the operating environments of these devices are typically highly dynamic and have complex spatial structures, placing extremely high demands on their path planning capabilities. However, traditional path planning methods for these devices rely primarily on manual control and experience-based judgment by operators, resulting in low levels of automation and intelligence. This fails to fully meet the urgent needs of modern construction machinery for efficient, safe, and intelligent operation in complex and dynamic environments. Summary of the Invention

[0003] The purpose of this application is to provide a method for planning the operation path of an operation device, an operation device, an electronic device, and a computer-readable storage medium, which can significantly improve the intelligence, safety, and efficiency of the operation process, while also enhancing the obstacle avoidance capability of the operation device in complex environments.

[0004] To achieve the above objectives: In a first aspect, embodiments of this application provide a method for planning the work path of a work device, the method comprising: Obtain a list of obstacles around the working device; Based on the obstacle list, a 3D grid map is constructed for path planning; Based on the real-time status of the working device, the target pose, and the three-dimensional grid map, a reinforcement learning algorithm is used to generate control commands for the working unit in real time, so as to drive the working device to move towards the target pose along a collision-free path.

[0005] In some embodiments, constructing a 3D raster map for path planning based on the obstacle list includes: Discretize the working space of the operating device to form a three-dimensional grid space; Transform and merge the bounding boxes of each obstacle in the obstacle list into a base coordinate system with the rotation center of the working device as the origin; The merged bounding boxes are mapped onto the 3D raster space to generate the 3D raster map.

[0006] In some embodiments, the step of transforming and fusing the bounding boxes of each obstacle in the obstacle list to a base coordinate system with the rotation center of the working device as the origin includes: Based on the real-time pose of the end of the working device, the bounding boxes of each obstacle are mapped to the base coordinate system with the rotation center of the working device as the origin through coordinate transformation; In the base coordinate system, determine whether volume overlap occurs between different oriented bounding boxes; The overlapping oriented bounding boxes are merged to generate a merged oriented bounding box that covers all overlapping areas. In some embodiments, the coordinate transformation is achieved by the following formula: C_new = R • (C_old - P_end) + P_end, where R is the coordinate system rotation matrix determined based on the motion of the working device, P_end is the position of the end of the current working device in the base coordinate system, C_old is the center point of the orientation bounding box before transformation, and C_new is the center point of the orientation bounding box after transformation.

[0007] In some embodiments, the real-time state includes the real-time pose of the end effector; the real-time pose of the end effector is determined in the following manner: Based on the wire encoder and tilt sensor installed on the working device, the extension and retraction of each working unit, as well as the pitch and yaw angles relative to the horizontal plane, are obtained. Based on the kinematic model of the working device, the theoretical pose of the end effector of the working device is determined according to the extension, pitch angle and yaw angle. The deformation compensation amount of the working device is determined by querying the deformation database of the working device based on its current extension dimensions and spatial attitude parameters; the deformation database is established through finite element analysis. The theoretical pose of the end effector of the working device is corrected based on the deformation compensation amount of the working device to obtain the real-time pose of the end effector. In some embodiments, the real-time generation of control instructions for the work unit using a reinforcement learning algorithm is implemented using a hierarchical decision architecture, including: In the global navigation layer, a global guidance path from the starting point to the target pose is planned in the three-dimensional grid map; In the local control layer, guided by the global guidance path, the action instructions of the work unit are calculated in real time through the reinforcement learning algorithm, and the action instructions are output as the control instructions of the work unit. In some embodiments, the reinforcement learning algorithm is implemented based on a Markov decision model; The state space of the Markov decision model includes: the grid coordinates of the end effector of the working device mapped in the three-dimensional grid map, the discretized combination of the angles of each working unit of the working device, and the obstacle occupancy information within a preset range around the working device. The action space of the Markov decision model is defined as the discrete set of the angular increments Δq_i of each working unit of the working device, where Δq_i is taken from the discrete set {-δ, 0, +δ}, and δ is the preset step angle. The state transition function of the Markov decision model is modeled as follows: based on the kinematic model of the working device, the predicted next state is calculated from the current state and the current action; random disturbances are introduced into the predicted next state to simulate control error. The reward function of the Markov decision model is a multi-objective weighted function, and the corresponding immediate reward is calculated by weighting the reward components based on obstacle avoidance safety, path efficiency, motion smoothness, and target guidance.

[0008] In some embodiments, the reinforcement learning algorithm includes a dual deep Q-network; the step of calculating the action instructions of the work unit in real time using the reinforcement learning algorithm includes: The three-dimensional grid map, the real-time status of each working unit of the working device, and the real-time pose and target pose of the end of the working device are used as status inputs; The Q-value of each action is calculated using a dual-depth Q-network; The optimal action is determined based on the Q value, and the optimal action is output as the action command of the work unit. In some embodiments, the dual-depth Q-network is trained by minimizing the error between the target Q-value and the current Q-value, where Q = r(s, a) + γ • Q target (s', a'), r(s, a) represents the immediate reward, γ represents the discount factor, and Q represents the discount factor. target This is the estimate for maximum profit.

[0009] Secondly, embodiments of this application provide a work device, including: a work apparatus communicatively connected to a controller; the work apparatus includes a work unit, and the controller executes a computer program to implement the work path planning method of the work apparatus as described in the first aspect.

[0010] Thirdly, embodiments of this application provide an electronic device, including: a storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the operation path planning method of the operating device as described in the first aspect.

[0011] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the job path planning method of the working device as described in the first aspect.

[0012] The embodiments of this application provide a method for planning the work path of a work device, a work equipment, an electronic device, and a computer-readable storage medium. The method includes: acquiring a list of obstacles around the work device; constructing a three-dimensional grid map for path planning based on the obstacle list; and generating control commands for the work unit in real time using a reinforcement learning algorithm based on the real-time status of the work device, the target pose, and the three-dimensional grid map, so as to drive the work device to move towards the target pose along a collision-free path. In this way, by digitally modeling obstacles in the environment surrounding the work device using a three-dimensional grid map, the reinforcement learning algorithm can search and make decisions in a defined three-dimensional obstacle space, ensuring the safety of the planned path, effectively avoiding collisions between the work device and dynamic or static obstacles, significantly improving the intelligence and safety of the work process, and the obstacle avoidance capability of the work device in complex environments. Attached Figure Description

[0013] Figure 1 A flowchart illustrating the operation path planning method for the operation device provided in an embodiment of the present invention. Detailed Implementation

[0014] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. In this application, "each" includes one or more items.

[0016] Figure 1 Figure 1 is a flowchart illustrating a work path planning method for a work device according to an embodiment. As shown in Figure 1, the work path planning method for the work device of this application includes the following steps: S101, Obtain a list of obstacles around the working device.

[0017] S102, Based on the list of obstacles, construct a 3D raster map for path planning.

[0018] S103, based on the real-time status of the working device, the target pose, and the three-dimensional grid map, generates control commands for the working unit in real time through reinforcement learning algorithms, so as to drive the working device to move towards the target pose along a collision-free path.

[0019] The obstacle list can be obtained from environmental sensing data collected by a single environmental sensing device deployed on the working device, or it can be generated by fusing environmental sensing data collected separately by multiple environmental sensing devices deployed on the working device. No specific limitation is made here. Environmental sensing devices refer to sensors deployed on the working device, such as a boom, used to detect the surrounding environment. In this embodiment, at least two different types of environmental sensing devices are used to detect obstacles around the working device, achieving complementarity in physical principles and detection characteristics, thereby improving the perception capability of the surrounding environment. In specific applications, environmental sensing devices can be installed at or near the end of the working device. In this embodiment, the environmental sensing devices can include at least two of the following: lidar, millimeter-wave radar, and ultrasonic radar. For example, after acquiring environmental perception data collected by at least two different types of environmental perception devices from the environment surrounding the working device, the environmental perception data collected by each device can be processed in parallel to generate corresponding obstacle detection results. These results are then fused to generate an obstacle list. By directly and effectively applying the results of multi-device fusion perception to work path planning, the working device's motion decisions are based on a comprehensive and accurate understanding of the environment, thereby further improving the intelligence and safety of the work process. The obstacle list is a structured dataset that summarizes information on all identified obstacles, typically including the location and orientation bounding box (including length, width, height, and direction) of each obstacle. Additionally, the obstacle list may also include information such as the speed of each obstacle. It should be noted that in this embodiment, the working device refers to any mechanical structure capable of being positioned, operated, or moved in space through controllable motion, including but not limited to the boom of construction machinery (such as the telescopic boom of an aerial work platform, the folding boom of a concrete pump truck), the robotic arm of an industrial robot, the boom and stick of an excavator, and even the robotic arm of a robot. A working unit refers to the basic components that constitute a working device and have relative motion functions, including but not limited to the joints between the sections of the boom, the rotary or traverse joints of the robotic arm, and any drive connection parts that can provide pitch, yaw, telescopic, or rotational motion.

[0020] In this context, a 3D grid map refers to a collection of volumetric units (voxels) discretized into a 3D space, with each grid marked as either "occupied" (obstacle) or "free" (passable). After uniformly dividing the workspace of the working device (a 3D cubic region) into small cubic grids, each obstacle in the obstacle list can be marked into a corresponding grid according to its geometry. Grids occupied by obstacles are marked as "occupied," thus constructing a 3D grid map for path planning. Finally, based on the real-time acquired status of the working device (such as the angles of each working unit and the pose of the device's end effector) and the given target pose, along with the constructed 3D grid map, a reinforcement learning algorithm (such as the DDQN algorithm) can output optimal control commands in real time (such as the rotation angle increment of each working unit). This drives the motors corresponding to each working unit, enabling the working device to move smoothly and efficiently towards the target pose while avoiding all marked obstacles. In this embodiment, the reinforcement learning algorithm can be viewed as a pre-trained reinforcement learning agent that can output optimal control commands in real time based on the currently input data such as the status of the working device.

[0021] It should be noted that, in this embodiment, the target pose refers to the final state that the end effector of the working device needs to reach, including its target position (i.e., three-dimensional coordinates) in three-dimensional space and its target orientation (i.e., rotation angles around each coordinate axis). In some embodiments, the purpose of the work path planning may be to drive the working device to move along a collision-free path to the target position and maintain the target orientation after the end effector of the working device reaches the target position. In some embodiments, the purpose of the work path planning may be to drive the working device to move along a collision-free path so that the end effector of the working device reaches the target position.

[0022] In summary, the operation path planning method for the operating device provided in the above embodiments uses a three-dimensional grid map to digitally model obstacles in the environment surrounding the operating device, enabling the reinforcement learning algorithm to search and make decisions in a defined three-dimensional obstacle space. This ensures the safety of the planned path, effectively avoids collisions between the operating device and dynamic or static obstacles, significantly improves the intelligence and safety of the operation process, and enhances the obstacle avoidance capability of the operating device in complex environments.

[0023] In some embodiments, obstacles in the obstacle list are represented by oriented bounding boxes; step S102, based on the obstacle list, constructing a three-dimensional raster map for path planning, including: Discretize the working space of the operating device to form a three-dimensional grid space; Transform and merge the bounding boxes of each obstacle in the obstacle list into a base coordinate system with the rotation center of the working device as the origin; The merged bounding boxes are mapped onto a 3D raster space to generate a 3D raster map.

[0024] Discretization refers to the process of dividing a continuous physical space into a finite number of regularly arranged units. The base coordinate system is a global reference coordinate system with the rotation center of the working device as its fixed origin; it serves as the benchmark for describing the absolute positions of the working device itself and obstacles. First, a three-dimensional space, i.e., the workspace, can be determined based on the maximum working range of the working device, and then divided into N sections along the X, Y, and Z axes. x * N y *N z The system uses a cubic grid space, i.e., a three-dimensional grid space. Since the working device is in motion during operation, the positions of obstacles relative to the device are constantly changing. Therefore, the oriented bounding boxes (OBBs) of the obstacles need to be unified to a fixed base coordinate system through coordinate transformation. This transformation also takes into account the rotation and translation of the working device. Finally, these OBBs in the base coordinate system are mapped to the three-dimensional grid space. Grids covered by OBBs are set to "occupied" (value 1), otherwise "idle" (value 0), thus generating a three-dimensional grid map that reflects the current obstacle distribution in real time. In this way, the dynamic fusion mechanism ensures that the three-dimensional grid map can reflect the relative motion between the working device and the environment in real time, ensuring that the environmental information used for path planning is always up-to-date, guaranteeing the real-time performance and accuracy of path planning.

[0025] In some embodiments, transforming and fusing the bounding boxes of each obstacle in the obstacle list to a base coordinate system with the rotation center of the working device as the origin includes: Based on the real-time pose of the end of the working device, the bounding boxes of each obstacle are mapped to the base coordinate system with the rotation center of the working device as the origin through coordinate transformation; In the base coordinate system, determine whether volume overlap occurs between different oriented bounding boxes; The overlapping oriented bounding boxes are merged to generate a merged oriented bounding box that covers all overlapping areas.

[0026] After acquiring the angles of each working unit of the working device in real time, the real-time pose (including position and orientation) of the end effector can be calculated using forward kinematics. For the bounding boxes of each obstacle in the obstacle list, coordinate transformation is performed using the real-time pose of the end effector to convert the center point and direction vector of the bounding box to the base coordinate system. Then, in the base coordinate system, it is detected whether there is volume overlap between all pairs of bounding boxes. Volume overlap refers to the spatial intersection of two or more bounding boxes. The judgment criteria can be: checking whether the eight vertices of one bounding box are inside another bounding box, or calculating whether the center distance between two bounding boxes is less than the sum of their semi-axis lengths. For all detected groups of overlapping bounding boxes, a new and larger bounding box that can completely enclose all bounding boxes in the group is calculated, and this new bounding box replaces the original multiple bounding boxes. That is, the overlapping bounding boxes are merged to generate a fused bounding box that covers all overlapping areas. Of course, this fused bounding box is then mapped onto a 3D raster map. In this way, by merging overlapping bounding boxes, the number of obstacle entities in the map is effectively reduced, the computational complexity of subsequent path search algorithms is reduced, and the efficiency of path planning is improved.

[0027] In some embodiments, coordinate transformation is achieved using the following formula: C_new = R•(C_old - P_end) + P_end, where R is the coordinate system rotation matrix determined based on the motion of the working device, P_end is the current position of the end of the working device in the base coordinate system, C_old is the center point of the orientation bounding box before transformation, and C_new is the center point of the orientation bounding box after transformation. Specifically, "C_old - P_end" transforms the obstacle center from the original coordinate system to a vector with the end of the working device as the origin; "R•(C_old - P_end)" rotates the coordinate system, with matrix R determined by the real-time rotation angle of the working device; and "R•(C_old - P_end) + P_end" transforms the rotated vector back to the base coordinate system. Using this formula, the new position of the obstacle in a global, fixed base coordinate system can be accurately calculated based on the motion of the working device itself.

[0028] In some embodiments, the real-time state includes the real-time pose of the end effector; the real-time pose of the end effector is determined in the following manner: Based on the wire encoder and tilt sensor installed on the working device, the extension and retraction of each working unit of the working device, as well as the pitch and yaw angles relative to the horizontal plane, are obtained. Based on the kinematic model of the working device, the theoretical attitude of the end effector is determined according to the extension, pitch and yaw angles. The deformation compensation amount of the working device is determined by querying the deformation database of the working device based on its current extension dimensions and spatial attitude parameters; the deformation database is established through finite element analysis. The theoretical pose of the end effector is corrected based on the deformation compensation of the working device to obtain the real-time pose of the end effector.

[0029] Among them, the wire encoder is a sensor used to measure the extension length of the working unit; the tilt sensor is a sensor used to measure the angle between the working unit and the horizontal plane; the deformation database is a database pre-established through finite element analysis, used to query the structural deformation of the working device caused by gravity and load under different extension scales and spatial attitude parameters.

[0030] It is understandable that factors such as deformation, joint clearance, and environmental disturbances may cause end-effector positioning errors in the working device. Therefore, a draw-wire encoder can be used to read the extension and retraction of each working unit, and a tilt sensor can be used to read the pitch and yaw angles of each working unit relative to the horizontal plane. These data are then input into the kinematic model of the working device (such as the Denavit-Hartenberg model) to calculate the theoretical pose of the end-effector. The kinematic model of the working device is a mathematical model that describes the geometric structure and motion relationships between the working units and is used to calculate the end-effector pose based on the driving quantities (such as joint angles and extension / retraction) of each working unit. This model includes, but is not limited to, the Denavit-Hartenberg model, the exponential product formula, or analytical models based on mechanism geometry. For different types of working devices, the specific parameters and structures of their kinematic models differ, but the technical means for establishing and applying this model are generally known in the art. Taking the working device as a boom and the working unit as a segment as an example, the kinematic chain can be solved based on the Denavit-Hartenberg model, calculating the theoretical pose of the end-effector in the base coordinate system step by step. The specific formula is as follows:

[0031] in, Let i be the design length of the i-th arm segment. Let be the extension / retraction amount of the i-th arm segment. Let be the pitch angle between the i-th arm segment and the horizontal plane. Let yaw angle be the yaw angle of the i-th arm segment relative to the horizontal plane.

[0032] Finally, based on the current extension dimension and spatial attitude parameters of the working device, a deformation database pre-established using finite element analysis software (such as Ansys) can be consulted to obtain the deformation compensation amount under the current attitude through interpolation. For boom-type equipment, the extension dimension can be specifically represented by the boom length, while for other equipment, it may be other forms of linear displacement. For example, taking the working device as a boom, the working unit as a boom segment, the extension dimension as the boom length, and the spatial attitude parameter as the amplitude angle, the deformation compensation amount under the current attitude can be obtained using the formula... Calculate deformation compensation , These are the fitting coefficients. For variable angle, The arm length is used. Furthermore, for the boom, spatial attitude parameters may include the amplitude angle and / or yaw angle; for the robot, these are directly referred to as joint angles. Finally, this compensation is used to correct the theoretical pose, that is, to correct the theoretical height of the end effector, thereby obtaining an accurate real-time pose of the end effector that reflects the actual structural deformation. Thus, by introducing deformation compensation, the influence of the flexible deformation of the end effector and joint clearance on positioning accuracy is effectively overcome, improving the end effector positioning accuracy.

[0033] In some embodiments, the real-time generation of control instructions for work units using reinforcement learning algorithms is implemented using a hierarchical decision architecture, including: In the global navigation layer, a global guidance path from the starting point to the target pose is planned in the 3D grid map; At the local control layer, guided by the global guidance path, the action instructions of the work unit are calculated in real time through reinforcement learning algorithms, and the action instructions are output as the control instructions of the work unit.

[0034] In this process, algorithms such as A* search can be used to plan a global guidance path from the starting point to the target pose in a 3D grid map. A* search is a classic graph search algorithm used to find the shortest path from the starting point to the destination in a 3D grid map. At the global navigation layer, given the target pose, the A* algorithm is run on a coarse-resolution 3D grid map to quickly plan a global guidance path from the current position of the working device to the target pose. This global guidance path may not be smooth and may not consider the kinematic details of the working device, but it indicates the general direction of movement. At the local control layer, reinforcement learning algorithms (such as DDQN) use this global guidance path as a guide, while combining high-resolution local grid maps and detailed joint states to calculate the optimal action command for each step in real time. This allows the working device to both follow the global guidance path and flexibly avoid unknown or newly appearing local obstacles along the global guidance path. Thus, by providing target guidance to the reinforcement learning algorithm through the global guidance path, it is possible to effectively prevent the reinforcement learning algorithm from getting stuck in local loops in complex obstacle environments. Meanwhile, the A* algorithm combines reinforcement learning algorithms for hierarchical decision-making, which can ensure real-time performance and generate safe and smooth motion trajectories.

[0035] In some embodiments, the reinforcement learning algorithm is implemented based on a Markov decision model; The state space of the Markov decision model includes: the grid coordinates of the end effector mapped in the three-dimensional grid map, the discretized combination of the angles of each working unit of the working device, and the obstacle occupancy information within a preset range around the working device. The action space of the Markov decision model is defined as the discrete set of the angular increments Δq_i of each working unit of the working device, where Δq_i is taken from the discrete set {-δ, 0, +δ}, and δ is the preset step angle; The state transition function of the Markov decision model is modeled as follows: based on the kinematic model of the working device, the predicted next state is calculated from the current state and the current action; random disturbances are introduced into the predicted next state to simulate control error. The reward function of the Markov decision model is a multi-objective weighted function, and the corresponding immediate reward is calculated by weighting the reward components based on obstacle avoidance safety, path efficiency, motion smoothness and target guidance.

[0036] The Markov Decision Model (MDP) is a mathematical model framework for sequential decision-making, comprising four core elements: state, action, state transition, and reward. This embodiment uses the Markov Decision Model to transform the continuous motion control problem of a work device into a discrete sequential decision-making problem. The immediate reward r(s,a) can be expressed as r(s,a) = w_1·r_obstacle avoidance + w_2·r_efficiency + w_3·r_smoothness + w_4·r_target, where w_i are weighting coefficients, each corresponding to the reward components for obstacle avoidance safety, path efficiency, motion smoothness, and target guidance, respectively. For obstacle avoidance safety rewards, a negative reward is given when the predicted new state leads to a collision; otherwise, it is zero. For path efficiency rewards, it is proportional to the distance the end-effector of the work device approaches the target point. For motion smoothness rewards, it is inversely proportional to the magnitude of the change in the work unit's action. For target guidance rewards, a positive reward is given when the end-effector of the work device reaches the target pose.

[0037] In some embodiments, the reinforcement learning algorithm includes a dual deep Q-network; the reinforcement learning algorithm calculates action instructions for the work unit in real time, including: The three-dimensional grid map, the real-time status of each working unit of the working device, and the real-time pose of the end of the working device and the target pose are used as status inputs. The Q-value of each action is calculated using a dual-depth Q-network; The optimal action is determined based on the Q value, and the optimal action is output as the action instruction for the work unit.

[0038] Among them, the Dual Deep Q-Network (DDQN) is an advanced deep reinforcement learning algorithm that solves the problem of overestimation of Q-values ​​in traditional DQN by decoupling action selection and value evaluation. Before inputting the 3D grid map, the real-time status of each working unit of the work device, and the real-time pose of the end effector and the target pose into the Dual Deep Q-Network, different preprocessing operations are required. For the 3D grid map, since it is a 3D matrix composed of 0s and 1s, representing the occupancy status of obstacles in the base coordinate system of the work device (0 = free, 1 = occupied), this 3D matrix is ​​input into a pre-trained 3D convolutional neural network. The 3D convolutional neural network extracts high-level spatial features of the environment and outputs a fixed-length environmental feature vector (e.g., 256-dimensional). This vector encodes geometric information crucial for planning, such as the spatial layout of obstacles and the width of passages; therefore, this environmental feature vector can be called the obstacle spatial embedding vector. For the real-time status of each working unit of the working device, the real-time status of each working unit can be regarded as the current angle of each working unit. A long, sparse binary vector, i.e., the working unit status vector, can be generated by one-hot encoding of the current angle of each working unit. Specifically, the continuous angle values ​​of each working unit are first discretized into a preset interval (e.g., dividing 0° to 180° into 20 intervals). Then, a binary vector is generated for each working unit (only the position corresponding to the current angle interval is 1, and the rest are 0). Finally, the one-hot encoded vectors of all working units are concatenated. For the real-time pose of the working device end effector and the target pose, the difference between the two can be calculated. This difference includes the position difference and the attitude difference. Simultaneously, these differences can be normalized, for example, by dividing the difference by the maximum range of the workspace, so that it falls within the interval [-1, 1], thereby generating a small, dense floating-point vector, i.e., the normalized target pose difference vector.

[0039] In the calculation process, the environmental feature vector from the 3D convolutional neural network, the one-hot encoded job unit state vector, and the normalized target pose difference vector are first concatenated to form a comprehensive state feature vector. This vector is then further fused and transformed through one or more fully connected layers to capture the complex relationship between the environment, the user's own state, and the task objective. Next, the fused state feature vector is input into the online network of the DDQN network. After forward propagation processing by the online network, a corresponding Q-value is output at the output layer for each possible action in the action space. The Q-value represents the estimated long-term expected reward obtained after performing the action in the current state; a higher Q-value indicates a better action. In this embodiment, a greedy strategy can be used, directly selecting the action with the highest Q-value from all output Q-values ​​as the optimal action. Since an action is a vector whose elements correspond to the angular increments of each job unit, the optimal action can be output as an action command. Upon receiving the action command, the servo controller of the job unit will correspondingly drive the motor or hydraulic valve to precisely rotate each job unit by the specified angle. Thus, the DDQN algorithm effectively alleviates the problem of Q-value overestimation, making action selection more reliable while ensuring the real-time nature of decision-making.

[0040] In some embodiments, the dual-depth Q-network is trained by minimizing the error between the target Q-value and the current Q-value, where Q = r(s, a) + γ • Q target (s', a'), r(s, a) represents the immediate reward, γ represents the discount factor, and Q represents the discount factor. target This is the estimate for maximum profit.

[0041] Where s' represents the next state the environment transitions to after action a is performed in the current state s. a' represents all possible candidate actions in the next state s'. Q target (s', a') represents a set of predicted long-term values ​​from the DDQN network for any candidate action a' in state s', with the maximum value being the predicted value. γ is a number between 0 and 1, used to weigh the importance of the current immediate reward against the potential future reward. During the training of a dual-deep Q network, an objective needs to be defined to guide the update of the network parameters; this objective is given by the formula Q = r(s, a) + γ •Q. target (s', a') is defined as follows: r(s, a) is the immediate reward obtained after performing action a, γ is the discount factor, and Q is the immediate reward obtained after performing action a. target(s',a') is the maximum Q-value estimate among all possible actions a' in the next state s', calculated by an independent, slowly updated target network. During training, the network parameters are gradually adjusted by minimizing the error (such as mean squared error) between the current network's predicted Q-value and the target Q-value, making its predictions increasingly accurate and eventually approaching the optimal policy.

[0042] Based on the same inventive concept as the foregoing embodiments, this application also provides a working device, including a controller and a working apparatus communicatively connected to the controller. The working apparatus includes a working unit, and the controller executes a computer program to implement the working path planning method of the working apparatus as described above. Optionally, the working apparatus includes a boom. Optionally, the working device may further include a first environmental sensing device and a second environmental sensing device deployed on the working apparatus.

[0043] Based on the same inventive concept as the foregoing embodiments, this application also provides an electronic device, including a storage medium and a controller, wherein a computer program is stored on the storage medium, and when the computer program is executed by the controller, it implements the operation path planning method of the operation device as described above.

[0044] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the operation path planning method of the operation device as described above.

[0045] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0046] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0047] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for planning the work path of a work device, the work device comprising work units, characterized in that, The method includes: Obtain a list of obstacles around the working device; Based on the obstacle list, a 3D grid map is constructed for path planning; Based on the real-time status of the working device, the target pose, and the three-dimensional grid map, a reinforcement learning algorithm is used to generate control commands for the working unit in real time, so as to drive the working device to move towards the target pose along a collision-free path.

2. The method according to claim 1, characterized in that, The construction of a 3D raster map for path planning based on the obstacle list includes: The working space of the operating device is discretized to form a three-dimensional grid space; Transform and merge the bounding boxes of each obstacle in the obstacle list into a base coordinate system with the rotation center of the working device as the origin; The merged bounding boxes are mapped onto the 3D raster space to generate the 3D raster map.

3. The method according to claim 2, characterized in that, The step of converting and fusing the bounding boxes of each obstacle in the obstacle list into a base coordinate system with the rotation center of the working device as the origin includes: Based on the real-time pose of the end of the working device, the bounding boxes of each obstacle are mapped to the base coordinate system with the rotation center of the working device as the origin through coordinate transformation; In the base coordinate system, determine whether volume overlap occurs between different oriented bounding boxes; The overlapping oriented bounding boxes are merged to generate a merged oriented bounding box that covers all overlapping areas.

4. The method according to claim 1, characterized in that, The real-time status includes the real-time pose of the end effector; the real-time pose of the end effector is determined in the following way: Based on the pull-wire encoder and tilt sensor installed on the working device, the extension and retraction of each working unit of the working device, as well as the pitch and yaw angles relative to the horizontal plane, are obtained respectively. Based on the kinematic model of the working device, the theoretical pose of the end effector of the working device is determined according to the extension, pitch angle and yaw angle. Based on the current extension scale and spatial attitude parameters of the working device, the deformation database of the working device is queried to determine the deformation compensation amount of the working device; The deformation database was established through finite element analysis; The theoretical pose of the end effector of the working device is corrected based on the deformation compensation amount of the working device to obtain the real-time pose of the end effector.

5. The method according to any one of claims 1 to 4, characterized in that, The real-time generation of control instructions for the work unit using reinforcement learning algorithms is implemented using a hierarchical decision architecture, including: In the global navigation layer, a global guidance path from the starting point to the target pose is planned in the three-dimensional grid map; In the local control layer, the global guidance path serves as a macroscopic guide. The reinforcement learning algorithm is used to calculate the action instructions of the work unit in real time, and the action instructions are output as the control instructions of the action unit.

6. The method according to claim 5, characterized in that, The reinforcement learning algorithm is implemented based on the Markov decision model; The state space of the Markov decision model includes: the grid coordinates of the end effector of the working device mapped in the three-dimensional grid map, the discretized combination of the angles of each working unit of the working device, and the obstacle occupancy information within a preset range around the working device. The action space of the Markov decision model is defined as the discrete set of the angular increments Δq_i of each work unit of the work device, where Δq_i is taken from the discrete set {-δ, 0, +δ}, and δ is a preset step angle. The state transition function of the Markov decision model is modeled as follows: based on the kinematic model of the working device, the predicted next state is calculated from the current state and the current action; random disturbances are introduced into the predicted next state to simulate control error. The reward function of the Markov decision model is a multi-objective weighted function, and the corresponding immediate reward is calculated by weighting the reward components based on obstacle avoidance safety, path efficiency, motion smoothness, and target guidance.

7. The method according to claim 6, characterized in that, The reinforcement learning algorithm includes a dual deep Q-network; the step of calculating the action instructions of the work unit in real time using the reinforcement learning algorithm includes: The three-dimensional grid map, the real-time status of each working unit of the working device, and the real-time pose and target pose of the end of the working device are used as status inputs; The Q-value of each action is calculated using a dual-depth Q-network; The optimal action is determined based on the Q value, and the optimal action is output as the action command of the work unit.

8. A working device, characterized in that, include: A controller and a working device communicatively connected to the controller; the working device includes a working unit, and the controller executes a computer program to implement the working path planning method of the working device as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, include: A storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the operation path planning method of the operating device as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device stores a computer program, which, when executed by a processor, implements the operation path planning method for the operating device as described in any one of claims 1 to 7.

Images