Mechanical device and its operation control method

Abstract

The application provides a mechanical device and a work control method thereof. The method comprises: obtaining observation data of the mechanical device for trajectory tracking control; searching, from a plurality of action combinations constructed for the mechanical device, a plurality of candidate action combinations corresponding to an executed action of the mechanical device at a current time; determining, based on the executed action at the current time, the plurality of candidate action combinations, and the observation data, a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations; and controlling the mechanical device to work according to a target action combination corresponding to an optimal predicted trajectory at a next time, the optimal predicted trajectory being a predicted trajectory with the highest matching degree to a target planned trajectory among the plurality of predicted trajectories. Through the application, the generalization of the algorithm can be improved, and the migration between different mechanical devices can be facilitated.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of engineering machinery, and in particular to a mechanical device and a work control method thereof. BACKGROUND

[0002] Engineering machinery devices, for example, excavators can be used to fill, dig or move materials, and loaders can be used to shovel, load, unload, transport and other bulk materials such as earth and stone, to carry out material collection, loading and other operations.

[0003] Taking an excavator as an example, it is a typical hydraulic control system, and its motion characteristics have problems such as high time delay, significant time-varying characteristics, and serious joint coupling. The control problem of such highly nonlinear systems is still a big challenge. At present, the control in the industry mostly adopts the PID control method, and a high tracking accuracy (such as current, speed, and position) is achieved on the mechanical arm through three-loop control, but this method needs complex PID parameter tuning, which is time-consuming and laborious and has poor portability, and a large amount of repeated operations are needed for another excavator.

[0004] Trajectory control is a common problem in robot systems, which requires the robot to move along a specific trajectory. A simplified trajectory control method is to track each point of the trajectory one by one, but this method is prone to action jamming because the subsequent trajectory is not considered. SUMMARY

[0005] Therefore, the embodiments of the present application provide at least a mechanical device and a work control method thereof to overcome at least one of the above-mentioned defects.

[0006] In a first aspect, the exemplary embodiments of the present application provide a work control method of a mechanical device, the method comprising: obtaining observation data of the mechanical device for trajectory tracking control, the mechanical device comprising a plurality of movable joints, the observation data comprising joint positions and joint velocities of each joint; searching, from a plurality of action combinations constructed for the mechanical device, a plurality of candidate action combinations corresponding to an executed action of the mechanical device at a current time; determining, based on the executed action at the current time, the plurality of candidate action combinations, and the observation data, a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations; and controlling the mechanical device to perform work according to a target action combination corresponding to an optimal predicted trajectory at a next time, the optimal predicted trajectory being a predicted trajectory with the highest matching degree to a target planning trajectory among the plurality of predicted trajectories.

[0007] In a second aspect, the embodiments of the present application further provide a mechanical device, comprising a processor configured to: acquire observation data of the mechanical device for trajectory tracking control, the mechanical device comprising a plurality of movable joints, and the observation data comprising joint positions and joint velocities of each joint; search, from a plurality of action combinations constructed for the mechanical device, a plurality of candidate action combinations corresponding to an executed action of the mechanical device at a current time; determine, based on the executed action at the current time, the plurality of candidate action combinations, and the observation data, a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations; and control the mechanical device to perform work according to a target action combination corresponding to an optimal predicted trajectory at a next time, the optimal predicted trajectory being a predicted trajectory with the highest matching degree to a target planning trajectory among the plurality of predicted trajectories.

[0008] In a third aspect, the embodiments of the present application further provide a computer-readable storage medium, having stored thereon a computer program, which, when executed by a processor, performs the steps of the work control method of the mechanical device.

[0009] The mechanical device and the work control method thereof provided by the embodiments of the present application can improve the generalization of the algorithm and facilitate the migration between different mechanical devices.

[0010] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following will describe preferred embodiments in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0012] Figure 1 A flowchart of the work control method of the mechanical device provided by the exemplary embodiments of the present application is shown;

[0013] Figure 2 A flowchart of the step of constructing a plurality of action combinations provided by the exemplary embodiments of the present application is shown;

[0014] Figure 3 A single-joint action schematic diagram provided by the exemplary embodiments of the present application is shown;

[0015] Figure 4 A flowchart of the step of determining a plurality of single-joint actions of a joint provided by the exemplary embodiments of the present application is shown;

[0016] Figure 5 Fig. 1 shows a schematic diagram of a trajectory tracking control process according to an example embodiment of the present application;

[0017] Figure 6 Fig. 2 shows a flowchart of a process for determining an optimal predicted trajectory according to an example embodiment of the present application;

[0018] Figure 7 Fig. 3 shows a schematic diagram of a control structure of a mechanical device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application merely serve the purpose of illustrating and describing and should not be used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application.

[0020] In the present specification, the terms “one”, “an”, “the” and “said” are used to indicate that there is one or more element / component / etc.; the terms “include” and “have” are used to indicate an open-ended inclusion and refer to the existence of additional elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as labels and are not a limitation on the number of objects.

[0021] It should be understood that in the embodiments of the present application, “at least one” means one or more, and “multiple” means two or more. “And / or” is merely a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character “ / ” generally represents an “or” relationship between the associated objects. “Including A, B and / or C” means including any one or any two or three of A, B and C.

[0022] It should be understood that in the embodiments of the present application, “B corresponding to A”, “B corresponding to A”, “A corresponding to B” or “B corresponding to A” means that B is associated with A and can be determined according to A. Determining B according to A does not mean that B is determined only according to A, but also can be determined according to A and / or other information.

[0023] In addition, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0024] Engineering machinery equipment, for example, excavators can be filled by digging or moving materials, loaders can be used to shovel, load, unload, transport and stone-like bulk materials such as earth and stone to carry out material collection, loading and other operations.

[0025] Taking an excavator as an example, it is a typical hydraulic control system, and its motion characteristics have problems such as high time delay, significant time-varying characteristics, and serious joint coupling. The control problem of such highly nonlinear systems is still a big challenge. At present, the control in the industry mostly adopts the PID control method, and a high tracking accuracy (such as current, speed, and position) is achieved on the mechanical arm through three-loop control, but this method needs complex PID parameter tuning, which is time-consuming and laborious and has poor portability, and a large amount of repeated operations are needed for another excavator.

[0026] Trajectory control is a common problem in robot systems, which requires the robot to move along a specific trajectory. A simplified trajectory control method is to track each point of the trajectory one by one, but this method is prone to action jamming because the subsequent trajectory is not considered.

[0027] In the related art, the trajectory control of mechanical equipment usually adopts the following control methods:

[0028] PID control method: PID is used to track each point of the trajectory, and the speed of each point in the trajectory is calculated by interpolation. The speed and position of each point in the trajectory are tracked as targets, which can achieve high tracking accuracy. However, this control method needs manual calibration, and lacks a standard process, and more relies on manual experience for trial and error. In addition, the migration of this method has problems, and a new excavator needs to repeat the above process for trajectory tracking, which is time-consuming and laborious. In addition, PID has good control effect on single point, but poor tracking effect on some complex curves.

[0029] MPC control method: build an explicit robot model, get the optimal action through optimization method according to the target trajectory, execute a few steps at a time, and re-plan to reduce the impact of errors. However, this control method requires the establishment of an explicit motion model of the excavator, and also requires manual participation in calibration. In addition, the optimization calculation of MPC is time-consuming, and real-time operation is difficult. The effect of trajectory control is improved compared with PID control.

[0030] RL control method: trajectory planning through reinforcement learning is also a popular approach. Data is obtained online or offline, and a reasonable reward function is set to let the robot learn the desired action through exploration and utilization strategies. This control method essentially also builds a model of the excavator, but only needs to collect data and does not require manual calibration. However, the RL control method is unstable and cannot guarantee optimality, and cannot guarantee control effect. At the same time, the interpretability is poor, and the application is limited in safety priority problems such as excavators.

[0031] To solve the above problems, the mechanical equipment and its operation control method are proposed. In at least one aspect, the problem of poor generalization of traditional PID control method, the need for manual calibration, and high cost is solved. In another aspect, the problem of large calculation amount of MPC control method and difficulty in real-time use is solved. In still another aspect, the problems of unstable training and poor interpretability of RL control method are solved.

[0032] To facilitate the understanding of the present application, the mechanical equipment and its operation control method provided by the embodiments of the present application are described in detail below.

[0033] Please refer to Figure 1 The flowchart of the operation control method of the mechanical equipment provided by the exemplary embodiments of the present application is generally applied to the controller of the mechanical equipment. The above control method can be integrated in the on-board controller of the mechanical equipment, or it can also be run in the independent controller on the mechanical equipment. Through the communication between the independent controller and the on-board controller of the mechanical equipment and / or the driving mechanism, the trajectory tracking and / or the shovel operation process are realized, but the present application is not limited thereto.

[0034] Please refer to Figure 1 In step S101, the observation data used for trajectory tracking control of the mechanical equipment is obtained.

[0035] Exemplarily, the above-mentioned mechanical equipment can include but is not limited to an excavator or a loader. For example, the excavator can fill, dig or move materials by digging, and the loader is an engineering mechanical equipment mainly used for loading, unloading, transporting and transporting various materials and goods.

[0036] The mechanical device provided in the exemplary embodiments of the present application has movable joints, which may, for example, include but are not limited to a cabin, a large arm, a small arm and a bucket. The cabin can be driven to rotate, and the large arm, the small arm and / or the bucket can be driven to perform a scooping action.

[0037] In the exemplary embodiments of the present application, the observation data of the mechanical device may, for example, include but are not limited to joint positions and joint speeds of each joint. For example, sensors can be arranged at the joints to obtain the observation data from the arranged sensors. For example, a heading angle sensor can be arranged at the cabin of the mechanical device to obtain a rotation angle and a rotation speed of the cabin as the joint position and the joint speed of the cabin. In addition, an inclination angle sensor can be arranged at each of the large arm, the small arm and the bucket to obtain an inclination angle value and a rotation speed of the large arm, the small arm and the bucket as the joint position and the joint speed of the large arm, the small arm and the bucket.

[0038] Preferably, the observation data used for the trajectory tracking control may, for example, include but is not limited to observation data at the current time and observation data at a historical time before the current time, i.e., the historical state of the mechanical device is considered in the trajectory tracking control of the mechanical device to facilitate the rapid migration of the trajectory tracking control scheme to different mechanical devices.

[0039] In an optional example, the historical time before the current time can refer to the last time, so that the observation data used for the trajectory prediction in the current round can include observation data Q Figure 5 For example, referring to the trajectory tracking control flowchart shown in the figure, assuming that the current time is t, the observation data used for the trajectory prediction in the current round can include observation data Q t and observation data Q t- 1 .

[0040] It should be understood that the observation data at the current time t and the historical time t-N can also be selected for trajectory prediction, where N is a natural number greater than 1, and the present application does not limit this.

[0041] In step S102, a plurality of candidate action combinations corresponding to the executed action of the mechanical device at the current time are searched from a plurality of action combinations constructed for the mechanical device.

[0042] Currently, the trajectory tracking control of the mechanical device usually involves the following two:

[0043] One is based on Cartesian space, for example, to track the end trajectory of a mechanical device, which may refer to the tooth tip trajectory of a control bucket of a mechanical device.

[0044] Another one is based on joint space, for example, to form the trajectory of a mechanical device based on the joint positions of a plurality of joints included in the mechanical device. The operation control method of the mechanical device of the present application is a trajectory tracking control scheme based on joint space.

[0045] In the embodiments of the present application, the observation data of the mechanical device is used to represent the state of the mechanical device from the joint side, for example, the observation data is used to represent the state of a plurality of joints of the mechanical device, for example, the state of the joint is represented by the joint position and the joint speed. The action of the mechanical device is used to represent the control of the mechanical device from the drive side, for example, the action is used to represent the control amount of the drive mechanism corresponding to each joint, for example, taking the hydraulic system as an example, the action refers to the control amount of the hydraulic system for driving the joint movement.

[0046] In a preferred embodiment of the present application, a plurality of action combinations can be constructed in advance, and the constructed plurality of action combinations are stored in the video memory to perform subsequent processing for determining the optimal predicted trajectory in the video memory. Here, considering that the number of action combinations is very large, by performing trajectory prediction processing in the video memory and directly retrieving a plurality of action combinations from the video memory, the calculation efficiency can be effectively improved. The process of constructing a plurality of action combinations will be introduced below. Figures 2-4

[0047] Figure 2 A flow chart showing the steps of constructing a plurality of action combinations provided by an exemplary embodiment of the present application.

[0048] As shown in Figure 2 , in step S201, for each joint of the movable mechanical device, a plurality of single-joint actions of the joint are determined.

[0049] Taking the above mechanical device including the cabin, the large arm, the small arm and the bucket as an example, for a single joint, a plurality of single-joint actions of the joint can be determined in advance.

[0050] Referring to Figure 3 the single-joint action diagram, the diagram represents the action space of a single joint, in this example, the action space is represented by a two-dimensional coordinate system, the horizontal coordinate of the action space represents the number of steps executed in time sequence, and the vertical coordinate of the action space represents the action of the joint, i.e. the control amount for driving the joint movement. For each joint, the control amount of the drive mechanism for driving the joint movement can be normalized to normalize the action of the joint to the range of [-1, 1].​

[0051] Figure 3 A curve shown represents the joint future multi-step action, taking 0 as the starting point, using a straight line with different slopes as the action, which represents the change of the action over time.

[0052] The steps of determining a multi-step single-joint action of a joint will be described below with reference to Figure 4 The multi-step single-joint action of each joint of the mechanical device can be determined by the processing steps of Figure 4 .

[0053] In the embodiments of the present application, the multi-step single-joint action of each joint can include an initial action and subsequent actions, for example, one initial action and multiple subsequent actions. In the trajectory tracking control of the present application, since only the first step action is performed subsequently, the determination of the initial action is more important.

[0054] As shown in Figure 4 , in step S301, a first data point projected by the joint on the action space is determined.

[0055] Taking the example shown in Figure 3 , the first data point can refer to the data point corresponding to the starting step 0 on the action space, and the initial control amount of the first data point is 0.5. It should be understood that it can also be at 0.2, 0.6, etc.

[0056] In step S302, the action within the neighborhood of the first data point is determined as the initial action of the joint.

[0057] Here, a plurality of initial actions can be determined from the neighborhood according to a preset traversal fineness. It should be understood that the finer the traversal fineness, the larger the data amount of the action combination, and correspondingly, the coarser the traversal fineness, the smaller the data amount. The traversal fineness can be determined according to actual needs. When determining the above plurality of actions and subsequently forming the action combination, the coupling of the joints of the mechanical device also needs to be considered.

[0058] Referring to Figure 3In the example shown, assuming that the first data point is at (0, 0.5), the neighborhood of the first data point (e.g., the step interval 0.5-1.5 corresponding to the black box in the diagram) is represented in this example, it should be understood that the neighborhood described above can refer to a step interval after the step 0 of the first data point, and those skilled in the art can select a step interval as the neighborhood of the first data point according to actual needs. In this example, the step interval around step 1 represented by the black box is selected as the neighborhood, and this selection method is only an example, and the present application is not limited thereto. Assuming that 5 data points are searched in the neighborhood according to the preset traversal fineness, the corresponding 5 initial actions are determined. Here, the traversal fineness can refer to the accuracy of constructing the action curve of a single joint. For example, taking the joint of a mechanical device as the seat, assuming that the movable range of the seat is 0-360 degrees, and for example, when 60 degrees is selected as the traversal fineness, a plurality of joint positions can be obtained within the movable range of 0-360 degrees starting from the joint position corresponding to the first data point according to the traversal fineness of 60 degrees. Mapping the corresponding control amount of the plurality of joint positions in the drive system to the joint space of the seat can obtain the plurality of data points searched in the neighborhood of the first data point. It should be understood that those skilled in the art can determine the traversal fineness according to actual needs, and a uniform traversal fineness can be selected for all joints, or different traversal finenesses can be selected for different joints, which is not limited by the present application.

[0059] In step S303, the action slope is determined. Here, the action slope can refer to the slope of the line segment from the first data point to the second data point projected on the action space of the joint.

[0060] As described above, after determining a plurality of initial actions on the action space, for each initial action, the second data point corresponding to the initial action is connected with the first data point to determine the slope of the connected line segment, i.e., a plurality of initial actions correspond to a plurality of line segments, and each line segment corresponds to a slope.

[0061] As described above Figure 3 In the example shown, the second data points corresponding to the 5 initial actions are connected with the first data point respectively to form 5 line segments, such as the green, orange, blue, red, and purple line segments shown in the figure, and further, the slope of each line segment is determined as the action slope.

[0062] In step S304, the action curve is obtained by extending from the first data point according to the action slope in the action space of the joint.

[0063] Preferably, the action range of the joint is constrained in the action space of the joint, such as the range normalized to [-1, 1] as described above, see Figure 3In the illustrated example, in the action space of the joint, starting from the first data point, the action is extended according to the action slope, and the part exceeding the action range is limited at the boundary value of the action range. For example, in the case of the green action curve, when the action is extended according to the action slope to the upper boundary limit value, the action is limited at the upper boundary limit value 1. The orange action curve is similar. For the purple action curve, when the action is extended according to the action slope to the lower boundary limit value, the action is limited at the lower boundary limit value -1, so as to represent that after the third step, the control amount of the driving mechanism corresponding to the joint remains unchanged, maintaining the -1 inverse normalized control amount.

[0064] In step S305, the subsequent action of the joint is determined based on the third data point falling on the action curve.

[0065] Here, the subsequent action of the joint can include multiple steps of actions, for example, in Figure 3 In the illustrated example, the action corresponding to each step on the abscissa can be found on an action curve to form the subsequent action of the joint. It can be understood that in Figure 3 An action curve in the joint constitutes a multi-step single-joint action of the joint.

[0066] In order to reduce the amount of calculation, the overall action can be modeled as a line segment with a fixed slope, and the value exceeding the boundary range is limited at the boundary.

[0067] Returning to Figure 2 In step S202, the multiple single-joint actions of the joints are combined to form multiple action combinations.

[0068] Here, a preset step number can be set for the action combination, and each action combination includes the same preset step number. At this time, each action combination includes multiple actions of the preset step number, and each action is composed of a single-joint action of each joint at the step number.

[0069] For example, an action combination is composed of 3 steps of actions, and for 4 movable joints of the mechanical device, each step of action includes 4 actions of the 4 joints at the step number. For example, each joint has the above Figure 3 The single-joint action diagram is illustrated, and joint A is taken as an example. Based on one action curve in the single-joint action diagram of joint A, an action at the step number is determined. Correspondingly, based on one action curve in the single-joint action diagram of joint B, an action at the step number is determined, and so on. The 4 actions respectively determined for the 4 joints at the step number are combined to form one step of action in the action combination. Referring to the above process, other steps of action in the action combination can be continuously determined. Here, each joint corresponds to multiple action curves, and the multiple action curves of the multiple joints are combined to form a very large number of action combinations.

[0070] A series of action combinations are pre-generated in the joint space, ensuring that the action combinations cover the entire joint space, and the number of action combinations affects the real-time performance of the overall algorithm. At the same time, when the joints are subjected to action combinations, the coupling of the joints of the mechanical device also needs to be considered.

[0071] In the embodiments of the present application, each candidate action combination includes a plurality of actions to be performed after the predicted execution action of the mechanical device at the current time. For example, for any action of the mechanical device, the plurality of actions to be performed after the action can be predicted in the manner as above, and here, a plurality of cases exist in which the action can be connected to the action, and one case can be formed into a candidate action combination.

[0072] For example, the plurality of action combinations constructed above can form an action combination sequence, and at this time, the required action combination sequence can be searched as a candidate action combination from the preloaded action combination sequence according to the execution action at the current time, such as cutting from the preloaded action combination sequence, requiring the starting point to be the execution action at the current time.

[0073] In order to speed up the operation, a fixed slope can be used when forming the action curve in the action space, and in addition, the action combination can be pre-generated in the video memory, and subsequent reading can be directly from the video memory, avoiding switching data from the memory to the video memory in the real-time calculation process, which can significantly speed up the calculation efficiency.

[0074] Return Figure 1 In step S103, based on the execution action at the current time, the plurality of candidate action combinations, and the obtained observation data, a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations are determined.

[0075] In a preferred embodiment of the present application, a pre-trained world model is used to determine the predicted trajectory corresponding to each candidate action combination. Preferably, to improve the prediction accuracy, the world model is a single-step prediction world model, and the predicted trajectory corresponding to each candidate action combination is obtained through multiple iterations, so as to predict the execution of the mechanical device under a given action. In the embodiments of the present application, the trajectory of the mechanical device can refer to the joint position of each joint.

[0076] Currently, for trajectory tracking control, the core lies in constructing a dynamic transfer model of the robot, and obtaining the optimal action according to the model. There are various ways to obtain the model, one is to construct an explicit model through a differential equation, the advantage of which is that the optimal action can be obtained through an optimization method, and the other way is to construct an implicit model. Considering that the motion characteristics of mechanical equipment (such as excavators) are relatively complex, the world model used in the embodiments of the present application is obtained through neural network training, which belongs to an implicit model.

[0077] The mechanical equipment trajectory control method based on the world model proposed in the present application can solve the problems of manual calibration, low calculation efficiency and poor result interpretability in mechanical equipment trajectory control by combining the trained world model with trajectory search to find the action execution corresponding to the optimal trajectory.

[0078] In addition, the control process also considers the time delay characteristics of the mechanical equipment to make the result more accurate. Moreover, the generalization of the model is improved by considering the historical state of the mechanical equipment, so that the control scheme can be quickly migrated to different mechanical equipment.

[0079] For each candidate action combination, the following processing is performed at each iteration: the executed action at the current time, the candidate action combination, the current time and the observation data at the historical time are input into the world model to obtain a one-step predicted action for the mechanical equipment.

[0080] In combination with Figure 5 the trajectory tracking control flowchart shown in the figure, for time t, the executed action at the current time is A t , the candidate action combination includes 3-step actions, such as A t , A t+ 1 , A t+ 2 , A t+ 3 , the observation data at the current time t is Q t , and the observation data at the historical time t-1 is Q t- 1 .

[0081] Based on this, at time t, A t , A t+ 1 , A t+ 2 , A t+ 3 , Q t , Q t- 1 can be input into the world model, at this time, the world model outputs a one-step prediction, such as S't+ 1 determining the predicted trajectory S' of the candidate action combination t+ 1 corresponding predicted action, e.g., A' t+ 1 taking the predicted action as the executed action at the next iteration, e.g., A' is taken as A t+ 1 A t+ A 1 A t+ A 2 A t+ Q 3 Q t Q t- 1 inputting into the world model to output the next predicted step, e.g., S' t+ 2 or, A' can also be inputted into the world model to output the next predicted step S' t A t+ A 1 A t+ A 2 A t+ Q 3 Q t Q t- 1 inputting into the world model to output the next predicted step S' t+ 2 iterating in this way, and the iteration is terminated when the number of predicted actions outputted by the world model reaches a preset number, thereby obtaining the predicted trajectory corresponding to the candidate action combination.

[0082] To accelerate the operation efficiency, the pre-generated actions can be normalized (which can be completed in advance and stored), and then the normalized candidate action combinations are inputted into the world model in batches to iteratively generate the future states, and finally the states outputted by the world model (i.e., the predicted trajectory) are de-normalized to obtain the result.

[0083] taking Figure 5 as an example, taking the action combination including 3 steps as an example, at time t, a predicted trajectory S' of a candidate action combination is outputted by the world model t+ 1 S' t+ S' 2 S' t+ S' 3 S' t+ 4 .

[0084] In an optional embodiment, the world model for trajectory prediction of the mechanical device in the embodiments of the present application can be obtained through neural network training. For example, the world model can be obtained through the following training method: training data of the mechanical device is obtained, which can be obtained based on historical actual operation data of the mechanical device. For example, the training data includes a plurality of training samples, each training sample includes training observation data of the mechanical device, a training action combination, and a work trajectory. The training observation data can be composed of observation data at adjacent time points in the historical actual operation data of the mechanical device, such as observation data at time t and time t+1. The training action combination can be composed of a plurality of actions at adjacent time points in the historical actual operation data of the mechanical device. The time point of the training observation data in a training sample corresponds to the training action combination. For example, the training action combination can include actions at time t+1, t+2, t+3, and t+4. Correspondingly, the work trajectory in the training sample can refer to the actual motion trajectory of the mechanical device at time t+2, t+3, t+4, and t+5 in the historical actual operation data.

[0085] After obtaining the plurality of training samples, the training observation data and the training action combination in each training sample are taken as the input of an initial neural network, and the work trajectory in each training sample is taken as the output of the initial neural network. The initial neural network is trained to obtain a world model that meets the trajectory prediction requirements.

[0086] Return Figure 1 In step S104, the mechanical device is controlled to perform work according to the target action combination corresponding to the optimal predicted trajectory at the next time point.

[0087] Here, the predicted trajectories obtained by different candidate action combinations are different, and a predicted trajectory that is closest to the target planning trajectory needs to be found as the target trajectory. That is, the optimal predicted trajectory is a predicted trajectory that has the highest matching degree with the target planning trajectory among the plurality of predicted trajectories.

[0088] For example, the target planning trajectory can be given by a planning module, for example, a trajectory determined by the planning module in real time / advance using any existing trajectory algorithm. In addition, the target planning trajectory can also be a trajectory artificially specified according to a trajectory tracking task, and the determination method of the target planning trajectory is not limited in the present application.

[0089] The process of searching for the optimal predicted trajectory from the plurality of predicted trajectories will be described below. Figure 6 It should be understood that Figure 6 The optimization method shown is only an optional example, and the present application is not limited thereto. The trajectory with the highest matching degree with the target planning trajectory can also be determined through other methods.

[0090] Figure 6 A flow chart illustrating the steps of determining the optimal predicted trajectory provided by the example embodiments of the present application is shown.

[0091] As shown in step S401, for each predicted trajectory, the mean square error value of each trajectory point on the predicted trajectory relative to the target planning trajectory is calculated. Figure 6

[0092] Here, the method of calculating the mean square error is well known in the art, and the present application will not repeat the details.

[0093] In an optional example, the sum of the mean square error values of all trajectory points on a predicted trajectory can be determined as the matching index of the predicted trajectory, to screen out the optimal predicted trajectory.

[0094] In addition to the above method, the hydraulic system of the mechanical equipment can also be considered to have a time delay characteristic, and a weighted mean square error can be introduced, in which the later the trajectory point, the higher the corresponding weight value.

[0095] For example, in step S402, for each predicted trajectory, a weight value is set for each trajectory point on the predicted trajectory according to the action execution order.

[0096] Here, the weight value corresponding to the trajectory point at the front of the action execution order is larger, and the weight value corresponding to the trajectory point at the back of the action execution order is smaller. For example, the weight value can be exponentially decreased by e.

[0097] In step S403, for each predicted trajectory, a matching index representing the matching degree of the predicted trajectory is calculated based on the mean square error value of each trajectory point on the predicted trajectory and the corresponding weight value.

[0098] In step S404, an optimal predicted trajectory is determined according to the matching index of all predicted trajectories.

[0099] Here, the predicted trajectory corresponding to the minimum matching index can be determined as the optimal predicted trajectory.

[0100] In a preferred embodiment of the present application, the target action combination corresponding to the optimal predicted trajectory is stored in the memory for calling.

[0101] Preferably, in the embodiments of the present application, the mechanical equipment is controlled to perform one step action at the next time, and for example, the first step action in the target action combination is extracted from the memory to control the mechanical equipment to perform the first step action at the next time, which can reduce the influence of system error compared with directly performing all actions.

[0102] ​Here, after controlling the mechanical device to perform the first action in the next time (e.g., t+1), the first action performed in the next time is taken as the executed action in the current time (e.g., t) as described above, and the processing flow described above in the embodiments of the present application is repeated to obtain the optimal predicted trajectory corresponding to the time after the next time (e.g., t+2), and the first action in the action combination corresponding to the optimal predicted trajectory is controlled to be performed by the mechanical device at t+2, and so on, until the mechanical device completes the movement of the target planning trajectory.

[0103] In the embodiments of the present application, the observation data includes joint speed and joint position, the action of the joint refers to the control amount of the hydraulic system for driving the joint movement, and the trajectory of the mechanical device includes the joint position of each joint. Therefore, it can be understood that, in the operation control scheme of the present application, the joint position (i.e., the predicted trajectory) of each joint of the mechanical device can be predicted based on the combination of the joint speed, the joint position, and the control amount of the driving system, and the control amount for each joint in the driving system is deduced based on the predicted joint position of each joint, so as to control the driving system to control the movement of each joint to the predicted joint position according to the control amount.

[0104] Based on the same application concept, the embodiments of the present application also provide a mechanical device corresponding to the method provided by the above-mentioned embodiments. Since the principle of solving the problem of the mechanical device in the embodiments of the present application is similar to the operation control method of the above-mentioned embodiments of the present application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described again.

[0105] Figure 7 A control structure schematic diagram of the mechanical device provided for the exemplary embodiments of the present application is shown in FIG. 2. As shown in FIG. 2, the mechanical device 200 includes a controller 210, which is configured to perform the following processing: Figure 7

[0106] Obtaining observation data used for trajectory tracking control of the mechanical device;

[0107] Searching, from a plurality of action combinations constructed for the mechanical device, a plurality of candidate action combinations corresponding to the executed action of the mechanical device at the current time;

[0108] For each candidate action combination, determining a predicted trajectory of the mechanical device under the candidate action combination based on the executed action at the current time, the candidate action combination, and the observation data;

[0109] Controlling the mechanical device to perform operation according to a target action combination corresponding to the optimal predicted trajectory at the next time, the optimal predicted trajectory being a predicted trajectory with the highest matching degree to the target planning trajectory among the plurality of predicted trajectories.

[0110] ​In a possible implementation of the present application, the mechanical device comprises a plurality of movable joints, and the observation data for trajectory tracking control comprises observation data of the plurality of joints at the current time and historical times before the current time, and the observation data comprises joint positions and joint velocities of each joint. For example, the joint positions and joint velocities of each joint can be detected by sensors 220 arranged at the joints.

[0111] In a possible implementation of the present application, the controller 210 is further configured to construct the plurality of action combinations by: determining, for each joint of the mechanical device, a plurality of single-joint actions of the joint after the preset action; and combining the plurality of single-joint actions of each joint to form the plurality of action combinations, each action combination comprising a plurality of actions of the preset number of steps, each action being combined by the single-joint actions of each joint at the number of steps.

[0112] In a possible implementation of the present application, the plurality of single-joint actions of each joint comprises an initial action and a subsequent action, and the controller 210 is further configured to determine the initial action of each joint by: determining a first data point on an action space of the joint, the first data point being projected by the preset action; and determining an action within a neighborhood of the first data point as the initial action of the joint.

[0113] In a possible implementation of the present application, the controller 210 is further configured to determine the subsequent action of each joint by: determining an action slope, the action slope being a slope of a line segment from the first data point to a second data point on the action space of the joint, the second data point being projected by the initial action; extending from the first data point according to the action slope to obtain an action curve on the action space of the joint; and determining the subsequent action of the joint based on a third data point falling on the action curve.

[0114] In a possible implementation of the present application, each joint of the mechanical device is driven by a hydraulic system 240, and the action of the joint refers to a control amount of the hydraulic system 240 for driving the joint to move, wherein the horizontal coordinate of the action space represents the number of steps, and the vertical coordinate of the action space represents an action range of the joint.

[0115] In a possible implementation of the present application, the controller 210 is further configured to form the action curve by: extending from the first data point according to the action slope on the action space of the joint, and limiting a part exceeding the action range at a boundary value of the action range.

[0116] In a possible implementation of the present application, each candidate action combination comprises a plurality of actions to be performed after a performed action of the mechanical device at the current time.

[0117] In a possible implementation of the present application, the controller 210 is further configured to determine the predicted trajectory corresponding to each candidate action combination by: obtaining the predicted trajectory corresponding to the candidate action combination through multiple iterations by using the pre-trained world model, and performing the following processing at each iteration: inputting the executed action, the candidate action combination, and the observation data into the world model to obtain a one-step predicted action for the mechanical device, wherein the predicted action output by the world model at the current iteration is used as the executed action at the next iteration, and the iteration is terminated when the number of steps of the predicted action output by the world model reaches a preset number of steps.

[0118] In a possible implementation of the present application, the controller 210 is further configured to determine the optimal predicted trajectory by: calculating, for each predicted trajectory, a mean square error value of each trajectory point on the predicted trajectory relative to the target planning trajectory; setting, for each predicted trajectory, a weight value for each trajectory point on the predicted trajectory in the action execution order, wherein the trajectory point earlier in the action execution order corresponds to a larger weight value; calculating, for each predicted trajectory, a matching index used to represent the matching degree of the predicted trajectory based on the mean square error value of each trajectory point on the predicted trajectory and the corresponding weight value; and determining an optimal predicted trajectory according to the matching index of all predicted trajectories.

[0119] In a possible implementation of the present application, the controller 210 is further configured to: store the constructed multiple action combinations into the video memory 230 to perform the operation for determining the optimal predicted trajectory in the video memory 230.

[0120] In a possible implementation of the present application, the controller 210 is further configured to: store the target action combination corresponding to the optimal predicted trajectory into a memory, wherein the controller 210 extracts a first-step action in the target action combination from the memory; and control the mechanical device to execute the first-step action at the next moment.

[0121] In a possible implementation of the present application, the controller 210 is further configured to: continue to perform the trajectory prediction for the mechanical device based on the executed first-step action at the next moment until the mechanical device completes the movement of the target planning trajectory.

[0122] Based on the above device, the generalization of the algorithm can be improved, and the migration between different mechanical devices can be facilitated.

[0123] The embodiment of the present application further provides a computer readable storage medium, and the storage medium stores a computer program. When the computer program is run by a processor, the steps of the job control method of the mechanical device in any of the above embodiments can be executed, and the steps are as follows:

[0124] Obtaining observation data of a mechanical device for trajectory tracking control, the mechanical device comprising a plurality of movable joints, the observation data comprising joint positions and joint velocities of each joint; searching, from a plurality of action combinations constructed for the mechanical device, a plurality of candidate action combinations corresponding to an executed action of the mechanical device at a current time; determining, based on the executed action at the current time, the plurality of candidate action combinations, and the observation data, a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations; controlling the mechanical device to perform work according to a target action combination corresponding to an optimal predicted trajectory at a next time, the optimal predicted trajectory being a predicted trajectory in the plurality of predicted trajectories that has the highest matching degree with a target planning trajectory.

[0125] Based on the above computer-readable storage medium, the generalization of the algorithm can be improved, and the migration between different mechanical devices can be facilitated.

[0126] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, and will not be described here. In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some communication interface, device or unit, and can be electrical, mechanical or other forms.

[0127] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0128] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0129] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0130] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A work control method of a machine device, characterized by, The method comprises: acquiring observation data for trajectory tracking control of a mechanical device, the mechanical device comprising a plurality of movable joints, the observation data comprising joint positions and joint velocities of each joint; from a plurality of action combinations constructed for the mechanical device, searching for a plurality of candidate action combinations corresponding to an executed action of the mechanical device at a current time; wherein the plurality of action combinations are constructed by: for each joint of the mechanical device that is movable, determining a plurality of multi-step single-joint actions of the joint; and combining the plurality of single-joint actions of each joint to form a plurality of action combinations, each action combination comprising a plurality of actions of a preset number of steps, each action being formed by combining single-joint actions of each joint at the number of steps; based on the executed action at the current time, the plurality of candidate action combinations, and the observation data, determining a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations; controlling the mechanical device to perform work according to a target action combination corresponding to an optimal predicted trajectory at a next time, the optimal predicted trajectory being a predicted trajectory in the plurality of predicted trajectories that has the highest matching degree with a target planned trajectory.

2. The method of claim 1, wherein, The observation data for trajectory tracking control comprises observation data for the plurality of joints at the current time and at historical times before the current time.

3. The method of claim 1, wherein, The plurality of multi-step single-joint actions of each joint comprises an initial action and a subsequent action, wherein the initial action of each joint is determined by: determining a first data point projected on an action space of the joint, determining, as the initial action of the joint, an action that is within a neighborhood of the first data point.

4. The method of claim 3, wherein, The subsequent action of each joint is determined by: determining an action slope, the action slope being a slope of a line segment from the first data point to a second data point projected on the action space of the joint from the initial action; extending from the first data point according to the action slope in the action space of the joint to obtain an action curve; determining the subsequent action of the joint based on a third data point falling on the action curve.

5. The method of claim 4, wherein, Each joint of the mechanical device is driven by a hydraulic system, and the action of the joint refers to a control amount of the hydraulic system for driving movement of the joint, wherein the action space is represented by a two-dimensional coordinate system, a horizontal coordinate of the action space representing a step number, and a vertical coordinate of the action space representing the control amount for driving movement of the joint.

6. The method of claim 5, wherein, The action curve is formed by: extending from the first data point according to the action slope in the action space of the joint, and limiting a portion exceeding an action range at a boundary value of the action range.

7. The method of claim 1, wherein, Each candidate action combination comprises a plurality of actions to be performed after the executed action at the current time is predicted for the mechanical device.

8. The method of claim 1, wherein, The predicted trajectory corresponding to each candidate action combination is determined by: The pre-trained world model is used to obtain a predicted trajectory corresponding to the candidate action combination through multiple iterations, and the following processing is performed at each iteration: the executed action, the candidate action combination, and the observation data are input into the world model to obtain a one-step predicted action for the mechanical device, wherein the predicted action output by the world model at the current iteration is used as the executed action at the next iteration, and the iteration is terminated when the number of steps of the predicted action output by the world model reaches a preset number of steps.

9. The method of claim 1, wherein, The optimal predicted trajectory is determined in the following manner: For each predicted trajectory, the mean square error value of each trajectory point on the predicted trajectory relative to the target planning trajectory is calculated; For each predicted trajectory, a weight value is set for each trajectory point on the predicted trajectory in the action execution order, wherein the weight value corresponding to a trajectory point earlier in the action execution order is greater; For each predicted trajectory, a matching index representing the matching degree of the predicted trajectory is calculated based on the mean square error value of each trajectory point on the predicted trajectory and the corresponding weight value; An optimal predicted trajectory is determined according to the matching indexes of all predicted trajectories.

10. The method of claim 1, wherein, Further comprising: storing the constructed multiple action combinations in the video memory to perform the operation for determining the optimal predicted trajectory in the video memory.

11. The method of claim 10, wherein, Further comprising: storing the target action combination corresponding to the optimal predicted trajectory in the memory, wherein the step of controlling the mechanical device to work according to the target action combination corresponding to the optimal predicted trajectory at the next time comprises: extracting the first action in the target action combination from the memory; controlling the mechanical device to execute the first action at the next time.

12. The method of claim 11, wherein, Further comprising: continuing to perform trajectory prediction for the mechanical device based on the executed first action at the next time until the mechanical device is controlled to complete the motion of the target planning trajectory.

13. A mechanical device, characterized by comprises a controller configured to: obtain observation data for trajectory tracking control of a mechanical device, the mechanical device comprising a plurality of movable joints, the observation data comprising joint positions and joint velocities of each joint; search for a plurality of candidate action combinations corresponding to the executed action of the mechanical device at the current time from a plurality of action combinations constructed for the mechanical device, wherein the plurality of action combinations are constructed in the following manner: for each movable joint of the mechanical device, a plurality of single-joint actions of the joint are determined; and a plurality of single-joint actions of each joint are combined to form a plurality of action combinations, each action combination comprising a plurality of actions of a preset number of steps, each action being formed by the combination of single-joint actions of each joint at the number of steps; determine a plurality of predicted trajectories of the mechanical device under the plurality of candidate action combinations based on the executed action at the current time, the plurality of candidate action combinations, and the observation data; control the mechanical device to work according to the target action combination corresponding to the optimal predicted trajectory at the next time, the optimal predicted trajectory being a predicted trajectory with the highest matching degree to the target planning trajectory among the plurality of predicted trajectories.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is run by the processor to execute the steps of the method in any one of claims 1 to 12.

Images