Method, device and equipment for motion planning and storage medium

Abstract

According to the embodiment of the invention, the invention provides a method and device for motion planning, equipment and a storage medium. The method comprises the following steps: acquiring a plurality of local features corresponding to a plurality of robots; and based on the plurality of local features, generating fusion features for the plurality of robots. And for a target robot in the plurality of robots, determining a target action of the target robot at the target moment based on the fusion feature and at least one reference action at the target moment, the at least one reference action being a planned action of at least one robot different from the target robot in the plurality of robots at the target moment. In this manner, a target action of a target robot is determined based on global information for a plurality of robots, thereby improving global knowledge of a current scene in action planning. The target action is generated based on the planning actions of other robots at the same moment, and the accuracy of action planning is further improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for motion planning. Background Art

[0002] In recent years, robotics has developed rapidly and has been widely used in many technical fields. For example, on the production line of a factory, robots can be used to perform various tasks such as sorting and handling. In order to ensure the normal operation of the robot, the robot's motion needs to be planned. However, the current robot motion planning method still has some problems that affect the accuracy of planning. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for motion planning is provided. The method comprises: acquiring a plurality of local features corresponding to a plurality of robots respectively, wherein the local features in the plurality of local features represent the destination of the corresponding robots and the environmental information related to the position of the corresponding robots; generating fusion features for the plurality of robots based on the plurality of local features; and determining, for a target robot among the plurality of robots, a target motion of the target robot at the target moment based on the fusion features and at least one reference motion at the target moment, wherein the at least one reference motion is a planned motion of at least one robot among the plurality of robots different from the target robot at the target moment.

[0004] In a second aspect of the present disclosure, a device for action planning is provided. The device includes: an acquisition module configured to acquire a plurality of local features corresponding to a plurality of robots, wherein the local features in the plurality of local features represent the destination of the corresponding robots and environmental information related to the positions of the corresponding robots; a generation module configured to generate fusion features for the plurality of robots based on the plurality of local features; and a determination module configured to determine, for a target robot among the plurality of robots, a target action of the target robot at the target moment based on the fusion features and at least one reference action at the target moment, wherein the at least one reference action is a planned action of at least one robot among the plurality of robots different from the target robot at the target moment.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure may be implemented;

[0010] Figure 2 A schematic diagram showing an example architecture for motion planning according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram showing an example architecture for generating local features according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic diagram showing an example architecture for model training according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A flowchart illustrating an example process of motion planning according to some embodiments of the present disclosure;

[0014] Figure 6 A schematic structural block diagram showing an example apparatus for action planning according to some embodiments of the present disclosure; and

[0015] Figure 7 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0019] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", which are used interchangeably in this article. A model may also include different types of processing units or networks.

[0020] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0021] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0022] Figure 1Schematic diagram of an example environment in which embodiments of the present disclosure can be implemented. Figure 1 As shown, one or more robots are deployed in the environment 100, for example, a first robot 110-1, a second robot 110-2, and a third robot 110-3. The first robot 110-1, the second robot 110-2, and the third robot 110-3 may also be collectively or individually referred to as robots 110. Hereinafter, an example embodiment of path planning will be described by taking the first robot 110-1 as an example. A control device may be deployed in each robot 110 to control the operation of the robot. For example, Figure 1 As shown, the first robot 110-1, the second robot 110-2 and the third robot 110-3 are respectively deployed with a control device 120-1, a control device 120-2 and a control device 120-3, which may also be collectively or individually referred to as a control device 120. In some embodiments, only one robot may be deployed in the environment 100, and only the moving path of this one robot needs to be planned. In some embodiments, multiple robots may be deployed in the environment 100. In this case, path planning may be performed for multiple robots in the environment 100 at the same time, or path planning may be performed for only any robot in the environment 100.

[0023] In the embodiments of the present disclosure, the robot 110 can be used for various suitable purposes, for example, it can be a transport robot for delivering goods, and the control device 120 can be used to plan the movement path of the robot 110. The robot 110 can, for example, perform a package delivery service. In this case, the control device 120 can obtain information such as the place of origin, destination, recipient, and contact information of the recipient of the package. As an example, the robot 110 works in an indoor environment, such as a storage environment.

[0024] exist Figure 1 In the environment 100, the control device 120 can be any type of microcontroller, programmable logic controller, industrial computer, single board computer, field programmable gate array, digital signal processor, multi-core processor or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0025] In some embodiments, the control device 120 can control the robot 110 based on the control instructions generated by itself. In some embodiments, the operation of the robot 110 can be controlled via the server 130. The server 130 can receive the relevant data of each task and control the operation of the robot 110 in a remote manner. The control device 120 collects the data of the robot 110 and communicates with the server 130 via the network 140 to send the data to the server 130. The server 130 determines the control instructions for the robot based on the acquired data (such as the target task of the robot and the environmental information of the robot's location) and sends the control instructions to the control device 120. The control device 120 controls the robot 110 based on the received control instructions. The server 130 can be various types of computing systems / servers that can provide computing capabilities, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0026] The server 130 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server 130 may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0027] A communication connection may be established between the server 130 and the robot 110 or the control device 120. The communication connection may be established by wire or wirelessly. The communication connection may include, but is not limited to, a Bluetooth connection, a mobile network connection, a Universal Serial Bus (USB) connection, a Wireless Fidelity (WiFi) connection, etc., and the embodiments of the present disclosure are not limited in this respect. In an embodiment of the present disclosure, the server 130 and the robot 110 may realize signaling interaction through a communication connection between the two. In some embodiments, the robot 110 may be connected to the server 130 via the network 140 and communicate with it. The robot 110 may send relevant data of each task performed by the robot 110 to the server 130 in real time.

[0028] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0029] As briefly mentioned above, robots are widely used in many technical fields. When multiple robots work together, in order to improve the working efficiency of the robots, it is necessary to plan the actions of the robots (for example, planning the movement paths of the robots, etc.).

[0030] Taking robot path planning as an example, most of the robot path planning technologies currently used are implemented based on reinforcement learning path planning models. Since multi-robot path planning in some scenarios (for example, warehousing scenarios) is a problem of sparse rewards, it affects the training effect of the path planning model. Therefore, the shared experience actor-critic algorithm ((Hierarchical) Shared Experience Actor Critic, (H)SEAC) can be used to implement the path planning model. During the training of the path planning model, the experience sharing between different robots is achieved by combining the gradients of different robots, thereby improving the learning efficiency of the path planning model in scenarios with sparse rewards. However, this technology requires an independent path planning model to be configured for each robot, which is costly. At present, the multi-robot path trajectory solution mainly adopts the "centralized training distributed execution (CTDE)" architecture, in which each robot implements path planning based on the same model.

[0031] However, in the path planning scheme based on the CTDE paradigm, each robot can only plan its own path based on its own information (i.e., local information) using the path planning model, and cannot obtain sufficient global information, resulting in a lack of global understanding of the current scene. In scenarios that require multi-robot collaboration, the accuracy of path planning may be affected.

[0032] Embodiments of the present disclosure provide a scheme for motion planning. According to various embodiments of the present disclosure, a plurality of local features corresponding to a plurality of robots are obtained, wherein the local features in the plurality of local features represent the destination of the corresponding robot and the environmental information related to the position of the corresponding robot. Based on the plurality of local features, fusion features for the plurality of robots are generated. For a target robot among the plurality of robots, a target action of the target robot at the target moment is determined based on the fusion features and at least one reference action at the target moment, wherein the at least one reference action is a planned action of at least one robot among the plurality of robots that is different from the target robot at the target moment.

[0033] In this way, the target action of the target robot is determined based on the global information (i.e., fusion features) for multiple robots, thereby improving the global understanding of the current scene in action planning and improving the accuracy of action planning. Furthermore, the target action is generated based on the planned actions of other robots at the same time, preventing interference between robots and improving the accuracy of action planning.

[0034] Example Architecture

[0035] Figure 2 FIG. 2 is a schematic diagram of an example architecture 200 for action planning according to some embodiments of the present disclosure. Figure 2 In the example of , the architecture 200 can be implemented or deployed in the server 130. In some embodiments, the server 130 first obtains local information corresponding to each robot. Exemplarily, the local information corresponding to each robot 110 includes environmental information related to the location of each robot and the robot's working status information (for example, the robot's target task or the robot's destination). The following takes the robot 110 performing path planning as an example to illustrate the architecture 200 for action planning. The path planning generated by the architecture 200 can be in the form of a target trajectory, or the action of the robot at each time step.

[0036] In some embodiments, when planning a path for the robot 110, the required information corresponding to the robot 110 includes the working state of the robot 110 (for example, whether the robot 110 is faulty, etc.), the current location of the robot 110, the destination of the robot 110 and the obstacle information of the robot's location (for example, the obstacle information within 1m around the location of the robot 110), the path distance data of the robot 110 (that is, the data of the minimum cumulative cost distance of the robot 110 from the current location to each location), etc. For the target robot among the multiple robots 110, the global information (for example, the global map, etc.) of the environment in which the multiple robots 110 are located at the current moment can be first obtained. Subsequently, based on the coordinates of the target robot, the location of the target robot in the global map is determined, so as to obtain the local information corresponding to the robot based on the global information. In some embodiments, the information corresponding to the target robot can be obtained based on the sensor (for example, image sensor, depth sensor, etc.) carried by the target robot. Exemplarily, a local map corresponding to the target robot can be created based on the image sensor carried by the target robot. Subsequently, the local information corresponding to the target robot, such as the obstacle information, is determined based on the local map.

[0037] In some embodiments, the architecture 200 needs to obtain local information corresponding to multiple robots 110, which may face the problem of too large input data dimensions. Therefore, a feature extraction model can be used to extract features from the local information corresponding to multiple robots. Figure 3 FIG. 3 is a schematic diagram of an example architecture 300 for obtaining local features according to some embodiments of the present disclosure. The architecture 300 may be implemented in the server 130. Figure 3As shown, the feature extraction network 320 can be implemented based on a VGG (Visual Geometry Group) convolutional neural network, and the feature extraction network 320 includes multiple convolutional layers and pooling layers. The robot local information includes first robot local information 310-1, second robot local information 310-2, third robot local information 310-3, and fourth robot local information 310-4, which can be individually or collectively referred to as robot local information 310. The obtained local information corresponding to the multiple robots can be provided to the VGG convolutional neural network to obtain multiple local features. Figure 3 As shown, the local features include a first local feature 210 - 1 , a second local feature 210 - 2 , a third local feature 210 - 3 , and a fourth local feature 210 - 4 , which may be individually or collectively referred to as robot local information 310 . Figure 3 The number of robot local information 310 and local features 330 shown in the figure is only exemplary and is not intended to be limiting. Each local feature 210 corresponds to a plurality of robot local information 310. In this way, the local features corresponding to each robot are better extracted through the VGG convolutional neural network to improve the accuracy and reliability of motion planning.

[0038] Continue to refer Figure 2 .like Figure 2 As shown, the multiple local features 210 corresponding to each robot 110 are converted into vector form through the vectorization unit. Subsequently, the multiple local features 210 in vector form are provided to the encoding model 220 to perform the encoding operation. The encoding model 220 applies a cross attention mechanism to the multiple local features 210 to achieve implicit communication between the multiple local features 210. Subsequently, based on the output of the encoding model 220, a fusion feature 230 is determined. Figure 2 As shown, the fused feature 230 includes a first feature code 231-1, a second feature code 231-2, a third feature code 231-3 and a fourth feature code 231-4, which may be individually or collectively referred to as feature codes 231. Each feature code 231 corresponds to each robot 110, and each feature code 231 fuses local features corresponding to other robots 110. Figure 2 The number of feature codes 231 shown in FIG. 2 is merely exemplary and is not intended to be limiting. In this way, the global understanding of the current scene in action planning is improved, the limitations brought by local information are overcome, and the accuracy of action planning is improved.

[0039] In some embodiments, the architecture 200 sequentially determines the planned actions of each robot 110 in the scene at the current moment. For any robot in the scene, the target action of the current robot can be determined based on the fusion feature 230 and the reference action. Figure 3As shown, the reference actions include a first reference action 240 - 1 , a second reference action 240 - 2 , and a third reference action 240 - 3 , which may be individually or collectively referred to as reference actions 240 . Figure 2 The number of reference actions 240 shown in is only exemplary and is not intended to be limiting. The reference action is a planned action of another robot among multiple robots that is different from the target robot at the target time. For example, if the scene includes three robots, namely, the first robot 110-1, the second robot 110-2, and the third robot 110-3. The target action 280 of the first robot 110-1 at the target time can be determined by first using the fusion feature 230. Subsequently, the target action of the first robot 110-1 at the target time is used as the first reference action 240-1. The target action 280 of the second robot 110-2 at the target time is determined based on the first reference action 240-1 and the fusion feature 230. Subsequently, the target action 280 of the second robot 110-2 is used as the second reference action 240-2, and the target action 280 corresponding to the third robot 110-2 is determined based on the first reference action 240-1, the second reference action 240-2, and the fusion feature 230.

[0040] In some embodiments, the reference motion 240 and the fused features 230 may be provided to a decoding model 250. Subsequently, a probability distribution of the target motion is determined based on the output of the decoding model 250. The probability distribution of the target motion is provided to a perception model 270 to determine a target motion 280.

[0041] The output of the architecture 200 is a single-step planning action, which may affect the control accuracy of the robot. In some embodiments, an auxiliary environment can be set based on the scene (i.e., the main environment) where the robot 110 is located. Subsequently, the trajectory of each robot 110 is generated by determining the target action in the auxiliary environment. Subsequently, the robot action is controlled based on the obtained trajectory in the main environment. Exemplarily, if the main environment is a real scene, the auxiliary environment is a deep simulation of the main environment. If the main environment is a virtual scene generated by a simulator, the auxiliary environment is obtained by copying the main environment.

[0042] In some embodiments, for a target robot among the multiple robots, a trajectory of the target robot can be generated according to the target action 280 of the target robot at a number of consecutive time steps (e.g., 5 time steps). Subsequently, the trajectory of the target robot is provided from the auxiliary environment to the main environment to control the movement of the robot.

[0043] In some embodiments, the environmental information related to the location of each robot 110 is received by the auxiliary environment from the main environment. If the main environment changes (for example, an obstacle in the main environment moves), the auxiliary environment is updated based on the change in the main environment, thereby ensuring the accuracy and reliability of the target action 280 and the robot trajectory. In this way, the trajectory of each robot is generated in the auxiliary environment, so as to better control the robot to perform the corresponding task.

[0044] In some embodiments, since each robot can only plan a path based on local information, the mutual interference between the robots increases the dynamics of the environment, affecting the efficiency of model training. At the same time, considering the complexity of the robot working scene and the scale of information that the architecture 200 needs to process (for example, robot local information, etc.), during the training process of the encoding model and the decoding model, the behavior cloning algorithm (BC) and the proximal policy optimization algorithm (PPO) can be used simultaneously to train the model to improve the convergence speed of the model.

[0045] Figure 4 A schematic diagram of an example architecture 400 for model training according to some embodiments of the present disclosure is shown. Figure 4 As shown, the architecture 400 includes a policy network (Policy Network) corresponding to the actor (Actor) and a value network (Value Network) corresponding to the critic (Critic). The policy network includes an encoding model 420 and a decoding model 450 for generating training actions. The value network (not shown) can be used to evaluate the value of the action. During the model training process, the model parameters are updated based on the gradients of both the reinforcement learning side and the imitation learning side. The encoding model 420 can be regarded as the encoding model 220 to be trained, and the decoding model 450 can be regarded as the decoding model 250 to be trained.

[0046] In some embodiments, the gradient can be calculated by behavioral cloning (BC) to update the policy network. Exemplarily, firstly, training data for multiple training robots are obtained. The training data includes robot training local information corresponding to each training robot. Subsequently, expert data corresponding to each training local information is generated using any suitable algorithm. The expert data includes multiple data tuples Among them t is the local information of the robot at time t, is the expert action of the robot at time t based on the algorithm planning. At the same time, multiple training local features 410 matching multiple training robots are determined. Subsequently, each training local feature 410 is provided to the encoding model 420 to generate a fusion feature 430 for multiple training robots. The fusion feature 430 is provided to the decoding model 450 to obtain the output of the decoding model 450. Subsequently, the output of the decoding model 450 is provided to the perception model 470 to determine multiple training actions 480 corresponding to the multiple training robots respectively. The perception model 470 can be regarded as the perception model 270 to be trained. For any training action among the multiple training actions, the action difference between the training action and the reference action 440 is determined. The reference action 440 and the training action are generated based on the same local information (that is, corresponding to the action of the same training robot at the same time).

[0047] In some embodiments, the action difference can be determined based on a cross entropy loss function. The action difference is calculated as follows:

[0048]

[0049] Where L bc is the cross entropy loss function, and θ is the parameter of the policy network.

[0050] In some embodiments, based on the action differences respectively determined for a plurality of training actions, parameters of the encoding model and parameters of the decoding model may be updated to maximize the probability that the policy network generates an expert action.

[0051] In some embodiments, the gradient can be calculated by the proximal policy optimization algorithm, and the policy network and the value network can be updated at the same time. Exemplarily, the training fusion features 430 for multiple training robots are provided to the value network to determine multiple training values ​​corresponding to multiple predetermined actions respectively. The predetermined action is an action that the training robot can perform in advance. At the same time, the training fusion features 430 are provided to the decoding model 450 to obtain multiple first training actions corresponding to the training robot. Subsequently, the parameters of the encoding model, the parameters of the decoding model, and the parameters of the value network are updated based on the obtained multiple training values. Using the updated encoding model, the encoding operation is performed again on the multiple training local features to update the training fusion features. The updated training fusion features are provided to the decoding model 450 to obtain multiple second training actions corresponding to each training robot respectively. By comparing the first training action and the second training action, the action to be updated in the multiple training actions is determined. The action update rate is determined based on the ratio of the number of actions to be updated to the number of multiple training actions, so as to update the parameters of the encoding model and the parameters of the decoding model based on the multiple action differences and the action update rate.

[0052] In some embodiments, the loss function value of the proximal policy optimization algorithm is calculated as follows:

[0053] L=L actor +L critic ; (2)

[0054] Where L is the loss function value of the proximal strategy optimization algorithm, L actor is the loss function value of the policy network, L critic is the loss function value of the value network.

[0055] In some embodiments, the loss function value of the policy network is calculated as follows:

[0056]

[0057] Where N is the total number of robots in the environment, r t (θ)=π θ (a t |o t ) / π θold (a t |o t ) is the action update rate, which represents the ratio of the strategy to be updated to the old strategy. θ (a t |o t ) is the output of the policy network in the current training process, π θold (a t |o t ) is the output of the policy network in the previous training process. t is the local information of the robot, a t is the target action generated by the model. θ is the parameter of the policy network. A t is the advantage function, which represents the action a taken at the current moment t Whether there are advantages.

[0058] In some embodiments, the advantage function A t The calculation formula is as follows:

[0059] A t =Q(o t , a t )-V(s t ); (4)

[0060] where s t Indicates the current state of the robot, Q(o t , a t ) means taking action a in the current state t The mass, V(s t ) are the actions a in the current state tThe average quality of the

[0061] The optimization direction of the policy model loss function is A t >0. If A t >0, indicating that action a is taken in the current state t Can increase the overall quality.

[0062] In some embodiments, the loss function value of the value network is calculated as follows:

[0063]

[0064] Where V θ (s t ) is the output of the value network when the parameter of the policy model is θ. critic is the loss function value of the value network, R t is the target value, which represents the cumulative reward. The optimization direction of the value network loss function is to approach R t .

[0065] In some embodiments, the policy network and the value network may be trained using an environment including a reward model 460 and a training reward 461. Figure 4 As shown, the training action 480 is provided to the environment as an "action". The environment provides the training reward 461 and the training local feature 410 as a "reward" and a "state" to the agent, respectively.

[0066] In some work scenarios (for example, warehousing scenarios), users are required to work with robots to perform tasks. Exemplarily, in the process of robots and workers cooperating to transport goods, the robot's actions are determined by a scheduler (for example, a control device). This type of data is applied to the model training process, which will affect the accuracy of the model. Therefore, in some embodiments, additional labels can be constructed for each training data. The label is used to reflect the type of action (for example, an action generated by a scheduler or an action generated by a model). During the model training process, the parameters of the encoding model and the parameters of the decoding model can be updated based on multiple action differences and multiple weights corresponding to the multiple action differences. The weight corresponding to each action difference is determined based on the type of reference action corresponding to the action difference. In some embodiments, the weight of the action difference corresponding to the action generated by the scheduler can be set to 0, and the weight of the action difference corresponding to the action generated by the model can be set to 1. Exemplarily, the obtained action differences can be weighted and summed, and then the overall loss of the model can be averaged. The calculation formula for the overall loss of the model is as follows:

[0067]

[0068] Where Loss is the overall loss of the model, i is the serial number of the reference action, and maski is the weight of the i-th reference action (i.e., expert action), loss i is the action difference corresponding to the i-th reference action.

[0069] In this way, the impact of the data generated by the scheduler on the model training process can be avoided, and the convergence speed of the model can be improved.

[0070] Figure 5 1 is a flow chart showing an example process 500 for action planning according to some embodiments of the present disclosure. Process 500 may be implemented at server 130. Figure 1 The process 500 is described below.

[0071] like Figure 5 As shown, in box 510, the server 130 obtains a plurality of local features respectively corresponding to a plurality of robots, wherein the local features in the plurality of local features represent the destination of the corresponding robot and the environmental information related to the position of the corresponding robot.

[0072] In block 520 , the server 130 generates fused features for the plurality of robots based on the plurality of local features.

[0073] In some embodiments, generating the fused feature includes: using the encoding model, applying a cross-attention mechanism to multiple local features to obtain an output of the encoding model; and determining the fused feature based on the output of the encoding model.

[0074] In box 530, the server 130 determines a target action of the target robot at a target time for a target robot among the multiple robots based on the fusion features and at least one reference action at the target time, where the at least one reference action is a planned action of at least one robot among the multiple robots that is different from the target robot at the target time.

[0075] In some embodiments, the acquisition of multiple local features and the determination of target actions are performed in an auxiliary environment, which is a simulation environment or an emulated environment corresponding to the main environment in which multiple robots are located, and the target moment is one of multiple moments within a predetermined time period, and process 500 also includes: determining the trajectory of the target robot based on the target actions determined respectively for multiple target moments; and providing the trajectory from the auxiliary environment to the main environment.

[0076] In some embodiments, environmental information related to the location of the robot is received by the auxiliary environment from the primary environment.

[0077] In some embodiments, determining a target action based on fused features and at least one reference action includes: generating a probability distribution of the target action based on the fused features and at least one reference action using a decoding model; and determining the target action based on the probability distribution of the target action using a perception model.

[0078] In some embodiments, an encoding model for determining a fusion feature and a decoding model for determining a target action are trained in the following manner: based on multiple training local features for multiple training robots, using an encoding model, training fusion features for multiple training robots are generated; based on the training fusion features, using a decoding model, multiple training actions corresponding to the multiple training robots are determined respectively; for a training action among the multiple training actions, an action difference between the training action and a reference action is determined, the reference action and the training action correspond to the same training robot, and the reference action is generated using a reference action planning model; and based on the action differences determined respectively for the multiple training actions, parameters of the encoding model and parameters of the decoding model are updated.

[0079] In some embodiments, updating the parameters of the encoding model and the decoding model based on action differences includes: based on training fusion features for multiple training robots, using a value network, determining multiple training values ​​corresponding to multiple predetermined actions respectively; based on the multiple training values, updating the parameters of the encoding model and the parameters of the decoding model; using the updated encoding model and the decoding model, based on multiple training local features, determining the actions to be updated among multiple training actions; and updating the parameters of the encoding model and the parameters of the decoding model based on multiple action differences and an action update rate, wherein the action update rate is determined based on the ratio of the number of actions to be updated to the number of multiple training actions.

[0080] In some embodiments, updating parameters of the encoding model and parameters of the decoding model based on multiple action differences includes: determining, for a target action difference among multiple action differences, a weight of the target action difference based on the type of target action corresponding to the target action difference; and updating parameters of the encoding model and parameters of the decoding model based on multiple action differences and multiple weights corresponding to the multiple action differences, respectively.

[0081] Example devices and equipment

[0082] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an example apparatus 600 for action planning according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the server 130. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0083] like Figure 6 As shown, the device 600 includes: an acquisition module 610, configured to acquire multiple local features corresponding to multiple robots respectively, the local features in the multiple local features represent the destination of the corresponding robot and environmental information related to the position of the corresponding robot; a generation module 620, configured to generate fusion features for multiple robots based on the multiple local features; and a determination module 630, configured to determine, for a target robot among the multiple robots, a target action of the target robot at a target moment based on the fusion features and at least one reference action at the target moment, the at least one reference action being a planned action of at least one robot among the multiple robots that is different from the target robot at the target moment.

[0084] In some embodiments, the acquisition of multiple local features and the determination of target actions are performed in an auxiliary environment, which is a simulation environment or an emulated environment corresponding to the main environment in which multiple robots are located, and the target moment is one of multiple moments within a predetermined time period, and the device 600 also includes a providing module, which is configured to: determine the trajectory of the target robot based on the target actions determined respectively for multiple target moments; and provide the trajectory from the auxiliary environment to the main environment.

[0085] In some embodiments, environmental information related to the location of the robot is received by the auxiliary environment from the primary environment.

[0086] In some embodiments, the device 600 also includes a model training module, which is configured to: generate training fusion features for multiple training robots based on multiple training local features for multiple training robots using a coding model; determine multiple training actions corresponding to the multiple training robots respectively based on the training fusion features using a decoding model; determine the action difference between the training action and a reference action for a training action among the multiple training actions, the reference action and the training action correspond to the same training robot, and the reference action is generated using a reference action planning model; and update the parameters of the coding model and the parameters of the decoding model based on the action differences determined respectively for the multiple training actions.

[0087] In some embodiments, the model training module is further configured to: determine multiple training values ​​corresponding to multiple predetermined actions respectively based on training fusion features for multiple training robots using a value network; update parameters of the encoding model and parameters of the decoding model based on the multiple training values; determine actions to be updated among multiple training actions based on multiple training local features using the updated encoding model and decoding model; and update parameters of the encoding model and parameters of the decoding model based on multiple action differences and action update rates, wherein the action update rate is determined based on the ratio of the number of actions to be updated to the number of multiple training actions.

[0088] In some embodiments, the model training module is further configured to: determine the weight of the target action difference among multiple action differences based on the type of target action corresponding to the target action difference; and update the parameters of the encoding model and the parameters of the decoding model based on the multiple action differences and the multiple weights corresponding to the multiple action differences respectively.

[0089] In some embodiments, the generation module 620 is further configured to: apply a cross-attention mechanism to multiple local features using the encoding model to obtain an output of the encoding model; and determine a fused feature based on the output of the encoding model.

[0090] In some embodiments, the determination module 630 is further configured to: generate a probability distribution of the target action based on the fused features and at least one reference action using a decoding model; and determine the target action based on the probability distribution of the target action using a perception model.

[0091] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0092] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0093] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0094] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0095] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0096] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0097] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0099] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0100] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0101] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. An action planning method, comprising: Acquire a plurality of local features respectively corresponding to the plurality of robots, wherein a local feature among the plurality of local features represents a destination of the corresponding robot and environmental information related to a position of the corresponding robot; Based on the multiple local features, generating fusion features for the multiple robots; as well as For a target robot among the multiple robots, a target action of the target robot at the target moment is determined based on the fusion feature and at least one reference action at the target moment, wherein the at least one reference action is a planned action of at least one robot among the multiple robots that is different from the target robot at the target moment.

2. The method according to claim 1, wherein the acquisition of the plurality of local features and the determination of the target action are performed in an auxiliary environment, the auxiliary environment being a simulated environment or an emulated environment corresponding to the main environment in which the plurality of robots are located, and the target moment is one of a plurality of moments within a predetermined time period, and The method further comprises: Determining a trajectory of the target robot based on the target actions respectively determined for the plurality of target moments; as well as The trajectory is provided from the secondary environment to the primary environment.

3. The method of claim 2, wherein the environmental information related to the location of the robot is received by the auxiliary environment from the primary environment.

4. The method according to claim 1, wherein the encoding model for determining the fusion feature and the decoding model for determining the target action are trained in the following manner: Based on a plurality of training local features for a plurality of training robots, generating training fusion features for the plurality of training robots using the encoding model; Based on the training fusion features, using the decoding model, determining a plurality of training actions corresponding to the plurality of training robots respectively; For a training action among the plurality of training actions, determining an action difference between the training action and a reference action, wherein the reference action and the training action correspond to the same training robot, and the reference action is generated using a reference action planning model; as well as Based on the action differences respectively determined for the plurality of training actions, parameters of the encoding model and parameters of the decoding model are updated.

5. The method according to claim 4, wherein updating the parameters of the encoding model and the decoding model based on the action difference comprises: Based on the training fusion features for the plurality of training robots, using a value network, determining a plurality of training values ​​corresponding to the plurality of predetermined actions respectively; Based on the multiple training values, updating parameters of the encoding model and parameters of the decoding model; Determine, using the updated encoding model and the decoding model, an action to be updated among the multiple training actions based on the multiple training local features; as well as Based on the multiple action differences and an action update rate, parameters of the encoding model and parameters of the decoding model are updated, and the action update rate is determined based on a ratio of the number of the actions to be updated to the number of the multiple training actions.

6. The method of claim 4, wherein updating the parameters of the encoding model and the parameters of the decoding model based on the plurality of motion differences comprises: For a target action difference among the plurality of action differences, determining a weight of the target action difference based on a type of a reference action corresponding to the target action difference; as well as Based on the plurality of action differences and a plurality of weights respectively corresponding to the plurality of action differences, a parameter of the encoding model and a parameter of the decoding model are updated.

7. The method according to claim 1, wherein generating the fusion feature comprises: Applying a cross-attention mechanism to the plurality of local features using the encoding model to obtain an output of the encoding model; as well as The fusion feature is determined based on the output of the encoding model.

8. The method according to claim 1, wherein determining the target action based on the fused feature and the at least one reference action comprises: Generate, using a decoding model, a probability distribution of the target action based on the fused features and the at least one reference action; as well as Based on the probability distribution of the target action, the target action is determined using a perception model.

9. An action planning device, comprising: an acquisition module configured to acquire a plurality of local features respectively corresponding to the plurality of robots, wherein a local feature among the plurality of local features represents a destination of the corresponding robot and environmental information related to a position of the corresponding robot; A generating module, configured to generate fusion features for the multiple robots based on the multiple local features; as well as A determination module is configured to determine, for a target robot among the multiple robots, a target action of the target robot at the target moment based on the fusion feature and at least one reference action at the target moment, wherein the at least one reference action is a planned action of at least one robot among the multiple robots that is different from the target robot at the target moment.

10. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images