A robot intelligent decision control method based on adversarial imitation learning

Abstract

The application provides a robot intelligent decision control method based on adversarial imitation learning, and belongs to the technical field of automatic control. The method comprises the following steps: obtaining expert demonstration data of a robot, and performing semantic feature extraction on image data corresponding to the expert demonstration data based on a visual language pre-training model to obtain semantic features of each trajectory data in the expert demonstration data; constructing a teacher model group based on the semantic features and a generative adversarial imitation learning strategy; obtaining a distillation loss between multiple teacher models in the teacher model group and a single student model; migrating the knowledge of the multiple teacher models to the single student model based on the distillation loss, and performing robot intelligent decision control based on the student model. The technical solution of the application can effectively overcome the limitations of a single strategy model, thereby improving the diversity and generalization ability of the robot intelligent decision strategy.

Description

Technical Field

[0001] This application relates to the field of automatic control technology, and in particular to a robot intelligent decision-making and control method based on adversarial imitation learning. Background Technology

[0002] In the field of robotics, the research and application of intelligent decision-making and control methods for robots are particularly important because robots need to make accurate decisions in real time in complex and dynamic environments. However, traditional intelligent decision-making and control methods for robots usually rely on expert data and preset rules, making it difficult to adapt to the dynamic changes of new situations and unexpected events.

[0003] To address the aforementioned issues, imitation learning has been proposed as a novel method for intelligent decision-making and control of robots. Imitation learning learns decision-making and control strategies from expert demonstrations, avoiding the reliance on preset rules and reward functions found in traditional methods. Behavior cloning is a classic imitation learning method that treats behavior learning as a supervised learning task, replicating expert behavior by establishing a mapping between environmental states and expert actions. However, behavior cloning can only simply copy state-action pairs from expert data; when faced with states not present in the expert data, the model cannot make reasonable decisions, resulting in poor generalization ability. Summary of the Invention

[0004] The main objective of this application is to propose a robot intelligent decision-making and control method based on adversarial imitation learning, which aims to improve the diversity and generalization ability of robot intelligent decision-making strategies.

[0005] To achieve the above objectives, a first aspect of this application proposes a robot intelligent decision-making and control method based on adversarial imitation learning, the method comprising:

[0006] Obtain expert demonstration data of the robot, and extract semantic features from the image data corresponding to the expert demonstration data based on the visual language-image pretraining (CLIP) model to obtain the semantic features of each trajectory data in the expert demonstration data;

[0007] A teacher model group is constructed based on the aforementioned semantic features and generative adversarial imitation learning strategy;

[0008] Obtain the distillation loss between multiple teacher models and a single student model in the teacher model group;

[0009] Based on the distillation loss, the knowledge of the multiple teacher models is transferred to the single student model, and the robot's intelligent decision-making and control are performed based on the student model.

[0010] In some embodiments, the image data corresponding to the expert demonstration data includes the image data corresponding to each trajectory data in the expert demonstration data;

[0011] The semantic feature extraction of the image data corresponding to the expert demonstration data based on the visual language pre-training model is obtained to obtain the semantic features of each trajectory data in the expert demonstration data, including:

[0012] Based on the visual language pre-trained model, semantic features are extracted from the image data corresponding to the target trajectory data in the expert demonstration data to obtain the mean value of the trajectory features of the target trajectory data, and the mean value of the trajectory features is used as the semantic features of the target trajectory data.

[0013] The target trajectory data is any one of the multiple trajectory data in the expert demonstration data.

[0014] In some embodiments, the method further includes:

[0015] Obtain environmental dimension information associated with the expert demonstration data;

[0016] Based on the environmental dimension information and the action information of each trajectory in the expert demonstration data, image rendering processing is performed on each trajectory data to obtain the image data corresponding to each trajectory data.

[0017] In some embodiments, constructing a teacher model set based on the semantic features and the generative adversarial imitation learning strategy includes:

[0018] The semantic features are divided into multiple clusters;

[0019] Based on the generative adversarial imitation learning strategy, each of the multiple cluster sets is trained independently to obtain a teacher model corresponding to each cluster set; the teacher model group includes the teacher model corresponding to each cluster set.

[0020] In some embodiments, obtaining the distillation loss between multiple teacher models in the teacher model group and a single student model includes:

[0021] Obtain the teacher actions generated by each of the multiple teacher models in the teacher model group, and obtain the student actions generated by a single student model;

[0022] The distillation loss between each of the multiple teacher models and the single student model is calculated based on the teacher actions and the student actions.

[0023] In some embodiments, obtaining student actions generated by a single student model includes:

[0024] Acquire the data generated by the robot's interaction with the environment;

[0025] The single student model is initialized based on the data generated by the robot's interaction with the environment, resulting in student actions generated by the single student model.

[0026] In some embodiments, transferring knowledge from the plurality of teacher models to the single student model based on the distillation loss includes:

[0027] The policy network parameters of the single student model are optimized based on the distillation loss and the reward signal of the preset discriminator, so as to transfer the knowledge of the multiple teacher models to the single student model.

[0028] The discriminator generates a reward signal based on the expert demonstration data and the student actions generated by the single student model.

[0029] To achieve the above objectives, a second aspect of this application proposes a robot intelligent decision-making and control device based on adversarial imitation learning, the device comprising:

[0030] The data acquisition and feature extraction module is used to acquire expert demonstration data of the robot, and extract semantic features from the image data corresponding to the expert demonstration data based on the visual language pre-trained model to obtain the semantic features of each trajectory data in the expert demonstration data.

[0031] An adversarial imitation learning module is used to construct a teacher model group based on the semantic features and to generate an adversarial imitation learning strategy.

[0032] The knowledge distillation module is used to obtain the distillation loss between multiple teacher models in the teacher model group and a single student model; and to transfer the knowledge of the multiple teacher models to the single student model based on the distillation loss, and to perform robot intelligent decision control based on the student model.

[0033] To achieve the above objectives, a third aspect of this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the robot intelligent decision control method based on adversarial imitation learning described in the first aspect.

[0034] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot intelligent decision-making and control method based on adversarial imitation learning described in the first aspect.

[0035] To achieve the above objectives, a fifth aspect of the present application provides a computer program product storing a computer program that, when executed by a processor, implements the robot intelligent decision-making and control method based on adversarial imitation learning described in the first aspect.

[0036] The robot intelligent decision-making and control method, device, computer equipment, computer-readable storage medium, and computer program product based on adversarial imitation learning proposed in this application embodiment acquire expert demonstration data of the robot, and extract semantic features from the image data corresponding to the expert demonstration data based on a visual language pre-trained model to obtain the semantic features of each trajectory data in the expert demonstration data; construct a teacher model group based on the semantic features and generate an adversarial imitation learning strategy; obtain the distillation loss between multiple teacher models in the teacher model group and a single student model; and transfer the knowledge of the multiple teacher models to the single student model based on the distillation loss, and perform robot intelligent decision-making and control based on the student model.

[0037] Thus, compared to the methods of robot intelligent decision-making and control based on simple behavior cloning in related technologies, the embodiments of this application perform image rendering processing on the collected expert demonstration data, extract semantic features using CLIP, and then train independent generative adversarial imitation learning strategies based on the extracted semantic features to construct a group of teacher models. Furthermore, knowledge distillation technology is used to transfer the knowledge of multiple teacher models to a single student model. In this way, the limitations of a single strategy model can be effectively overcome by adopting a multi-teacher knowledge distillation mechanism, thereby significantly improving the diversity and generalization ability of the robot's intelligent decision-making strategies.

[0038] Furthermore, by combining image rendering processing with CLIP feature extraction, the embodiments of this application can also achieve efficient representation of multimodal data. Attached Figure Description

[0039] Figure 1 A flowchart illustrating the steps of the robot intelligent decision-making and control method based on adversarial imitation learning provided in some embodiments of this application;

[0040] Figure 2 A flowchart illustrating the steps of the robot intelligent decision-making and control method based on adversarial imitation learning provided in some other embodiments of this application;

[0041] Figure 3 for Figure 1 A detailed flowchart of step S102;

[0042] Figure 4for Figure 1 A detailed flowchart of step S103;

[0043] Figure 5 A flowchart of a complete embodiment of the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application;

[0044] Figure 6 A flowchart of the generative adversarial imitation learning method involved in a complete embodiment of the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application;

[0045] Figure 7 A flowchart of the knowledge distillation method involved in a complete embodiment of the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application;

[0046] Figure 8 These are screenshots of the graphical interfaces for three experimental tasks of the robot intelligent decision-making and control method based on adversarial imitation learning proposed in the embodiments of this application.

[0047] Figure 9 This is a schematic diagram showing the comparison curves between the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application and other methods in the HalfCheetah physical environment.

[0048] Figure 10 This is a schematic diagram showing the comparison curves between the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application and other methods in the Hopper physical environment.

[0049] Figure 11 This is a schematic diagram showing the comparison curves between the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application and other methods in the Walker physical environment;

[0050] Figure 12 A schematic diagram of the structure of a robot intelligent decision-making and control device based on adversarial imitation learning provided in an embodiment of this application;

[0051] Figure 13 This is a schematic diagram of the hardware structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] First, a brief explanation of the technical terms used in the embodiments of this application will be given.

[0056] Intelligent decision control:

[0057] Intelligent decision control refers to the technology of analyzing and processing large amounts of data through artificial intelligence and intelligent algorithms to arrive at optimal decision results. The core of intelligent decision control lies in integrating advanced algorithms such as machine learning, deep learning, and reinforcement learning to mimic the human decision-making process and achieve automated and intelligent decision-making.

[0058] Next, the overall concept of the embodiments of this application will be described.

[0059] In the field of robotics, the research and application of intelligent decision-making and control methods for robots are particularly important because robots need to make accurate decisions in real time in complex and dynamic environments. However, traditional intelligent decision-making and control methods for robots typically rely on expert data and preset rules, making it difficult to adapt to the dynamic changes of new situations and unexpected events. Furthermore, the models in related technologies have low scalability and computational efficiency in high-dimensional state spaces, making it difficult to simultaneously ensure both real-time performance and accuracy in decision-making.

[0060] To address the aforementioned issues, imitation learning offers a novel approach for intelligent decision-making and control of robots. By learning decision-making and control strategies from expert demonstrations, imitation learning avoids the reliance on pre-defined rules and reward functions found in traditional methods. Behavior cloning, a classic imitation learning method, treats behavior learning as a supervised learning task, replicating expert behavior by establishing a mapping between environmental states and expert actions. However, behavior cloning can only simply copy state-action pairs from expert data; when faced with states not present in the expert data, the model cannot make reasonable decisions, resulting in poor generalization ability. Inverse reinforcement learning is another imitation learning method. Its core principle is to infer a potential reward function from expert demonstrations and use this reward function to train the policy. However, the reward function inferred by inverse reinforcement learning relies excessively on expert data, making it difficult to generalize to new environments. Furthermore, the process of setting the reward function is complex and computationally expensive, limiting its widespread use in practical applications.

[0061] Generative adversarial imitation learning combines the advantages of generative adversarial networks (GANs) and imitation learning. It learns policies directly from expert data through an adversarial game between a generator and a discriminator. The generator attempts to generate policies that match the expert data, while the discriminator distinguishes between generated data and real expert data. Through this adversarial mechanism, GAN can effectively avoid the complexity of reward function design while improving the generalization ability of the policy. However, when dealing with high-dimensional, multimodal data, GAN often struggles to effectively capture the complex relationships between state-action sequences, and single-policy models are ill-suited to adapt to diverse decision-making needs when facing complex tasks.

[0062] Based on this, this application proposes a robot intelligent decision-making and control method based on adversarial imitation learning, which aims to achieve efficient representation of multimodal data and improve the diversity and generalization ability of decision-making strategies.

[0063] In some embodiments, the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application first performs image rendering processing on the collected expert demonstration data and extracts semantic features using the visual language pre-trained model CLIP. Secondly, it uses the K-means clustering algorithm to divide the extracted semantic feature data into multiple clusters, and trains independent generative adversarial imitation learning strategies for each cluster to construct a teacher model group. Then, it uses knowledge distillation technology to transfer the knowledge of multiple teacher models to a single student model and integrates the discriminator reward signal to optimize the student strategy. Thus, by combining image rendering with CLIP feature extraction, efficient representation of multimodal data is achieved; and the multi-teacher knowledge distillation mechanism effectively overcomes the limitations of a single policy model, significantly improving the diversity and generalization ability of decision-making strategies.

[0064] The robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application can be widely applied in the field of robot intelligent decision-making and control, and has important practical value.

[0065] Next, based on the overall concept of the embodiments of this application described above, specific embodiments of the robot intelligent decision-making and control method, the robot intelligent decision-making and control device, the computer device, the computer-readable storage medium, and the computer program product based on adversarial imitation learning provided in the embodiments of this application are proposed. First, the various specific embodiments of the robot intelligent decision-making and control method based on adversarial imitation learning in the embodiments of this application are described in detail.

[0066] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0067] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0069] Furthermore, the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a terminal device capable of controlling the robot's operation, such as a smartphone, tablet, laptop, or desktop computer. In some embodiments, the terminal can also be the robot itself. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the robot intelligent decision-making and control method based on adversarial imitation learning, but is not limited to the above forms.

[0070] Alternatively, embodiments of this application can also be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: robots, robotic systems, personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0071] For ease of understanding and explanation, the following descriptions will use the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application as an example. The implementation of any of the above-described subject matter using the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application can refer to the process described below for applying the robot intelligent decision-making and control method based on adversarial imitation learning to a terminal device.

[0072] Please refer to Figure 1 , Figure 1 The flowchart illustrates the steps of the robot intelligent decision-making and control method based on adversarial imitation learning provided in some embodiments of this application. It should be understood that, although... Figure 1The figure shows the execution order of some method steps, but based on different design needs of practical applications, the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application embodiment can of course adopt a different execution order of method steps than shown in the figure. That is, Figure 1 The order of the method steps shown does not constitute a limitation on the execution logic order of the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application. Any other method based on... Figure 1 Reasonable changes to the sequence of steps shown should be included within the protection scope of the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application.

[0073] like Figure 1 As shown, in some embodiments, the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application may include, but is not limited to, steps S101 to S104.

[0074] Step S101: Obtain expert demonstration data of the robot, and extract semantic features from the image data corresponding to the expert demonstration data based on the visual language pre-trained model to obtain the semantic features of each trajectory data in the expert demonstration data.

[0075] Before performing intelligent decision-making and control of the robot, the terminal device first acquires the expert demonstration data of the robot, and then performs image rendering processing on the expert demonstration data to render the expert demonstration data into an image (the image data corresponding to the expert demonstration data). Then, the visual language pre-trained model CLIP is used to extract semantic features from the image data to obtain the semantic features of each trajectory data in the expert demonstration data.

[0076] It should be noted that in the expert demonstration data acquired by the terminal device, each trajectory data consists of a series of continuous states and actions. For example, the expert demonstration data is {obs:[s1,s2,…,s...]} T ],action:[a1,a2,…,a T ]}, where s t This is the state at step t, a t This is the action at step t.

[0077] Step S102: Construct a teacher model group based on the semantic features and generative adversarial imitation learning strategy.

[0078] After obtaining the semantic features of each trajectory data in the expert demonstration data, the terminal device can train independent generative adversarial imitation learning (GAIL) strategies based on these semantic features, thereby constructing a teacher model group that includes multiple teacher models.

[0079] It should be noted that the Generative Adversarial Imitation Learning (GAIL) strategy can also be called Adversarial Generative Imitation Learning. The terminal device trains independent teacher models based on these independent GAIL strategies, enabling it to capture features of different decision-making patterns. Since the multiple teacher models in the teacher model group are each obtained by training independent GAIL strategies, their individual knowledge is not entirely the same.

[0080] In some embodiments, when the terminal device trains an independent teacher model based on a training-independent generative adversarial learning strategy, the generative adversarial learning strategy GAIL may include, but is not limited to, steps S1 to S6 as shown below.

[0081] S1: Data from expert presentations Mid-sampled state-action pairs are generated through interactions between the agent (robot) and the environment.

[0082] S2: Initialize the policy network π θ Discriminator Network D φ and value network Parameters;

[0083] S3: Calculate the discriminator loss Update the discriminator parameter φ;

[0084] Among these, the discriminator D φ The goal is to distinguish between expert behavior and agent behavior, and its loss function is the binary cross-entropy loss:

[0085]

[0086] For example, the discriminator parameter φ can be updated via gradient descent:

[0087]

[0088] Among these, η D It is the learning rate of the discriminator.

[0089] S4: Calculate the network loss of the strategy Update the policy parameter θ;

[0090] Among these, the policy network π θThe goal is to maximize the reward signal provided by the discriminator:

[0091]

[0092] For example, the policy parameter θ can be updated using the Trust Region Policy Optimization (TRPO) algorithm:

[0093]

[0094] Among these, η π It is the learning rate of the policy network.

[0095] S5: Computational Value Network Loss Update the value network parameters ψ;

[0096] Among these, value networks The goal is to accurately estimate the state-value function, and the loss error is the mean squared error loss:

[0097]

[0098] Here, R(s) is the cumulative discount reward for state s.

[0099] For example, the value network parameters can be updated via gradient descent. :

[0100]

[0101] Among these, η V It is the learning rate of the value network.

[0102] S6: Repeat steps S2-S5 until the policy network converges.

[0103] Step S103: Obtain the distillation loss between multiple teacher models and a single student model in the teacher model group.

[0104] The terminal device constructs a teacher model group, and for each of the multiple teacher models in the teacher model group, calculates the distillation loss between each of the multiple teacher models and a single student model.

[0105] Step S104: Based on the distillation loss, transfer the knowledge of the multiple teacher models to the single student model, and perform intelligent decision control of the robot based on the student model.

[0106] After calculating the distillation loss between each of the multiple teacher models and the single student model, the terminal device uses knowledge distillation technology to transfer the knowledge of the multiple teacher models to the single student model based on each distillation loss. Then, the terminal device can perform intelligent decision-making and control of the robot based on the student model.

[0107] In some embodiments, after transferring knowledge from multiple teacher models to a single student model using knowledge distillation technology, the terminal device can determine whether the student model has learned well. If the student model has learned well, the training ends, and intelligent decision-making and control of the robot are performed based on the student model. If the student model has not yet learned well, the policy of the student model is updated, and the aforementioned steps are repeated until it is determined that the learning model has learned well.

[0108] It should be noted that the terminal device uses knowledge distillation technology to transfer knowledge from multiple teacher models to a single student model, thereby improving the generalization ability and decision diversity of the student model. This knowledge distillation technology may include, but is not limited to, steps S1 to S3 as shown below.

[0109] S1: Obtain the action prediction of each teacher model on the expert trajectory from multiple teacher models;

[0110] S2: Calculate the difference in action predictions between the student model and each teacher model on the expert trajectory;

[0111] S3: Update the policy network parameters of the student model by accumulating the distillation loss of multiple teacher models.

[0112] In this embodiment, before performing intelligent decision-making and control of the robot, the terminal device first acquires expert demonstration data of the robot and performs image rendering processing on the expert demonstration data to obtain the corresponding image data. Then, the visual language pre-trained model CLIP is used to extract semantic features from the image data, thereby obtaining the semantic features of each trajectory data in the expert demonstration data. Based on these semantic features, the terminal device trains independent generative adversarial imitation learning strategies GAIL to construct a teacher model group containing multiple teacher models. For the multiple teacher models in the teacher model group, the terminal device calculates the distillation loss between each of the multiple teacher models and a single student model. Then, using knowledge distillation technology, the knowledge of the multiple teacher models is transferred to the single student model based on each distillation loss. In this way, the terminal device can perform intelligent decision-making and control of the robot based on the student model.

[0113] Compared to methods in related technologies that rely on simple behavior cloning for intelligent robot decision-making and control, this application's embodiments perform image rendering processing on collected expert demonstration data, extract semantic features using the visual language pre-trained model CLIP, and then train independent generative adversarial imitation learning strategies based on the extracted semantic features to construct a group of teacher models. Furthermore, knowledge distillation technology is used to transfer the knowledge of multiple teacher models to a single student model. In this way, the limitations of a single strategy model can be effectively overcome by adopting a multi-teacher knowledge distillation mechanism, thereby significantly improving the diversity and generalization ability of the robot's intelligent decision-making strategies.

[0114] Furthermore, by combining image rendering processing with CLIP feature extraction, the embodiments of this application can also achieve efficient representation of multimodal data.

[0115] In some embodiments, the image data corresponding to the expert demonstration data may be the image data corresponding to each trajectory data in the expert demonstration data. In this case, the step S101 above, "extracting semantic features from the image data corresponding to the expert demonstration data based on a visual language pre-trained model to obtain the semantic features of each trajectory data in the expert demonstration data," may include the following steps:

[0116] Based on the visual language pre-trained model, semantic features are extracted from the image data corresponding to the target trajectory data in the expert demonstration data to obtain the mean value of the trajectory features of the target trajectory data, and the mean value of the trajectory features is used as the semantic features of the target trajectory data.

[0117] The target trajectory data is any one of the multiple trajectory data in the expert demonstration data.

[0118] When the terminal device uses CLIP to extract semantic features from the image data corresponding to the expert demonstration data, it can use CLIP to extract the mean value of the trajectory features of each target trajectory data in the image data corresponding to each target trajectory data in the expert demonstration data, and then use the mean value of the semantic features as the semantic features of each target trajectory data.

[0119] In this embodiment, the terminal device uses CLIP to extract the mean semantic features of each trajectory, which can enhance the data representation capability.

[0120] In some embodiments, the terminal device may perform image rendering processing on each trajectory data in the expert demonstration data before extracting the mean value of trajectory features from the image data corresponding to each target trajectory data in the expert demonstration data using CLIP, thereby obtaining the image data corresponding to each trajectory data.

[0121] Please refer to Figure 2 , Figure 2 The flowcharts of the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application are shown in some other embodiments.

[0122] like Figure 2 As shown, in some embodiments, the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application may further include steps S201 and S202 as shown below.

[0123] Step S201: Obtain environmental dimension information associated with the expert demonstration data.

[0124] When the terminal device performs image rendering processing on each trajectory data in the expert demonstration data, it can obtain the corresponding environmental dimension information by initializing the robot's simulation environment (such as the HalfCheetah, Hopper, and Walker environments), thus preparing for setting the environmental state during subsequent image rendering processing. Furthermore, the terminal device uses this environmental dimension information as the environmental dimension information associated with the robot's expert demonstration data.

[0125] It should be noted that the environmental dimension information obtained by the terminal device may include: position dimension nq (such as joint angle) and velocity dimension nv (such as joint angular velocity).

[0126] Step S202: Based on the environmental dimension information and the action information of each trajectory in the expert demonstration data, perform image rendering processing on each trajectory data to obtain the image data corresponding to each trajectory data.

[0127] After obtaining the environmental dimension information, the terminal device combines the environmental dimension information with the action information of each trajectory in the expert demonstration data, sets the environmental state and executes actions to render the current frame image, thereby realizing image rendering processing of each trajectory data and obtaining the image data corresponding to each trajectory data.

[0128] For example, the terminal device can iterate through each trajectory data in the expert demonstration data. That is, the input is all the trajectory data in the expert demonstration data, and the device iterates through each trajectory data and displays a progress bar. In this process, for each trajectory data, the terminal device initializes the feature accumulator, then iterates through each action in the trajectory data, and uses environmental dimension information to set the environmental state, execute actions, and render the current frame image, thus obtaining the image data corresponding to each trajectory data.

[0129] Furthermore, after rendering the current frame image, the terminal device can use the CLIP model to extract image features, then accumulate the features and calculate the mean of trajectory features. The mean of trajectory features can be regarded as the "overall features" of the trajectory (i.e., the aforementioned semantic features).

[0130] Please refer to Figure 3 , Figure 3 for Figure 1 A detailed flowchart of step S102.

[0131] like Figure 3 As shown, in some embodiments, step S102 above: constructing a teacher model group based on the semantic features and the generative adversarial imitation learning strategy may include, but is not limited to, steps S301 and S302 as shown below.

[0132] Step S301: Divide the semantic features into multiple clusters.

[0133] Step S302: Train each cluster in the plurality of cluster sets independently based on the generative adversarial imitation learning strategy to obtain the teacher model corresponding to each cluster set; the teacher model group includes the teacher model corresponding to each cluster set.

[0134] When a terminal device trains independent generative adversarial learning strategies based on semantic features to construct a teacher model group, it can first perform cluster analysis on the semantic features, that is, use the K-means clustering algorithm to divide the semantic feature data into multiple clusters. Then, the terminal device trains an independent generative adversarial learning strategy for each cluster, thereby training a teacher model corresponding to each cluster. The teacher models corresponding to each cluster together form a teacher model group.

[0135] It should be noted that after the terminal device uses the K-means clustering algorithm to divide the extracted semantic features into multiple clusters, each cluster represents a specific decision-making pattern. The K-means algorithm iteratively optimizes the cluster center u. k The system calculates the distance between the semantic features of each trajectory data point and the cluster center, assigns it to the nearest cluster, and finally returns the cluster label for each trajectory.

[0136]

[0137] Among them, C k It is the k-th cluster, u k It is the center of the k-th cluster.

[0138] Please refer to Figure 4 , Figure 4 for Figure 1 A detailed flowchart of step S103.

[0139] like Figure 4 As shown, in some embodiments, step S103 above: obtaining the distillation loss between multiple teacher models and a single student model in the teacher model group may include, but is not limited to, steps S401 and S402 as shown below.

[0140] Step S401: Obtain the teacher actions generated by each of the multiple teacher models in the teacher model group, and obtain the student actions generated by a single student model.

[0141] When the terminal device obtains the distillation loss between each of the multiple teacher models in the teacher model group and a single student model, it can load multiple teacher models separately to obtain the teacher actions generated by each of the multiple teacher models.

[0142] For example, the terminal device can use a policy network that loads multiple teacher models. Value Network and discriminator Thus, for each teacher model Generate teacher actions:

[0143]

[0144] Among these, a i It is the action generated by the i-th teacher model.

[0145] While the terminal device acquires the teacher actions generated by multiple teacher models, it can also obtain the student actions generated by the student model by initializing the student model.

[0146] In some embodiments, the step of "obtaining student actions generated by a single student model" in step S401 above may include, but is not limited to, the following steps:

[0147] Acquire the data generated by the robot's interaction with the environment;

[0148] The single student model is initialized based on the data generated by the robot's interaction with the environment, resulting in student actions generated by the single student model.

[0149] When the terminal device initializes the student model to generate student actions, it can first obtain the data generated by the robot's interaction with the environment, and then initialize the network parameters of the student model based on the data generated by the robot's interaction with the environment, thereby obtaining the student actions generated by the student model.

[0150] For example, the terminal device can initialize the policy network of the student model using data generated from the robot's interaction with the environment. Value Network and discriminator Thus, the student's actions are obtained. Where a s These are actions generated by the student model.

[0151] Step S402: Calculate the distillation loss between each of the multiple teacher models and the single student model based on the teacher actions and the student actions.

[0152] After acquiring the teacher's actions and the student's actions, the terminal device can calculate the distillation loss between multiple teacher models and student models based on the Mean Square Error (MSE) loss function, using the teacher's actions and the student's actions as parameters.

[0153] For example, the terminal device can transmit student actions a s Teacher's actions a i As input, the MSE distillation loss between the student model and the teacher model is calculated according to the formula shown below:

[0154]

[0155] Where N represents the number of teacher models.

[0156] In some embodiments, when calculating the MSE distillation loss between the student model and the teacher model, the terminal device can also update the policy network parameters θ of the student model through backpropagation and the optimizer. s ,satisfy:

[0157]

[0158] Where, η s It is the learning rate of the student model.

[0159] In some embodiments, during the process of transferring knowledge from multiple teacher models to a single student model using knowledge distillation technology, the terminal device can also integrate discriminator reward signals to optimize student strategies.

[0160] It should be noted that the pre-defined discriminator's reward signal is used to optimize the student model's strategy. The discriminator's input consists of student actions generated by the student model and expert demonstration data. The objective of the discriminator's reward signal is to minimize the negative log probability of the discriminator for student actions.

[0161]

[0162] In this way, by combining distillation loss and discriminator reward signals, the student model can learn diverse behavioral patterns from multiple teacher models.

[0163] Based on this, the step of "transferring the knowledge of the multiple teacher models to the single student model based on the distillation loss" in step S104 above may include, but is not limited to, the following steps:

[0164] The policy network parameters of the single student model are optimized based on the distillation loss and the reward signal of the preset discriminator, so as to transfer the knowledge of the multiple teacher models to the single student model.

[0165] The discriminator generates a reward signal based on the expert demonstration data and the student actions generated by the single student model.

[0166] After calculating the distillation loss between the student model and the teacher model, the terminal device can use knowledge distillation technology to optimize the policy network parameters of a single student model based on the distillation loss and the reward signal from the discriminator, thereby transferring the knowledge of multiple teacher models to a single student model.

[0167] In some embodiments, after transferring knowledge from multiple teacher models to a single student model, the terminal device can further optimize the strategy for that student model during robot-environment interaction. For example, the student model's strategy can be updated using data generated from the robot's interaction with the environment until the student model achieves the predetermined learning effect.

[0168] Next, a complete embodiment of the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application is presented.

[0169] Please refer to Figure 5 , Figure 5 The flowchart of a complete embodiment of the robot intelligent decision-making and control method based on adversarial imitation learning provided in this application is shown.

[0170] like Figure 5 As shown, the robot intelligent decision control method based on adversarial imitation learning provided in this application embodiment may include steps (1) to (8) as shown below.

[0171] Step (1): Render the collected expert demonstration data into an image and use CLIP to extract the mean value of trajectory features for each trajectory data.

[0172] Step (1) includes steps (1.1) to (1.3).

[0173] Step (1.1): Load expert demonstration data: The input is expert demonstration data {obs:[s1,s2,…,s T ],action:[a1,a2,…,aT The expert demonstration data was loaded using the pickle module `pickle.load` in Python. This data contains multiple trajectories, each consisting of a series of continuous states and actions, where s... t This is the state at step t, a t This is the action at step t.

[0174] Step (1.2): Initialize the environment: Initialize the three environments HalfCheetah, Hopper and Walker, and obtain the dimensional information of the environment, including the position dimension (nq, such as joint angle) and the velocity dimension (nv, such as joint angular velocity), to prepare for setting the environment state later.

[0175] Step (1.3): Traverse each trajectory: The input is all trajectories in the expert demonstration data. Traverse each trajectory τ = {(s1,a1),(s2,a2),…,(s...} T ,a T The process involves initializing a feature accumulator for each trajectory, iterating through each step t in the trajectory, setting the environment state, executing actions, rendering the current frame image, extracting image features using the CLIP model, accumulating features, and calculating the mean of trajectory features. The mean of trajectory features can be considered as the "overall features" of the trajectory, which is used for subsequent cluster analysis.

[0176] Step (2): Using the overall features of the K-means clustering trajectory, generate multiple cluster sets, where the KMeans algorithm iteratively optimizes the cluster center u. k The location of each trajectory feature is calculated, the distance between each feature and the cluster center is determined, and the feature is assigned to the nearest cluster. Finally, the cluster label for each trajectory is returned.

[0177]

[0178] Among them, C k It is the k-th cluster, u k It is the center of the k-th cluster.

[0179] Step (3): Train independent generative adversarial imitation learning strategies for each cluster to construct a teacher model group.

[0180] like Figure 6 As shown, step (3) includes steps (3.1) to (3.6).

[0181] Step (3.1): From expert demonstration data Mid-sampled state-action pairs are generated through interactions between the agent and the environment.

[0182] Step (3.2): Initialize the policy network π θ Discriminator Network D φ and value network Parameters;

[0183] Step (3.3): Calculate the discriminator loss Update the discriminator parameter φ;

[0184] Discriminator D φ The goal is to distinguish between expert behavior and agent behavior, and its loss function is the binary cross-entropy loss:

[0185]

[0186] Update the discriminant parameter φ using gradient descent:

[0187]

[0188] Where, η D It is the learning rate of the discriminator.

[0189] Step (3.4): Calculate the policy network loss Update the policy parameter θ using the TRPO algorithm;

[0190] Policy Network π θ The goal is to maximize the reward signal provided by the discriminator:

[0191]

[0192] Update the policy parameter θ using the TRPO algorithm:

[0193]

[0194] Where, η π It is the learning rate of the policy network.

[0195] Step (3.5): Calculate the value network loss Update the value network parameters ψ;

[0196] Value Network The goal is to accurately estimate the state-value function, and the loss error is the mean squared error loss:

[0197]

[0198] Where R(s) is the cumulative discount reward for state s.

[0199] Update the value network parameters using gradient descent. :

[0200]

[0201] Where, η V It is the learning rate of the value network.

[0202] Step (3.6): Repeat steps (3.2)-(3.5) until the policy network converges.

[0203] Step (4): Load the policy network of multiple teacher models Value Network and discriminator For each teacher model Generate teacher actions Where a i It is the action generated by the i-th teacher model.

[0204] Step (5): Initialize the policy network of the student model using data generated from interaction with the environment. Value Network and discriminator Generate student actions Where a s These are actions generated by the student model.

[0205] Step (6): Input student action a s Teacher's actions a i Calculate the MSE distillation loss between the student model and the teacher model;

[0206]

[0207] Where N represents the number of teacher models.

[0208] Step (7): Transfer knowledge from multiple teacher models to a single student model using knowledge distillation techniques, and optimize the student policy by integrating the reward signal from the discriminator. First, update the student policy parameter θ through backpropagation. s :

[0209]

[0210] Where, η s This is the learning rate of the student model. Furthermore, the reward signal r(s,a) provided by the discriminator is also used to optimize the policy, with the goal of minimizing the negative log probability of the discriminator for the student's actions:

[0211]

[0212] By combining distillation loss and discriminator reward signals, the student model can learn diverse behavioral patterns from multiple teacher models and further optimize its strategies through interaction with the environment, such as... Figure 7 As shown.

[0213] Step (8): Determine whether the student model has learned well. If so, end the process; otherwise, update the strategy and repeat the above steps.

[0214] In this embodiment, image rendering technology is used to transform environmental states (such as robot sensor data and scene information) into visual information, enhancing environmental perception capabilities. Furthermore, the CLIP feature extraction technology enables unified representation of multimodal data, improving data utilization efficiency. The use of a multi-teacher knowledge distillation mechanism effectively overcomes the limitations of single-strategy models, significantly enhancing the diversity and generalization ability of decision-making strategies.

[0215] Next, we will present a detailed experimental analysis of the robot intelligent decision-making and control method based on adversarial imitation learning provided in the embodiments of this application.

[0216] The robot intelligent decision control method based on adversarial imitation learning provided in this application was tested in three control tasks in a physics-based high-dimensional and continuous action space. The experimental results show that the method proposed in this application can learn a better policy than policy GAIL and policy OODIL (Observation-Orientation-Decision Iterative Learning) under the same conditions.

[0217] The technical effects of the robot intelligent decision-making and control method based on adversarial imitation learning proposed in the embodiments of this application are described in detail below with reference to experiments.

[0218] Please refer to Figure 8 , Figure 8 These are screenshots of the graphical interfaces for three experimental tasks of the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application.

[0219] like Figure 8 As shown, the three experimental tasks of the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application include:

[0220] HalfCheetah PyBullet: The goal is to make HalfCheetah run forward quickly. This task has a 26-dimensional continuous state space and a 6-dimensional continuous action space. The reward is given based on each step the object takes, considering its motion relative to the floor, control costs, etc.

[0221] Hopper PyBullet: The goal is to enable Hopper to move forward continuously without falling. This task has an eleven-dimensional continuous state space and a three-dimensional continuous action space. The reward is given for each step taken, based on the object's motion relative to the floor, control costs, etc.

[0222] Walker PyBullet: The goal is to enable Walker to walk forward while maintaining balance. This task has a 17-dimensional continuous state space and a 6-dimensional continuous action space. The reward is given based on each step taken, considering the object's motion relative to the floor, control costs, etc.

[0223] The experimental setup for the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application is as follows:

[0224] Feedforward neural networks were used as approximators with different architectures: the policy network consisted of two hidden layers, each with 128 units, with tanh activations in between, and a softmax output layer; the value network and discriminator network in the TRPOs algorithm each consisted of two 128-unit hidden layers and one output layer, with tanh activations in between. All networks were randomly initialized at the start of each trial. In each task, three random seeds were used for random initialization of the environment simulator and the model, and policy learning curves for the robot under the three methods were plotted and compared in Half-Cheetah, Hopper, and Walker environments.

[0225] The experimental results of the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application are as follows: Figure 9 , Figure 10 and Figure 11 As shown, where, Figure 9 This is a comparative curve diagram of the three methods in the Half Cheetah physical environment. Figure 10 This is a comparative curve diagram of the three methods in the Hopper physics environment. Figure 11 This is a comparative curve diagram of the three methods in the Walker physical environment. The results from the three graphs show that the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application has a faster learning speed and better learning effect compared to Gail and OODIL. This indicates that by introducing CLIP feature extraction and knowledge distillation, the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application can learn policies more effectively, thereby obtaining higher rewards. The OODIL method, through comparative learning clustering and weight allocation, also shows better performance than the traditional Gail, but not as good as the robot intelligent decision-making and control method based on adversarial imitation learning proposed in this application. Overall, introducing additional feature extraction and knowledge distillation steps can significantly improve the effect of imitation learning, enabling the model to perform better in complex tasks.

[0226] Next, please refer to Figure 12This application also provides a robot intelligent decision-making and control device based on adversarial imitation learning, which can realize the above-mentioned robot intelligent decision-making and control method based on adversarial imitation learning.

[0227] like Figure 12 As shown, in some embodiments, the robot intelligent decision-making and control device based on adversarial imitation learning provided in this application may include:

[0228] The data acquisition and feature extraction module is used to acquire expert demonstration data of the robot, and extract semantic features from the image data corresponding to the expert demonstration data based on the visual language pre-trained model to obtain the semantic features of each trajectory data in the expert demonstration data.

[0229] An adversarial imitation learning module is used to construct a teacher model group based on the semantic features and to generate an adversarial imitation learning strategy.

[0230] The knowledge distillation module is used to obtain the distillation loss between multiple teacher models in the teacher model group and a single student model; and to transfer the knowledge of the multiple teacher models to the single student model based on the distillation loss, and to perform robot intelligent decision control based on the student model.

[0231] In some embodiments, the image data corresponding to the expert demonstration data includes the image data corresponding to each trajectory data in the expert demonstration data;

[0232] The data acquisition and feature extraction module is also used to extract semantic features from the image data corresponding to the target trajectory data in the expert demonstration data based on the visual language pre-trained model, to obtain the mean value of the trajectory features of the target trajectory data, and to use the mean value of the trajectory features as the semantic features of the target trajectory data.

[0233] The target trajectory data is any one of the multiple trajectory data in the expert demonstration data.

[0234] In some embodiments, the data acquisition and feature extraction module is further configured to acquire environmental dimension information associated with the expert demonstration data; and, based on the environmental dimension information and the action information of each trajectory in the expert demonstration data, perform image rendering processing on each trajectory data to obtain image data corresponding to each trajectory data.

[0235] In some embodiments, the adversarial imitation learning module is further configured to divide the semantic features into multiple clusters; and to independently train each of the multiple clusters based on the generative adversarial imitation learning strategy to obtain a teacher model corresponding to each cluster; the teacher model group includes the teacher model corresponding to each cluster.

[0236] In some embodiments, the knowledge distillation module is further configured to obtain teacher actions generated by each of the multiple teacher models in the teacher model group, and to obtain student actions generated by a single student model; and to calculate the distillation loss between each of the multiple teacher models and the single student model based on the teacher actions and the student actions.

[0237] In some embodiments, the knowledge distillation module is further configured to acquire data generated by the robot's interaction with the environment; and to initialize the single student model based on the data generated by the robot's interaction with the environment, thereby obtaining student actions generated by the single student model.

[0238] In some embodiments, the knowledge distillation module is further configured to optimize the policy network parameters of the single student model based on the distillation loss and the reward signal of the preset discriminator, so as to transfer the knowledge of the multiple teacher models to the single student model.

[0239] The discriminator generates a reward signal based on the expert demonstration data and the student actions generated by the single student model.

[0240] It should be noted that the specific implementation of the robot intelligent decision-making and control device based on adversarial imitation learning provided in this application is basically the same as the specific implementation of the robot intelligent decision-making and control method based on adversarial imitation learning described above, and will not be repeated here.

[0241] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned robot intelligent decision-making and control method based on adversarial imitation learning. This computer device can be any intelligent terminal device, including tablet computers and personal computers.

[0242] Please see Figure 13 , Figure 13 The illustrations show the hardware structure of a computer device in some embodiments. The computer device may include:

[0243] The processor 1301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0244] The memory 1302 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1302 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1302 and is called and executed by the processor 1301 to execute the robot intelligent decision-making and control method based on adversarial imitation learning according to the embodiments of this application.

[0245] The input / output interface 1303 is used to implement information input and output;

[0246] The communication interface 1304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0247] Bus 1305 transmits information between various components of the device (e.g., processor 1301, memory 1302, input / output interface 1303, and communication interface 1304);

[0248] The processor 1301, memory 1302, input / output interface 1303 and communication interface 1304 are connected to each other within the device via bus 1305.

[0249] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described robot intelligent decision-making and control method based on adversarial imitation learning.

[0250] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0251] This application also provides a computer program product that stores a computer program. When the computer program is executed by a processor, it implements the above-described robot intelligent decision-making and control method based on adversarial imitation learning.

[0252] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0253] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0254] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0256] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0257] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0259] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0260] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0261] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0262] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A robot intelligent decision-making and control method based on adversarial imitation learning, characterized in that, The method includes: Acquire expert demonstration data for the robot; the image data corresponding to the expert demonstration data includes the image data corresponding to each trajectory data in the expert demonstration data; Based on a visual language pre-trained model, semantic features are extracted from the image data corresponding to the target trajectory data in the expert demonstration data to obtain the mean trajectory feature of the target trajectory data, and the mean trajectory feature is used as the semantic feature of the target trajectory data; the target trajectory data is any one of the multiple trajectory data in the expert demonstration data. A teacher model group is constructed based on the aforementioned semantic features and generative adversarial imitation learning strategy; Obtain the distillation loss between multiple teacher models and a single student model in the teacher model group; Based on the distillation loss, the knowledge of the multiple teacher models is transferred to the single student model, and the robot's intelligent decision-making and control are performed based on the student model.

2. The robot intelligent decision-making and control method based on adversarial imitation learning according to claim 1, characterized in that, The method further includes: Obtain environmental dimension information associated with the expert demonstration data; Based on the environmental dimension information and the action information of each trajectory in the expert demonstration data, image rendering processing is performed on each trajectory data to obtain the image data corresponding to each trajectory data.

3. The robot intelligent decision-making and control method based on adversarial imitation learning according to claim 1, characterized in that, The teacher model group constructed based on the semantic features and generative adversarial imitation learning strategy includes: The semantic features are divided into multiple clusters; Based on the generative adversarial imitation learning strategy, each of the multiple cluster sets is trained independently to obtain a teacher model corresponding to each cluster set; the teacher model group includes the teacher model corresponding to each cluster set.

4. The robot intelligent decision-making and control method based on adversarial imitation learning according to claim 1, characterized in that, The step of obtaining the distillation loss between multiple teacher models in the teacher model group and a single student model includes: Obtain the teacher actions generated by each of the multiple teacher models in the teacher model group, and obtain the student actions generated by a single student model; The distillation loss between each of the multiple teacher models and the single student model is calculated based on the teacher actions and the student actions.

5. The robot intelligent decision-making and control method based on adversarial imitation learning according to claim 4, characterized in that, The process of obtaining student actions generated by a single student model includes: Acquire the data generated by the robot's interaction with the environment; The single student model is initialized based on the data generated by the robot's interaction with the environment, resulting in student actions generated by the single student model.

6. The robot intelligent decision-making and control method based on adversarial imitation learning according to claim 1, characterized in that, The process of transferring knowledge from the multiple teacher models to the single student model based on the distillation loss includes: The policy network parameters of the single student model are optimized based on the distillation loss and the reward signal of the preset discriminator, so as to transfer the knowledge of the multiple teacher models to the single student model. The discriminator generates a reward signal based on the expert demonstration data and the student actions generated by the single student model.

7. A robot intelligent decision-making and control device based on adversarial imitation learning, characterized in that, The device includes: The data acquisition and feature extraction module is used to acquire expert demonstration data of the robot, wherein the image data corresponding to the expert demonstration data includes image data corresponding to each trajectory data in the expert demonstration data; and, based on a visual language pre-trained model, to extract semantic features from the image data corresponding to the target trajectory data in the expert demonstration data, to obtain the mean trajectory feature of the target trajectory data, and to use the mean trajectory feature as the semantic feature of the target trajectory data; wherein the target trajectory data is any one of the multiple trajectory data in the expert demonstration data. An adversarial imitation learning module is used to construct a teacher model group based on the semantic features and to generate an adversarial imitation learning strategy. The knowledge distillation module is used to obtain the distillation loss between multiple teacher models in the teacher model group and a single student model; and to transfer the knowledge of the multiple teacher models to the single student model based on the distillation loss, and to perform robot intelligent decision control based on the student model.

8. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the robot intelligent decision-making and control method based on adversarial imitation learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot intelligent decision-making and control method based on adversarial imitation learning as described in any one of claims 1 to 6.

Images