Method for optimizing selection of sensors by a robotic agent facing heterogeneous data streams

Abstract

The application discloses a method for robot agent to optimize selection of sensors for heterogeneous data flow. The application builds an expert data set representing the behavior of a driver, and splits the data of different sensor sets contained in the expert data set. The robot agent uses generative adversarial imitation learning to uniformly imitate the data flow of all sensor sets, thereby obtaining a basic strategy for making actions. Finally, a mask module is introduced, and a wake-sleep two-stage training process is used, so that the agent strategy can autonomously learn to select the sensors relied on in various situations. The technical scheme of the application not only provides a general adaptive learning scheme for processing heterogeneous sensor data and a solution for decision-making tasks with heterogeneous data flow as input, but also improves the decision-making efficiency of the agent and reduces redundant data calculation. Moreover, by focusing on key information and reducing irrelevant information interference, the decision-making accuracy and robustness are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent robot agents, and particularly relates to a method for optimizing selection of sensors of a robot agent facing heterogeneous data streams. BACKGROUND

[0002] Intelligent robot agent control problems have gained more and more attention in the industries of intelligent driving, medical services, etc. In the field of intelligent robot agent control, a motion robot agent usually has multiple different types of sensors, such as image sensors responsible for collecting visual data, laser radar sensors, sound sensors (microphone arrays) responsible for collecting auditory data, IMU sensors, pressure sensors, etc. responsible for monitoring motion states. The autonomous mobile task of a robot agent is usually modeled as a decision-making problem according to states, and the "state" is the data collected by the sensors at each place of the robot agent, and the "action" is the control signal of the robot agent. Since the interaction behavior of the robot agent with the environment is difficult to be quantified as a reward signal, the strategy learning of the motion robot agent often adopts imitation learning (IL) to solve it. In imitation learning, there are usually two types of roles: expert and learner robot agents. The expert is responsible for providing behavior data, which is usually a human expert or a trained AI robot agent; the learner will use the behavior data provided by the expert to train the strategy to complete the decision-making task, which is usually an AI robot agent to be trained. Common imitation learning methods include: behavior cloning (BC), inverse reinforcement learning (IRL), and generative adversarial imitation learning (GAIL).

[0003] However, most imitation learning tasks typically assume that the expert's and the robot's observation spaces are identical, meaning they are isomorphic. In the field of intelligent robot control, this translates to the assumption that the robot and the expert have the exact same set of sensors, resulting in isomorphic data flows during decision-making. In reality, however, different human experts often use different sensor sets when guiding a robot's actions, meaning their observation spaces are inconsistent. For example, humans might focus more on the robot's pose and visual signals from the surrounding environment when making decisions, often disregarding sensors like pressure and sound. Therefore, the human expert ultimately provides the learner with heterogeneous sensor data flows. Heterogeneous state spaces present greater challenges to the learning process. In such cases, the expert's strategy may differ from the robot's strategy. Using generic imitation learning methods to allow the robot to learn within heterogeneous sensor data flows may lead to poor learning outcomes.

[0004] Therefore, in practical applications, it is necessary to propose a sensor active selection strategy to solve the dilemma of low decision-making efficiency and accuracy faced by robot agents in policy learning when the state space is heterogeneous. Summary of the Invention

[0005] The purpose of this invention is to address the lack of existing imitation learning methods for heterogeneous data streams by providing a sensor active selection strategy training method based on heterogeneous data streams.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A first aspect of the present invention: a method for optimizing sensor selection for a robotic agent oriented to heterogeneous data streams, comprising the following steps:

[0008] (1) Construct a dataset representing expert behavior; the expert dataset contains behavioral data of multiple experts under various sensor configurations, the behavioral data including sensor readings and corresponding expert decision-making behavior data;

[0009] (2) The data of different sensor sets contained in the expert dataset are split, and for the data stream under each sensor set after splitting, the discriminator model of the sensor set is trained by generative adversarial imitation learning;

[0010] (3) Using a generative adversarial imitation learning framework, the outputs of the discriminators under each sensor set are summed as the reward signal, and the robot agent is trained to maximize the reward signal until the reward signal converges; that is, the robot agent obtains a basic strategy for generating behavior by uniformly imitating all sensor set data streams to obtain comprehensive multi-sensor information;

[0011] (4) Based on the basic strategy obtained in step (3), a mask module is introduced and randomly initialized; the mask module is used to output the weights of different sensors when the robot agent moves in the scene; the robot agent is guided to establish a movement strategy based on several sensors with a weight greater than 0.3; after introducing the initialized mask module, the wake-sleep two-stage algorithm is used for iterative optimization until the strategy converges stably, so that the robot agent strategy can learn to select the dependent sensors in multiple scenes.

[0012] Specifically, the plurality of sensors in the step (1) includes a pressure sensor at each joint of the robot agent, a speed sensor of the entire robot agent, a binocular camera / laser radar visual sensor, and a microphone sensor.

[0013] Further, the step (1) is specifically implemented by the following sub-steps:

[0014] (1.1) Using the interaction system of the open source platform MuJoCo as the environment, the observation space of the robot agent on the platform includes sensor position readings and speed readings of each robot agent component;

[0015] (1.2) Using the PPO algorithm, train several expert motion robot agents with different observation spaces in the MuJoCo environment;

[0016] (1.3) Using the trained expert robot agent to interact in the MuJoCo environment, collecting sensor readings and expert actions during the interaction, and constructing an expert dataset.

[0017] Further, the step (2) is specifically implemented by the following sub-steps:

[0018] (2.1) First, according to the sensor set contained in the expert dataset, the data stream is split, and the data stream of the same sensor set is split into the same group;

[0019] (2.2) Using generative adversarial imitation learning, train a discriminator for each group of data streams; wherein the discriminator is used to determine whether a piece of data belongs to the real data stream of the sensor set, and the output of the discriminator is between 0 and 1. The closer to 1 means the greater the probability that the data belongs to the sensor set, and vice versa.

[0020] Furthermore, step (3) is implemented through the following sub-steps:

[0021] (3.1) Discriminators from multiple data streams participate together in the generative adversarial learning task;

[0022] (3.2) The robot agent interacts with the MuJoCo environment based on its initial policy function. The interaction process generates action data including sensor readings at each decision and control signals of the actions taken, and transmits the action data to all discriminators.

[0023] (3.3) The discriminator provides the judgment result on whether the data comes from the real data stream; and feeds the result back to the robot agent;

[0024] (3.4) The robot agent directly sums the reward signals of all discriminators and uses it as its own reward signal;

[0025] (3.5) Using the generated data stream information and reward signal, with the goal of maximizing the reward signal, train the robot agent through the Proximal Policy Optimization (PPO) algorithm until the policy converges, that is, until the change in the reward signal is less than a predetermined threshold; and obtain the basic policy of the robot agent.

[0026] Furthermore, step (4) is implemented through the following specific sub-steps:

[0027] (4.1) Randomly initialize the mask module. The mask module will output a mask of the same length according to the sensor data. The mask represents the sensor that the robot agent should pay attention to when making decisions in the current state. The size of the mask value is positively correlated with the importance of the sensor data at this time.

[0028] (4.2) First, enter the awakening phase, freeze the training parameters of the mask module, and the mask module provides the mask in the current state according to the current data stream;

[0029] (4.3) The robot agent masks the sensor data according to the mask and gives behavioral data based on the current strategy and the masked sensor data;

[0030] (4.4) Transmit the unmasked sensor data and the behavioral data output by the robot agent to each discriminator, and the discriminator outputs the discrimination result to the robot agent;

[0031] (4.5) The robot agent performs generative adversarial imitation learning according to the discrimination result, specifically, according to the mask given by the mask module and the sensor set of each discriminator, the reward signals of all discriminators are added by weight as the reward signal of the robot agent, the weight of each discriminator is determined by the sensor of the corresponding data stream and the mask of the current state, the calculation method is the intersection union ratio of the sensor set and the mask or other any kind of index measuring the similarity of the set, the weight is positively correlated with the importance of the reward signal of the discriminator to the robot agent;

[0032] (4.6) Using the data stream information and the reward signal, the robot agent is trained by the proximal policy optimization algorithm PPO to maximize the sum of the weighted reward signals;

[0033] (4.7) Enter the sleep stage, unfreeze the parameters of the mask module, freeze the parameters of the robot agent, and the robot agent interacts with the environment according to the current policy to obtain the sensor data and behavior data under the current policy;

[0034] (4.8) Enumerate each piece of data in the data stream, including sensor data and behavior data, and give it to the mask module; the mask module outputs a mask according to the data, and the data of the mask is input into each discriminator respectively, and the discriminator gives the discrimination result as the reward signal;

[0035] (4.9) According to the reward signals of each discriminator, it is judged whether there is a difference between the discriminators; if the difference between the reward signals is greater than a predetermined threshold, there is a difference; otherwise, if the reward signals are consistent or very small or very large, there is no difference;

[0036] (4.10) If there is a difference, the data is passed through the curiosity module to determine whether to access the authoritative expert and finally obtain the real mask according to the curiosity module test result; if there is no difference or the curiosity module test is not passed, the real mask is determined by the sensor set corresponding to the discriminator;

[0037] (4.11) Use the data stream and the real mask to train the mask module for several rounds, and the training target is to make the mask module output the closest real mask, and the loss function of the training is set as the cross entropy error of the output result of the mask module and the real result;

[0038] (4.12) Use the data stream to train the curiosity module for several rounds, and the training target is to make the curiosity module more accurately determine whether a piece of sensor data has appeared before, and the loss function of the training is the mean square error of the prediction result of each module and the true value.

[0039] Specifically, the authoritative expert in step (4.10) comprises the following features:

[0040] The authority expert is an externally provided interface / tool, which is a pre-trained neural network module or a module defined based on artificial rules, and is used to give a real mask in a state.

[0041] Specifically, the mask module comprises the following features:

[0042] The mask module is a deep neural network or a trainable parameter equal to the number of sensors; the output bit number of the mask module is consistent with the number of sensors, and each bit is a floating point number between 0 and 1, representing the weight of different sensors of the robot agent when moving in the environment, guiding the robot agent to establish a moving strategy on the basis of a plurality of sensors with a weight greater than 0.3.

[0043] The second aspect of the present application: an electronic device comprising a memory and a processor, the memory being coupled with the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the method for optimizing selection of sensors of a robot agent facing heterogeneous data streams.

[0044] The third aspect of the present application: a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize the method for optimizing selection of sensors of a robot agent facing heterogeneous data streams.

[0045] The beneficial effects of the present application are as follows:

[0046] The present application provides a general adaptive learning scheme for processing heterogeneous sensor data by introducing a mask module in generative adversarial imitation learning, and provides a solution for decision-making tasks with heterogeneous data streams as input. The mask module can help the robot agent to select sensors, better establish a strategy on the basis of important sensor data, focus on key information, reduce irrelevant information interference, and improve decision-making accuracy and robustness; at the same time, it can guide the learner to imitate different experts according to different weights, improve the decision-making efficiency of the robot agent, and reduce redundant data calculation. In addition, the curiosity module controls the number of queries to the authority expert, effectively reducing the dependence on the authority expert in the learning process. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The flowchart for the present application is executed;

[0048] Figure 2 The robot agent imitation learning diagram under the heterogeneous sensor data stream is briefly described for the present application;

[0049] Figure 3is a structure diagram of a sensor active selection strategy training method based on heterogeneous data streams;

[0050] Figure 4 is a structure diagram of a curiosity module;

[0051] Figure 5 is an example schematic diagram of the present application. DETAILED DESCRIPTION

[0052] The present application will be described in detail below with reference to the accompanying drawings.

[0053] As shown in the figure, the execution flow of the present application has six steps, which are as follows: Figure 1

[0054] Step one, train expert robot agents on the MuJoCo platform. Specifically, use the interaction system of the open-source platform MuJoCo as the environment, the observation space of the robot agent on the platform includes the sensor position reading, speed reading of each robot agent component; use PPO algorithm to train several motion robot agents with different observation spaces in the MuJoCo environment as experts.

[0055] Step two, use the trained expert robot agent to build an expert dataset. Specifically, use the trained expert robot agent to interact in the MuJoCo environment, collect sensor readings and expert actions during the interaction, and build an expert dataset.

[0056] Step three, group the data in the dataset according to the sensor set. Specifically, according to the sensor set contained in the expert dataset, split the data stream, and split the data stream of the same sensor set into the same group.

[0057] Step four, train a discriminator for each group of sensor sets. Specifically, for the data stream under each sensor set after splitting, use generative adversarial imitation learning to train the discriminator model of the sensor set; specifically including the following sub-steps:

[0058] (3.1) Multiple data streams participate in the generative adversarial imitation learning task together;

[0059] (3.2) The robot agent interacts with the environment according to its initial policy function, and the interaction process generates action data including sensor readings at each decision and control signals of the action taken, and transmits the action data to all discriminators;

[0060] (3.3) The discriminators respectively give the judgment result of whether the data comes from the real data stream; and feedback the result to the robot agent;

[0061] ​(3.4) The robot agent directly adds all the reward signals of the discriminators as its own reward signal;

[0062] (3.5) The generated data stream information and reward signal are used to train the robot agent by a proximal policy optimization (PPO) algorithm to maximize the reward signal until the policy converges, and a basic policy of the robot agent is obtained.

[0063] Step five, initialize the mask module. Specifically, the following sub-steps are implemented:

[0064] (5.1) Randomly initialize the mask module, which will output masks of the same length according to the sensor data. The mask represents the sensors that the robot agent should focus on when making decisions in the current state. The larger the mask value, the higher the importance of the sensor data at this time;

[0065] (5.2) First, enter the wake-up phase, freeze the training parameters of the mask module, and the mask module gives the mask in the current state according to the current data stream;

[0066] (5.3) The robot agent masks the sensor data according to the mask, and gives the behavior data according to the current policy and the masked sensor data;

[0067] (5.4) The unmasked sensor data and the behavior data output by the robot agent are transmitted to each discriminator, and the discriminators output the discrimination results as reward signals, which are fed back to the robot agent;

[0068] (5.5) The robot agent adds all the reward signals of the discriminators according to the mask given by the mask module and the sensor set of each discriminator, as its own reward signal. The weight of each discriminator is determined by the sensor of the corresponding data stream and the mask of the current state. The calculation method is the intersection-union ratio (or other arbitrary index measuring the similarity of the set) of the sensor set and the mask. The higher the weight, the more important the reward signal of the discriminator to the robot agent;

[0069] (5.6) Use the data stream information and the reward signal to maximize the sum of the weighted reward signals by the proximal policy optimization (PPO) algorithm to train the robot agent;

[0070] (5.7) Enter the sleep phase, unfreeze the parameters of the mask module, and freeze the parameters of the robot agent. The robot agent interacts with the environment according to the current policy to obtain the sensor data and behavior data under the current policy;

[0071] (5.8) Enumerate each piece of data in the data stream (including sensor data and behavior data), and give it to the mask module; the mask module outputs a mask according to the data, and the data after the mask is input into each discriminator respectively, and the discriminator gives the discrimination result as the reward signal;

[0072] (5.9) According to the reward signal of each discriminator, it is judged whether there is a difference between the discriminators, if the reward signal difference is large, there is a difference; otherwise, if the reward signal is consistent or consistent, there is no difference;

[0073] (5.10) If there is a difference, the data is passed through the curiosity module, and the true mask is obtained according to the curiosity module test result; if there is no difference or the curiosity module test is not passed, the true mask is determined by the sensor set corresponding to the discriminator;

[0074] (5.11) Use the data stream and the true mask to train the mask module for several rounds, and the training target is to make the mask module output as close to the true mask as possible, and the loss function of the training is set as the cross entropy error of the mask module output result and the true result;

[0075] (5.12) Use the data stream to train the curiosity module for several rounds, and the training target is to make the curiosity module more accurately determine whether a piece of sensor data has appeared, and the loss function of the training is the mean square error of the predicted result and the true value of each module.

[0076] Step six, iterate the training of the wake-up stage and the sleep stage until the robot agent motion strategy converges. Specifically, the following sub-steps are included:

[0077] (6.1) Wake-up stage: use the mask module to optimize the generative adversarial imitation learning process of the robot agent. Collect the behavior data of the robot agent in the environment, and provide it to the discriminator. Then calculate the weight of different data streams according to the mask given by the mask module, reweight the output of the discriminator in the generative adversarial imitation learning, and train each discriminator and the robot agent;

[0078] (6.2) Sleep stage: first, the robot agent acts in the environment according to its current strategy, and collects the sensor data stream of the current strategy; after the collection is completed, it is observed whether there is a difference in the judgment result of the data stream by each discriminator. If there is no difference, the true mask is determined by the discriminator score and each sensor set; if there is a difference, the authoritative expert is consulted to obtain the true mask, and then the mask module is trained. The authoritative expert is an externally provided interface / tool for giving a true mask in a state, but the query cost is high, so the query times are limited;

[0079] (6.3) Iteratively perform the steps of the wake stage and sleep stage described in step five until the policy converges, completing the entire training process. Alternating iteration of the above wake-sleep stages can enable the robot agent's behavior policy to perform well and remain stable for a long time.

[0080] As shown in Figure 2 , a robot agent imitation learning diagram under a heterogeneous sensor data stream. The sensors loaded on the robot agent include radar signal sensors, video signal sensors, audio sensors, and pressure sensors, while the sensors possessed by the three experts are less. After the experts interact with the real environment, expert data is obtained. Finally, the expert data set is provided to the robot agent for training.

[0081] As shown in Figure 3 and Figure 4 , the present application is a sensor active selection policy training method based on heterogeneous data streams. In the wake stage, the mask module remains unchanged. In each step of the interaction between the robot agent and the environment, the mask module is used to mask the data stream fed back by the environment, and the robot agent gives the action based on the masked data stream and interacts with the environment, and the discriminator of the generated adversarial imitation learning module gives the discrimination result according to the current data stream and the action of the robot agent, and the robot agent weights the discrimination result according to the mask as a reward signal to train itself. The training goal is to maximize the reward signal, that is, to train the discriminator to output as large as possible.

[0082] In the sleep stage, the robot agent and the discriminator remain unchanged. According to the divergence between the discriminators, each piece of data stream in the sampled trajectory of the robot agent is labeled with the real mask. If there is no divergence between the discriminators, the real mask is selected by the discriminators together; if there is a divergence between the discriminators, the authoritative expert is queried according to the detection result of the curiosity module. The mask module is trained according to the state of the robot agent sampling trajectory and the corresponding real mask.

[0083] The following will be described in detail:

[0084] First, the experts of different sensor sets respectively sample the expert's policy trajectory based on the optimal strategy under their own used sensors, and regard the current data stream as the state in the trajectory. The sampling in this stage is to provide training samples for the subsequent imitation learning process of the robot agent. Each expert interacts with the environment. The expert makes actions according to the environment state under its own perspective, and feeds back to the environment, and the environment gives the next state according to the expert action. In this step, the new state returned by the environment is recorded at all times and projected to the expert's own perspective, and then "state-action-next state" is stored as an expert trajectory;

[0085] After obtaining enough expert trajectory data, the robot agent is trained to learn a unified strategy from all experts by using generative adversarial imitation learning (GAIL) based on the strategy trajectory from the expert's perspective. The goal is to let the robot agent learn a basic strategy for interacting with the environment. The goal of each GAIL is to wherein represents the output of the i-th discriminator, the value of which is between 0 and 1; represents a piece of data in the data set, represents the parameters of all discriminator neural networks, is the parameter of the i-th discriminator; represents the parameters of the robot agent strategy network; is the expert data set, is the robot agent data set; represents the probability expectation, which represents the expected value of the above formula calculated according to the sampling probability when the data is sampled from the data set. In this step, the weights of all experts are consistent, and all experts guide the learning process of the robot agent through the discriminator. The robot agent is trained using the Proximal Policy Optimization (PPO) algorithm, and the learning goal of the robot agent is to maximize the total reward signal obtained by the action, i.e. .

[0086] Then, a mask module is introduced to perform the two-stage training process of the wake-sleep algorithm. The mask module is responsible for outputting a mask according to the current state, and the mask has the same dimension as the state space but the value range is between 0 and 1; the mask represents the information that the robot agent should focus on when making decisions in the current state. In the wake-up phase: the mask module is used to adjust the generative adversarial imitation learning. In this step, when the robot agent learns the strategy through generative adversarial imitation learning, different experts will determine their own weights according to the output of the mask and the differences in their observation spaces. The specific way is to calculate the intersection over union of the mask and the observation space, and use the intersection over union as the weight for the generative adversarial training. In this process, the mask module parameters are fixed, and the discriminators of each expert and the robot agent are further trained through the process of generative adversarial imitation learning. The robot agent is trained using the Proximal Policy Optimization (PPO) algorithm, but the reward signal of the robot agent is obtained by multiplying the original reward signal by the corresponding weight and then adding it, and the training goal becomes wherein is the value of the i-th bit output by the mask module.

[0087] Then is the sleep stage, first using the current strategy of the robot agent, sampling in the environment, obtaining the trajectory of the current mixed strategy. Enumerate each transition equation (state-action-next state) in the trajectory data, respectively mask the current state to each expert perspective and input into each discriminator, according to the output result of the discriminator, judge whether the discriminator has divergence on the policy trajectory. Specifically, if the output result of the discriminator is higher than a certain threshold, it is believed that the perspective of the expert is selected; if it is lower than a certain threshold, it is judged by the curiosity module whether to ask the authoritative expert. The curiosity module predicts the hidden code of the next state according to the state and action, and calculates the hidden code of the next state, if the calculated hidden code and the predicted hidden code are greatly different, the state mask is given by the authoritative expert; otherwise the state mask is selected by expert voting. In this way, the real mask under each state is obtained, and the mask module is trained, and the loss function of the mask module is defined as the mean square error between the real mask and the mask output by the module, that is , wherein represents the output of the mask module when the sensor reading is , is the real mask. At the same time, the transition equation is used to train the curiosity module, which includes the encoding module and the prediction module. The encoding module is responsible for converting the state into state encoding, and the prediction module is responsible for predicting the next state encoding from the state encoding and action. Its loss function is designed as the mean square error of the predicted next state encoding and the real next state encoding, that is , wherein is the encoding module, which is a neural network structure; is the prediction module, which is also a neural network structure; , , respectively represent the current sensor reading, the sensor reading of the next state, and the action made by the robot agent under the current sensor reading.

[0088] Repeat the awake stage-sleep stage training until the policy converges.

[0089] Embodiment 1: as Figure 5As shown, it is a simplified diagram of a robot agent training instance on a MuJoCo platform. The complete robot agent in the figure includes position sensors and velocity sensors at each joint. The experts are experts without right leg sensors, without left arm sensors, and without right arm sensors. In theory, the expert lacking the right leg sensor has a lower guiding effect on the robot agent learning to walk, and the data of the other two experts has a greater effect on the robot agent learning how to walk. Therefore, after training, the walking strategy of the robot agent is mainly generated by the guidance of the latter two experts. The robot agent before training cannot complete the walking task and will fall down after standing in the environment for a short time. The robot agent after training can complete the task of walking alternately with the left and right feet.

[0090] The embodiment of the present application also provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the method for optimizing selection of sensors of a robot agent facing heterogeneous data streams in the above embodiment.

[0091] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, of any of the preceding embodiments. The computer readable storage medium can also be any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Further, the computer readable storage medium can also include an internal storage unit of any data processing capable device and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0092] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0093] The above embodiments are only used to illustrate the design ideas and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it, and the protection scope of the present application is not limited to the above embodiments. Therefore, any equivalent change or modification made according to the disclosed principles and design ideas is within the protection scope of the present application.

Claims

1. A method for optimizing sensor selection in a robotic agent oriented towards heterogeneous data streams, characterized in that, The method comprises the following steps: (1) constructing a data set representing expert behavior; the expert data set contains the behavior data of multiple experts under multiple sensor configurations, including sensor reading data and corresponding expert decision-making behavior data; (2) splitting the data of different sensor sets contained in the expert data set, and for each data stream under a split sensor set, training a discriminator model for the sensor set using generative adversarial imitation learning; (3) using a generative adversarial imitation learning framework, summing the discriminator outputs under each sensor set as a reward signal, and training the robot agent to maximize the reward signal until the reward signal converges; that is, the robot agent obtains a basic strategy for generating behavior by uniformly imitating all sensor set data streams to obtain comprehensive multi-sensor information; (4) based on the basic strategy obtained in step (3), introducing a mask module and randomly initializing it; the mask module is used to output the weights of different sensors when the robot agent moves in the scene; guiding the robot agent to establish a movement strategy based on several sensors with a weight greater than 0.3; after introducing the initialized mask module, the wake-sleep two-stage algorithm is used for iterative optimization until the strategy converges stably, enabling the robot agent strategy to learn to select dependent sensors in multiple scenarios.

2. The method for optimizing selection of sensors for a heterogeneous data flow oriented robotic agent according to claim 1, wherein, The sensors in step (1) include pressure sensors at each joint of the robot agent, speed sensors for the entire robot agent, binocular camera / laser radar visual sensors, and microphone sensors.

3. The method for robot agent optimization selection sensor for heterogeneous data stream according to claim 1, wherein, Step (1) is implemented through the following sub-steps: (1.1) using the interaction system of the open-source platform MuJoCo as the environment, the observation space of the robot agent on the platform includes sensor position readings and speed readings of each robot agent component; (1.2) using the PPO algorithm to train several motion robot agents with different observation spaces in the MuJoCo environment as experts; (1.3) using the trained expert robot agent to interact in the MuJoCo environment, collecting sensor readings and expert actions during the interaction to construct an expert data set.

4. The method for robot agent optimization selection sensor for heterogeneous data stream according to claim 1, characterized in that, Step (2) is implemented through the following sub-steps: (2.1) first, according to the sensor sets contained in the expert data set, the data stream is split, and the data stream of the same sensor set is split into the same group; (2.2) using generative adversarial imitation learning, training a discriminator for each group of data streams; wherein the discriminator is used to determine whether a data belongs to the true data stream of the sensor set, and the discriminator output is between 0 and 1, the closer to 1 means the greater the probability that the data belongs to the sensor set, and vice versa.

5. The method for robot agent optimization selection sensor for heterogeneous data stream according to claim 1, characterized in that, Step (3) is implemented through the following sub-steps: (3.1) multiple data streams participate in the generative adversarial imitation learning task together; (3.2) The robot agent interacts with the MuJoCo environment based on its initial policy function. The interaction process generates action data including sensor readings at each decision and control signals of the actions taken, and transmits the action data to all discriminators. (3.3) The discriminator provides the judgment result on whether the data comes from the real data stream; and feeds the result back to the robot agent; (3.4) The robot agent directly sums the reward signals of all discriminators and uses it as its own reward signal; (3.5) Using the generated data stream information and reward signal, with the goal of maximizing the reward signal, train the robot agent through the Proximal Policy Optimization (PPO) algorithm until the policy converges, that is, until the change in the reward signal is less than a predetermined threshold; and obtain the basic policy of the robot agent.

6. The method for robot agent optimization selection sensor for heterogeneous data stream according to claim 1, characterized in that, Step (4) is implemented through the following specific sub-steps: (4.1) Randomly initialize the mask module. The mask module will output a mask of the same length according to the sensor data. The mask represents the sensor that the robot agent should pay attention to when making decisions in the current state. The size of the mask value is positively correlated with the importance of the sensor data at this time. (4.2) First, enter the awakening phase, freeze the training parameters of the mask module, and the mask module provides the mask in the current state according to the current data stream; (4.3) The robot agent masks the sensor data according to the mask and gives behavioral data based on the current strategy and the masked sensor data; (4.4) Transmit the unmasked sensor data and the behavioral data output by the robot agent to each discriminator, and the discriminator outputs the discrimination result to the robot agent; (4.5) The robot agent performs generative adversarial learning based on the discrimination results. Specifically, based on the mask provided by the mask module and the sensor set of each discriminator, the reward signals of all discriminators are weighted and summed as its own reward signal. The weight of each discriminator is determined by the sensor of the corresponding data stream and the mask of the current state. The calculation method is the intersection-union ratio of the sensor set and the mask. The weight is positively correlated with the importance of the reward signal of the discriminator to the robot agent. (4.6) Using data stream information and reward signals, the robot agent is trained by maximizing the sum of weighted reward signals through the Proximal Policy Optimization (PPO) algorithm. (4.7) Entering the sleep stage, the parameters of the mask module are unfrozen, the parameters of the robot agent are frozen, and the robot agent interacts with the environment according to the current strategy to obtain sensor data and behavioral data under the current strategy. (4.8) Enumerate each data item in the data stream, including sensor data and behavioral data, and pass it to the masking module; The mask module outputs a mask based on the data, and then inputs the mask data into each discriminator. The discriminator gives the discrimination result as a reward signal. (4.9) Determine whether there is a discrepancy between the discriminators based on the reward signals of each discriminator; if the difference in reward signals is greater than a preset threshold, then there is a discrepancy. Otherwise, if the reward signals are consistently very small or consistently very large, then there is no disagreement; (4.10) If there is a difference, the data passes through the curiosity module, and according to the curiosity module test result, it is determined whether to access the authoritative expert, and finally obtain the real mask; If there is no difference or the curiosity module test fails, the real mask is determined by the sensor set corresponding to the discriminator; The authoritative expert is an externally provided interface / tool for giving a real mask in a state; (4.11) The mask module is trained for several rounds using the data stream and the real mask, and the training target is to make the mask module output closest to the real mask, and the loss function of the training is set as the cross-entropy error of the mask module output result and the real result; (4.12) The curiosity module is trained for several rounds using the data stream, and the training target is to make the curiosity module more accurately determine whether a piece of sensor data has appeared, and the loss function of the training is the mean square error of the prediction result of each module and the real value.

7. The method for robot agent optimization selection sensor for heterogeneous data stream according to claim 6, characterized in that, The authoritative expert in step (4.10) comprises the following features: The authoritative expert is an externally provided interface / tool, which is a pre-trained neural network module or a module defined based on artificial rules, for giving a real mask in a state, and since its query has a certain cost, the number of queries needs to be controlled to control the cost.

8. The method for robot agent optimization selection sensor for heterogeneous data stream according to claim 1, characterized in that, The mask module comprises the following features: The mask module is a deep neural network; the output bit number of the mask module is consistent with the number of sensors, and each bit is a floating point number between 0 and 1, representing the weight of different sensors when the robot agent moves in the environment, guiding the robot agent to establish a moving strategy on the basis of several sensors with a weight greater than 0.

3.

9. An electronic device comprising a memory and a processor, characterized in that The memory and the processor are coupled; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the method for optimizing selection of sensors of a robot agent facing heterogeneous data stream according to any one of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the method for optimizing selection of sensors of a robot agent facing heterogeneous data stream according to any one of claims 1-8.

Images