Human-computer interaction likelihood value evaluation method and device based on deep RNN and medium

Abstract

The invention discloses a human-computer interaction likelihood value evaluation method and device based on depth RNN and a medium, and relates to the technical field of human-computer hybrid intelligence, and the method comprises the steps: obtaining an environment depth image of a robot during autonomous navigation movement; inputting the environment depth image into an action value estimation network module to obtain action value estimation; inputting the environment depth image into an interaction likelihood value estimation network module to obtain an environment state hidden feature vector; inputting the action value estimation into an interactive likelihood value estimation network module to obtain an action value estimation hidden feature vector; and inputting the environment state hidden feature vector and the action value estimation hidden feature vector into a man-machine interaction likelihood estimation module to obtain interaction likelihood estimation values of the robots, and selecting the robot with the maximum estimation value according to the interaction likelihood estimation values of the robots. According to the invention, active selection can be carried out on a multi-robot system in a dynamic complex environment.

Description

Technical Field

[0001] The present application relates to the field of human-machine hybrid intelligent technology, and in particular to a method, device and medium for evaluating the likelihood value of human-machine interaction based on deep RNN. Background Art

[0002] As intelligent robots become more prevalent in complex and dynamic environments, they often encounter emergency situations that lead to mission failures. For example, during autonomous navigation, drones lack sufficient prediction space for moving objects that suddenly appear in the picture due to the limited camera field of view. Remote human intervention to guide robots out of these dilemmas has proven to be an effective solution, that is, control of the robot that needs the most help can be assigned to the monitoring personnel. However, with the increase in the number of unmanned devices deployed, a single operator often needs to monitor multiple robots that perform independent tasks at the same time. Humans are challenged to quickly filter and select the robot that needs the most control from the status information of multiple machines. In order to reduce the burden on operators and improve the efficiency of the overall task, human-machine decision-making power autonomous allocation algorithms can be developed and deployed to predict whether the robot needs the operator's help based on the robot's current state, environmental conditions, task requirements, and historical interaction data. Currently, the most common way to achieve autonomous selection in sliding autonomy settings is to install complex sensor systems on the robot and select priorities through manually adjusted thresholds. However, this approach equips each robot with too many measurement devices and requires experienced personnel to adjust the thresholds according to different tasks. Another class of methods uses scheduling algorithms to infer human intentions or assess the danger level of the robot's environment. However, these methods usually require accurate robot dynamic models and surrounding environment models, which are usually inadequate in dynamic environments. As a result, the effectiveness of these methods is often limited. Summary of the invention

[0003] The purpose of this application is to provide a method, device and medium for evaluating the likelihood value of human-computer interaction based on deep RNN, which can effectively draw on the action value estimation that conforms to the robot's own action decision model, and actively select multi-robot systems in dynamic and complex environments.

[0004] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for evaluating the likelihood value of human-computer interaction based on a deep RNN, comprising: Acquire the depth image of the environment when the robot is moving in autonomous navigation.

[0005] The environmental depth image is input into an action value estimation network module to obtain an estimation of the action value of the robot during autonomous navigation movement.

[0006] The environmental depth image is input into an interactive likelihood estimation network module to obtain a hidden feature vector of the environmental state of the robot during autonomous navigation movement.

[0007] The action value estimation is input into the interaction likelihood estimation network module to obtain the action value estimation hidden feature vector.

[0008] The environment state hidden feature vector and the action value estimation hidden feature vector are input into a human-machine interaction likelihood estimation module to obtain an interaction likelihood estimation value of the robot.

[0009] According to the interaction likelihood estimation values ​​of each robot, the robot with the largest estimation value is selected.

[0010] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the above-mentioned methods for evaluating the likelihood value of human-computer interaction based on a deep RNN.

[0011] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the above-mentioned methods for evaluating the likelihood value of human-computer interaction based on a deep RNN.

[0012] According to the specific embodiments provided in this application, this application discloses the following technical effects: The present application provides a method, device and medium for evaluating the likelihood of human-computer interaction based on deep RNN. First, a specific device is used to capture the environmental depth image of the robot during autonomous navigation movement. This step ensures that the data source on which the evaluation process depends is accurate and real-time. Subsequently, the acquired environmental depth image is input into a pre-trained action value estimation network module; then, the same environmental depth image is input again into another pre-trained interaction likelihood estimation network module. This module focuses on extracting and encoding hidden feature vectors from environmental images, which reflect the key information of the robot's environmental state. At the same time, the action value estimation obtained in the previous step is used as input and also passed to the interaction likelihood estimation network module. The module further processes these action value estimation information to generate corresponding action value estimation hidden feature vectors, which represent the potential value of the robot's action under a specific environmental state. Finally, the environmental state hidden feature vector and the action value estimation hidden feature vector are jointly input into the human-computer interaction likelihood estimation module. Based on the deep RNN architecture, the module comprehensively analyzes the intrinsic relationship between the environmental state and the action value, and calculates the interaction likelihood estimation value of the robot. When evaluating the interaction likelihood values ​​of multiple robots, the robot with the largest estimated value is selected as the optimal choice by comparing the interaction likelihood estimates of each robot. This step ensures the objectivity and accuracy of the decision-making process and helps improve the overall performance of the human-computer interaction system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 This is an application environment diagram of a method for evaluating the likelihood of human-computer interaction based on deep RNN in one embodiment of the present application.

[0015] Figure 2 A flow chart of a method for evaluating the likelihood of human-computer interaction based on a deep RNN is provided in accordance with an embodiment of the present application.

[0016] Figure 3 A schematic flow chart of a deep recurrent neural network method for estimating the likelihood of active human-computer interaction provided in one embodiment of the present application.

[0017] Figure 4 A schematic diagram of the training process of an action value estimation network module of a human-computer interaction likelihood estimation network provided in one embodiment of the present application.

[0018] Figure 5 A schematic diagram of the training process of a human-computer interaction likelihood estimation network module for human-computer active interaction likelihood estimation provided in one embodiment of the present application.

[0019] Figure 6 A schematic diagram of a multi-machine interactive verification scenario provided in an embodiment of the present application Figure 1 .

[0020] Figure 7 A schematic diagram of a multi-machine interactive verification scenario provided in an embodiment of the present application Figure 2 .

[0021] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The embodiment of the present application provides a method for evaluating the likelihood value of human-computer interaction based on a deep RNN, which can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the environmental depth image to be processed to the server 104. After the server 104 receives the environmental depth image to be processed, for the environmental depth image to be processed, the server 104 obtains the environmental depth image of the robot during autonomous navigation movement; the environmental depth image is input into the action value estimation network module to obtain the action value estimation of the robot during autonomous navigation movement; the environmental depth image is input into the interactive likelihood estimation network module to obtain the hidden feature vector of the environmental state of the robot during autonomous navigation movement; the action value estimation is input into the interactive likelihood estimation network module to obtain the action value estimation hidden feature vector; the environmental state hidden feature vector and the action value estimation hidden feature vector are input into the human-computer interaction likelihood estimation module to obtain the interactive likelihood estimation value of the robot; according to the interactive likelihood estimation value of each robot, the robot with the largest estimated value is selected. The server 104 may feed back the selected robot with the largest estimated value to the terminal 102. In addition, in some embodiments, the human-machine interaction likelihood value evaluation method based on the deep RNN may also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 may directly perform image processing on the environmental depth image to be processed, or the server 104 may obtain the environmental depth image to be processed from the data storage system and perform image processing on the environmental depth image to be processed.

[0025] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0026] In an exemplary embodiment, Figure 2 As shown, a method for evaluating the likelihood value of human-computer interaction based on a deep RNN is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 206. Among them: Step 201, obtaining a depth image of the environment when the robot is in autonomous navigation motion.

[0027] Step 202: Input the environment depth image into an action value estimation network module to obtain an action value estimation of the robot during autonomous navigation movement.

[0028] Step 203: input the environment depth image into an interactive likelihood estimation network module to obtain a hidden feature vector of the environment state of the robot during autonomous navigation movement.

[0029] Step 204: input the action value estimate into the interactive likelihood value estimation network module to obtain an action value estimate hidden feature vector.

[0030] Step 205: input the environment state hidden feature vector and the action value estimation hidden feature vector into a human-machine interaction likelihood estimation module to obtain an interaction likelihood estimation value of the robot.

[0031] Step 206: According to the interaction likelihood estimation values ​​of the robots, the robot with the largest estimation value is selected.

[0032] When executing steps 201-203, the specific steps may be as follows: like Figure 3 As shown, when multiple robots perform autonomous navigation, each robot inputs the denoised and resized environment depth image into the deep action value estimation network module for action value estimation, and the interactive likelihood value estimation network module to obtain the hidden feature vector of the environment state.

[0033] Specifically, the environment depth image is input into the action value estimation network module to obtain the action value estimation of the robot during autonomous navigation movement, specifically including: The time series environment perception data is preprocessed to obtain a state matrix; the time series environment perception data is a data representation of an environment depth image; Figure 3 As shown in the figure, when predicting the interaction likelihood estimation value of the robot, firstly, the time series environment perception data is input into the action value estimation network module and the interaction likelihood estimation network module respectively and preprocessed to obtain the input state matrix , T is the number of time nodes in the set time series data.

[0034] The state matrix is ​​input into the action value estimation network module to calculate the value estimation and action advantage estimation.

[0035] The value estimate is added to the action advantage estimate, and the mean of the value estimates corresponding to each discrete action in the action performed by the robot selected by the action value estimation network is subtracted to obtain the action value estimate.

[0036] The depth camera acquires the environment depth images of multiple time steps (T), and then performs image denoising and size transformation to adapt to the network model in this embodiment.

[0037] In the action value estimation network module, the input state , the action value estimation network module calculates the value estimate and action advantage estimation , then add and subtract Discrete actions The corresponding value estimates are averaged and then fused to obtain the action value estimate .

[0038]

[0039] in is the input state, The actions performed by the robot selected for the action value estimation network. A set of trainable parameters for stacked convolutional layers and bidirectional gated recurrent units. is the trainable parameter of the first fully connected layer estimated by the corresponding value; similarly, are the trainable parameters of the second fully connected layer corresponding to the action advantage estimation, is the number of discrete output actions of the policy model, An estimate of the action value; For value estimation and is the action advantage estimate, and a is each specific action in A. Q here should be a matrix, which is the value estimate corresponding to all a in A, and small a is each specific action in A. The sum of it here means to find the mean of the advantage estimate P corresponding to all a, so this is the data regularization of the advantage estimate matrix corresponding to A.

[0040] Specifically, Figure 3 In the example, stacked convolutional layers and bidirectional gated recurrent units are used to extract the input state. The spatial and temporal characteristics of . First, the data is compressed through the stacked convolutional layers and the state feature matrix is ​​extracted through the convolution kernel. Then, the state features are sequentially input into the bidirectional gated recurrent unit to obtain the time feature state values ​​of multiple time steps. The bidirectional gated recurrent unit mainly includes two gating structures, the update gate and the reset gate, and the candidate hidden state at the current moment. and the current hidden state Two calculated variables.

[0041] For the bidirectional gated recurrent unit operation at a certain moment in the T-step sequence, the state feature vector at this moment is First, after the reset gate operation, we get , as shown in formula (2): (2).

[0042] in, is the hidden state vector of the depth map at time t-1 calculated by the bidirectional gated recurrent unit operation at the previous moment. If it is the first moment, then is a zero vector of the same dimension, and To reset the parameters of the fully connected layer in the gate, is the bias vector, As the activation function, the Sigmoid function is used here, and its expression is: (3).

[0043] At the same time, the state eigenvector After the operation of the update gate, we get , as shown in Formula 4: (4).

[0044] in and To reset the parameters of the fully connected layer in the gate, is the bias vector.

[0045] Candidate hidden states The calculation of is shown in formula (5). The current time step resets the gate The hidden state at the previous time step Element-wise multiplication is performed by multiplying two vectors or matrices of the same dimension one by one. Then, the result of element-wise multiplication is added to the hidden state of the current time step, and the candidate hidden state is calculated through a fully connected layer with an activation function tanh. . and is the fully connected layer weight vector of the corresponding variable, is the bias vector.

[0046] (5).

[0047] Reset gate output Controls the hidden state of the previous time step How to flow into the candidate hidden state of the current time step. The hidden state of the previous time step may contain all the historical information of the time series up to the previous time step. Therefore, the reset gate can be used to discard historical information that is irrelevant to prediction.

[0048] The expression of the tanh function is: (6).

[0049] The update gate output As shown in formula (7), the hidden state of the current time step can be controlled How should the candidate hidden state containing the information of the current time step be Updated.

[0050] (7).

[0051] The above is the calculation process of the forward gated recurrent unit. In order to make the output of each time node contain the complete past and future environmental state perception information of the current time node in the input sequence, the bidirectional gated recurrent unit cited in this application includes a reverse gated recurrent unit. First, Reverse the time sequence in , and then repeat formula (2) to formula (7) for each time node in the set to obtain .

[0052] (8).

[0053] in, The environmental status information from time T to the current time node can be extracted; The extracted information is the environmental state information from the initial time node (t=0) to the current time node. As shown in formula (8), the and By weighted summation, we can consider the past and future environmental state perception information of the current time node and improve the perception ability of the network model.

[0054] When executing step 204, the specific steps may be as follows: The action value estimate is input into the data normalization layer in the interaction likelihood estimation network module to obtain the normalized action value estimate.

[0055] The standardized action value estimate is input into the third fully connected layer in the interactive likelihood estimation network module to obtain the augmented action value estimate hidden feature vector.

[0056] Specifically, the action value estimate is input into the data normalization layer in the interactive likelihood estimation network module to obtain the normalized action value estimate, which specifically includes: The action value estimate is normalized according to formula (9).

[0057] (9).

[0058] in, is the average estimate of the action value.

[0059] The standardized action value estimate is input into the third fully connected layer in the interactive likelihood estimation network module to obtain the augmented action value estimate hidden feature vector.

[0060] The specific steps of executing step 205 are as follows: like Figure 3 As shown, the hidden feature vector of the environment state and the hidden feature vector of the action value estimate are concatenated and input into the human-computer interaction likelihood estimation module to obtain the interaction likelihood estimation value of the robot, and then the interaction likelihood estimation values ​​of multiple robots are compared, and the robot with the largest estimation value is selected to interact with the human.

[0061] The augmented action value estimation hidden feature vector is concatenated with the environment state hidden feature vector obtained by stacking convolutional layers, bidirectional gated recurrent units and temporal state fusion layers in the interaction likelihood estimation network module, and then input into the fourth fully connected layer to obtain the final interaction likelihood estimation value after calculation.

[0062] In some embodiments, the training of the action value estimation network module and the interaction likelihood estimation network module in the human-computer interaction likelihood estimation network will be carried out using the temporal difference error loss function and the maximum likelihood loss function respectively. As shown in formula (10), the temporal difference loss function is the maximum target action value estimate for the next time node state The reward for interacting with the environment at this time point minus the actual action value estimate .

[0063] (10).

[0064] The maximum likelihood estimation loss function is shown in formula (11). Likelihood estimation network module Trainable network parameters, The environmental state quantity perceived by the robot selected by the person in the training database at this time point, and The environmental state quantity perceived by other robots.

[0065] (11).

[0066] The present embodiment is divided into two stages to train the network module. The two-stage process adopted is introduced below. In the first stage, the action value estimation network module is trained. The algorithm training flow chart is as follows: Figure 4As shown. After starting training, the number of iterations is first initialized, the robot is randomly initialized in the training environment, and a random number between 0 and 1 is generated. When the number is greater than an initial threshold ε, the action value estimation network module is used to generate the action corresponding to the maximum value estimate among the value estimates of all actions. Otherwise, an action is randomly selected for execution. The robot executes the selected action and interacts with the training environment to generate the next state and calculate the environmental reward. The states before and after the interaction, the executed action, and the environmental reward are added to the playback data memory as a training sample. According to the preset conditions such as the robot task and safety, it is determined whether to end this round of training. If it is not ended, it is determined whether the number of training samples stored in the playback data memory exceeds the set threshold. If it is greater, a batch of training samples is extracted from it, and the action value estimate is obtained by forward propagation. The loss function is calculated according to the environmental reward and the action value estimation network module parameters are updated by back propagation. If the data is insufficient, a random number is regenerated and returned to 204 to select the next action. At the same time, if it is determined that this round of training is over, the current round number is increased by 1, and a new round of training is started by returning to 203. The training ends when the training rounds have reached the set number of rounds.

[0067] In some embodiments, the network module loss function is set to the time difference error, and the Adam optimizer is used for training, with the learning rate and momentum parameters being 0.003 and (0.9, 0.99) respectively; the model is trained in batch training mode, with the number of data samples used for each back propagation being 128, and a total of 2000 rounds of environmental interaction data collection and training. After the training is completed, the action value estimation network module parameters will be frozen and will not change in the next stage of training.

[0068] In this embodiment, the action value network module is first trained using the temporal difference error loss function. After training, the action value estimation network module parameters are frozen. Figure 3 Each inference is performed using the process in Figure 3 Process, obtain the forward propagation data, and then use the maximum likelihood loss function to train the interaction likelihood value estimation network module.

[0069] In the second stage, the interactive likelihood estimation network module is trained. The algorithm training flow chart is as follows: Figure 5As shown. After starting training, the number of iteration steps is first initialized, and then training is started, and the divided samples are input according to batches and the preset batch training sample size. Each batch of samples first calculates the action value estimate corresponding to each sample in the batch through the action value estimation network module, and then inputs the action value estimate and the batch training samples into the human-computer interaction likelihood estimation network module for forward propagation, calculates the human-computer interaction likelihood estimation network loss function, and performs back propagation and updates the network parameters of the human-computer interaction likelihood estimation network module according to the loss function. Determine whether to traverse the training set. If not, return to 304 to continue. If the data set has been traversed, add 1 to the current round number and return to 303 to start a new round of training. The training ends when the training rounds have reached the set number of rounds.

[0070] According to one embodiment of the present application, the network module loss function is set to a likelihood estimation function, and the Adam optimizer is used for training, with a learning rate and momentum of 0.001, 0.9, and 0.99, respectively; the model is trained in a batch training mode, with a data batch size of 64, and a total of 50 rounds of training are performed.

[0071] In another embodiment of the present application, the network module loss function is set to a likelihood estimation function, and the Adam optimizer is used for training, with a learning rate and momentum of 0.001, 0.9, and 0.99, respectively; the model is trained in a batch training mode, with a data batch size of 64, and a total of 50 rounds of training are performed.

[0072] In one embodiment of the present application, a battlefield exploration or search and rescue mission scenario is set up in a multi-machine unknown denied environment. Imagine that in a corridor with multiple complex dynamic environmental features, in order to improve the battlefield search and rescue mission, a commander monitors multiple light quad-rotor drones to detect the environment in the corridor and collect intelligence. The drone needs to autonomously avoid obstacles and navigate to the vicinity of the set target point coordinates based on the perception and positioning equipment (depth camera) it carries. According to the above task introduction, the design is as follows Figure 6 and Figure 7 As shown in the verification scenario 1 and verification scenario 2, when there is no GPS navigation indoors, the shape and position of obstacles in the environment are randomly generated during each training and verification task. At the same time, there are multiple moving and static people in the scene as mobile obstacles, and the trajectory and appearance time of the mobile obstacles are unknown to the drone. In the above environment, two, three and five drones are placed for human-computer interaction experiments. For the navigation algorithm, safe flight is the primary goal. This embodiment assumes that the key situation of active interaction is the collision between the drone and surrounding objects. Therefore, this embodiment counts the matching accuracy between the collision between the drone and the surrounding objects and the interactive selection of the active interaction algorithm. In order to conduct comparative verification, this embodiment implements multiple human-computer active interaction algorithms, which are: 1) Distribution variance based on DuelingDQN: Based on the standard twin deep Q network, the network model is trained through the temporal difference algorithm (TD-error). The uncertainty of the policy model is represented by calculating the variance of the output probability distribution of the trained network model. A large output distribution variance means that the model is uncertain about what action to take for the input state, that is, the corresponding state-action pair is missing in the training data.

[0073] 2) Random variance based on NoisyDQN: The output of the noisy deep Q network will be superimposed with a trainable noise distribution, which is usually assumed to be a normal distribution, and its variance can approximate the distribution variance of the Bayesian network, which can represent the uncertainty of the distribution.

[0074] 3) Uncertainty estimation based on REM: The random ensemble hybrid method is an improved method for integrating deep Q-learning. This method uses multiple parameterized Q networks to estimate Q values, and regards the convex combination of multiple Q-value estimates as the Q-value estimate itself, enforcing the best Bellman consistency method. The outputs of multiple Q networks can be considered as multiple distributions, and the method of integrating networks in deep learning can be used to analyze uncertainty from multiple perspectives. For example, the information entropy (REM-Entropy) of the Q-value network distribution of the final convex combination, the mean of the variance of different Q-value estimates for the same action prediction probability (REM-Var), and the JS divergence (REM-JS) between different Q-value estimate distributions.

[0075] 4) Imitation learning based on maximum likelihood loss function (MLE-Net): This method consists of a commander observing three drones performing tasks. When a certain drone is found to require human intervention and control, the human intervenes and marks the intervention state with a label that requires interaction. 6,000 sample label data are collected, including 3,000 label data that require interaction and 3,000 label data that do not require interaction. Then, the maximum likelihood loss function is used to train the interaction model.

[0076] The experimental comparison results are shown in Table 1, where the method proposed in this application is represented by DQN-MLE: Table 1 Statistics of active interaction accuracy of interactive models (%)

[0077] As can be seen from Table 1, the active interaction algorithm proposed in this application still achieved an interaction accuracy of 91.87% in the environment of the five drone experiments, and 92.95% in the scenario of three drones, which is significantly higher than the six compared methods. Especially when the number of monitored drones is increased, the DQN-MLE method proposed in this application shows stronger prediction robustness and less decrease in interaction accuracy.

[0078] The present application provides a method for predicting the timing of active interaction between humans and robots based on a deep recurrent network of state value estimation. It includes a deep action value estimation network module and an interaction likelihood value estimation network module, and both modules include a bidirectional gated recurrent unit. The deep action value estimation network module generates an action value estimation based on the action strategy of a single robot. The present application adopts a twin deep Q-value estimation network as the strategy network of the robot, that is, decomposing the action value estimation into two reasoning processes: value estimation and action advantage estimation. The network collects reward functions through interaction with the environment and trains its value estimation for each strategy action. When applied, the network selects the action with the largest value estimation as the action performed by the robot according to the state of the environment. The present application introduces an action value estimation network module, that is, the twin deep Q-value estimation network after the above training, which can provide a value estimation that conforms to the robot's action strategy for the human-computer interaction likelihood value estimation network module and provide a reference for the interaction likelihood value estimation, that is, when the action value estimation is generally low, it means that the robot is in a more complex or unfamiliar environment; and when the estimated action value is high, it means that the model has a high confidence in the action decision of the current state, and the robot's autonomous decision reliability is stronger.

[0079] Secondly, the human-machine interaction likelihood estimation network module proposed in this application integrates the value estimation calculated by the action value estimation module with the hidden feature vector of the robot environment state, and infers the robot interaction likelihood estimation value. This module collects the state-selection data pairs when people monitor the multi-robot system to perform tasks, and trains the module to imitate people's selection tendency among multiple robots through the maximum likelihood loss function. This module can effectively utilize the robot's own action value estimation suggestions, as well as imitate people's tendency in multi-robot selection. While making reasonable judgments on the robot's strategy, only a small amount of human monitoring data is needed to train an accurate interaction likelihood estimation network.

[0080] Finally, the present application introduces a recurrent neural network module in both of the above-mentioned network modules, specifically a bidirectional gated recurrent unit. A recurrent neural network is a deep neural network with a cyclic structure and mechanism. When the network cyclically processes the transmitted information content, it can learn the timing information in the data through data propagation. The introduction of this module in the present application can improve the robot's action decision-making ability in a dynamic environment. However, the standard recurrent neural network has the problem of gradient vanishing when processing long sequence data. The long short-term memory network improves the network's ability to extract long-term timing information by introducing gating structures such as memory gates and forget gates. The bidirectional gated recurrent unit greatly reduces the number of parameters that need to be trained and ensures its advantages in solving the problems of long-term memory and back-propagation gradient vanishing by redesigning the gating units in the long short-term memory network.

[0081] When a human monitors the multi-robot system, each robot records the environmental status information in real time, and estimates the likelihood that it needs human monitoring through the human-machine interaction likelihood estimation network designed in this application. The interactive system summarizes the interaction likelihood estimation values ​​of all robots, selects the robot with the largest value to communicate with the human, and the human controls the robot.

[0082] The present application provides a deep recurrent network human-robot active interaction timing prediction system based on state value estimation, the system includes a deep action value estimation network module and an interaction likelihood value estimation network module, and the two modules also include a bidirectional gated recurrent unit. When multiple robots perform autonomous navigation movements, each robot inputs the denoised and resized environmental depth image into the deep action value estimation network module for action value estimation, and the interaction likelihood value estimation network module to obtain the hidden feature vector of the environmental state. The action value estimation is sequentially calculated through the normalization layer and the fully connected layer to obtain the action value estimation hidden feature vector. The above two hidden feature vectors are spliced ​​and input into the human-computer interaction likelihood estimation module to obtain the interaction likelihood estimation value of the robot, and then the interaction likelihood estimation values ​​of multiple drones are compared, and the robot with the largest estimation value is selected to interact with humans.

[0083] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a human-computer interaction likelihood value evaluation method based on a deep RNN is implemented.

[0084] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0085] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0086] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0087] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0088] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0089] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0090] In summary, this application has the following technical effects: (1) The action value estimation of the human-machine interaction likelihood estimation network proposed in this application is based on the action value estimation of the action value estimation network module. On the one hand, it can effectively refer to the action value estimation that conforms to the action decision model of the robot itself, and assist the likelihood estimation network in evaluating the urgency of the state at different times. On the other hand, it can avoid the likelihood estimation model from having to learn to evaluate the different states of the robot from scratch, greatly reducing the demand for human experience data and the cost of collecting data sets.

[0091] (2) Based on the collected human robot selection experience data and likelihood loss function, this application adopts a data-driven model to train the human-robot interaction likelihood estimation network end-to-end. That is, for the active interaction problem of multi-robot systems, the method proposed in this application does not need to rely on accurate environmental dynamic models and robot motion models. Compared with traditional interaction prediction methods, it shows better prediction accuracy and the ability to be applied to a wider range of different types of robots and task types. Especially for long-term tasks in dynamic and complex environments, it has important practical engineering significance.

[0092] (3) This application introduces a lightweight deep recurrent neural network into two modules of the human-computer interaction likelihood estimation network, improving the network model's ability to extract temporal information features and further improving its ability to cope with dynamic environments and long-term tasks.

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for evaluating the likelihood value of human-computer interaction based on deep RNN, characterized in that: The method for evaluating the likelihood value of human-computer interaction based on deep RNN includes: Obtaining the depth image of the environment when the robot is moving in autonomous navigation; Inputting the environmental depth image into an action value estimation network module to obtain an action value estimation of the robot during autonomous navigation movement; Inputting the environmental depth image into an interactive likelihood estimation network module to obtain a hidden feature vector of the environmental state of the robot during autonomous navigation movement; Inputting the action value estimation into the interactive likelihood estimation network module to obtain an action value estimation hidden feature vector; Inputting the environment state hidden feature vector and the action value estimation hidden feature vector into a human-machine interaction likelihood estimation module to obtain an interaction likelihood estimation value of the robot; According to the interaction likelihood estimation values ​​of each robot, the robot with the largest estimation value is selected.

2. According to claim 1, a method for evaluating the likelihood of human-computer interaction based on deep RNN is characterized in that: The environment depth image is input into the action value estimation network module to obtain the action value estimation of the robot during autonomous navigation movement, specifically including: Preprocessing the time series environment perception data to obtain a state matrix; the time series environment perception data is a data representation of an environment depth image; Input the state matrix into the action value estimation network module to calculate the value estimation and action advantage estimation; The value estimate is added to the action advantage estimate, and the mean of the value estimates corresponding to each discrete action in the action performed by the robot selected by the action value estimation network is subtracted to obtain the action value estimate.

3. According to claim 2, a method for evaluating the likelihood of human-computer interaction based on deep RNN is characterized in that: The value estimate is added to the action advantage estimate, and the mean of the value estimates corresponding to each discrete action in the action performed by the robot selected by the action value estimation network is subtracted to obtain the calculation formula for the action value estimate: ； in, is the input state, The actions performed by the robot selected for the action value estimation network; A set of trainable parameters for stacked convolutional layers and bidirectional gated recurrent units; The trainable parameters of the fully connected layer estimated for the corresponding values; are the trainable parameters of the fully connected layer corresponding to the action advantage estimation, is the number of discrete output actions of the strategy model; An estimate of the action value; For value estimation and is the action advantage estimate, and a is each specific action in A.

4. According to claim 3, a method for evaluating the likelihood of human-computer interaction based on deep RNN is characterized in that: The environment depth image is input into the interactive likelihood estimation network module to obtain the hidden feature vector of the environment state of the robot during autonomous navigation movement, specifically including: According to the stacked convolutional layers, bidirectional gated recurrent units and temporal state fusion layers in the interactive likelihood estimation network module, data preprocessing is performed on the environmental depth image to obtain a hidden feature vector of the environmental state of the robot during autonomous navigation movement.

5. According to claim 4, a method for evaluating the likelihood of human-computer interaction based on deep RNN is characterized in that: Inputting the action value estimation into the interactive likelihood estimation network module to obtain the action value estimation hidden feature vector specifically includes: Inputting the action value estimate into the data normalization layer in the interaction likelihood estimation network module to obtain the normalized action value estimate; The standardized action value estimate is input into the third fully connected layer in the interactive likelihood estimation network module to obtain the augmented action value estimate hidden feature vector.

6. The method for evaluating the likelihood of human-computer interaction based on deep RNN according to claim 5, characterized in that: The action value estimate is input into the data normalization layer in the interactive likelihood estimation network module to obtain the normalized action value estimate, which specifically includes: According to the formula Normalize action value estimates; in, is the average estimate of the action value.

7. The method for evaluating the likelihood of human-computer interaction based on deep RNN according to claim 6, characterized in that: The training method of the action value estimation network module specifically includes: The action value estimation network module is trained using a temporal difference error loss function, specifically including: According to the formula , training the action value estimation network module; Among them, the temporal difference loss function is the maximum target action value estimate for the next time node state The reward for interacting with the environment at the current time node minus the actual action value estimate .

8. The method for evaluating the likelihood of human-computer interaction based on deep RNN according to claim 1, characterized in that: The training method of the interactive likelihood estimation network module specifically includes: The maximum likelihood loss function is used to train the interactive likelihood estimation network module, including: According to the formula Training the interaction likelihood estimation network module; in, Likelihood estimation network module Trainable network parameters; The environmental state quantity perceived by the robot selected by the person in the training database at the current time node; The environmental state quantity perceived by other robots.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for evaluating the likelihood of human-computer interaction based on a deep RNN as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements a method for evaluating the likelihood of human-computer interaction based on a deep RNN as described in any one of claims 1 to 8.

Images