A water pouring service robot control method based on a multi-modal reinforcement learning algorithm

Abstract

The application discloses a water pouring service robot control method based on a multi-modal reinforcement learning algorithm, which uses words, sounds and monocular RGB images as inputs to provide rich multi-modal information for the robot, adopts a neural network to perform representation, alignment, correction and fusion on the multi-modal information, so that the robot can quickly and self-learnedly complete a service water pouring task by using the reinforcement learning algorithm, and the following problems of training the robot to complete the water pouring task by using the reinforcement learning in a real life scene are solved: time-consuming and labor-consuming, unstable training process, unrobust training model and low task completion degree. The application can make the robot self-learn to complete the water pouring task more quickly, and execute the water pouring action more accurately.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a control method for a water-pouring service robot based on a multimodal reinforcement learning algorithm. Background Technology

[0002] Whether in daily life or in industrial production, the ability of robots to independently complete the task of pouring water is of great significance. Compared to simple tasks such as grabbing and placing objects, pouring water requires considering a long trajectory. Each action in the trajectory has a significant impact on subsequent actions. Furthermore, when a robot performs the task of pouring water, the actions are more complex, the uncertainties are greater, the precision requirements are higher, and the environmental influences are more severe. This requires the algorithm to be able to handle various interactive scenarios and make fast and correct actions under various conditions, making the implementation of a water-pouring algorithm more difficult.

[0003] Various methods have been proposed to solve the robot's water-pouring task. These technologies can be divided into two main categories: traditional control methods and learning-based methods. Traditional control methods are mainly based on programming, that is, setting a goal in a specific scenario and implementing a certain function of the robot through planning and programming. Such methods have weak generalization ability, environmental noise can have a serious impact, or when the scenario changes, it is necessary to re-plan or reprogram for the new scenario, which greatly increases the manual labor and is very inefficient. Learning-based methods are represented by a series of algorithms such as reinforcement learning, especially the deep reinforcement learning method, which has developed rapidly in recent years and has been proven effective in various fields. It has great potential in the field of control. Compared with traditional control methods, deep reinforcement learning takes various noisy raw data such as RGB images as input and outputs the robot's actions. The robot generates a series of states and action trajectory sequences by interacting with the environment and receives rewards from the environment. Through trajectory sequences and reward functions, the robot can obtain an action policy through deep reinforcement learning. With this action policy, the robot can complete the given task in various environments. Even if the environment changes, the action policy can quickly adapt to the new scenario through "self-learning fine-tuning", so that the robot has the characteristics of "intelligence".

[0004] Although reinforcement learning methods have the above advantages, their unstable learning results and the need for a large amount of interactive data to learn seriously hinder their application to the water-pouring task. Summary of the Invention

[0005] The purpose of this invention is to propose a control method for a water-pouring service robot based on a multimodal reinforcement learning algorithm, so as to solve the problems of time-consuming and laborious training, unstable training process, unromantic training model and low task completion rate when using reinforcement learning to train robots to complete water-pouring tasks in real life scenarios.

[0006] To achieve the above objectives, this invention proposes a control method for a water-pouring service robot based on a multimodal reinforcement learning algorithm, comprising the following steps:

[0007] S1: Constructing a visual perception module: Based on a monocular camera, the spatial position information of the end effector about objects in the scene is obtained, and the position information feature vector in the RGB image is extracted by a convolutional neural network;

[0008] S2: Encode the textual description of the target container's attributes into a bag-of-words model;

[0009] S3: Model the water-pouring task using a Markov decision process and construct a strategy control module to guide the robot in learning the water-pouring task; input the obtained position information feature vector and the bag-of-words model encoding of the target container into the strategy control module, output the robot's control actions, and guide the robot to complete the water-pouring task in real time;

[0010] S4: Construct a sound correction module; use sound signals to correct the spatial location feature vector obtained by the visual perception module, obtain a more accurate description of visual state information, and guide the strategy control model to quickly adapt to changes in environmental factors such as the position of the water cup and camera disturbances.

[0011] Furthermore, in step S1, the monocular camera acquires the RGB image I∈R 3×H×W The convolutional neural network extracts features from the RGB image, and the convolutional neural network extracts spatial information feature maps of different granularities at each stage. The feature maps from each stage are downsampled and concatenated, and finally a 1024-dimensional spatial feature vector f∈R is obtained through 3 fully connected layers and 2 activation layers. 1024 The spatial feature vector contains a description of the spatial positional relationship between the robot's end effector and various objects in the scene.

[0012] Furthermore, in step S2, the text description records the state, category, color, shape, and texture of the target container in an unstructured form. It is a human language description of the various attributes of the target object. Each attribute is positionally encoded in a one-hot manner. Keywords are extracted from the text description, and the values ​​corresponding to the extracted keywords are added to the positions of the position encoding in an unordered manner to form a "bag-of-words" description D∈Rk of the target container, where k is the encoding length, which is determined by the number of attributes to be described. The bag-of-words description is a structured description of the target container processed by a computer.

[0013] Furthermore, in step S3, the service robot's water-pouring task is modeled as a Markov decision process, and a strategy control module is constructed.

[0014] Furthermore, the method for establishing the policy control module is as follows:

[0015] The process of pouring water can be described as having a state S at the current moment. t Robot Action A t The next state S t+1 Current time step reward R t The set {A1, A2, ..., A} t}, denoted as U t {S t A t S t+1 R t The strategy control module is mathematically represented using a neural network.

[0016] S31: The agent interacts with the environment, receiving feedback through a reward function, enabling the agent to learn the water-pouring task through self-learning; the specific form of the reward function is as follows:

[0017]

[0018] S32: The state representation S of the environment is a combination of spatial feature vector F and bag-of-words description D;

[0019] S33: The robot's motion A represents the vector representation of the robot's 6 joint angular offsets;

[0020] S34: The strategy control module is divided into a modality fusion module and an action generation module. The fusion module takes the spatial feature vector F and the bag-of-words description D as inputs and fuses the two modal information at the feature level to obtain the fused control semantic vector U.

[0021] The action generation module takes a control semantic vector U as input and outputs the robot's action A.

[0022] Furthermore, in step S4, the ambient sound fed back by the robot during the water-pouring task is acquired using an auditory sensor, the ambient sound is filtered for noise, and the Mel spectrum M of the sound is extracted. The Mel spectrum M is then used to correct the feature maps fi obtained at each stage of the convolutional neural network of the visual perception module at different granularity information scales.

[0023] Furthermore, the sound correction module is divided into a sound feature extraction module and a sound-to-image feature correction module;

[0024] The sound feature extraction module takes into account the digital signal sequence obtained in the first three time steps, and extracts the Mel spectrum M for each sound segment using a sliding window with a size w = 1024 and a step size s = 64. i (i = 1, 2, 3), the Mel spectrum M∈R is obtained by superimposing the three Mel spectra along the time dimension. 3×h×w ;

[0025] The sound-to-image correction module consists of multiple correction modules: Moditiers, where the input to each Moditier is the feature map obtained from the previous stage's convolutional neural network. And Mel spectrum M, Moditier uses Mel spectrum M to analyze feature maps Make the following corrections;

[0026]

[0027] Obtain the corrected feature map Where O(*) represents two convolution operations, mapping the Mel spectrum to a convolution map of the same size as the current granularity; * represents the element-wise multiplication operation at the corresponding position in Hadamard; b i It is the corrected offset for each Moditier.

[0028] Furthermore, the Modifier module corrects the spatial position information feature vector of scene objects obtained from the RGB image through cross-modal information, so that the perception module obtains accurate position feature information, guiding and promoting the strategy control module to guide the robot to complete the water pouring task.

[0029] Compared with existing technologies, the advantages of this invention are as follows: This method utilizes the complementarity between text, sound, and image modal information, eliminates redundancy between modal information, and learns a perception module and control strategy model within a reinforcement learning framework that can guide the robot to complete the water-pouring task, thereby controlling the service robot to efficiently complete the water-pouring task. Compared with existing technologies, this invention can enable the robot to learn to complete the water-pouring task more quickly and execute the water-pouring action more accurately. Attached Figure Description

[0030] Figure 1 This is a flowchart illustrating the robot's water-pouring task in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of a robot pouring water in a real-world scenario, as shown in an embodiment of the present invention.

[0032] Figure 3 This is a diagram illustrating the overall framework of the multimodal reinforcement learning algorithm in this embodiment of the invention.

[0033] Figure 4 This is a diagram of the convolutional neural network structure of the visual perception module in an embodiment of the present invention.

[0034] Figure 5 This is a neural network structure diagram of the strategy control module fusion module and the action generation module in an embodiment of the present invention.

[0035] Figure 6 This is a network structure diagram showing how the sound correction module corrects the visual perception module at each stage in this embodiment of the invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be further described below.

[0037] This invention proposes a control method for a water-pouring service robot based on multimodal reinforcement learning, the process of which is as follows: Figure 1 As shown, it includes:

[0038] A vision sensor (RealSense camera) acquires RGB images, and a convolutional neural network in the vision perception module extracts visual features of objects in the scene from the RGB images. The network structure of the convolutional neural network is as follows: Figure 4 As shown, the entire network can be divided into n stages. Preferably, in this embodiment, n=3. The input of the first stage of the network is the original RGB image, and each subsequent stage outputs a location feature map containing information of different granularities. Specifically, each stage will input... After 5x5 convolution, consistent padding, and ReLU activation, we obtain... Between the two stages, a dilated convolution with a dilation rate of 2 and pooling operations are used to downsample the feature map to 1 / 4 of its original size. The final positional feature map... Perform a flatten operation, and then pass the vector through three fully connected layers and two activation layers to obtain a 1024-dimensional spatial feature vector f∈R. 1024 This spatial feature vector contains a description of the spatial positional relationship between the robot's end effector and various objects in the scene.

[0039] Provide a text description of the target container, such as "Pour water into a blue, open-topped, cylindrical glass with a handle placed on the table until it is 60% full." Encode this text description as a k-dimensional bag-of-words description vector D∈R. k Preferably, in this embodiment, k=8 is selected, so the above text description can be encoded as [0, 0, 255, 1, 1, 0, 0, 60] where the 1st to 3rd dimensions represent the container color, the 4th dimension represents the container position (on a table or in a person's hand), the 5th dimension represents whether there is a handle, the 6th dimension represents the container shape, the 7th dimension represents whether the container has a lid, and the 8th dimension represents the target water volume.

[0040] The spatial feature vector f∈R obtained above 1024 The bag-of-words description of the target container is input into the modal fusion network of the policy control module. The modal fusion network structure is as follows: Figure 5 As shown, specifically, the spatial feature vector and the bag-of-words vector are concatenated into a 1032-dimensional vector. This vector is then input into a fully connected layer to obtain a 516-dimensional intermediate vector. The original spatial feature vector and bag-of-words vector are residually concatenated with the intermediate vector, and then sequentially input into two fully connected layers and one activation layer to obtain a control semantic vector U. The control semantic vector U is then input into the action generation module to output the robot's actions. Specifically, the control semantic vector is sequentially input into 128-dimensional and 6-dimensional fully connected layers, where the first layer uses the ReLU activation function and the second layer uses the tanh activation function, resulting in the final 6-dimensional action vector (Δθ1, Δθ2, Δθ3, Δθ4, Δθ5, Δθ6). This action vector is then input into the robot's built-in drive to guide the robot's movement. Each time the robot performs an action, it receives a reward signal from the environment, such as pouring water into a cup or spilling water on a table. Specifically, the reward signal is expressed as follows:

[0041]

[0042] To encourage the agent to attempt pouring water, it receives a reward of -1 if it fails to do so. Through these reward signals, the agent can begin to learn the task of pouring water.

[0043] Finally, as Figure 6 As shown, a sound correction module is added to the visual perception module. A sound sensor (such as a microphone) is placed in a suitable position, and the raw digital sound signal is acquired using a program. A sliding window of fixed size w and sliding step size s is used to extract the Mel spectrum of the raw sound signal. This process is repeated for c steps to obtain the Mel spectrum M. i ∈R h×cPreferably, w=1024, s=64, c=1000, h=256 are selected, and the Mel spectrograms obtained from the previous L frames, including the current frame, are superimposed in the time dimension to obtain the location feature map for correction. Mel spectrum M∈R L×h×c Preferably, L = 3. The location feature map... Mel spectrum M∈R L×h×c Inputting into the modifier yields the corrected location feature map. Specifically, convolution operations are used to downsample the Mel spectrum, so that the downsampled Mel spectrum map matches the location feature map. With the same size, the downsampled Mel spectrogram and the location feature map are multiplied at corresponding positions, and the correction offset of the current modifier is added to obtain... The above operations can be represented by the following operations:

[0044]

[0045] Where O(*) represents two convolution operations, the purpose of which is to map the Mel spectrum to a convolution map of the same size as the current granularity; * represents the element-wise multiplication operation at the corresponding position in Hadamard; b i It is the correction offset for each modifier.

[0046] The corrected final location feature map Following the process described above, we obtain the spatial feature vector f∈R. 1024 The water-pouring action can be accurately obtained by fusing the bag-of-words vectors with the vectors and inputting them into the policy control module. Figure 2 As shown, this guides the robot to precisely perform the water-pouring operation.

[0047] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A control method for a water-pouring service robot based on a multimodal reinforcement learning algorithm, characterized in that, Includes the following steps: S1: Constructing a visual perception module: Based on a monocular camera, the spatial position information of the end effector about objects in the scene is obtained, and the position information feature vector in the RGB image is extracted by a convolutional neural network; S2: Encode the textual description of the target container's attributes into a bag-of-words model; S3: Model the water-pouring task using a Markov decision process and construct a strategy control module to guide the robot in learning the water-pouring task; input the obtained position information feature vector and the bag-of-words model encoding of the target container into the strategy control module, output the robot's control actions, and guide the robot to complete the water-pouring task in real time; S4: Construct the sound correction module; The spatial position feature vector obtained by the visual perception module is corrected by using sound signals to obtain a more accurate description of visual state information, which guides the strategy control module to quickly adapt to changes in environmental factors such as the position of the water cup and camera disturbances. In step S2, the text description records the state, category, color, shape, and texture of the target container in an unstructured form. It is a human language description of the various attributes of the target object. Each attribute is positionally encoded in a one-dimensional manner. Keywords are extracted from the text description, and the values ​​corresponding to the extracted keywords are added to the corresponding positions of the positional encoding in an unordered manner, thus forming a "bag-of-words" description of the target container. It is the encoding length, determined by the number of attributes that need to be described; bag-of-words description is a structured description of a target container that is processed by a computer. In step S3, the service robot's water-pouring task is modeled as a Markov decision process, and a strategy control module is constructed. The method for establishing the policy control module is as follows: The process of pouring water can be described as a state at the current moment. Robot Actions Next moment state Current time step reward set , recorded as The strategy control module is mathematically represented using a neural network. S31: The agent interacts with the environment, receiving feedback through a reward function, enabling the agent to learn the water-pouring task through self-learning; the specific form of the reward function is as follows: ； S32: The state representation S of the environment is a combination of spatial feature vector F and bag-of-words description D; S33: The robot's motion A represents the vector representation of the robot's 6 joint angular offsets; S34: The strategy control module is divided into a modality fusion module and an action generation module. The modality fusion module takes the spatial feature vector F and the bag-of-words description D as inputs and fuses the two modal information at the feature level to obtain the fused control semantic vector U. The action generation module takes a control semantic vector U as input and outputs the robot's action A. In step S4, an auditory sensor is used to acquire the ambient sound feedback received by the robot during the water-pouring task. The ambient sound is then filtered for noise, and the Mel spectrum M of the sound is extracted. The Mel spectrum M is then used to analyze the feature maps obtained from each stage of the convolutional neural network of the visual perception module. Corrections are made at different granularity information scales; The sound correction module is divided into a sound feature extraction module and a sound-to-image feature correction module; The sound feature extraction module takes into account the digital signal sequence obtained in the first three time steps, and extracts the Mel spectrum of each sound segment using a sliding window with a size of w=1024 and a step size of s=64. The Mel spectrum is obtained by superimposing the three Mel spectra along the time dimension. ; The sound-to-image correction module consists of multiple correction modules: Modifiers, where the input to each Modifier is the feature map obtained from the previous stage of the convolutional neural network. And Mel spectrum M, the modifier uses Mel spectrum M to modify the feature map. Make the following corrections; ； Obtain the corrected feature map ,in This represents two convolution operations, mapping the Mel spectrum to a convolutional map of the same size as the current granularity; * represents the element-wise multiplication operation at the corresponding position in Hadamard. It is the correction offset for each modifier; The Modifier module corrects the spatial position information feature vector of scene objects obtained from the RGB image by using cross-modal information, so that the perception module obtains accurate position feature information, which guides and facilitates the strategy control module to guide the robot to complete the water pouring task.

2. The water-pouring service robot control method based on multimodal reinforcement learning algorithm according to claim 1, characterized in that, In step S1, the monocular camera acquires RGB images. The convolutional neural network extracts features from the RGB image, and the convolutional neural network extracts spatial information feature maps of different granularities at each stage. The feature maps from each stage are downsampled and concatenated, and finally a 1024-dimensional spatial feature vector is obtained through 3 fully connected layers and 2 activation layers. The spatial feature vector contains a description of the spatial positional relationship between the robot's end effector and various objects in the scene.

Images