A cognitive behavior modeling method for operators of heavy-load robotic arms

Abstract

The present invention discloses a method for modeling operator cognitive behavior of heavy-duty robotic arms in the field of heavy-duty robotic arm control technology. The method comprises the following steps: using operating environment image information, heavy-duty robotic arm state parameters, and human-machine coupling interaction torque as feature data input, and outputting target action instructions of the manipulation direction, displacement, and speed of the joystick; extracting features of input information of different dimensions based on a deep convolutional neural network; and establishing a cognitive mechanism of "perception processing-understanding memory-judgment and decision-making" for operators manipulating heavy-duty robotic arms by fusing visual perception and long-short-term memory neural networks; constructing a data-driven operator cognitive behavior model, characterizing the manipulation styles of different operators, and laying the foundation for designing a heavy-duty robotic arm controller that can adapt to different operator styles.

Description

Technical Field

[0001] The present invention relates to the technical field of heavy-load robotic arm control, and in particular to a method for modeling operator cognitive behavior of a heavy-load robotic arm. Background Art

[0002] Heavy-duty manipulators, with their strong load capacity, high power density, and high operational intensity, are widely used in fields such as construction and emergency rescue, becoming indispensable equipment. When performing tasks such as heavy object handling and wall demolition, operators must continuously interact with the joystick for force feedback to maintain real-time perception of the manipulator's state. Frequent joystick force feedback can easily cause operator fatigue, leading to variations in operator reaction speed. Furthermore, during operations, the moment the end effector and the load come into contact with the heavy-duty manipulator's end is subject to intense impact, causing varying degrees of tension and panic in the operator. This, in turn, results in varying torques (magnitude and direction) applied to the joystick and speeds at which the joystick is manipulated, exacerbating differences in operator control styles. This variability complicates the design of controllers that adapt to diverse control styles, making it difficult to optimize the control based on the operator's individual characteristics and manipulation preferences. This significantly limits the manipulator's performance. Therefore, it is imperative to construct operator models to characterize the control styles of different operators, laying the foundation for designing controllers that can adapt to these styles.

[0003] Cognitive behavioral models are the most crucial component of operator models, as their cognitive processes and outcomes directly influence the operator's behavior and decision-making. By modeling operator cognitive behavior, we can simulate the operator's perception and understanding of different operating environments, as well as their judgment and decision-making regarding operational task requirements, thereby analyzing the operator's cognitive behavioral characteristics. Currently, domestic and international scholars primarily use methods such as queuing networks and adaptive control thinking-rationality to construct operator cognitive behavioral models to characterize the operator's understanding of the environment and decision-making regarding tasks. However, in unstructured scenarios such as emergency rescue and construction, emergencies such as building collapse and landslides can easily cause operators to experience cognitive biases, resulting in highly nonlinear cognitive behavior. However, the operator cognitive behavioral models constructed using existing methods are mostly linear models, which struggle to accurately describe the nonlinear cognitive behavioral characteristics of operators in unstructured environments. Therefore, there is an urgent need to construct a data-driven operator cognitive behavioral model that accurately characterizes the nonlinear cognitive behavioral characteristics of operators operating heavy-duty robotic arms. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid blurring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] Therefore, the purpose of the present invention is to provide an operator cognitive behavior modeling method for heavy-duty manipulators to characterize the manipulation styles of different operators and lay the foundation for designing a heavy-duty manipulator controller that can adapt to different operator styles.

[0006] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0007] A cognitive behavior modeling method for operators of heavy-duty manipulators is proposed. The steps are as follows:

[0008] S1. Acquire input data of different dimensions, wherein the input data of different dimensions include image information of the working environment, state parameters of the heavy-load manipulator, and human-machine coupling interaction torque;

[0009] S2. Constructing an operator cognitive behavior model based on a long short-term memory neural network and an attention mechanism. The operator cognitive behavior model includes a perception processing layer, a comprehension memory layer, and a judgment and decision-making layer.

[0010] S3. The perception processing layer extracts and fuses the spatiotemporal features of input data of different dimensions, and then passes the processed information to the understanding and memory layer. The understanding and memory layer retrieves the scene-strategy knowledge base in the long-term memory based on the processed work scene and operation task information, and combines the associative memory method to call the manipulation experience corresponding to similar scenes. Based on the cognitive results, the decision-making layer focuses on the strategies related to the current work task through the attention mechanism module, and dynamically decides the manipulation direction, displacement, and speed target instructions of the joystick at future moments under the operator's cognition, thereby realizing decision-making control of the heavy-loaded robotic arm.

[0011] As a preferred solution of the cognitive behavior modeling method for operators of heavy-duty manipulators described in the present invention, the perception processing layer uses upsampling operations to compensate for data acquisition losses when processing image information of the working environment. The processing effect is optimized by designing a visual perception loss function. The ReLU activation function is used to perform nonlinear transformation on key feature data, retaining the original feature information, and a convolutional network is used to extract local feature information. The visual perception loss function is:

[0012] ;

[0013] in, is the original input data, is the original prediction data, is the upsampling operation, , , is the input data after upsampling, is the predicted data after upsampling, and their lengths are .

[0014] As a preferred embodiment of the method for modeling operator cognitive behavior for a heavy-duty manipulator described in the present invention, the perception processing layer, when processing the state parameters of the heavy-duty manipulator, first extracts features related to the state of the heavy-duty manipulator (joint posture, attachment-load contact force, etc.) through a self-attention mechanism module. Then, an average pooling operation is performed on the extracted environmental image features and the human-machine coupling interaction torque features. Based on the spatiotemporal distribution of information of different dimensions, the dimensionality of the multi-dimensional feature data is unified.

[0015] Then, a Sigmoid activation function is applied to ensure that the feature data has a consistent numerical range when output. After feature processing, data of different dimensions are converted into three-dimensional features. Finally, feature fusion is performed to generate a feature map for output and pass it to the understanding and memory layer.

[0016] As a preferred embodiment of the method for modeling operator cognitive behavior of a heavy-duty manipulator according to the present invention, the understanding memory layer is composed of a multi-layer LSTM structure. In the multi-layer LSTM structure, the LSTM units of each layer jointly complete the processing of various types of information in the working environment through intra-layer and inter-layer interactions.

[0017] Inside the LSTM unit, the current unstructured scene feature information and the working memory of the previous moment pass through the forget gate, discarding information irrelevant to the operation task. For important information, it will continue to be retained and multiplied with the long-term memory of the previous moment to form experience for coping with different types of operation tasks.

[0018] At the same time, the input gate is multiplied by the newly added long-term memory to determine the degree of retention of the newly added memory. Subsequently, the retained new memory is added to the long-term memory retained at the previous moment, and the long-term memory for storage is output and passed to the next LSTM unit. The long-term memory at the current moment is first passed through the tanh function to capture and display the nonlinear relationship between the operator's action instructions and the current working environment, and then multiplied by the output gate to obtain the working memory at the current moment.

[0019] As a preferred solution of the method for modeling operator cognitive behavior of a heavy-duty robotic arm described in the present invention, in which, within the first layer of LSTM, the environmental image, the state of the robotic arm, and the human-machine coupling are processed time step by time. Through the time step transmission mechanism and associative memory method within the layer, the processed various types of information are associated and retrieved in the long-term memory scene-strategy knowledge base. If the experiential knowledge matching the current task is retrieved, the operation task information at the current moment is combined with the historical experience, and the joystick direction, displacement and speed instructions after the operation environment and task are understood and memorized by the first layer of LSTM are output; if not retrieved, it will be passed to the next layer of LSTM, and then it will be decided whether to forget or remember.

[0020] As a preferred solution of the method for modeling operator cognitive behavior of a heavy-duty manipulator according to the present invention, in the second layer LSTM, the working memory of the first layer is further forgotten and stored, and the second layer LSTM not only receives the working memory output by the first layer, but also , and also combined with the working memory of the previous time step of this layer and long-term memory On this basis, the second layer LSTM extracts the long-term memory, working memory and higher-level feature information of the current input information of the first layer LSTM, and compares it with similar scenarios. If there is no operation experience related to the operation task in the long-term memory, the overall goal of the current operation task is comprehensively considered to predict the joystick direction, displacement and speed instructions required to complete the operation task. The joystick direction, displacement and speed instructions after memory are understood by the second layer LSTM and output to the third layer LSTM.

[0021] As a preferred solution of the operator cognitive behavior modeling method for a heavy-load manipulator according to the present invention, the steps of executing the judgment and decision-making layer are as follows:

[0022] First, let the output of the multi-layer LSTM structure be the working memory vector , the specific formula is as follows;

[0023] ;

[0024] Where, It is with The input corresponding to The output working memory vector of the LSTM unit, is the input, is the number of units in each recurrent layer of the LSTM network;

[0025] Second, for each working memory vector First, a linear transformation is performed, and then a nonlinear transformation is performed by the tanh activation function to capture the nonlinear characteristics between the operation task and the operator's action instructions in each memory vector, and obtain the importance of each working memory vector. , the specific formula is as follows;

[0026] ;

[0027] Where, and are the weight matrix and bias respectively;

[0028] Then, use the above formula to calculate each working memory vector Attention weight in the current operation task , the weight reflects the relative importance of different working memory vectors. The specific formula is as follows:

[0029] ;

[0030] in, is the importance of each working memory vector The exponential operation result of the exponential function can map any real number into the positive range and can amplify different The difference between them makes the more important working memory vectors get higher weights, is the sum of the exponential scores of all working memory vectors, used as the denominator for normalization, is the number of all working memory vectors.

[0031] Finally, the working memory vector in the decision layer is determined and attention weights Multiply to get the target instruction vector , the instruction vector represents the operator's instruction on the direction, displacement, and speed of the joystick in order to complete the operation task. The specific formula is as follows:

[0032] .

[0033] Compared with the existing technology, the present invention has the following beneficial effects: in order to solve the problem that existing methods are difficult to describe the nonlinear cognitive behavioral characteristics of operators, the present invention studies the operator's "perception processing-understanding memory-judgment and decision-making" cognitive mechanism, analyzes the operator's nonlinear cognitive behavioral characteristics, constructs a data-driven operator cognitive behavior model, characterizes the manipulation styles of different operators, and then designs a heavy-duty robotic arm controller that can adapt to different operator styles. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0035] Figure 1 This is a flow chart of a method for modeling operator cognitive behavior of a heavy-duty robotic arm according to the present invention;

[0036] Figure 2 This is a structural diagram of the operator cognitive behavior model provided by the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0038] The present invention provides a method for modeling operator cognitive behavior of heavy-duty manipulators. Figure 1 As shown in the figure, the model uses image information of the working environment, state parameters of the heavy-duty manipulator, and torque characteristics of human-machine interaction as input, and the joystick's control direction, displacement, and speed target action instructions as output. A deep convolutional neural network extracts features from the input information of different dimensions. By integrating visual perception with a long-short-term memory neural network, a cognitive mechanism of "perception processing - understanding memory - judgment and decision-making" for the operator's heavy-duty manipulator operation is established, thus constructing a data-driven nonlinear cognitive behavior model for the operator. Specifically, the perception processing layer extracts and integrates key feature data from the working environment. The understanding memory layer, based on the processed working scene and operation task information, uses associative memory to retrieve the scene-strategy knowledge base in long-term memory and invokes corresponding operation experience for similar scenarios. Based on the cognitive results, the judgment and decision-making layer uses an attention mechanism module to focus on strategies related to the current working task and dynamically determine the joystick's control direction, displacement, and speed target instructions for the future moment based on the operator's cognition, thus achieving decision-making control of the heavy-duty manipulator.

[0039] The operator cognitive behavior model framework is as follows Figure 2As shown in Figure 1, the model employs a hierarchical and progressive architecture. The perception processing layer simulates a real operator's multiple senses, such as vision, hearing, and touch, to perceive feature information from different dimensions of the work environment. Using a deep convolutional neural network, it extracts and fuses spatiotemporal features from data such as the work environment image, the heavy-duty manipulator's state, and the human-robot interaction torque. This processed feature information is then passed to the understanding and memory layer, which utilizes a multi-layer LSTM network. Information such as key parameters of the current operational task and dynamic environmental characteristics is temporarily stored in the working memory of the first LSTM layer. A keyword mapping algorithm then associates and retrieves experience from the long-term memory scenario-strategy knowledge base. If knowledge matching the current task exists, the corresponding action command is directly output. If no matching content is retrieved, it is treated as new knowledge and passed to the next LSTM layer, where it is then determined whether to be forgotten or remembered. During the downward transfer of working memory, information not essential to the task is forgotten. The LSTM units simultaneously capture both long-term and short-term dependencies within the memory sequence, transforming the heavy-duty manipulator's experience into formalized knowledge that is then passed to the judgment and decision layer. Finally, at the judgment and decision-making layer, the working memory vector and the attention weight matrix are weighted and fused through the attention mechanism, dynamically focusing on the manipulation actions that match the current task, and deciding the manipulation direction, displacement, and speed target instructions of the joystick at future moments under the operator's cognition.

[0040] (1) Perception processing layer

[0041] The perception processing layer simulates the sensory response of a heavy-duty robotic arm operator to knowledge, extracting features from the multidimensional feature information collected from the work environment (such as the work environment image, the heavy-duty robotic arm state, and the human-machine interaction torque). Based on the differences in input feature dimensions, different branches of processing are performed on the feature information, aiming to convert the multi-dimensional feature information into three-dimensional data, thereby achieving unified feature fusion.

[0042] For feature data such as work environment images, an upsampling operation is employed. The number of elements in feature vectors, such as those for work environment images, is increased to compensate for information loss during data acquisition and ensure that the feature data dimension requirements of the feature fusion step are met. To optimize feature processing, a visual perception loss function is designed to simulate the operator's visual perception mechanism and accurately measure the quality of feature processing, ensuring that the generated feature data better meets actual operational needs. Furthermore, a ReLU activation function is used to perform nonlinear transformations on key feature data after visual perception, preserving the original feature information to the greatest extent possible. Furthermore, a convolutional network (conv) is used to gradually extract local feature information after expansion processing.

[0043] Visual Perception Loss based on upsample and mean squared error (MSE) The definition is as follows:

[0044] (1)

[0045] in, is the original input data, is the original prediction data, is the upsampling operation, , , is the input data after upsampling, is the predicted data after upsampling, and their lengths are .

[0046] For feature information such as the state of a heavy-duty manipulator, a self-attention mechanism module is first used to extract features related to the heavy-duty manipulator's state, such as joint posture and "attachment-load" contact force. Then, an average pooling operation is performed on the environment image and other features. Based on the spatiotemporal distribution of information of different dimensions, the dimensionality of the multi-dimensional feature data is unified. The Sigmoid activation function is then applied to ensure that the key feature data has a consistent numerical range when output. Finally, the data of different dimensions in the operating environment are converted into three-dimensional features after feature processing. Feature fusion is then performed to generate a feature map for output and pass it to the understanding and memory layer.

[0047] (2) Understanding the Memory Layer

[0048] The comprehension memory layer consists of a multi-layer LSTM structure, which in turn is composed of multiple interconnected LSTM units. These units store information related to the operational task, the heavy-duty manipulator's manipulation strategy, and more. This multi-layer LSTM structure not only continuously accumulates and iterates knowledge and experience in long-term memory, but also addresses the issue of long-term memory being lost or forgotten during the knowledge transfer process. Furthermore, when a scenario is replayed or a specific operational task is required, this structure rapidly learns from past operational experience by associating old and new memories. This allows the system to automatically extract the heavy-duty manipulator's manipulation strategy corresponding to the current task and environmental state from the scenario-strategy knowledge base.

[0049] In a multi-layer LSTM structure, LSTM units in each layer jointly process various types of information from the operating environment through intra- and inter-layer interactions. Within the LSTM unit, current unstructured scene feature information and the previous working memory pass through a forget gate, discarding information irrelevant to the task. However, important information, such as the joystick motion characteristics corresponding to the gripping angle of a heavy-duty robot arm under a specific angle configuration, is retained and multiplied with the previous long-term memory to generate experience for handling different types of tasks. Simultaneously, the input gate multiplies the newly added long-term memory to determine the degree of retention of the added memory. This added long-term memory specifically refers to the action commands dynamically generated by the operator during the operation based on experience and existing knowledge to complete a specific task (such as manipulating a heavy-duty robot arm to grasp and remove an obstacle in its path). The retained new memory is then added to the previous long-term memory, outputting a long-term memory for storage and passing it to the next LSTM unit. The current long-term memory is first processed through a tanh function to capture and represent the nonlinear relationship between the operator's action commands and the current operating environment. It is then multiplied by the output gate to generate the current working memory.

[0050] In the first layer of LSTM, various types of information in the working environment (such as environmental images, the status of the robotic arm, and the human-machine coupling interaction torque) are processed time step by time step. Through the time step transmission mechanism and associative memory method within the layer, the various types of processed information are associated and retrieved in the long-term memory scene-strategy knowledge base. If the experiential knowledge that matches the current task is retrieved, the operation task information at the current moment is combined with historical experience, and the joystick direction, displacement, and speed instructions of the working environment and task are output after being understood and memorized by the first layer of LSTM; if not retrieved, it will be passed to the next layer of LSTM, and then it will be decided whether to forget or remember. In the second layer of LSTM, the working memory of the first layer is further forgotten and stored. The second layer of LSTM not only receives the working memory output by the first layer , and also combines the working memory of the previous time step of this layer and long-term memory On this basis, the second LSTM layer extracts higher-level feature information from the long-term memory and working memory of the first LSTM layer, as well as the current input information, and compares it with similar scenarios. If there is no operational experience related to the operation task in the long-term memory, the overall goal of the current operation task is comprehensively considered to predict the joystick direction, displacement, and speed instructions required to complete the operation task. The joystick direction, displacement, and speed instructions are combined with the second LSTM layer to understand the memorized joystick direction, displacement, and speed instructions, and are output to the third LSTM layer.

[0051] By increasing the network depth, each LSTM layer can further capture the relationship between operation tasks based on the previous layer, and combine the current operation task with previous experience knowledge to generate the relevant joystick direction, displacement, and speed instruction set.

[0052] (3) Judgment and decision-making level

[0053] In the process of using the attention mechanism to determine the target instructions that can efficiently and accurately complete the operation task, the specific process of the attention mechanism and the LSTM network working together is as follows.

[0054] First, let the output of the multi-layer LSTM structure be the working memory vector. As shown in formula (2), each working memory vector integrates the working memory content output by the understanding memory layer, including the joystick direction, displacement and angle related to the current operation task and past operation experience.

[0055] (2)

[0056] Where, It is with The input corresponding to The output working memory vector of the LSTM unit, is the input, is the number of units in each recurrent layer of the LSTM network.

[0057] Secondly, each working memory vector is first linearly transformed and then nonlinearly transformed by the tanh activation function to capture the nonlinear characteristics between the operation task and the operator's action instructions in each memory vector, and obtain the importance of each working memory vector. , the specific formula is shown in (3).

[0058] (3)

[0059] Where, and are the weight matrix and bias respectively.

[0060] Then, use formula (3) to calculate the attention weight of each working memory vector in the current operation task , the weight reflects the relative importance of different working memory vectors, and the specific formula is shown in (4).

[0061] (4)

[0062] in, is the importance of each working memory vector The exponential operation result of the exponential function can map any real number into the positive range and can amplify different The difference between them makes the more important working memory vectors get higher weights, is the sum of the exponential scores of all working memory vectors, used as the denominator for normalization, is the number of all working memory vectors;

[0063] Finally, the working memory vector in the decision layer is determined and attention weights Multiply to get the target instruction vector The instruction vector represents the direction, displacement, and speed instructions that the operator should implement on the joystick in order to complete the operation task. The specific formula is shown in (5):

[0064] (5)

[0065] Working Memory Vector Action commands that closely match the current task (such as the joystick's direction, displacement, and speed) are weighted more heavily and prominently reflected in the decision output; conversely, they are weighted less heavily. Simultaneously, as the task progresses, the working memory vector, updated in real time, continuously participates in new rounds of weight calculation and command generation, ensuring that the target commands output by the decision-making layer always match the dynamically changing operating environment and task requirements. This continuous optimization process dynamically adjusts the controller's parameters and strategies, thereby improving the maneuverability of the heavy-duty robotic arm.

[0066] Although the present invention has been described above with reference to embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of such combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for modeling operator cognitive behavior of a heavy-duty manipulator, characterized in that: Here are the steps: S1. Acquire input data of different dimensions, wherein the input data of different dimensions include image information of the working environment, state parameters of the heavy-load manipulator, and human-machine coupling interaction torque; S2. Constructing an operator cognitive behavior model based on a long short-term memory neural network and an attention mechanism. The operator cognitive behavior model includes a perception processing layer, a comprehension memory layer, and a judgment and decision-making layer. S3. The perception processing layer extracts and fuses the spatiotemporal features of input data of different dimensions, and then transmits the processed information to the understanding and memory layer. The understanding and memory layer retrieves the scene-strategy knowledge base in long-term memory based on the processed work scene and operation task information, and combines the associative memory method to call the manipulation experience corresponding to similar scenes. Based on the cognitive results, the judgment and decision-making layer uses the attention mechanism module to focus on the strategy related to the current work task, and dynamically determines the manipulation direction, displacement, and speed target instructions of the joystick at the future moment under the operator's cognition, thereby realizing decision-making control of the heavy-loaded robotic arm; The understanding memory layer is composed of a multi-layer LSTM structure. In the multi-layer LSTM structure, the LSTM units of each layer jointly complete the processing of various types of information in the working environment through intra-layer and inter-layer interactions; Within the LSTM unit, the current operating environment feature information and the previous moment's working memory pass through the forget gate, discarding information irrelevant to the operation task. However, important information is retained and multiplied with the previous moment's long-term memory to form experience for coping with different types of operation tasks. At the same time, the input gate is multiplied by the newly added long-term memory to determine the degree of retention of the newly added memory. Subsequently, the retained new memory is added to the long-term memory retained at the previous moment, and the long-term memory for storage is output and passed to the next LSTM unit. The long-term memory at the current moment is first passed through the tanh function to capture and display the nonlinear relationship between the operator's action instructions and the current working environment, and then multiplied by the output gate to obtain the working memory at the current moment.

2. The method for modeling operator cognitive behavior of a heavy-duty manipulator according to claim 1, characterized in that: When processing the working environment image information, the perception processing layer uses upsampling operations to compensate for data acquisition losses. The processing effect is optimized by designing a visual perception loss function. The ReLU activation function is used to perform nonlinear transformation on key feature data, retaining the original feature information, and a convolutional network is used to extract local feature information. The visual perception loss function is: Among them, x i is the original input data, y i is the original prediction data, P(·) is the upsampling operation, is the input data after upsampling, are the predicted data after upsampling, and their length is m.

3. The method for modeling operator cognitive behavior of a heavy-duty manipulator according to claim 1, characterized in that: When processing the state parameters of the heavy-duty manipulator, the perception processing layer first extracts features related to the joint posture, "attachment-load" contact force, and the heavy-duty manipulator state through a self-attention mechanism module. It then performs an average pooling operation based on the extracted environmental image features and the human-machine coupling interaction torque features. Based on the spatiotemporal distribution of information of different dimensions, the dimensionality of the multi-dimensional feature data is unified. Then, the Sigmoid activation function is used to ensure that the feature data has a consistent numerical range when it is output; data of different dimensions are converted into three-dimensional features after feature processing, and finally feature fusion is performed to generate a feature map for output and pass it to the understanding and memory layer.

4. The method for modeling operator cognitive behavior of a heavy-duty manipulator according to claim 1, characterized in that: In the first LSTM layer, the working environment image, the status of the heavy-loaded robotic arm, and the human-machine coupling interaction torque are processed time step by time. Through the time step transmission mechanism and associative memory method within the layer, the processed information is associated and retrieved in the long-term memory scenario-strategy knowledge base. If the experiential knowledge matching the current task is retrieved, the current operation task information is combined with historical experience, and the joystick direction, displacement, and speed instructions after the working environment and task are understood and memorized by the first LSTM layer are output; if not retrieved, it will be passed to the next LSTM layer to decide whether to forget or remember.

5. The method for modeling operator cognitive behavior of a heavy-duty manipulator according to claim 1, characterized in that: In the second layer of LSTM, the working memory of the first layer is further forgotten and stored. The second layer of LSTM not only receives the working memory output of the first layer, but also Also combined with the working memory of the previous time step of this layer and long-term memory On this basis, the second-layer LSTM extracts the long-term memory, working memory and higher-level feature information of the current input information of the first-layer LSTM, and compares it with similar scenarios. If there is no operational experience related to the operation task in the long-term memory, the overall goal of the current operation task is comprehensively considered to predict the joystick direction, displacement and speed instructions required to complete the operation task. The joystick direction, displacement and speed instructions after memory are understood by the second-layer LSTM and output together to the third-layer LSTM.

6. The method for modeling operator cognitive behavior of a heavy-duty manipulator according to claim 1, characterized in that: The steps executed by the judgment and decision-making layer are as follows: First, let the output of the multi-layer LSTM structure be the working memory vector h i , the specific formula is as follows; h i =LSTM(s i ),i∈[1,L] Where h i is the output working memory vector of the i-th LSTM unit corresponding to the i-th input, s i is the input, L is the number of units in each recurrent layer of the LSTM network; Second, for each working memory vector h i First, perform linear transformation, and then perform nonlinear transformation by tanh activation function to capture the nonlinear characteristics between the operation task and the operator's action instructions in each memory vector, and obtain the importance u of each working memory vector i , the specific formula is as follows; u i =tanh(W s h i +b s ) Where W s and b s are the weight matrix and bias respectively; Then, use the above formula to calculate each working memory vector h i The attention weight α in the current operation task i , the weight reflects the relative importance of different working memory vectors. The specific formula is as follows: in, is the importance u of each working memory vector i The exponential operation result of the exponential function can map any real number into the positive range and can amplify different u i The difference between them makes the more important working memory vectors get higher weights, The exponential scores of all working memory vectors are summed and used as the denominator for normalization, and j is the number of all working memory vectors. Finally, the working memory vector h in the decision layer is determined i With attention weight α i Multiply them to get the target command vector v, which represents the direction, displacement, and speed instructions that the operator should implement on the joystick in order to complete the operation task. The specific formula is as follows:

Images