Operator cognitive behavior modeling method for heavy-load mechanical arm

Abstract

The invention discloses an operator cognitive behavior modeling method for a heavy-load mechanical arm in the technical field of heavy-load mechanical arm control, and the method comprises the following steps: taking operation environment image information, heavy-load mechanical arm state parameters and man-machine coupling interaction torque as characteristic data input, and taking the control direction, displacement and speed of a control lever as target action instruction output; feature extraction is performed on input information of different dimensions based on a deep convolutional neural network, a cognitive mechanism of'perception processing-understanding memory-judgment decision 'of an operator for heavy-load mechanical arm manipulation is established by fusing visual perception and a long-short-term memory neural network, and an operator cognitive behavior model based on data driving is constructed. The control styles of different operators are represented, and a foundation is laid for designing a heavy-load mechanical arm controller capable of adapting to the styles of different operators.

Description

Technical Field

[0001] The present invention relates to the technical field of heavy-duty manipulator control, and particularly to a method for modeling the cognitive behavior of an operator for a heavy-duty manipulator. Background Art

[0002] With the advantages of strong load capacity, high power density, and high operation intensity, heavy-duty manipulators are widely used in fields such as construction and emergency rescue, and have become an indispensable important operation equipment. When performing operation tasks such as heavy object handling and wall demolition, the operator needs to continuously perform force feedback interaction with the joystick to perceive the state of the manipulator in real time. However, the frequent joystick force feedback is likely to cause fatigue of the operator, resulting in changes in the reaction speeds of different operators. Moreover, at the moment of contact between the end effector and the load during the operation process, the heavy-duty manipulator is extremely vulnerable to severe impacts, leading to different degrees of tension, panic and other emotions of the operator. As a result, the torques (magnitude, direction) applied by different operators on the joystick and the speeds when operating the joystick are also different, exacerbating the differences in the operating styles of the operators. This difference increases the difficulty of designing a controller that adapts to different operating styles, making it difficult for the controller to achieve the optimal operating effect according to the individual characteristics and operating tendency characteristics of the operator, and greatly restricting the operating performance of the heavy-duty manipulator. Therefore, it is urgent to construct an operator model to characterize the operating styles of different operators and lay a foundation for designing a controller that can adapt to different operator styles.

[0003] As the most important part of the operator model, the cognitive process and results of the cognitive behavior model directly affect the behavior and decision-making of the operator. By modeling the cognitive behavior of the operator, it is possible to simulate the operator's perception and understanding of different operating environments and the judgment and decision-making of the operation task requirements, and then analyze the cognitive behavior characteristics of the operator. At present, domestic and foreign scholars mainly use methods such as queuing networks and adaptive control thinking-rationality to construct an operator cognitive behavior model to characterize the operator's understanding of the environment and decision-making behavior for tasks. However, in unstructured scenarios such as emergency rescue and construction, sudden situations such as building collapses and landslides of mountain rocks are likely to cause cognitive biases of the operator, resulting in highly nonlinear characteristics of their cognitive behavior. Most of the existing operator cognitive behavior models constructed by existing methods are linear models, which are difficult to accurately describe the nonlinear cognitive behavior characteristics of the operator in an unstructured environment. Therefore, it is urgent to construct a data-driven operator cognitive behavior model to accurately characterize the nonlinear cognitive behavior characteristics of the operator for a heavy-duty manipulator. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Simplifications or omissions may be made in this section, the abstract of the specification, and the title of the invention to avoid obscuring the purpose of this section, the abstract of the specification, and the title of the invention, and such simplifications or omissions shall not be used to limit the scope of the present invention.

[0005] Therefore, an object of the present invention is to provide a method for modeling the cognitive behavior of operators for heavy-duty manipulators, which can characterize the manipulation styles of different operators and lay a foundation for designing a heavy-duty manipulator controller that can adapt to different operator styles.

[0006] To solve the above technical problems, according to one aspect of the present invention, the following technical solutions are provided: A method for modeling the cognitive behavior of operators for heavy-duty manipulators, the steps are as follows: S1. Obtain input data of different dimensions, where the input data of different dimensions includes operation environment image information, heavy-duty manipulator state parameters, and human-machine coupling interaction torque; S2. Construct an operator cognitive behavior model based on a long short-term memory neural network and an attention mechanism, and the operator cognitive behavior model includes a perception processing layer, an understanding and memory layer, and a judgment and decision-making layer; S3. The perception processing layer performs spatio-temporal feature extraction and fusion processing on the input data of different dimensions, and then transmits the processed information to the understanding and memory layer. The understanding and memory layer retrieves the scenario-strategy knowledge base in long-term memory according to the processed operation scenario and operation task information, and calls the manipulation experience corresponding to the similar scenario. Based on the cognitive result, the judgment and decision-making layer focuses on the strategies related to the current operation task through the attention mechanism module, and dynamically decides the manipulation direction, displacement, and speed target instructions of the joystick at the future moment under the operator's cognition, so as to realize the decision control of the heavy-duty manipulator.

[0007] As a preferred solution of the method for modeling the cognitive behavior of operators for heavy-duty manipulators according to the present invention, when the perception processing layer processes the operation environment image information, upsampling operation is used to compensate for data acquisition loss, the processing effect is optimized by designing a visual perception loss function, the ReLU activation function is used for non-linear transformation of key feature data to retain the original feature information, and a convolutional network is used to extract local feature information, where the visual perception loss function is: ; Where is the original input data, is the original prediction data, is the upsampling operation, , , is the input data after upsampling, is the predicted data after upsampling, and their lengths are both .

[0008] As a preferred solution of an operator cognitive behavior modeling method for a heavy-duty manipulator according to the present invention, when the perception processing layer processes the state parameters of the heavy-duty manipulator, first, through the self-attention mechanism module, features related to the state of the heavy-duty manipulator (joint pose, "attachment-load" contact force, etc.) are extracted, and then the extracted environmental image features and the human-machine coupling interaction torque features are combined for average pooling operation. According to the spatio-temporal distribution law of different-dimensional information, the dimensional unification of multi-dimensional feature data is realized; Then, the Sigmoid activation function is used to make the feature data have a consistent numerical range when output. After feature processing, data of different dimensions all become three-dimensional features. Finally, feature fusion is performed to generate a feature map for output and transmitted to the understanding and memory layer.

[0009] As a preferred solution of an operator cognitive behavior modeling method for a heavy-duty manipulator according to the present invention, the understanding and memory layer is composed of multiple LSTM structures. In the multiple LSTM structures, each LSTM unit jointly completes the processing of various types of information in the operation environment through intra-layer and inter-layer interactions; Inside the LSTM unit, the current unstructured on-site feature information and the working memory of the previous moment pass through the forget gate to discard information irrelevant to the operation task. For important information, it will continue to be retained and multiplied by the long-term memory of the previous moment to form experience for dealing with different types of operation tasks; At the same time, the input gate is multiplied by the newly added long-term memory to determine the retention degree of the newly added memory. Subsequently, the remaining newly added memory is added to the long-term memory retained in the previous moment to output the long-term memory for storage and transmitted to the next LSTM unit. The long-term memory at the current moment first passes through the tanh function to capture and show the non-linear relationship between the operator's action instructions and the current operation environment, and then is multiplied by the output gate to obtain the working memory at the current moment.

[0010] As a preferred solution of an operator cognitive behavior modeling method for a heavy-duty manipulator according to the present invention, within the first-layer LSTM, the environmental image, the state of the manipulator, and the human-machine coupling are processed step by step in time. Through the time-step transfer mechanism and the associative memory method within the layer, various processed information is associated and retrieved in the scenario-strategy knowledge base of long-term memory. If the empirical knowledge matching the current task is retrieved, the operation task information at the current moment is combined with historical experience, and the joystick direction, displacement, and speed commands after the operation environment and task are understood and memorized by the first-layer LSTM are output. If not retrieved, it will be passed to the next layer of LSTM to decide whether to forget or memorize.

[0011] As a preferred solution of an operator cognitive behavior modeling method for a heavy-duty manipulator according to the present invention, within the second-layer LSTM, the working memory of the first layer is further forgotten and stored. The second-layer LSTM not only receives the working memory output by the first layer but also combines the working memory of the previous time step of this layer and the long-term memory On this basis, the second-layer LSTM extracts the higher-level feature information from the long-term memory, working memory, and current input information of the first-layer LSTM, and compares it with similar scenarios. If there is no operation experience related to the operation task in the long-term memory, the overall goal of the current operation task is comprehensively considered, and the joystick direction, displacement, and speed commands required to complete the operation task are predicted. Combined with the joystick direction, displacement, and speed commands after the second-layer LSTM understands and memorizes them, they are jointly output to the third-layer LSTM.

[0012] As a preferred solution of an operator cognitive behavior modeling method for a heavy-duty manipulator according to the present invention, the specific steps of the judgment and decision-making layer are as follows: First, let the output of the multi-layer LSTM structure be the working memory vector , and the specific formula is as follows; ; In the formula, is the output working memory vector of the th LSTM unit corresponding to the th input, is the input, is the number of units in each recurrent layer of the LSTM network; Secondly, each working memory vector is first linearly transformed and then non-linearly transformed by the tanh activation function to capture the non-linear characteristics between the operation task and the operator's action commands in each memory vector, and the importance degree of each working memory vector , the specific formula is as follows; ; In the formula, and are the weight matrix and bias respectively; Then, use the above formula to calculate the attention weight of each working memory vector in the current operation task. This weight reflects the relative importance of different working memory vectors, and the specific formula is as follows: ; Among them, is the exponential operation result of the importance degree of each working memory vector . Using the exponential function can map any real number to the positive range and can amplify the differences between them, so that more important working memory vectors obtain higher weights. is the sum of the exponential scores of all working memory vectors, which is used as the denominator for normalization. is the number of all working memory vectors.

[0013] Finally, judge that the working memory vector in the decision-making layer is multiplied by the attention weight to obtain the target instruction vector . This instruction vector represents the manipulation direction, displacement, and speed action instructions that the operator should implement on the joystick to complete the operation task. The specific formula is as follows: .

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention aims at the problem that the prior art is difficult to describe the non-linear cognitive behavior characteristics of the operator, studies the operator's cognitive mechanism of "perception processing - understanding and memory - judgment and decision-making", analyzes the non-linear cognitive behavior characteristics of the operator, constructs a data-driven operator cognitive behavior model, represents the manipulation styles of different operators, and further designs a heavy-duty manipulator controller that can adapt to different operator styles. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the drawings and detailed embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. Among them: Figure 1 is the flowchart of a method for modeling the cognitive behavior of an operator for a heavy-duty manipulator according to the present invention; Figure 2 This is the structural diagram of the operator cognitive behavior model provided by the present invention. Detailed implementation manners

[0016] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific implementation manners of the present invention will be given in conjunction with the accompanying drawings.

[0017] The present invention provides a method for modeling the operator cognitive behavior for a heavy-duty manipulator. As Figure 1 shown, taking the operation environment image information, heavy-duty manipulator state parameters, and human-machine coupling interaction torque characteristic data as inputs, and the manipulation direction, displacement, and speed target action instructions of the joystick as outputs, feature extraction is performed on the input information of different dimensions based on a deep convolutional neural network, and by fusing visual perception and a long short-term memory neural network, an operator "perception processing - understanding and memory - judgment and decision-making" cognitive mechanism for heavy-duty manipulator operation is established, and a data-driven operator non-linear cognitive behavior model is constructed. Among them, the perception processing layer extracts and fuses the key feature data in the operation environment; the understanding and memory layer, according to the processed operation scenario and operation task information, and in combination with the associative memory method, retrieves the scenario - strategy knowledge base in the long-term memory, and invokes the manipulation experience corresponding to the similar scenario; based on the cognitive result, the judgment and decision-making layer, through the attention mechanism module, focuses on the strategies related to the current operation task, and dynamically decides the manipulation direction, displacement, and speed target instructions of the joystick at the future moment under the operator's cognition, so as to realize the decision control of the heavy-duty manipulator.

[0018] The framework of the operator cognitive behavior model is as Figure 2As shown in the figure, the model adopts a hierarchical progressive architecture. The perception processing layer simulates various senses of a real operator, such as vision, hearing, and touch, to perceive the feature information of different dimensions in the operating environment, and based on a deep convolutional neural network, it extracts and fuses the spatio-temporal features of data such as the operating environment image, the state of the heavy-duty manipulator, and the human-machine coupling interaction torque. The processed feature information is transmitted to the understanding and memory layer. This layer adopts a multi-layer LSTM network structure. The key parameters of the current operation task and information such as the dynamic features of the environment are temporarily stored in the working memory of the first layer of LSTM, and through the keyword mapping algorithm, empirical association and retrieval are performed in the scenario-strategy knowledge base of the long-term memory. If there is knowledge that matches the current task, the corresponding action instruction is directly output; if no matching content is retrieved, it will be regarded as new knowledge and passed to the next layer of LSTM to decide whether to forget or remember. During the downward transmission of the working memory, memory information that is not important for the task will be forgotten. The LSTM unit synchronously captures the long-term and short-term dependencies in the memory sequence, transforms the heavy-duty manipulator operation experience into formalized knowledge, and transmits it to the judgment and decision-making layer. Finally, in the judgment and decision-making layer, through the attention mechanism, the working memory vector and the attention weight matrix are weighted and fused to dynamically focus on the manipulation actions that match the current task, and the target instructions for the manipulation direction, displacement, and speed of the joystick at the future moment under the operator's cognition are decided.

[0019] (1) Perception processing layer The perception processing layer simulates the response process of the sensory center of the heavy-duty manipulator operator to knowledge, and extracts the feature information of multi-dimensional features collected in the operating environment (such as the operating environment image, the state of the heavy-duty manipulator, and the human-machine coupling interaction torque). According to the differences in the input feature dimensions, different branch processing operations are performed on the feature information, aiming to transform the feature information of different dimensions into three-dimensional data, and then achieve unified feature fusion.

[0020] For the feature data of the operating environment image type, upsampling operation is adopted. The number of elements is increased in the feature vectors such as the operating environment image to compensate for the impact of information loss during the data acquisition process, so that it meets the requirements of the feature data dimension for the feature fusion step. To optimize the feature processing effect, on the one hand, a visual perception loss function is designed to simulate the operator's visual perception mechanism of things and accurately measure the quality of feature processing, so that the generated feature data is more in line with the actual operation requirements. At the same time, the ReLU activation function is used to perform non-linear transformation on the key feature data after visual perception, and the original feature information is retained to the greatest extent; on the other hand, the local feature information after the extended processing is gradually extracted through the convolutional network (conv).

[0021] Visual Perception Loss based on Upsample and Mean Squared Error (MSE) is defined as follows: (1) where is the original input data, is the original predicted data, is the upsampling operation, , , is the input data after upsampling, is the predicted data after upsampling, and their lengths are both .

[0022] For feature information such as the state of the heavy-duty manipulator, first, a self-attention mechanism module is used to extract features related to the state of the heavy-duty manipulator, such as joint poses and the contact force between the attachment and the load. Then, average pooling operation is performed in combination with features such as environmental images. According to the spatio-temporal distribution law of different-dimensional information, the dimensional unification of multi-dimensional feature data is achieved. Then, the Sigmoid activation function is used to make the output of the key feature data have a consistent numerical range. Finally, the data of different dimensions in the working environment become three-dimensional features after feature processing, and then feature fusion is performed to generate a feature map for output and transfer to the understanding and memory layer.

[0023] (2) Understanding and Memory Layer The understanding and memory layer consists of multiple LSTM structures, and each multiple LSTM structure is composed of multiple interconnected LSTM units that store information related to operation tasks, heavy-duty manipulator control strategies, etc. The multiple LSTM structures can not only continuously accumulate and iterate knowledge and experience in long-term memory but also solve the problem of loss or forgetting of long-term memory during the process of transmitting knowledge. In addition, when the scene is reproduced or there is a need for a specific operation task, this structure can quickly learn past operation experience through the association of new and old memories and can automatically extract the heavy-duty manipulator control strategy corresponding to the current task and environmental state from the scene-strategy knowledge base.

[0024] In a multi-layer LSTM structure, the LSTM units of each layer jointly process various types of information in the operation environment through intra-layer and inter-layer interactions. Inside the LSTM unit, the current unstructured on-site feature information and the working memory of the previous moment pass through the forget gate to discard information irrelevant to the operation task. For important information, such as the joystick movement features corresponding to the grasping angle of the heavy-duty manipulator under a special included angle configuration, etc., it will continue to be retained and multiplied by the long-term memory of the previous moment to form experience for coping with different types of operation tasks. At the same time, the input gate is multiplied by the newly added long-term memory to determine the retention degree of the newly added memory. The newly added long-term memory specifically refers to the action instructions dynamically generated by the operator based on experience and existing knowledge during the operation process to complete a certain task (such as operating the heavy-duty manipulator to grasp and remove obstacles on the path, etc.). Subsequently, the remaining newly added memory is added to the long-term memory retained at the previous moment, and the long-term memory for storage is output and passed to the next LSTM unit. The long-term memory at the current moment first passes through the tanh function to capture and present the non-linear relationship between the operator's action instructions and the current operation environment, and then is multiplied by the output gate to obtain the working memory at the current moment.

[0025] Inside the first layer of LSTM, various types of information in the operation environment (such as environmental images, the state of the manipulator, and the human-machine coupling interaction torque) are processed step by step in time. Through the time-step transfer mechanism and associative memory method within the layer, the processed various types of information are associated and retrieved in the scenario-strategy knowledge base of the long-term memory. If the empirical knowledge matching the current task is retrieved, the operation task information at the current moment is combined with the historical experience, and the joystick direction, displacement, and speed instructions after the operation environment and task are understood and memorized by the first layer of LSTM are output; if not retrieved, it will be passed to the next layer of LSTM to decide whether to forget or remember. In the second layer of LSTM, the working memory of the first layer is further forgotten and stored. The second layer of LSTM not only receives the working memory output by the first layer but also combines the working memory of the previous time step of this layer and the long-term memory . On this basis, the second layer of LSTM extracts the higher-level feature information from the long-term memory, working memory, and current input information of the first layer of LSTM and compares it with similar scenarios. If there is no operation experience related to the operation task in the long-term memory, considering the overall goal of the current operation task comprehensively, the joystick direction, displacement, and speed instructions required to complete the operation task are predicted, and combined with the joystick direction, displacement, and speed instructions after the second layer of LSTM understands and memorizes, they are jointly output to the third layer of LSTM.

[0026] By increasing the network depth, each layer of LSTM can further capture the associations between operation tasks based on the previous layer, and combine the current operation task and past experience knowledge to generate relevant joystick direction, displacement, and speed instruction sets.

[0027] (3) Judgment decision-making layer In the process of using the attention mechanism to decide the target instructions that can efficiently and accurately complete the operation task, the specific process of the collaborative work of the attention mechanism and the LSTM network is as follows.

[0028] First, let the output of the multi-layer LSTM structure be the working memory vector. As shown in formula (2), each working memory vector integrates the working memory content output by the understanding memory layer, including the joystick direction, displacement, and angle related to the current operation task, as well as past operation experience.

[0029] (2) In the formula, is the output working memory vector of the th LSTM unit corresponding to the th input, is the input, is the number of units in each recurrent layer of the LSTM network.

[0030] Secondly, perform a linear transformation on each working memory vector first, and then perform a non-linear transformation by the tanh activation function to capture the non-linear characteristics between the operation task and the operator's action instructions in each memory vector, and obtain the importance of each working memory vector , and the specific formula is as shown in (3).

[0031] (3) In the formula, and are the weight matrix and bias respectively.

[0032] Then, use formula (3) to calculate the attention weight of each working memory vector in the current operation task. This weight reflects the relative importance of different working memory vectors, and the specific formula is as shown in (4).

[0033] (4) Among them, is the exponential operation result of the importance of each working memory vector. Using the exponential function can map any real number to the positive range and can amplify the differences between different , so that more important working memory vectors obtain higher weights. It is to sum the exponential fractions of all working memory vectors as the denominator for normalization. is the number of all working memory vectors; Finally, judge the working memory vectors within the decision-making layer and the attention weights are multiplied to obtain the target instruction vector . This instruction vector represents the manipulation direction, displacement, and speed action instructions that the operator should apply to the joystick to complete the operation task. The specific formula is as shown in (5): (5) The action instructions with a high degree of matching with the current operation task in the working memory vector (such as the manipulation direction, displacement, and speed of the joystick) will be multiplied by a larger weight and prominently reflected in the decision output; conversely, they will be multiplied by a smaller weight. At the same time, as the operation task progresses, the continuously updated working memory vectors continuously participate in a new round of weight calculation and instruction generation to ensure that the target instructions output by the decision-making layer always match the dynamically changing working environment and task requirements. This continuously optimized operation process can dynamically adjust the parameters and strategies of the controller, thereby improving the manipulation performance of the heavy-duty manipulator.

[0034] Although the present invention has been described above with reference to the embodiments, various improvements can be made to it and its components can be replaced with equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the disclosed embodiments of the present invention can be combined with each other in any way, and the cases of these combinations are not exhaustively described in this specification only for the sake of saving space and resources. Therefore, the present invention is not limited to the specific embodiments disclosed in the text, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for modeling the cognitive behavior of an operator for an overloaded robotic arm, characterized in that, The steps are as follows: S1. Obtain input data of different dimensions, where the input data of different dimensions includes operation environment image information, heavy-duty manipulator state parameters, and human-machine coupling interaction torque; S2. Construct an operator cognitive behavior model based on a long short-term memory neural network and an attention mechanism. The operator cognitive behavior model includes a perception processing layer, an understanding and memory layer, and a judgment and decision-making layer; S3. The perception processing layer performs spatio-temporal feature extraction and fusion processing on the input data of different dimensions, and then transmits the processed information to the understanding and memory layer. The understanding and memory layer, according to the processed operation scenario and operation task information, combines the associative memory method to retrieve the scenario-strategy knowledge base in long-term memory, calls the operation experience corresponding to similar scenarios, and based on the cognitive result, the judgment and decision-making layer, through the attention mechanism module, focuses on the strategies related to the current operation task, and dynamically decides the operation direction, displacement, and speed target instructions of the joystick at the future moment under the operator's cognition, so as to realize the decision control of the heavy-duty manipulator.

2. The operator cognitive behavior modeling method for a heavy-duty robotic arm according to claim 1, characterized in that When the perception processing layer processes the operation environment image information, it uses upsampling operation to compensate for data acquisition loss, optimizes the processing effect by designing a visual perception loss function, uses the ReLU activation function to perform non-linear transformation on the key feature data, retains the original feature information, and uses a convolutional network to extract local feature information. The visual perception loss function is: ； Among them, is the original input data, is the original predicted data, is the upsampling operation, , , is the input data after upsampling, is the predicted data after upsampling, and their lengths are both .

3. The operator cognitive behavior modeling method for a heavy-duty robotic arm according to claim 1, characterized in that When the perception processing layer processes the heavy-duty manipulator state parameters, first, it extracts the features related to the joint pose, "attachment-load" contact force, and heavy-duty manipulator state through the self-attention mechanism module; secondly, it performs average pooling operation in combination with the extracted environmental image features and human-machine coupling interaction torque features, and realizes the dimension unification of multi-dimensional feature data according to the spatio-temporal distribution law of different dimension information; Then, the Sigmoid activation function is used to make the feature data have a consistent numerical range when output; finally, the data of different dimensions become three-dimensional features after feature processing, and then feature fusion is performed at this time to generate a feature map and transmit it to the understanding and memory layer.

4. A method for modeling the cognitive behavior of an operator for a heavy-duty manipulator according to claim 1, characterized in that, The understanding and memory layer consists of multiple LSTM structures. In the multiple LSTM structures, each LSTM unit jointly completes the processing of various types of information in the operation environment through intra-layer and inter-layer interactions; Inside the LSTM unit, the current operation environment feature information and the working memory of the previous moment pass through the forget gate to discard the information irrelevant to the operation task. For important information, it will continue to be retained and multiplied by the long-term memory of the previous moment to form the experience for dealing with different types of operation tasks; At the same time, the input gate is multiplied by the newly added long-term memory to determine the retention degree of the newly added memory. Subsequently, the remaining newly added memory is added to the long-term memory retained in the previous moment to output the long-term memory for storage and transmit it to the next LSTM unit. The long-term memory at the current moment first passes through the tanh function to capture and show the non-linear relationship between the operator's action instruction and the current operation environment, and then is multiplied by the output gate to obtain the working memory at the current moment.

5. A method for modeling the cognitive behavior of an operator for a heavy-duty manipulator according to claim 4, characterized in that, Within the first-layer LSTM, the job environment image, the state of the heavy-duty robotic arm, and the human-machine coupling interaction torque are processed step by step in time. Through the time-step transfer mechanism and associative memory method within the layer, various types of processed information are associated and retrieved in the scenario-strategy knowledge base of long-term memory. If empirical knowledge matching the current task is retrieved, the operation task information at the current moment is combined with historical experience, and the joystick direction, displacement, and speed commands after the job environment and tasks are understood and memorized by the first-layer LSTM are output. If no match is found, it is passed to the next layer of LSTM to decide whether to forget or remember.

6. The operator cognitive behavior modeling method for a heavy-duty robotic arm according to claim 4, wherein In the second-layer LSTM, the working memory of the first layer is further forgotten and stored. The second-layer LSTM not only receives the working memory output by the first layer , but also combines the working memory of the previous time step of this layer and the long-term memory . On this basis, the second-layer LSTM extracts the higher-level feature information from the long-term memory, working memory, and current input information of the first-layer LSTM, and compares it with similar scenarios. If there is no operation experience related to the operation task in the long-term memory, then considering the overall goal of the current operation task comprehensively, it predicts the joystick direction, displacement, and speed commands required to complete the operation task, and combines them with the joystick direction, displacement, and speed commands after the second-layer LSTM understands and memorizes them, and jointly outputs them to the third-layer LSTM.

7. A method for modeling the cognitive behavior of an operator for a heavy-duty robotic arm according to claim 4, characterized in that, The specific steps of the judgment and decision-making layer are as follows: First, let the output of the multi-layer LSTM structure be the working memory vector , and the specific formula is as follows; ； wherein, is the output working memory vector of the -th LSTM cell corresponding to the -th input, is the input, is the number of cells in each recurrent layer of the LSTM network; Secondly, for each working memory vector first perform a linear transformation, and then perform a non-linear transformation by the tanh activation function to capture the non-linear characteristics between the operation tasks and the operator's action instructions in each memory vector, and obtain the importance degree of each working memory vector , and the specific formula is as follows; ； In the formula, and are the weight matrix and the bias respectively; Then, use the above formula to calculate each working memory vector The attention weight in the current operation task , which reflects the relative importance of different working memory vectors. The specific formula is as follows: ； Among them, is the exponential operation result of the importance of each working memory vector . Using the exponential function can map any real number to the positive range and can magnify the differences between them, so that more important working memory vectors obtain higher weights. is the sum of the exponential scores of all working memory vectors and is used as the denominator for normalization. is the number of all working memory vectors; Finally, judge the working memory vector in the decision-making layer and the attention weight are multiplied to obtain the target instruction vector . This instruction vector represents the manipulation direction, displacement, and speed action instructions that the operator should implement on the joystick to complete the operation task. The specific formula is as follows: 。

Images