An intelligent robot arm autonomous grabbing method based on episodic memory

Abstract

The application discloses a kind of intelligent mechanical arm autonomous grabbing methods based on episodic memory, belongs to artificial intelligence and automatic mechanical arm field.First, the hardware facilities of multi-modal perception are built to robot system and teaching system, and the depth camera in robot system provides visual positioning of object for the last mechanical arm autonomous grabbing process, and the tracker and flexible film pressure sensor in teaching system provide the continuous action sequence dataset of mechanical arm for episodic memory base;Second, by controlling computer to configure depth reinforcement learning environment and based on this environment, establish episodic memory base model;Action sequence data in dataset are processed, and teaching data is stored in the episodic memory base model built to provide the basis reference for mechanical arm autonomous grabbing decision;Finally, it is verified on operation platform.The application can accurately grasp the position and characteristics of the grabbing target by accurately grasping the perception information and comprehensively analyzing the episodic memory, quickly formulate the optimal grabbing strategy and realize accurate grabbing action, and provide protection for mechanical arm to complete diversified grabbing tasks in complex environment.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and automated robotic arms, and relates to an autonomous grasping method for an intelligent robotic arm based on contextual memory. Background Technology

[0002] With the widespread application of artificial intelligence technology and automated robotic arms in industry, production, and service sectors, the flexibility and intelligence of robotic arms have become key focuses of technological development. However, there are currently two main robotic arm grasping solutions: pre-programmed grasping and vision-guided grasping. Pre-programmed grasping involves pre-setting the grasping trajectory and action sequence of the robotic arm in a fixed scenario. This solution lacks adaptability to different scenarios and objects; if the environment changes, reprogramming and resetting are required. Vision-guided grasping utilizes visual sensors to acquire the appearance and position information of objects, and uses image processing and deep learning technologies to achieve object recognition and grasping planning. This solution has high requirements for visual recognition and is very difficult to accurately locate transparent, reflective, complex-shaped, or complex-background objects.

[0003] However, these solutions share a common problem: they cannot leverage prior grasping experience to handle new grasping scenarios. This necessitates reprogramming or relying on more advanced perception and processing to achieve grasping when faced with different environments or objects. This lack of prior experience limits the flexibility and intelligence of robotic arms in practical applications. Therefore, researching how robotic arms can learn from historical experience and make intelligent decisions to complete complex grasping tasks is particularly important. Summary of the Invention

[0004] To address the problems existing in current technologies, this invention provides an autonomous grasping method for intelligent robotic arms based on contextual memory. Contextual memory can accumulate and store experiential data on various grasping actions, and can make more adaptive grasping decisions based on the stored contextual memories. This method enables the robotic arm to have human-like task skill learning and acquisition capabilities, allowing it to autonomously and flexibly perform grasping tasks in different environments using contextual memory.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An autonomous grasping method for an intelligent robotic arm based on contextual memory is disclosed. This autonomous grasping method is implemented using an autonomous grasping system, which consists of a control computer (Ubuntu), a remote server (Windows), a collaborative robotic arm, an electric gripper, a depth camera, a tracker, a positioning base station, a flexible thin-film pressure sensor, an operating platform, and experimental materials, including conical flasks, beakers, plastic cups, and plastic bowls. Specifically: the collaborative robotic arm is equipped with an electric gripper and a depth camera at its end, forming a robotic system controlled by ROS under the control computer (Ubuntu); the control computer runs a PyTorch deep reinforcement learning environment, establishing four modules: data acquisition and preprocessing, contextual memory storage and retrieval, model training and inference, and decision-making and execution, thus forming the contextual memory library of the autonomous grasping system; the instructor wears a tracker and a flexible thin-film pressure sensor on their hand to simulate the movement of the robotic arm and gripper, forming a teaching system with the cooperation of the positioning base station, transmitting continuous teaching action datasets through the remote server (Windows) and communicating with the control computer via UDP protocol; the operating platform and experimental materials serve as the experimental verification part of the autonomous grasping system.

[0007] The autonomous grasping method first requires the construction of multimodal perception hardware for the robot system and teaching system. The depth camera in the robot system provides visual localization of the object during the final autonomous grasping process, while the tracker and flexible thin-film pressure sensor in the teaching system provide a dataset of continuous motion sequences from the robot arm for the next step of establishing a context memory library. Secondly, a deep reinforcement learning environment is configured on a control computer (Ubuntu), and a context memory library model is built based on this environment. This model stores the dataset generated during the next step of robot arm grasping teaching and filtering out redundant data to reduce noise. Then, the LSTM algorithm is used to process the motion sequence data in this dataset, providing intelligent inference capabilities based on previously learned motion patterns, sequence order, and time dependencies for the final autonomous grasping decision. Thirdly, robot arm grasping teaching is performed based on the multimodal perception hardware built in the first step. The teaching data is stored in the context memory library model built in the second step, providing a basic reference for the robot arm's autonomous grasping decision, enabling it to better evaluate and select appropriate grasping strategies based on previous grasping experience and scene information. Finally, after the groundwork laid in the first three steps, a tilting experiment was conducted on the operating platform to verify the autonomous grasping method. This experiment aimed to simulate the robotic arm's ability to autonomously grasp objects based on contextual memory. By simulating the grasping and tilting of different objects in specific scenarios, the experiment verified whether the robotic arm could flexibly adjust its grasping actions and correctly tilt objects based on data stored in its contextual memory bank. Specifically, the following steps were included:

[0008] The first step, multimodal perception: This involves building the hardware infrastructure for multimodal perception to ensure the robotic arm can accurately perceive objects and the surrounding environment. This provides the necessary data foundation and environmental information for subsequent contextual memory modeling, robotic arm grasping and learning, and ultimately, autonomous grasping. The completion of this hardware construction provides the following perception capabilities for subsequent steps:

[0009] ① During the robotic arm grasping and teaching learning phase, the remote server obtains real-time continuous pose data of the teacher's wrist through the tracker and positioning base station. This pose data is transmitted to the control computer via UDP and stored in the context memory of the control computer in JSON format, which serves as a reference for path planning in the subsequent autonomous grasping process of the robotic arm.

[0010] ② During the robotic arm grasping and teaching learning phase, the remote server acquires the pressure data between the teacher's fingers through a flexible thin-film pressure sensor. This pressure data is transmitted to the control computer via UDP and stored in the control computer's context memory bank in JSON format along with the continuous pose data at the same time in ①. This serves as a reference for the opening and closing of the end gripper and the gripping force during the subsequent autonomous grasping process of the robotic arm.

[0011] ③ During the autonomous grasping stage of the robotic arm, the control computer obtains the current feedback information of the gripper through the electric gripper installed at the end of the robotic arm and uses the ROS topic communication mechanism to control the gripper's grasping force in order to avoid damage to the object to be grasped.

[0012] ④ During the autonomous grasping phase of the robotic arm, the control computer uses the depth camera installed at the end of the robotic arm and the ROS topic communication mechanism to obtain the RGB and depth information of the object to be grasped. At the same time, the YOLOv8 object detection algorithm is used under the control computer to accurately locate the object to be grasped.

[0013] The second step is scenario memory bank modeling: Building upon the multimodal sensing hardware setup from the first step, a scenario memory bank model is established via a control computer to collect and store data from different scenarios. This data includes environmental information, taught action data, and object attributes, enabling the robotic arm to learn and make correct decisions. This step includes the following four modules:

[0014] ① Data Acquisition and Preprocessing: The main task of this module is to acquire data from multimodal sensing devices and robotic arm teaching, and to preprocess the data to clean and format it for subsequent processing. This preprocessed data will be used in the context memory storage and retrieval stage.

[0015] ② Contextual Memory Storage and Retrieval: The established contextual memory database model will receive and store preprocessed data. This module is responsible for classifying and indexing various data according to environmental information, teaching action data, and object attributes, etc., for subsequent access and retrieval.

[0016] ③ Model Training and Inference: Data from the contextual memory bank will be used to train the model. This stage utilizes LSTM to process the action sequence data. The LSTM model provides the basis for the robotic arm's grasping action decisions by learning and understanding patterns in the action sequences.

[0017] ④ Decision Making and Execution: This stage uses a Deep Q-Network (DQN) to make grasping action decisions based on data from the context memory. These decisions guide the robotic arm in performing grasping actions, enabling it to adapt to different scenarios and perform correct movements.

[0018] The third step, robotic arm grasping and teaching: Based on the modeling results of the context memory database in the previous step, and leveraging the advantages of robotic arm teleoperation teaching, this stage records and stores real-time continuous motion sequence data into the context memory database. This allows the robotic arm to comprehensively learn and understand accurate grasping actions to adapt to various different scenarios and objects. This step includes the following:

[0019] ① First, a specific grasping scenario is set up on the robotic arm operating platform. The instructor controls the robotic arm to move to the object to be grasped through the teaching device and makes the optimal grasping strategy based on human judgment to achieve accurate grasping and movement of the object.

[0020] ② While ① the optimal grasping strategy is taught manually, the multimodal sensing device generates a JSON dataset containing environmental information and grasping action sequence data. This dataset is transmitted in real time to the context memory database established on the control computer in the second step via the UDP transmission protocol on the remote server.

[0021] ③ Whenever a new grasping scenario is changed or the similarity between the object to be grasped and the existing dataset in the context memory is less than 80%, the robotic arm should be initialized with grasping experience training via a teaching device. New robotic arm grasping datasets generated during teaching will be categorized and stored in the context memory. In the fourth step of the robotic arm's autonomous grasping process, the control computer will use a depth camera to search the current environment and compare it with the existing dataset in the context memory. If the comparison is successful, the decision-making and execution module in the context memory will be invoked for autonomous grasping; otherwise, it will return that no similar context memory was found, and initial teaching is required.

[0022] The fourth step is for the robotic arm to achieve autonomous grasping: Through previous teleoperation teaching and learning, and the modeling information in the context memory bank, the robotic arm can perform independent grasping operations. It can formulate precise and efficient grasping strategies based on environmental information obtained through multimodal perception and data stored in the context memory bank. This process involves the following key steps:

[0023] ① First, the position of the object to be grabbed on the operating platform is randomly arranged. The host computer will obtain the current environmental information through the depth camera installed at the end of the robotic arm.

[0024] ②The computer control system will call the scene memory database to match and search for action scenes with a similarity of more than 80% based on the obtained environmental information, and select the action scene with the highest similarity.

[0025] ③Based on the best contextual memory experience data obtained in ②, the control computer trains the inference and decision-making module through the contextual memory bank model to formulate the optimal action strategy for the robotic arm in the current scenario;

[0026] ④ The robotic arm transforms the motion strategy set by the control computer into a specific sequence of motions, thereby achieving precise target grasping and movement;

[0027] ⑤ The control computer provides feedback and adjustments based on the execution results, continuously optimizing the robotic arm's grasping ability to ensure its efficient and stable performance in complex environments.

[0028] The beneficial effects of this invention are as follows:

[0029] This invention combines the precise information acquisition capabilities of multimodal perception, the efficient storage and retrieval mechanism of contextual memory, the intelligent analysis and inference capabilities of machine learning, and the flexible execution strategy of autonomous decision-making. This enables the robotic arm to achieve more precise and efficient autonomous grasping tasks through continuous learning and adaptation. By accurately grasping the perceived information and comprehensively analyzing the contextual memory, the robotic arm can accurately determine the position and characteristics of the grasping target, thereby quickly formulating the optimal grasping strategy and achieving precise grasping actions. This provides strong technical support and intelligent assurance for the robotic arm to complete diverse grasping tasks in complex environments. Attached Figure Description

[0030] Figure 1 This is an overall flowchart of the method of the present invention;

[0031] Figure 2 Modeling process for contextual memory banks;

[0032] Figure 3 A teaching and learning system for robotic arms to grasp objects. Detailed Implementation

[0033] The following description, in conjunction with the accompanying drawings and specific embodiments, further illustrates the autonomous grasping method of the intelligent robotic arm:

[0034] The overall process of the method of this invention is described in [link to invention]. Figure 1 This method first requires the deployment of multimodal sensing:

[0035] 1) Take pictures of the objects to be grasped in the experiment and label them. Amplify the samples through image transformation and train the samples using the YOLOv8 object detection algorithm to prepare for the subsequent acquisition of the RGB and depth information of the objects to be grasped by the robotic arm through the depth camera of its end flange.

[0036] 2) By obtaining the current information fed back by the electric gripper through the ROS robot operating system under the control computer, the force of the robotic arm when autonomously grasping objects through context memory can be accurately controlled, so as to effectively grasp objects of different materials.

[0037] 3) Under the remote server system, the wrist posture data of the teacher can be obtained in real time using positioning base stations and trackers and transmitted to the control computer via Ethernet UDP;

[0038] 4) The pressure feedback data between the teacher's two fingers is transmitted via the USB serial port of the control computer using a flexible thin film pressure sensor, thereby controlling the force with which the gripper holds different objects during the teaching process.

[0039] Next, we will establish a contextual memory bank:

[0040] 1) See Figure 2 First, data acquisition and preprocessing are required via a control computer. This includes acquiring multimodal perception data, robotic arm motion data, and robotic arm teaching data. Various perceptual data from the environment are collected through sensors, including scene information in depth images, object distribution data, robotic arm motion data, and teaching data, providing the necessary foundation for subsequent modeling and analysis.

[0041] 2) Next, the scenario memory storage and retrieval module is established. During manual teleoperation teaching, environmental information acquired by the camera and images of the object to be grasped are preprocessed and stored in the scenario memory storage and retrieval module, and a graph database index is established. The storage and retrieval of scenario memories are achieved through the graph database index. Using the graph database index structure, environmental information, motion trajectories, and sensory data of the robot when performing specific actions are stored. With the support of the index structure, rapid retrieval and access to the stored memory data are achieved.

[0042] 3) Next, a model training and inference module is established for the stored contextual memory. The pose information (X, Y, Z, Roll, Pitch, Yaw) of the robotic arm's end effector TCP, the data fed back by the pressure sensor, and the environmental information acquired by the depth camera are stored in JSON format in the model training and inference module of the contextual memory library. PyTorch is used for model training, and a Long Short-Term Memory (LSTM) network is used to process the action sequence data. In the contextual memory library, the output gate of the LSTM structure selectively outputs the robotic arm's grasping action decision in a specific context based on the grasping action pattern and scene information learned by the LSTM model, so that the robotic arm can complete the grasping task more accurately and intelligently. The output gate formula is:

[0043] o t =σ(W o ·[h t-1 ,x t ]+b o ).

[0044] in:

[0045] o t It is the output of the output gate;

[0046] W o It is the weight matrix of the output gate;

[0047] h t-1 It is the hidden state of the previous time step;

[0048] x t This is the input for the current time step;

[0049] b o It is the bias term of the output gate;

[0050] σ is the sigmoid activation function.

[0051] The output gate is represented in the context memory bank as follows:

[0052] Filtering hidden states: Output of the output gate o t The hidden state h is determined t Which parts of the information are passed to the next step, thus selectively transmitting important memory information. This helps to filter the information most useful for the current grasping decision, enabling the robotic arm to make grasping decisions based on important information in contextual memory.

[0053] Decision output: Output of the output gate o t With hidden state h tMultiplying these values ​​yields the network's final output. This output, based on the features extracted by the LSTM during learning and memorization, serves as the decision-making basis for the robotic arm to perform a grasping action. Therefore, the output gate determines the robotic arm's grasping behavior at the current moment, based on the LSTM's learning and understanding of contextual memory.

[0054] Using the training framework provided by PyTorch, the stored memory data is inferred and classified to identify and parse different scenes and action sequences. LSTM can effectively capture long-term dependencies in action sequences, improving the model's ability to understand and model action patterns.

[0055] 4) Finally, the Deep Q-Network (DQN) algorithm is used to implement the decision-making and execution module for the context memory database model. By training the intelligent robotic arm agent to interact with the environment, it learns the optimal grasping strategy and adjusts the strategy in real time based on environmental feedback to achieve the grasping operation of random objects. The DQN algorithm can make fast and effective decisions on the strategy, improving the execution efficiency of the robotic arm in grasping tasks. Finally, during the robotic arm's autonomous grasping process, the context memory storage and retrieval module retrieves similar action segments from the teaching process, while the decision-making and execution module retrieves the corresponding JSON data for these actions and uses the DQN algorithm to make the final grasping action decision.

[0056] After completing the model building of the contextual memory bank, the robotic arm will be used for grasping and teaching.

[0057] 1) See Figure 3 During the remote teaching of the robotic arm's grasping action, the instructor wears a tracker on his wrist between two positioning base stations. The remote server obtains the real-time pose information of the instructor's wrist and transmits it to the control computer via UDP to control the synchronous TCP movement of the robotic arm's end effector. At the same time, the instructor can simulate the grasping action of the robotic arm's end effector gripper through a thin-film pressure sensor worn on his fingertip.

[0058] 2) The instructor actively controls the wrist posture to make the robotic arm gradually approach the object to be grasped, while applying different forces to the pressure sensor between the fingers to control the gripper to grasp objects of different materials.

[0059] 3) To make the robotic arm run more stably and smoothly during the teaching process, the data transmitted via UDP needs to be optimized: data filtering and smoothing are used to remove noise and jitter, and a cyclic redundancy check (CRC) mechanism is added during data transmission to ensure the integrity and correctness of the data;

[0060] 4) When changing the scene or grasping an object, the control computer should first use the depth camera at the end of the robotic arm to retrieve the new scene information and compare it with the existing dataset in the scene memory bank. If the similarity is less than 80%, the robotic arm needs to be initialized with grasping experience teaching through the teaching device.

[0061] 5) The new robotic arm grasping dataset generated during teaching will be classified and stored in the context memory bank. During the final autonomous grasping process of the robotic arm, the control computer will search the current environment through the depth camera and compare it with the existing dataset in the context memory bank. If the comparison is successful, the decision-making and execution module in the context memory will be called to perform autonomous grasping. Otherwise, it will return that no similar context memory was found and teaching initialization is required.

[0062] Finally, the autonomous grasping of the robotic arm is achieved using the configured multimodal perception, context memory library, and robotic arm grasping teaching system.

[0063] 1) This invention uses a tilting experiment to verify the feasibility and effectiveness of the autonomous grasping method of the intelligent robotic arm based on context memory. The experimental scenario is set up with a plastic cup filled with half water and an empty plastic bowl randomly placed on the operating platform.

[0064] 2) In order to enable the robotic arm to autonomously grasp the plastic cup and move it above the plastic bowl to pour the liquid, a combination of conical flask and beaker with high similarity to the current scene was first selected to teach the robotic arm.

[0065] 3) Similarly, a conical flask filled halfway with water and an empty beaker are randomly placed on the operating platform. The instructor remotely teaches the robotic arm through the teaching system to perform continuous actions of grasping, lifting and pouring liquid into the beaker. The continuous action sequence of the teaching process is segmented, classified and stored in the context memory bank.

[0066] 4) Next, the operating platform is set to the experimental scenario in 1). The control computer obtains the current environmental information through the depth camera at the end of the robotic arm. After the similarity is compared by the context memory storage and retrieval module, the similarity is 93%. After the retrieval, the LSTM algorithm in the context memory and DQN decision support are used to output the action strategy to the robotic arm.

[0067] 5) Finally, the robotic arm autonomously grasps the plastic cup containing liquid in the current scene by using the motion strategy output by the control computer and the visual positioning of the end-effector depth camera, and accurately lifts it up and pours the liquid into the plastic bowl.

[0068] This invention makes full use of contextual memory and algorithmic decision-making, enabling the robotic arm to successfully complete a variety of complex grasping tasks through autonomous learning and intelligent decision-making, thereby improving operational efficiency and demonstrating the innovation and practicality of this invention.

[0069] The above-described embodiments are merely illustrative of the implementation methods of the present invention, but should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the protection scope of the present invention.

Claims

1. A method for autonomous grasping by an intelligent robotic arm based on contextual memory, characterized in that, The autonomous grasping method is based on an autonomous grasping system, which consists of a control computer, a remote server, a collaborative robotic arm, an electric gripper, a depth camera, a tracker, a positioning base station, a flexible thin-film pressure sensor, an operating platform, and experimental materials. Specifically, the collaborative robotic arm is equipped with an electric gripper and a depth camera at its end, forming a robot system controlled by ROS under the control computer. The control computer runs a PyTorch deep reinforcement learning environment, establishing four modules: data acquisition and preprocessing, context memory storage and retrieval, model training and inference, and decision making and execution, thus forming the context memory bank of the autonomous grasping system. The instructor wears a tracker and a flexible thin-film pressure sensor on their hand to simulate the movement of the robotic arm and gripper, forming a teaching system with the cooperation of the positioning base station. Continuous teaching action datasets are transmitted through the remote server and communicate with the control computer via UDP protocol. The operating platform and experimental materials serve as the experimental verification part of the autonomous grasping system. The autonomous grasping method first requires the construction of multimodal perception hardware for the robot system and teaching system. The depth camera in the robot system provides visual localization of the object during the final autonomous grasping process, while the tracker and flexible thin-film pressure sensor in the teaching system provide a dataset of continuous action sequences of the robot arm for the context memory. Second, a deep reinforcement learning environment is configured on the control computer, and a context memory model is built based on this environment. This model stores the dataset generated during the next step of robot arm grasping teaching and filtering out redundant data. Then, the LSTM algorithm is used to process the action sequence data in this dataset, providing intelligent inference capabilities based on previously learned action patterns, sequence order, and time dependencies for the final autonomous grasping decision. Third, robot arm grasping teaching is performed based on the constructed multimodal perception hardware, and the teaching data is stored in the constructed context memory model, providing a basic reference for the robot arm's autonomous grasping decision, enabling it to evaluate and select appropriate grasping strategies based on previous grasping experience and scene information. Finally, the method is validated on an operating platform. Specifically, the following steps are included: The first step, multimodal perception: First, establish the hardware foundation for multimodal perception to ensure the robotic arm can accurately perceive objects and the surrounding environment; after the hardware is built, the following perception content will be provided: ① During the robotic arm grasping and teaching learning phase, the remote server obtains real-time continuous pose data of the instructor's wrist through the tracker and positioning base station. The pose data is transmitted to the control computer via UDP and stored in the control computer's context memory in JSON format for reference in the subsequent path planning during the robotic arm's autonomous grasping process. ② During the robotic arm grasping and teaching learning stage, the remote server acquires the pressure data between the teacher's fingers through a flexible thin film pressure sensor. The pressure data is transmitted to the control computer via UDP and stored in the control computer's context memory bank in JSON format along with the continuous pose data at the same time in ①. This serves as a reference for the opening and closing of the end gripper and the gripping force during the subsequent autonomous grasping process of the robotic arm. ③ During the autonomous grasping stage of the robotic arm, the control computer obtains the current feedback information of the gripper through the electric gripper installed at the end of the robotic arm and uses the ROS topic communication mechanism to control the gripper's grasping force in order to avoid damage to the object to be grasped. ④ During the autonomous grasping stage of the robotic arm, the control computer obtains the RGB and depth information of the object to be grasped by using the depth camera installed at the end of the robotic arm and the ROS topic communication mechanism. At the same time, the YOLOv8 target detection algorithm is used under the control computer to accurately locate the object to be grasped. The second step is scenario memory bank modeling: Building upon the multimodal sensing hardware setup in the first step, a scenario memory bank model is established via a control computer to collect and store data from different scenarios. This data includes environmental information, teaching action data, and object attributes, enabling the robotic arm to learn and make correct decisions. This step includes the following four modules: ① Data acquisition and preprocessing: Acquire data from multimodal sensing devices and robotic arm teaching, and preprocess, clean and format the data. The preprocessed data will be used for the context memory storage and retrieval stage. ② Contextual memory storage and retrieval: Receive and store preprocessed data through the contextual memory bank model; classify and index various data according to environmental information, teaching action data, and object attributes for access and retrieval; ③ Model training and inference: The model is trained using data from the context memory bank, and LSTM is used to process the action sequence data; the LSTM model provides the basis for the robotic arm's grasping action decision by learning and understanding the patterns of the action sequence. ④ Decision making and execution: A deep Q-network is used to make grasping action decisions based on data in the context memory bank; the decisions will guide the robotic arm to perform grasping actions, enabling it to adapt to different scenarios and perform correct actions; The third step, robotic arm grasping and teaching: Based on the modeling results of the context memory bank, and leveraging the advantages of robotic arm teleoperation teaching, by recording and storing real-time continuous motion sequence data into the context memory bank, the robotic arm can comprehensively learn and understand accurate grasping actions to adapt to various different scenarios and objects; this step includes the following: ① First, a specific grasping scenario is set up on the robotic arm operating platform. The instructor controls the robotic arm to move to the object to be grasped through the teaching device and makes the optimal grasping strategy based on human judgment to achieve accurate grasping and movement of the object. ② While ① the optimal grasping strategy is taught manually, the multimodal sensing device generates a JSON dataset containing environmental information and grasping action sequence data. This dataset is transmitted in real time to the context memory database established on the control computer via the UDP transmission protocol on the remote server. ③ Whenever a new grasping scene is changed or the similarity between the object to be grasped and the existing dataset in the context memory is less than 80%, the robotic arm is initialized with grasping experience teaching through the teaching device; the teaching generates a new robotic arm grasping dataset, which is then classified and stored in the context memory. In the fourth step of the robotic arm's autonomous grasping process, the control computer will use the depth camera to search the current environment and compare it with the existing dataset in the context memory. If the comparison is successful, the decision-making and execution module in the context memory will be called to perform autonomous grasping; otherwise, it will return that no similar context memory was found and perform teaching initialization. The fourth step is for the robotic arm to achieve autonomous grasping: Through the teleoperation teaching and learning in the third step and the modeling information in the context memory bank, the robotic arm can perform independent grasping operations; it can formulate precise and efficient grasping strategies based on environmental information obtained from multimodal perception and data stored in the context memory bank; the key steps are as follows: ① The position of the object to be grabbed on the random placement operation platform will be obtained by the host computer through the depth camera installed at the end of the robotic arm to obtain the current environmental information; ② The computer controls the scene memory database to match and retrieve actions with a similarity of more than 80% based on the obtained environmental information, and selects the action with the highest similarity. ③Based on the best contextual memory experience data obtained in ②, the control computer trains the inference and decision-making module through the contextual memory bank model to formulate the optimal action strategy for the robotic arm in the current scenario; ④ The robotic arm transforms the motion strategy set by the control computer into a specific sequence of motions, thereby achieving precise target grasping and movement; ⑤ The control computer provides feedback and adjustments based on the execution results, continuously optimizing the robotic arm's grasping ability to ensure its efficient and stable performance in complex environments.

Images