Robot intelligent agent control method and system based on memory mechanism

Abstract

The invention provides a robot intelligent agent control method and system based on a memory mechanism, and is applied to the technical field of robot control. Determining a target object mask of the target object and an adjacent environment mask of an adjacent environment of the target object; based on the target object mask and the depth image of the current environment, determining three-dimensional space information of the target object in a robot coordinate system in combination with preset camera parameters and a current robot pose; based on the target object mask and the adjacent environment mask, performing retrieval from an adaptive specification to obtain target historical experience; and inputting the RGB image of the current environment, the three-dimensional space information and the target historical experience into the multi-modal large model for decision making, and obtaining an executable action sequence corresponding to a control instruction output by the multi-modal large model. According to the invention, the physical feasibility and safety of the robot intelligent agent control decision can be improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot intelligent agent control method and system based on a memory mechanism. Background Technology

[0002] Traditional robot control devices typically rely on closed-set state machines or predefined rule bases, making them rigid and lacking in generalization ability when facing unstructured open environments such as homes and offices. For example, while traditional robotic vacuum cleaners can build high-precision two-dimensional maps, they cannot understand the essential semantic and physical differences between "cleaning up spilled coffee" and "tidying up scattered building blocks."

[0003] To endow robots with stronger environmental understanding and logical reasoning capabilities, existing technologies are beginning to explore using Large Language Models (LLMs) as the robot's "brain" or high-level planner. These methods typically follow a "perception-planning-execution" paradigm: first, a visual model is used to convert environmental images into text descriptions; then, task instructions and environmental descriptions are input into the LLM to generate a series of action steps in natural language (such as "pick up the cup" or "go to the kitchen"); finally, these steps are executed using a predefined skill set.

[0004] However, despite the superior performance of LLM-based robotic agents at the inference level, they still face significant challenges in practical physical deployments, particularly in cleaning tasks involving physical contact and multi-tool operation. These challenges primarily stem from a severe disconnect between decision-making and the physical context. Summary of the Invention

[0005] This invention provides a robot intelligent agent control method and system based on a memory mechanism, which solves the defect of control decision-making being disconnected from the physical carrier in existing robot control methods, thereby improving the physical feasibility and safety of robot intelligent agent control decisions.

[0006] This invention provides a memory-based control method for robot intelligent agents, applied to robot intelligent agents, comprising: In response to receiving a control command for a target object in the current environment, the target object mask and the neighboring environment mask of the target object's neighboring environment are determined. Based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose, the three-dimensional spatial information of the target object in the robot coordinate system is determined. Based on the target object mask and the adjacent environment mask, the target historical experience is obtained by retrieving from the preset adaptive specification, wherein the adaptive specification includes: the static physical parameters of the robot agent and the historical interaction records of the robot agent; The RGB image of the current environment, the three-dimensional spatial information, and the target's historical experience are input into the multimodal large model for decision-making, resulting in an executable action sequence corresponding to the control command output by the multimodal large model.

[0007] According to a memory-based robot agent control method provided by the present invention, before determining the target object mask and the neighboring environment mask of the target object's neighboring environment in response to receiving a control command for a target object in the current environment, the method further includes: Obtain the RGB and depth images of the current environment; The RGB image is input into a multimodal large model for open vocabulary recognition, and the names of multiple objects in the current environment are obtained from the output of the multimodal large model. The name of each of the multiple objects is input into the open-world object detection model to obtain the detection bounding box of each object output by the open-world object detection model; The detection bounding box of each object is input into the segmentation model to obtain the segmentation mask of each object output by the segmentation model.

[0008] According to a memory-based robot agent control method provided by the present invention, the method for determining the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose, includes: Based on the target object mask and the depth image of the current environment, combined with preset camera parameters, the 3D point cloud coordinates of the target object in the camera coordinate system are determined: ,in ; Wherein, the three-dimensional point cloud coordinates of the target object in the camera coordinate system are: , Let be the transpose matrix, and let the pixel coordinates in the depth image be . The corresponding depth value is , and Indicates that the camera is in direction and Normalized focal length in direction and This represents the x-coordinate and y-coordinate of the principal point of the camera; Determine the extrinsic parameter matrix of the camera relative to the current robot pose; Based on the extrinsic parameter matrix, the coordinates of the three-dimensional point cloud are transformed to the robot coordinate system to obtain the three-dimensional spatial information of the target object in the robot coordinate system. The three-dimensional spatial information includes the geometric center coordinates and the three-dimensional bounding box size.

[0009] According to the present invention, a robot intelligent agent control method based on a memory mechanism is provided, wherein the step of retrieving target historical experience from a preset adaptive specification based on the target object mask and the adjacent environment mask includes: The union between the target object mask and the adjacent environment mask is input into the contrastive language image pre-training model to obtain the query feature vector output by the contrastive language image pre-training model; Based on the query feature vector and the memory feature vector of historical interaction records in the preset adaptive specification, a comprehensive score for each historical experience is determined: ; in, Indicates the first A comprehensive score based on historical experience. Represents the similarity function. This represents the query feature vector. Indicates the first Each memory feature vector Indicates the first Risk coefficient based on historical experience; The historical experience of the highest number of targets with the highest comprehensive scores is used as the target historical experience.

[0010] According to a memory-based robot agent control method provided by the present invention, after inputting the RGB image of the current environment, the three-dimensional spatial information, and the target historical experience into a multimodal large model for decision-making to obtain the executable action sequence corresponding to the control command output by the multimodal large model, the method further includes: Determine the end effector corresponding to the executable action sequence; Based on the three-dimensional spatial information of the target object in the robot coordinate system, the target end-effector pose required for the end effector to execute the executable action sequence is determined. Based on the inverse kinematics algorithm, the target joint angle vector that enables the end effector to reach the target end pose is determined.

[0011] According to a memory-based robot intelligent agent control method provided by the present invention, the step of determining the target joint angle vector that enables the end effector to reach the target end pose based on an inverse kinematics algorithm includes: ; in, Represents the target joint angle vector. Represents the inverse kinematic function. This indicates the pose of the target end effector. This represents the parameter that minimizes the objective function. Represents the forward kinematic function. Represents the weight parameters. This represents the input joint angle vector. This represents the current joint angle vector.

[0012] The present invention also provides a robot intelligent agent control device based on a memory mechanism, comprising the following modules: A mask module is used to determine the target object mask and the adjacent environment mask of the target object's adjacent environment in response to receiving a control command for a target object in the current environment. The determination module is used to determine the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose. The experience module is used to retrieve target historical experience from a preset adaptive specification based on the target object mask and the adjacent environment mask. The adaptive specification includes: the static physical parameters of the robot agent and the historical interaction records of the robot agent. The action module is used to input the RGB image of the current environment, the three-dimensional spatial information, and the target historical experience into the multimodal large model for decision-making, and obtain the executable action sequence corresponding to the control command output by the multimodal large model.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot intelligent agent control method based on the memory mechanism as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the memory-based robot intelligent agent control method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the memory-based robot intelligent agent control method as described above.

[0016] The present invention provides a robot intelligent agent control method and system based on a memory mechanism. First, by acquiring the mask of the target object and its surrounding environment, a clear distinction between the target and its context is achieved. Next, by combining depth information and camera parameters, the target object is transformed into the robot coordinate system, providing precise three-dimensional spatial positioning and ensuring the spatial accuracy of operations. Furthermore, by retrieving physical parameters and historical interaction records from the adaptive specification, knowledge of the specific robot body and past experience is introduced, enhancing the individual adaptability and experience reusability of decision-making. Finally, by integrating RGB images, three-dimensional spatial information, and historical experience, a multimodal large model is used for decision-making, generating reliable and executable action sequences, thereby achieving efficient and intelligent object control in complex environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the robot intelligent agent control method based on a memory mechanism provided by the present invention.

[0019] Figure 2 This is an example diagram of the robot intelligent agent control method based on memory mechanism provided by the present invention in a practical application.

[0020] Figure 3 This is a schematic diagram of the technical path for retrieval, updating, and specification provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the mobile cleaning robot provided by the present invention.

[0022] Figure 5 This is a schematic diagram of a module of the robot intelligent agent control device based on a memory mechanism provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Despite the superior performance of LLM-based robotic agents at the inference level, they still face significant challenges in practical physical deployment, particularly in cleaning tasks involving physical contact and multi-tool operations. These challenges primarily stem from a severe disconnect between decision-making and the physical environment, specifically manifested in the following aspects: First, a lack of self-awareness leads to unenforceable decisions. Most existing intelligent agent frameworks focus on making decision-making modules pay attention to tasks, user preferences, external knowledge, or environmental context, but often neglect to integrate the decision-making module with the robot's own physical capabilities. Decision-making modules often assume the robot possesses ideal operational capabilities, ignoring physical constraints such as the maximum opening width of the gripper and its reachability. For example, a robot equipped with only a 70mm opening gripper might be instructed by the decision-making module to grasp a larger cardboard box. This lack of physical constraints results in logically sound decisions that are physically infeasible, leading to an extremely high task execution failure rate.

[0026] Second, the lack of an effective memory mechanism based on real physical feedback leads to repeated errors. Existing robot memory modules typically directly adopt retrieval enhancement techniques from the field of natural language processing, i.e., retrieval based on the semantic similarity of text blocks. However, physical world interaction experiences are multimodal and have extremely high safety weights. An object that seems similar in text description may produce completely different consequences in actual operation due to different surrounding environments. Existing text similarity-based retrieval methods cannot effectively distinguish these important physical differences and often ignore those low-frequency but high-risk key experiences (such as spilling liquid when handling beverage cans). The lack of memory and weighting of consequences and risks makes it difficult for robots to learn from mistakes.

[0027] Third, there is a disconnect between the perception and execution layers, resulting in a lack of precise spatial reasoning. Large models excel at processing semantic information, but their ability to quantitatively perceive three-dimensional spatial relationships is weak. Most existing perception modules remain at the 2D bounding box or semantic segmentation level, lacking an understanding of the spatial positional relationships of objects in the scene.

[0028] In summary, how to build a robot task planning system that can deeply understand its own physical limitations, continuously learn and accumulate experience from real physical interactions, possess spatial reasoning capabilities, and is safe and reliable is the core technical problem that urgently needs to be solved in this field.

[0029] The core idea of ​​this invention is to view the task planning process of a robotic agent not only as a response to the external environment, but also as a scheduling of its own capabilities. By constructing an adaptive specification that includes static physical parameters and dynamic interaction experience, decision-making and control are integrated with the robot itself. Simultaneously, a risk-weighted retrieval enhancement generation mechanism is introduced, enabling the robot to focus more on high-risk records when utilizing the adaptive specification; enhanced perception is used to understand the spatial relationships of key objects in the scene; and a multi-expert parallel mechanism ensures the physical feasibility and safety of the decisions.

[0030] This invention combines intelligent agent decision-making with the physical limitations of the robot itself, thereby improving the feasibility of decision-making; through an adaptive instruction manual mechanism, it can summarize experience from real-world feedback after operation execution and update the instruction manual content; and it can be deployed on real cleaning robots to complete various scene-level cleaning tasks.

[0031] Figure 1 This is a flowchart illustrating the robot intelligent agent control method based on a memory mechanism provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps.

[0032] Step 101: In response to receiving a control command for a target object in the current environment, determine the target object mask and the neighboring environment mask of the target object's neighboring environment.

[0033] Before performing any task, the robotic agent needs to comprehensively perceive and understand its environment through its vision system. Specifically, the robotic agent simultaneously acquires RGB and depth images of the current environment using its onboard RGBD camera. The RGB image provides rich color and texture information, while the depth image records the distance information between each pixel in the scene and the camera.

[0034] After acquiring the environmental image, the robot agent initiates a semantic perception process based on a multimodal large-scale model. The RGB image is input into a pre-trained multimodal large-scale model, which possesses powerful open-vocabulary recognition and scene understanding capabilities. After analyzing the image, the multimodal large-scale model outputs a list of names of all noteworthy objects in the scene. For example, in a typical office scene, this list might include "paper ball," "can," "water stain," "keyboard," and "vase." This process completes the initial semantic parsing of the environment, identifying all potential operational objects and distractions.

[0035] When a robotic agent receives a control command from a user or upper-level system targeting a specific object, such as "clean up the spilled coffee," it enters the target locking phase. The agent first parses the command to determine the specific target object to be manipulated, such as "coffee stains." Next, the agent retrieves the specific segmentation mask corresponding to "coffee stains" from the generated complete scene segmentation map and identifies it as the target object mask.

[0036] After determining the target object mask, the robot agent further analyzes its spatial context to determine the neighboring environment mask. The specific process is as follows: In the scene segmentation map, the spatial distance between the segmentation masks of all other objects and the target object mask on the image plane is calculated. Masks of other objects whose edge distance to the target object mask is less than a preset threshold (e.g., the masks of the "cup" or "tabletop" near the coffee stain), along with the background regions between these masks and the target object mask, are collectively determined as the neighboring environment. Finally, the union of these neighboring object masks and the background regions between them is taken to generate a complete neighboring environment mask that covers the local area surrounding the target object.

[0037] Step 102: Based on the target object mask and the depth image of the current environment, combined with the preset camera parameters and the current robot pose, determine the three-dimensional spatial information of the target object in the robot coordinate system.

[0038] After successfully obtaining the target object mask, the robot agent needs to convert the semantic information in the two-dimensional image into precise geometric information in the three-dimensional physical space.

[0039] The robotic agent reads the target object mask, where each white pixel represents a projection position of the target object on the image. For each pixel in the mask, its pixel coordinates are taken, and its depth value is read from the corresponding position in the depth image. Using a pre-calibrated camera intrinsic matrix, the back-projection formula of the pinhole camera model is used for calculation until all pixels in the target object mask are converted into 3D point cloud coordinates in the camera coordinate system.

[0040] By querying pre-calibrated hand-eye transformation parameters, the transformation relationship from the camera coordinate system to the robot coordinate system is obtained. This transformation relationship is represented by a transformation matrix that includes rotation and translation components. Applying this transformation matrix to the 3D point cloud coordinates in the camera coordinate system transforms them to the robot coordinate system.

[0041] After obtaining the 3D point cloud in the robot coordinate system, physical property analysis is performed, that is, based on the 3D point cloud data, the geometric center coordinates and 3D dimension information of the target object are calculated. The geometric center is obtained by calculating the average coordinates of all points in the point cloud, and the 3D dimensions are obtained by calculating the extreme differences of the point cloud along each coordinate axis.

[0042] Step 103: Based on the target object mask and the adjacent environment mask, retrieve the target's historical experience from the preset adaptive specification.

[0043] The adaptive instruction manual includes: the static physical parameters of the robot agent and the historical interaction records of the robot agent.

[0044] The adaptive instruction manual, serving as the robot's knowledge base, comprises two main parts: first, static physical parameters, which record the inherent properties of the robot itself, such as the reachable workspace of the robotic arm, the maximum opening width of the gripper, and the maximum load capacity; and second, historical interaction records, which store in chronological order the environmental information, actions, results, and corresponding risk assessments of the robot's past tasks.

[0045] The target object mask is merged with its adjacent environment masks to form a comprehensive context mask that represents the current operating scenario. This mask includes not only the target object itself but also key environmental information surrounding it. Subsequently, a visual feature extraction model is used to process this context mask, generating a high-dimensional feature vector, which serves as the query feature vector for this retrieval.

[0046] For each historical experience stored in the instruction manual, its corresponding memory feature vector is retrieved (this vector is generated in the same way when the historical experience is recorded). The basic matching score is obtained by calculating the cosine similarity between the query feature vector and the memory feature vector of each historical experience.

[0047] To improve the security and usability of search results, this invention introduces a risk-weighted mechanism. Each historical experience is assigned a risk coefficient during storage, which is determined based on the security level of the historical operation results. The base matching score is multiplied by the corresponding risk coefficient to obtain a weighted comprehensive score. Historical experiences with higher risk coefficients (such as records that have caused task failures or security risks) are given higher weights, thus obtaining a higher ranking in the search results.

[0048] Historical experiences are sorted from highest to lowest based on their overall scores, and the top-ranked records are selected as target historical experiences.

[0049] Through the above retrieval process, the robot agent can quickly locate the most relevant and valuable historical information for the current scenario from its own experience, especially lessons learned in high-risk scenarios, thereby significantly improving the success rate and safety of task execution.

[0050] Step 104: Input the RGB image of the current environment, the three-dimensional spatial information, and the target's historical experience into the multimodal large model for decision-making, and obtain the executable action sequence corresponding to the control command output by the multimodal large model.

[0051] The current environment's RGB image, 3D spatial information (including the target object's geometric attributes and spatial relationship descriptions), and retrieved historical experience are combined to construct a structured cue information that the multimodal large model can understand. This structured cue information includes the context of the task instructions (control instructions), physical constraints, and relevant historical lessons learned.

[0052] Structured prompts are input into a multimodal large model for decision-making and reasoning. The multimodal large model, leveraging its powerful natural language understanding and logical reasoning capabilities, comprehensively analyzes the input information. The decision-making and reasoning process considers several key factors: first, the degree of matching between control commands and the current environment; second, the compatibility between the physical properties of the target object and the robot's own static physical parameters (such as the maximum opening size of the gripper and the tool's functional range); and third, the success patterns and risk warnings recorded in the target's historical experience.

[0053] Based on the above comprehensive analysis, the multimodal large model generates an executable sequence of actions corresponding to the control commands. This sequence of actions explicitly specifies the specific operation, the object being operated on, and the type of tool required. For example, for the command "clean up spilled coffee," the possible generated action is: "use a suction tool to suck up the coffee stains."

[0054] Through the above decision-making process, the robot agent combines visual perception, physical constraints, and historical experience to generate a sequence of actions that meets task requirements while also taking into account physical feasibility and safety.

[0055] Through the embodiments of this invention, firstly, by acquiring the mask of the target object and its surrounding environment, a clear distinction between the target and its context is achieved. Next, by combining depth information and camera parameters, the target object is transformed into the robot coordinate system, providing precise 3D spatial positioning and ensuring the spatial accuracy of operations. Furthermore, by retrieving physical parameters and historical interaction records from the adaptive specification, knowledge of the specific robot body and past experience is introduced, enhancing the individual adaptability and experience reusability of decision-making. Finally, by integrating RGB images, 3D spatial information, and historical experience, a multimodal large model is used for decision-making, generating reliable and executable action sequences, thereby achieving efficient and intelligent object control in complex environments.

[0056] According to a memory-based robot agent control method provided by the present invention, before determining the target object mask and the neighboring environment mask of the target object's neighboring environment in response to receiving a control command for a target object in the current environment, the method further includes: Obtain the RGB and depth images of the current environment; The RGB image is input into the multimodal large model for open vocabulary recognition, and the names of multiple objects in the current environment are obtained from the output of the multimodal large model. Input the name of each object from a set of multiple objects into the open-world object detection model to obtain the bounding box of each object output by the open-world object detection model. The detection bounding box of each object is input into the segmentation model to obtain the segmentation mask of each object output by the segmentation model.

[0057] In this embodiment of the invention, the aim is to use the RGBD camera of the robot agent to perform preliminary semantic understanding of the unstructured environment and extract key operational objects.

[0058] Specifically, the robotic agent acquires RGB and depth images of the current viewpoint (current environment) using an RGBD camera. The RGB images are then input into a pre-trained multimodal large model, and combined with task instructions, the open vocabulary recognition capability of the multimodal large model is used to output a list of noteworthy high-value object names in the scene (e.g., "can", "paper ball", "water stain").

[0059] Subsequently, the extracted object names are input as text prompts into an open-world object detection model (such as GroundingDINO) to obtain the detection box corresponding to each object.

[0060] Next, the detection bounding boxes are used as inputs to the segmentation model (such as the Segment Anything Model, SAM) to generate a precise pixel-level segmentation mask for each object.

[0061] Through the embodiments of the present invention, by fusing RGBD images with multimodal large models, open-world detection and segmentation models, open vocabulary recognition and accurate mask extraction of objects in unstructured environments are achieved, significantly improving the semantic understanding of unknown targets by robot agents.

[0062] According to the present invention, a robot intelligent agent control method based on a memory mechanism determines the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose, including: Based on the target object mask and the depth image of the current environment, combined with preset camera parameters, the 3D point cloud coordinates of the target object in the camera coordinate system are determined: ,in ; Wherein, the 3D point cloud coordinates of the target object in the camera coordinate system are: , The transpose of the matrix is ​​given by the pixel coordinates in the depth image. The corresponding depth value is , and Indicates the camera is in direction and Normalized focal length in direction and Represents the x-coordinate and y-coordinate of the camera's principal point; Determine the extrinsic parameter matrix of the camera relative to the current robot pose; Based on the extrinsic parameter matrix, the coordinates of the 3D point cloud are transformed to the robot coordinate system to obtain the 3D spatial information of the target object in the robot coordinate system. The 3D spatial information includes the coordinates of the geometric center and the size of the 3D bounding box.

[0063] In this embodiment of the invention, the aim is to map two-dimensional semantic information to three-dimensional physical space in order to understand the spatial relationships and physical properties between each object.

[0064] Specifically, using the target object mask, combined with the depth image, camera intrinsic matrix, and current robot pose, the pixel coordinates of the target object mask are back-projected into 3D space. Assume the pixel coordinates of the target object are... The corresponding depth value (Acquired directly from the depth camera), the camera intrinsic parameter matrix is: Then the 3D point cloud coordinates of that point in the camera coordinate system The following can be calculated using the formula for a pinhole camera model: ,in ; Subsequently, the extrinsic parameter matrix of the camera relative to the current robot pose is obtained through hand-eye calibration. The 3D point cloud coordinates of the target object in the camera coordinate system are transformed to the robot coordinate system to obtain the transformed point cloud data. : ; in, This represents the three-dimensional point cloud data of the target object in the robot coordinate system (i.e., the transformed point cloud data). This represents the rotation matrix from the camera coordinate system to the robot coordinate system. This represents the 3D point cloud coordinates of the target object in the camera coordinate system. This represents the translation vector from the camera coordinate system to the robot coordinate system.

[0065] Based on the transformed point cloud data It calculates the geometric center coordinates and 3D bounding box dimensions (length, width, and height) of objects. Simultaneously, by analyzing the relative positions of point clouds of different objects, it generates spatial relationship descriptions in natural language (e.g., "the blue block is on top of the green block"), providing physical constraints for subsequent decision-making.

[0066] For example, the geometric center coordinates of the target object are calculated. These coordinates are obtained by averaging all three-dimensional points in the point cloud, representing the position of the object's centroid. Simultaneously, the three-dimensional bounding box dimensions of the target object are calculated. By analyzing the distribution range of the point cloud along each coordinate axis, the object's physical dimensions, such as length, width, and height, are determined.

[0067] Through the embodiments of the present invention, the robot agent successfully transforms visual perception information into accurate three-dimensional spatial information that can be used for motion planning by performing the three-dimensional reconstruction, coordinate system transformation and physical attribute analysis processes in sequence. This provides a key physical environment understanding foundation for subsequent decision-making and physical interaction.

[0068] According to the present invention, a robot intelligent agent control method based on a memory mechanism retrieves target historical experience from a preset adaptive specification based on a target object mask and a mask of the adjacent environment, including: The union between the target object mask and the adjacent environment mask is input into the contrastive language image pre-training model to obtain the query feature vector output by the contrastive language image pre-training model; Based on the query feature vector and the memory feature vector of historical interaction records in the preset adaptive specification, a comprehensive score is determined for each historical experience: ; in, Indicates the first A comprehensive score based on historical experience. Represents the similarity function. This represents the query feature vector. Indicates the first Each memory feature vector Indicates the first Risk coefficient based on historical experience; The historical experience of the target quantity with the highest comprehensive score is used as the target historical experience.

[0069] In this embodiment of the invention, the core of retrieving historical experience using an adaptive specification lies in the introduction of environmental context and risk weights. The adaptive specification includes a static part (static physical parameters of the robot agent) and a dynamic part (historical interaction records of the robot agent). The retrieval process is as follows: Based on the target object mask and its adjacent environment mask, the union of the two is input into the Contrastive Language–Image Pre-training (CLIP) model to extract query feature vectors. .

[0070] To prioritize the recall of high-risk historical lessons, this invention designs a risk-weighted ranking score calculation formula. For the first [item / parameter] in the adaptive specification... The historical experience, its memory feature vector is The risk factor is The final retrieval ranking score The calculation is as follows: ; Represents cosine similarity. .in accordance with Retrieve the top-k historical experiences with the highest overall scores to ensure that high-risk scenarios (such as "liquid spills") are given priority.

[0071] In this embodiment of the invention, the target object mask is merged with the adjacent environment mask to form a complete scene context mask. This mask not only includes the target object itself, but also covers key environmental information around it, thus more comprehensively representing the visual features of the current operating scene.

[0072] The merged scene context mask is input into a pre-trained contrastive language image pre-training model. Leveraging its powerful visual feature extraction capabilities, the model analyzes the input mask image, generating a high-dimensional numerical vector as a query feature vector. This query feature vector effectively encodes the visual characteristics of the current scene, providing a foundation for subsequent similarity comparisons.

[0073] After extracting the query feature vector, feature matching is performed in the dynamic database of the adaptive specification. Each historical interaction record stored in the adaptive specification contains a corresponding memory feature vector, which is generated in the same way when historical experiences are recorded. The similarity between the current query feature vector and each memory feature vector in the database is calculated. This similarity value reflects the degree of matching between the current scene and historical scenes in terms of visual features.

[0074] To improve the security and usability of search results, a risk-weighted mechanism is introduced. Each historical experience in the adaptive specification is associated with a risk coefficient, which is determined based on the safety level of the historical operation results. The calculated similarity value is multiplied by the corresponding risk coefficient to obtain a weighted comprehensive score. Through this weighting method, historical records containing high-risk operations or failed experiences are prioritized for retrieval, and even if their visual similarity is not the highest, they can still obtain a higher ranking due to their importance.

[0075] All historical experiences are sorted from highest to lowest based on their overall scores, and a specified number of the top-ranked records are selected as target historical experiences. These experiences will serve as important contextual information input into subsequent decision-making processes, helping the robotic agent make safer and more effective decisions in similar scenarios.

[0076] Through the embodiments of the present invention, the robot agent can quickly locate the most relevant and valuable historical information in its own experience base, especially the lessons learned in high-risk scenarios, thereby significantly improving the success rate and safety of task execution.

[0077] In some embodiments, the robot agent combines its perception of the current environment with retrieved historical experience to generate a safe and executable sequence of actions through a multi-expert parallel mechanism. The RGB image of the current viewpoint, the target object mask and its adjacent environment mask, the retrieved historical experience of the target, and the static tool parameters in the adaptive specification are collectively constructed into structured prompts, which are then input into the multimodal large model. This information is then reasoned about in parallel by the multimodal large model from multiple different directions and finally summarized. Feasibility experts determine the feasibility of an operation based on the physical parameters of the tools and the dimensions of the object. Consequence projection experts: predict the potential consequences of actions based on historical experience; Security experts: Assess potential security risks.

[0078] Expert Synergy: Summarize the opinions of various experts to generate a specific sequence of actions that includes tool selection, operation actions, and target object (i.e., target object).

[0079] According to the present invention, a robot intelligent agent control method based on a memory mechanism retrieves target historical experience from a preset adaptive specification based on a target object mask and a mask of the adjacent environment, including: The union between the target object mask and the adjacent environment mask is input into the contrastive language image pre-training model to obtain the query feature vector output by the contrastive language image pre-training model; Based on the query feature vector and the memory feature vector of historical interaction records in the preset adaptive specification, a comprehensive score is determined for each historical experience: ; in, Indicates the first A comprehensive score based on historical experience. Represents the similarity function. This represents the query feature vector. Indicates the first Each memory feature vector Indicates the first Risk coefficient based on historical experience; The historical experience of the target quantity with the highest comprehensive score is used as the target historical experience.

[0080] According to the memory-based robot agent control method provided by the present invention, after inputting the RGB image of the current environment, three-dimensional spatial information, and target historical experience into a multimodal large model for decision-making and obtaining the executable action sequence corresponding to the control command output by the multimodal large model, the method further includes: Identify the end effector corresponding to the executable action sequence; Based on the three-dimensional spatial information of the target object in the robot coordinate system, the target end pose required for the end effector to execute the sequence of executable actions is determined. Based on the inverse kinematics algorithm, the target joint angle vector that enables the end effector to achieve the target end pose is determined.

[0081] In this embodiment of the invention, semantic-level action sequences are transformed into control commands at the lower level of the robot agent. Based on the decision result (i.e., the executable action sequence), the robot agent automatically switches to the appropriate end effector. Based on the acquired three-dimensional spatial information of the target object in the robot coordinate system, the target end-effector pose (matrix) required for the end effector to perform the operation is determined. The joint angle vectors satisfying the target pose are solved using inverse kinematics. .

[0082] According to the present invention, a robot intelligent agent control method based on a memory mechanism, based on an inverse kinematics algorithm, determines the target joint angle vector that enables the end effector to reach the target end pose, including: ; in, Represents the target joint angle vector. Represents the inverse kinematic function. Indicates the target's end-effector pose. This represents the parameter that minimizes the objective function. Represents the forward kinematic function. Represents the weight parameters. This represents the input joint angle vector. This represents the current joint angle vector.

[0083] In this embodiment of the invention, the above-mentioned plan is encapsulated using high-level atomic operations. For example, for the grasping atomic operation, the grasping trajectory is directly generated based on the object's center of mass; for the water absorption atomic operation, a plane equation is fitted based on the point cloud of the object's surface. This generates a coverage path that covers the plane.

[0084] Through the embodiments of the present invention, by integrating multimodal decision results with robot kinematics models, high-level semantic actions are accurately converted into target poses and joint angles of the end effector, realizing end-to-end closed-loop control from perception to execution, and improving the accuracy and reliability of robot agents in operating target objects in complex environments.

[0085] In some embodiments, lifelong learning of the robotic agent is achieved, forming a closed loop of perception-decision-learning.

[0086] After the operation is completed, the robot agent uses an RGBD camera to acquire environmental images again in the same pose. The images before and after the operation are simultaneously input into a multimodal large-scale model for reflection. The model compares the changes in the two images and analyzes the actual effect of the operation (e.g., "the ground becomes clean" or "liquid is spilled"). Based on preset evaluation criteria, the model scores the operation and calculates a risk coefficient.

[0087] The scenario feature vector (extracted by CLIP), operation description, result summary, and risk coefficient of this operation will be recorded as a new experience record and updated into the dynamic database of the adaptive specification for future task retrieval.

[0088] The following describes an example of the robot intelligent agent control method based on the memory mechanism provided by this invention in a practical application. Specific steps include: Step 1: The robot agent uses an RGBD camera to observe the area that needs to be cleaned and inputs the RGB information into a multimodal large model to extract the names of noteworthy high-value objects; Step 2: Input the extracted name into the GroundingDINO model to extract the object detection box corresponding to the name, and then input the detection box into SAM (Segment Anything Model) to obtain the segmentation mask of the corresponding object; Step 3: Calculate the pixel coordinates corresponding to the mask, and combine them with the camera parameters and the robot's pose to obtain the point cloud coordinates of the object relative to the robot's coordinate system. Using these coordinates, we can roughly calculate the object's coordinates and length. Step 4: Based on the SAM segmentation results, take the union of the object mask and the environment mask adjacent to the object, and use it as retrieval information to retrieve several historical experiences with the highest comprehensive scores from the instruction manual. Step 5: Input the image, the relative positional relationships of objects obtained from point cloud computing, and historical experience into the multimodal large model for decision-making, and obtain the sequence of actions to be executed; Step Six: Based on the object's point cloud information, the robot agent calculates the required end-effector pose and generates a sequence of actions. It then switches to the appropriate tool head and executes the operation. Step Six: The robot agent uses the RGBD camera to acquire images again at the same location. It inputs both the previous image and the new image into the multimodal large model, analyzes the actual effect of the operation, summarizes the experience and risk factors, and updates the instruction manual. This forms a loop.

[0089] refer to Figure 2 , Figure 2 This is an example diagram of the robot intelligent agent control method based on memory mechanism provided by the present invention in a practical application.

[0090] Figure 2 The demonstration showcases the control process of a robot intelligent agent based on a memory mechanism: the left side represents the task discovery phase, where the robot perceives the real environment through an RGBD camera, identifies target objects, and extracts key information; the right side represents the planner module, which combines a large model approach to perform semantic analysis of the state before and after the operation, evaluates the feasibility, consequences, and safety of the action based on the instruction manual, and summarizes and generates highly semantic operation instructions; the bottom reflection module analyzes the changes before and after the operation based on a large language model, summarizes causal relationships (such as "tipping over a can will cause liquid to spill" in the example), and updates the instruction manual accordingly, forming an intelligent closed loop from perception, decision-making, execution to continuous learning.

[0091] refer to Figure 3 , Figure 3 This is a schematic diagram of the technical path for retrieval, updating, and specification provided by the present invention.

[0092] Figure 3 Using "bottle" as an example input label, the retrieval process is initiated. This process first calculates similarity from the adaptive instruction manual using a visual matching model, filtering relevant historical memory entries and simultaneously acquiring the objective factual record associated with each memory and a pre-assessed risk coefficient. Subsequently, the system weights and sorts the retrieval results, combining similarity and risk coefficients to calculate a comprehensive score. This prioritizes the most relevant and cautionary "facts n" for use by the decision-making module. Simultaneously, the system has an update mechanism that returns new operational results and experiences as facts to the instruction manual, enabling continuous accumulation and updating of memories, forming a closed-loop experience learning and retrieval system.

[0093] To verify the effectiveness of the memory-based robot intelligent agent control method proposed in this invention in a real physical environment, an experimental platform was built in an actual office setting.

[0094] Experimental environment: The scene included realistic background elements as well as various naturally placed furniture and objects. The objects to be cleaned included crumpled paper, aluminum cans, and liquid stains. Distracting objects such as vases and mobile phones were also placed in the scene to test the robustness of perception.

[0095] Hardware platform: The experiment used a self-developed mobile cleaning robot system. The system consists of an automated guided vehicle chassis, a 6-DOF robotic arm, an RGBD camera, and various automatically switchable end-effector cleaning tools (including a two-finger gripper, a vacuum cleaner, and a water sprayer).

[0096] refer to Figure 4 , Figure 4 This is a schematic diagram of the mobile cleaning robot provided by the present invention, which includes: a 6-DOF robotic arm, a central service module, a mobile chassis, a depth camera, grippers, and a quick-release device.

[0097] Baseline Comparison: To evaluate the performance of the method of this invention, it was compared with three mainstream embodied intelligence methods: (1) React method: The adaptive specification module of this invention was removed, and decision-making was carried out only through the reasoning-action loop; (2) CoT method: The adaptive specification was removed, and the large model was guided to generate decisions through multi-step reasoning prompts; (3) Text-based RAG method: The "environment-aware visual retrieval" of this invention was replaced with "text description-based retrieval", that is, the text description of the image was generated first, and then the historical experience was retrieved through text similarity.

[0098] Evaluation metrics: The experiment mainly focuses on task success rate, false alarm rate / false negative rate, number of invalid decisions (measuring safety and physical feasibility), and number of inference steps (measuring decision efficiency).

[0099] Multiple rounds of real-world scenario tests were conducted under the above experimental setup, and the specific experimental data statistics are shown in Table 1 below.

[0100] Table 1 The data on the left side of this statistical table represents the agent's performance during the initial trial phase. At this stage, the agent's adaptive specification is initially blank, lacking any historical experience reserves. Its decision-making process relies entirely on real-time environmental perception and logical reasoning capabilities, gradually accumulating specification content. The data on the right side represents the agent's performance in subsequent trial phases. The agent's specification content begins with the specification at the end of the initial trial. Arrows indicate the experimental setup on the left side, and experimental results can be directly referenced from the left side.

[0101] Experimental results show that the method proposed in this invention outperforms the baseline method in all key indicators. Specific qualitative analysis is as follows: Significantly improved physical feasibility and safety: Compared to React and CoT methods, this invention significantly reduces invalid decisions by introducing an adaptive specification that includes the tool's physical parameters. For example, when faced with a large cardboard box exceeding the gripper's range, baseline methods often attempt to force a grip, resulting in failure. In contrast, this invention's method can identify physical limitations based on the parameters recorded in the specification and refuse to perform the operation, or proactively switch to a more suitable tool. This demonstrates the importance of integrating the decision-making module with the robot's own capabilities for physical world tasks.

[0102] Effectiveness of Risk-Weighted Retrieval: Compared to the text-based RAG method, the environment-aware visual retrieval mechanism employed in this invention exhibits stronger robustness. Text descriptions often lose key spatial details in a scene, while visual feature vectors that preserve the environmental context can more accurately recall similar scenes. More importantly, the risk-weighted mechanism ensures that high-risk historical lessons (such as "a can tipping over and causing liquid spillage") are prioritized for retrieval. Experiments showed that when the robot encounters a similar high-risk scenario again, it can successfully recall previous failures and proactively choose safer tools, thereby avoiding safety accidents.

[0103] Lifelong Learning and Efficiency Optimization: Experimental data shows that as the number of experimental rounds increases, the number of reasoning steps in the method of this invention decreases, while the success rate continues to rise. This indicates that the robot successfully internalizes the experience of each operation (whether successful or unsuccessful) into the knowledge in the instruction manual by reflecting on and summarizing the steps. In subsequent tasks, the robot no longer needs to perform tedious reasoning from scratch, but can directly draw on historical experience to generate the optimal strategy, demonstrating that the algorithm has the ability to continuously learn and evolve from real-world feedback.

[0104] In summary, by constructing an adaptive specification and combining it with a risk-weighted retrieval mechanism, this invention effectively solves the problem of traditional embodied intelligent agents ignoring their own physical limitations, and achieves safer, more efficient, and evolutionary robot task planning.

[0105] The memory-based robot intelligent agent control device provided by the present invention will be described below. The memory-based robot intelligent agent control device described below can be referred to in correspondence with the memory-based robot intelligent agent control method described above.

[0106] refer to Figure 5 , Figure 5 This is a schematic diagram of a module of the robot intelligent agent control device based on a memory mechanism provided by the present invention.

[0107] The mask module 501 is used to determine the target object mask and the adjacent environment mask of the target object's adjacent environment in response to receiving a control command for a target object in the current environment. The determination module 502 is used to determine the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose. The experience module 503 is used to retrieve the target's historical experience from a preset adaptive specification based on the target object mask and the adjacent environment mask. The adaptive specification includes: the static physical parameters of the robot agent and the historical interaction records of the robot agent. Action module 504 is used to input the RGB image of the current environment, three-dimensional spatial information and target historical experience into the multimodal large model for decision-making, and obtain the executable action sequence corresponding to the control command output by the multimodal large model.

[0108] Specifically, the robot intelligent agent control device based on the memory mechanism provided by the present invention can realize all the method steps implemented in the above-mentioned robot intelligent agent control method embodiment based on the memory mechanism, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0109] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communications bus 640. The processor 610 can call logic instructions in the memory 630 to execute a memory-based robot agent control method. This method includes: in response to receiving a control command for a target object in the current environment, determining the target object mask and the adjacent environment mask of the target object; based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose, determining the three-dimensional spatial information of the target object in the robot coordinate system; based on the target object mask and the adjacent environment mask, retrieving the target's historical experience from a preset adaptive specification, wherein the adaptive specification includes: the robot agent's static physical parameters and the robot agent's historical interaction records; inputting the RGB image of the current environment, the three-dimensional spatial information, and the target's historical experience into a multimodal large model for decision-making, obtaining an executable action sequence corresponding to the control command output by the multimodal large model.

[0110] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the memory-based robot agent control method provided by the above methods. The method includes: in response to receiving a control command for a target object in the current environment, determining a target object mask and a neighboring environment mask of the target object; determining the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose; retrieving the target's historical experience from a preset adaptive specification based on the target object mask and the neighboring environment mask, wherein the adaptive specification includes: static physical parameters of the robot agent and historical interaction records of the robot agent; inputting the RGB image of the current environment, the three-dimensional spatial information, and the target's historical experience into a multimodal large model for decision-making, and obtaining an executable action sequence corresponding to the control command output by the multimodal large model.

[0112] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the memory-based robot agent control method provided by the above methods. The method includes: in response to receiving a control command for a target object in the current environment, determining a target object mask and a neighboring environment mask of the target object's neighboring environment; determining the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose; retrieving the target's historical experience from a preset adaptive specification based on the target object mask and the neighboring environment mask, wherein the adaptive specification includes: static physical parameters of the robot agent and historical interaction records of the robot agent; inputting the RGB image of the current environment, the three-dimensional spatial information, and the target's historical experience into a multimodal large model for decision-making, and obtaining an executable action sequence corresponding to the control command output by the multimodal large model.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling a robot intelligent agent based on a memory mechanism, characterized in that, Applied to robotic intelligent agents, including: In response to receiving a control command for a target object in the current environment, the target object mask and the neighboring environment mask of the target object's neighboring environment are determined. Based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose, the three-dimensional spatial information of the target object in the robot coordinate system is determined. Based on the target object mask and the adjacent environment mask, the target historical experience is obtained by retrieving from the preset adaptive specification, wherein the adaptive specification includes: the static physical parameters of the robot agent and the historical interaction records of the robot agent; The RGB image of the current environment, the three-dimensional spatial information, and the target historical experience are input into the multimodal large model for decision-making, and the executable action sequence corresponding to the control command output by the multimodal large model is obtained.

2. The robot intelligent agent control method based on memory mechanism according to claim 1, characterized in that, Before determining the target object mask and the neighbor environment mask of the target object's neighbor environment in response to receiving a control command for a target object in the current environment, the method further includes: Obtain the RGB and depth images of the current environment; The RGB image is input into a multimodal large model for open vocabulary recognition, and the names of multiple objects in the current environment are obtained from the output of the multimodal large model. The name of each of the multiple objects is input into the open-world object detection model to obtain the detection bounding box of each object output by the open-world object detection model; The detection bounding box of each object is input into the segmentation model to obtain the segmentation mask of each object output by the segmentation model.

3. The robot intelligent agent control method based on memory mechanism according to claim 1, characterized in that, The process of determining the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose, includes: Based on the target object mask and the depth image of the current environment, combined with preset camera parameters, the 3D point cloud coordinates of the target object in the camera coordinate system are determined: ,in ; Wherein, the three-dimensional point cloud coordinates of the target object in the camera coordinate system are: , Let be the transpose matrix, and let the pixel coordinates in the depth image be . The corresponding depth value is , and Indicates that the camera is in direction and Normalized focal length in direction and This represents the x-coordinate and y-coordinate of the principal point of the camera; Determine the extrinsic parameter matrix of the camera relative to the current robot pose; Based on the extrinsic parameter matrix, the coordinates of the three-dimensional point cloud are transformed to the robot coordinate system to obtain the three-dimensional spatial information of the target object in the robot coordinate system. The three-dimensional spatial information includes the geometric center coordinates and the three-dimensional bounding box size.

4. The robot intelligent agent control method based on memory mechanism according to claim 1, characterized in that, The step of retrieving target historical experience from a preset adaptive specification based on the target object mask and the adjacent environment mask includes: The union between the target object mask and the adjacent environment mask is input into the contrastive language image pre-training model to obtain the query feature vector output by the contrastive language image pre-training model; Based on the query feature vector and the memory feature vector of historical interaction records in the preset adaptive specification, a comprehensive score for each historical experience is determined: ； in, Indicates the first A comprehensive score based on historical experience. Represents the similarity function. This represents the query feature vector. Indicates the first Each memory feature vector Indicates the first Risk coefficient based on historical experience; The historical experience of the highest number of targets with the highest comprehensive scores is used as the target historical experience.

5. The robot intelligent agent control method based on memory mechanism according to claim 1, characterized in that, After inputting the RGB image of the current environment, the three-dimensional spatial information, and the target historical experience into the multimodal large model for decision-making, and obtaining the executable action sequence corresponding to the control command output by the multimodal large model, the method further includes: Determine the end effector corresponding to the executable action sequence; Based on the three-dimensional spatial information of the target object in the robot coordinate system, the target end-effector pose required for the end effector to execute the executable action sequence is determined. Based on the inverse kinematics algorithm, the target joint angle vector that enables the end effector to reach the target end pose is determined.

6. The robot intelligent agent control method based on memory mechanism according to claim 5, characterized in that, The determination of the target joint angle vector that enables the end effector to reach the target end pose, based on the inverse kinematics algorithm, includes: ； in, Represents the target joint angle vector. Represents the inverse kinematic function. This indicates the pose of the target end effector. This represents the parameter that minimizes the objective function. Represents the forward kinematic function. Represents the weight parameters. This represents the input joint angle vector. This represents the current joint angle vector.

7. A robot intelligent agent control device based on a memory mechanism, characterized in that, include: A mask module is used to determine the target object mask and the adjacent environment mask of the target object's adjacent environment in response to receiving a control command for a target object in the current environment. The determination module is used to determine the three-dimensional spatial information of the target object in the robot coordinate system based on the target object mask and the depth image of the current environment, combined with preset camera parameters and the current robot pose. The experience module is used to retrieve target historical experience from a preset adaptive specification based on the target object mask and the adjacent environment mask. The adaptive specification includes: the static physical parameters of the robot agent and the historical interaction records of the robot agent. The action module is used to input the RGB image of the current environment, the three-dimensional spatial information, and the target historical experience into the multimodal large model for decision-making, and obtain the executable action sequence corresponding to the control command output by the multimodal large model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the robot intelligent agent control method based on the memory mechanism as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot intelligent agent control method based on the memory mechanism as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot intelligent agent control method based on the memory mechanism as described in any one of claims 1 to 6.

Images