Multi-scene mechanical arm sensing and control method and system based on hierarchical decoupling architecture

Abstract

The invention discloses a multi-scene mechanical arm sensing and control method and system based on a hierarchical decoupling architecture, and relates to the technical field of mechanical arm sensing and control. According to the method, the expandability and compatibility of the system are remarkably improved by constructing a layered architecture in which the sensing layer, the planning layer and the control layer are completely decoupled. According to the invention, the camera abstraction layer is utilized to shield the driving difference of heterogeneous hardware, and the standardized data stream is utilized to realize the modular communication among the hierarchies. Therefore, on the premise that kernel codes of a control layer are not modified, a developer can replace a visual model with higher precision at will, different types of image input sources are accessed or new task logic is expanded, the secondary development cost and maintenance difficulty of the system are greatly reduced, and the development efficiency is improved. The problems that an existing mechanical arm system is high in software and hardware coupling degree and difficult to adapt to the rapid iteration requirement are solved.

Description

Technical Field

[0001] This invention relates to the field of robotic arm sensing and control technology, specifically to a multi-scenario robotic arm sensing and control method and system based on a hierarchical decoupling architecture. Background Technology

[0002] With the rapid development of artificial intelligence and embodied intelligence technologies, robotic arm vision grasping systems have been widely used in various fields such as industrial sorting, home services, logistics warehousing, and education and scientific research. Traditional robotic arm control systems usually achieve grasping through teach pendant programming or preset fixed trajectories. However, when facing unstructured environments (such as objects with random placement and varying postures), it is often necessary to combine computer vision technology to achieve autonomous positioning and grasping.

[0003] Although existing robotic arm vision grasping technology has made some progress, the following significant technical defects and problems still exist in practical applications and secondary development:

[0004] Most existing robotic arm vision systems typically adopt a "siloed" or "monolithic" development model. That is, visual perception algorithms (such as object detection), business decision logic (such as grasping order), and underlying motion control code (such as motor drive) are tightly intertwined, resulting in severe hard-coding.

[0005] This tightly coupled architecture makes the system extremely difficult to maintain and expand. First, hardware replacement is difficult and compatibility is poor. If a user needs to replace a local USB camera with a webcam (RTSP stream) or replace the robotic arm itself, it often involves modifying the entire chain of code from the underlying driver to the upper-level logic, resulting in a huge workload. Second, algorithm upgrades are limited. When it's necessary to upgrade the visual model from an older version (such as YOLOv5) to a newer version, or replace a general detection algorithm with a specific grasping detection algorithm (such as GraspNet), the lack of a unified data interface standard forces developers to refactor a large amount of data processing logic. Simultaneously, task reusability is poor. For example, code developed for a "color sorting" scenario cannot be directly used for "chess playing" or "semantic grasping" scenarios, leading to low development efficiency and extremely low code reusability.

[0006] In the field of object detection, current technologies for visual grasping tasks often use axis-aligned bounding boxes (AABBs) to locate objects. However, in real-world scenarios, objects (such as screws, long blocks, pens, etc.) are often placed at arbitrary angles. Using AABBs results in the detection area containing a large amount of background redundancy and makes it impossible to accurately obtain the object's rotation angle (yaw angle). When robotic arms grasp such objects, they can often only grasp at a fixed angle, which can easily lead to slippage or collisions with surrounding objects. Furthermore, existing multi-camera systems often rely on complex 3D hand-eye calibration, which is computationally intensive and cumbersome to deploy, making it difficult to meet the requirements for lightweight and rapid deployment.

[0007] In robotic arm control, traditional inverse kinematics (IK) algorithms are typically based on mathematical solutions with rigid constraints. The goal is to find a solution that perfectly satisfies both the target position (x, y, z) and the target orientation (roll, pitch, yaw). However, in real-world multi-scenario applications, "soft constraint" requirements often exist. For example, when the target object is at the edge of the robotic arm's workspace, the theoretically perfect orientation may be unattainable (unsolvable), but in practice, a slight tilt of the gripper is sufficient for grasping; or in certain handling tasks, high positional accuracy is required, while low orientation accuracy is acceptable. Existing control systems lack the flexibility of inverse kinematics solutions, failing to flexibly adjust the priority of position and orientation according to task requirements under specific conditions (such as at the edge of the workspace or near singularities), easily leading to control failures or unsmooth movements. The lack of a weighted dynamic adjustment mechanism makes the robotic arm prone to planning failures or motion jitter near extreme positions or singularities, significantly limiting its effective working range.

[0008] Furthermore, traditional robotic arm systems are mostly controlled based on predefined category labels (such as "grab the object with category ID 1"). These systems cannot understand human natural language commands (such as "hand me that red, long, thin thing"), nor can they handle objects of unknown categories (open-voabulary). With the rise of Multimodal Large Models (MLLMs), how to organically combine the semantic understanding capabilities of large models with the physical control system of robotic arms, and achieve flexible configuration of task logic, is a major technological gap currently facing the field of embodied intelligence.

[0009] Therefore, there is an urgent need for a robotic arm perception and control system based on a hierarchical decoupling architecture that supports configuration-driven and multi-algorithm hot-swappable architecture, in order to address the shortcomings of existing technologies in terms of scalability, adaptability to irregular shape grasping, and intelligent interaction. To this end, we propose a novel multi-scenario robotic arm perception and control method and system based on a hierarchical decoupling architecture. Summary of the Invention

[0010] The purpose of this invention is to provide a multi-scenario robotic arm perception and control method and system based on a layered decoupled architecture. It constructs a system architecture in which the perception layer, planning layer and control layer are decoupled from each other. Each layer transmits and interacts with each other through a standardized data flow in a one-way manner, so as to solve the problem that the high coupling between vision algorithm, task logic and motion control in the prior art leads to the difficulty of system expansion and poor adaptability to multiple scenarios.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a multi-scenario robotic arm perception and control method based on a layered decoupled architecture, wherein the method is based on a system architecture in which the perception layer, planning layer, and control layer are decoupled, and includes the following steps:

[0012] S10, System Initialization and Establishment of Perception Data Stream: After the system starts, communication channels are established between each layer. The perception layer initializes the visual sensor through the camera abstraction layer, collects environmental images or video stream data, and encapsulates it into a standardized perception data stream for transmission to the planning layer, providing raw input for subsequent processing.

[0013] S20, Multi-scenario algorithm reasoning and decision generation: The planning layer receives the perception data stream, calls the corresponding vision and decision algorithm modules according to the currently set task scenario, and obtains the feature information or decision results of the target on the image plane; the task scenario includes at least one of the following: item classification and grasping scenario, chess game scenario, and semantic interaction scenario.

[0014] S30, Spatial mapping and standardized pose calculation: The planning layer inputs the target feature information or decision results obtained in step S20 into the two-dimensional calibration adapter, uses the pre-calibrated homography matrix to perform coordinate transformation, maps the pixel coordinate system of the image plane to the physical coordinate system of the robot arm's base, generates standardized pose data containing spatial position coordinates and end-effector posture angles, and sends the data to the control layer.

[0015] S40, Weighted Inverse Kinematics Control and Action Execution: The control layer receives the standardized pose data and generates action commands; the weighted inverse kinematics solver assigns corresponding position weights and attitude weights according to the different requirements of the current task for position accuracy and attitude accuracy, solves the combination of robot arm joint angles, and drives the robot arm to execute actions after trajectory smoothing interpolation.

[0016] Preferably, in step S10, the perception layer provides a camera abstraction interface; this interface encapsulates the underlying drivers of different image acquisition devices, dynamically binds specific driver implementation classes by reading fields in the configuration file, and provides a unified data acquisition method to enable switching between local camera input and network video stream input.

[0017] Preferably, in step S20, when in an item classification or grasping scenario, the visual algorithm module called by the planning layer is a rotating target detection model; the prediction result output by the rotating target detection model includes the center point pixel coordinates, length and width dimensions of the target, and the rotation angle relative to the image coordinate system; in step S30, the planning layer maps the rotation angle to the rotation attitude angle of the robotic arm end effector through coordinate system transformation, thereby realizing adaptive grasping of objects placed at any angle.

[0018] Preferably, the rotating target detection model uses the YOLOv11-OBB network or a variant thereof.

[0019] Preferably, in step S20, when in a semantic interaction scenario, the planning layer activates the large model adapter; the large model adapter receives the user's natural language instructions and calls a local or cloud-based multimodal large language model, combines the perceptual data stream to perform reference expression understanding or open set target detection, and outputs the bounding box or center coordinates of the target object in the image.

[0020] Preferably, in step S30, the homography matrix is ​​a 3×3 matrix obtained by collecting the pixel coordinates of multiple pairs of feature points in the image and the corresponding point coordinates in physical space, and solving it using the least squares method; the specific calculation process of the coordinate transformation is as follows: the pixel coordinates output by the visual algorithm are linearly transformed into physical coordinates through matrix multiplication, and a preset height offset is superimposed in the Z-axis direction.

[0021] Preferably, in step S40, the weighted inverse kinematics solver uses a sequential least squares programming algorithm; the objective function constructed by the system includes a position error term and an attitude error term, and weight coefficients are assigned to them respectively; when the target object is located at the edge of the workspace of the robotic arm or there is a risk of kinematic singularity, the solver calculates the joint angle that prioritizes ensuring the end of the robotic arm reaches the target position by reducing the attitude weight and increasing the position weight.

[0022] This invention also provides a multi-scenario robotic arm perception and control system based on a hierarchical decoupling architecture to implement the above method. The system includes:

[0023] The perception subsystem is used to establish communication connections with external vision sensors, shielding the differences in underlying hardware through the camera abstraction interface, and acquiring and outputting standardized image or video stream data in real time.

[0024] The planning subsystem is used to receive the data output by the perception subsystem, load and run the vision and decision algorithm module according to the configuration file, and use a two-dimensional calibration adapter to map the pixel features output by the algorithm into standardized pose data in the physical coordinate system; the vision and decision algorithm module includes a rotating target detection model, a large model adapter, and a game decision model.

[0025] The control subsystem is used to receive the standardized pose data, calculate the joint angles using a weighted inverse kinematics solver, and generate smooth motion commands through a trajectory smoothing interpolation module to drive the robotic arm to perform actions.

[0026] Preferably, the planning subsystem includes a configuration management module, a multi-scenario algorithm integration module, and a two-dimensional calibration and adaptation module. The configuration management module is used to parse the global configuration file, extract task scenario parameters and algorithm weight paths, and schedule corresponding algorithm resources. The multi-scenario algorithm integration module integrates a YOLO rotation detection unit, an LLM semantic parsing unit, and a game decision-making unit, and activates specific units according to the instructions of the configuration management module. The two-dimensional calibration and adaptation module stores a pre-calibrated homography matrix, which is used to perform linear transformations from the pixel coordinate system to the physical coordinate system of the robotic arm base.

[0027] Preferably, the trajectory smoothing interpolation module in the control subsystem is configured to: receive discrete joint angles obtained from inverse kinematics, use linear interpolation or S-shaped velocity planning algorithm to generate a series of smooth transition intermediate control points between the current angle and the target angle, and send the intermediate control points to the underlying driver.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. This invention significantly improves the scalability and compatibility of the system by constructing a layered architecture that completely decouples the perception layer, planning layer, and control layer. In traditional technologies, vision algorithms are often directly embedded in the underlying control code, requiring the entire system to be rebuilt when changing camera hardware or upgrading algorithms. This invention utilizes a camera abstraction layer to shield the driver differences of heterogeneous hardware and uses standardized data flows to achieve modular communication between layers. This means that developers can freely replace higher-precision vision models, connect different types of image input sources, or extend new task logic without modifying the control layer kernel code, greatly reducing the secondary development cost and maintenance difficulty of the system, and solving the problem of high hardware-software coupling and difficulty in adapting to rapid iteration requirements in existing robotic arm systems.

[0030] 2. This invention integrates multiple algorithm modules, including object detection, game theory decision-making, and a multimodal large model adapter, at the planning layer. This enables the same robotic arm system to perform both deterministic rule-based object classification and chess-playing tasks, as well as semantic understanding-based grasping tasks based on unstructured instructions. In particular, the introduction of the multimodal large model adapter gives the system the ability to understand fuzzy natural language instructions, achieving a leap from traditional instruction execution to perceptual and cognitive execution, greatly improving the naturalness of human-computer interaction and the system's intelligence level. This invention achieves unified support and flexible switching between multiple complex application scenarios within a single system.

[0031] 3. This invention effectively solves the challenges of accurate grasping and motion stability in unstructured environments by employing a two-dimensional calibration adapter and a weighted inverse kinematics optimization strategy. The two-dimensional spatial mapping using a homography matrix requires less computation and has lower operational complexity compared to complex three-dimensional hand-eye calibration, enabling rapid coordinate system alignment in desktop scenarios. Simultaneously, the weighted inverse kinematics solver allows the system to adjust the weights of position and attitude based on task priority. When the target object is located at the edge of the workspace, the system can sacrifice some attitude accuracy to prioritize position attainment, thus avoiding grasping failures sometimes caused by singularities or unreachability. This significantly expands the effective working range of the robotic arm and improves the grasping success rate.

[0032] 4. This invention optimizes the motion execution quality of the robotic arm through trajectory smoothing interpolation technology and a standardized pose control interface. By performing smoothing interpolation on the joint angles calculated from inverse kinematics at the control layer, mechanical jitter that may be caused by discrete commands is eliminated. This makes the robotic arm more stable and continuous when performing high-speed grasping or precision placement actions, ensuring smooth motion transitions during multi-scenario task switching and improving the overall system's operational efficiency and stability. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the data transmission process of the method described in this invention;

[0034] Figure 2 This is a schematic diagram illustrating the operating principle of the system described in this invention in an item classification scenario;

[0035] Figure 3 This is an F1-Confidence curve of the target detection model used in the item classification scenario of the system described in this invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Example 1

[0038] Please see Figure 1 - Figure 3 This invention provides a multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture, comprising the following steps:

[0039] S10, System Initialization and Establishment of Perception Data Stream: After the system starts, communication channels are established between each layer. The perception layer initializes the visual sensor through the camera abstraction layer, collects environmental images or video stream data, and encapsulates it into a standardized perception data stream for transmission to the planning layer, providing raw input for subsequent processing.

[0040] In step S10, the perception layer includes a camera abstraction interface; this interface encapsulates the underlying drivers for different image acquisition devices (such as OpenCV VideoCapture, RealSense SDK, etc.). Specific driver implementation classes are dynamically bound by reading fields from the configuration file, and a unified `get_frame()` data acquisition method is provided to enable switching between local camera input and network video stream input. This design allows the system to seamlessly switch between different perception information input devices, such as local USB camera input, depth camera input, or network video stream input, without modifying the code, making it suitable for different deployment architectures, including local computing and cloud computing.

[0041] S20, Multi-scenario algorithm reasoning and decision generation: The planning layer receives the perception data stream, calls the corresponding vision and decision algorithm modules according to the currently set task scenario, and obtains the feature information or decision results of the target on the image plane; the task scenario includes at least one of the following: item classification and grasping scenario, chess game scenario, and semantic interaction scenario.

[0042] For example, in an item classification or grasping scenario, the object detection model is called to identify the object's features and extract its pixel region and rotation angle on the image plane; in a chess-playing scenario, the game decision model is called to calculate the move position; in a semantic interaction scenario, a local or cloud-based multimodal large model is connected through a large model adapter to parse the user's natural language commands and locate the target object accordingly.

[0043] Furthermore, in step S20, when in an item classification or grasping scenario, the visual algorithm module invoked by the planning layer is a rotating target detection model, which employs a YOLOv11-OBB network or a variant thereof. Unlike traditional horizontal bounding box detection, the prediction result output by the rotating target detection model includes the pixel coordinates of the target's center point. Dimensions (length and width) and rotation angle relative to the image coordinate system While mapping pixel coordinates to physical coordinates, the system also rotates the pixel coordinates by the specified angle. The orientation of the long side of the object, obtained from its length and width dimensions, is mapped to the rotational attitude angle (Yaw) of the end effector of the robotic arm through coordinate system transformation. This enables adaptive grasping of long or asymmetrical objects placed at arbitrary angles, solving the problem of inaccurate grasping posture caused by traditional horizontal box detection.

[0044] In semantic interaction scenarios, the planning layer activates the large model adapter. The large model adapter receives the user's natural language commands (such as "Give me the leftmost red block") and invokes a local or cloud-based multimodal large language model to achieve visual localization. The large model combines the input image data with the user's commands, using reference expression comprehension or open-set object detection to output the bounding box or center coordinates of the target object in the image. The bounding box information output by the large model is passed to the 2D calibration adapter, converted into a physical grasping pose, thereby achieving general semantic grasping without pre-training for specific categories, as well as grasping based on complex logical reasoning in real-world scenarios.

[0045] S30, Spatial Mapping and Standardized Pose Calculation: The planning layer inputs the target feature information or decision results obtained in step S20 into the two-dimensional calibration adapter. The two-dimensional calibration adapter adopts a planar mapping method based on the homography matrix. It uses the pre-calibrated homography matrix to perform coordinate transformation, mapping the pixel coordinate system of the image plane to the physical coordinate system of the robot arm's base, generating standardized pose data containing spatial position coordinates and end-effector posture angles, and sending the data to the control layer.

[0046] The homography matrix is ​​a 3×3 matrix obtained by collecting the pixel coordinates of multiple pairs of feature points in the image and their corresponding coordinates in physical space, and then solving for them using the least squares method. The specific calculation process of the coordinate transformation is as follows: the pixel coordinates output by the visual algorithm are linearly transformed into physical coordinates through matrix multiplication, and a preset height offset is superimposed in the Z-axis direction. The planning layer maps the rotation angle to the rotation attitude angle of the robotic arm's end effector through coordinate system transformation, realizing adaptive grasping of objects placed at arbitrary angles.

[0047] Specifically, for scenarios where the robotic arm operates on a fixed plane (such as a desktop), the system establishes a perspective transformation relationship between the image pixel coordinate system and the physical coordinate system of the robotic arm base. By collecting the pixel coordinates of multiple pairs of specific points in the image and their corresponding coordinates in physical space, the least squares method is used to solve for the perspective transformation. homography matrix During system operation, the planning layer uses matrix multiplication to process the pixel coordinates output by the visual algorithm. Linear transformation to physical coordinates and in The method superimposes a preset height offset along the axis. Compared to complex hand-eye matrix calibration, this method requires less computation, has a simpler and more efficient calibration process, and supports rapid adaptation to cameras with different viewpoints.

[0048] S40, Weighted Inverse Kinematics Control and Action Execution: The control layer receives the standardized pose data and generates action commands; the weighted inverse kinematics solver assigns corresponding position weights and attitude weights according to the different requirements of the current task for position accuracy and attitude accuracy, solves the combination of joint angles of the robotic arm, and then smooths the generated joint motion sequence through the trajectory smoothing interpolation module to eliminate motion jitter and drive the robotic arm to perform actions, namely grasping, placing or dropping actions.

[0049] The weighted inverse kinematics solver employs a sequential least squares programming algorithm; the objective function constructed by the system includes a position error term. And attitude error term Er, and assign weight coefficients to them respectively. and When the target object is located at the edge of the robot arm's workspace or there is a risk of kinematic singularities, the solver reduces the pose weights. And increase position weight Calculate the priority to ensure the robotic arm's end effector reaches the target position. The joint angle allows for a certain deviation between the end effector posture and the target posture (e.g., tilting the gripper), thereby maximizing the effective working range of the robotic arm.

[0050] The principle and effect of the above technical solution are as follows: Through the process from S10 to S40, this invention realizes a closed-loop control flow that is completely decoupled between "perception-planning-control". The perception layer shields hardware differences, the planning layer focuses on handling complex visual and decision-making logic (such as YOLO or large models), and the control layer focuses on motion calculation and smooth execution. This layered design enables the system to flexibly adapt to various scenarios, from simple color sorting to complex semantic understanding, and has extremely high hardware compatibility and scalability.

[0051] Application Example 1

[0052] In one embodiment, S10 includes:

[0053] S101 reads the global configuration file and parses the parameters. Upon system startup, it first loads the configuration file (e.g., config.yaml) in the root directory. This file defines the system's operating parameters, including hardware parameters such as the robotic arm's serial port number (e.g., / dev / ttyUSB0 or ​​COM3), camera index number, or network stream address (camera_ip: URL / IP); model parameters such as the path to the object detection weight file (e.g., yolo11n-obb.pt) and confidence threshold (conf_thres); and scene-related parameters such as the placement location of each item category.

[0054] S102, Instantiate the perception driver. Based on the `camera_ip` field in the configuration file, the camera abstraction interface (PAI) of the perception layer dynamically binds to the specific driver class. If the field is empty, the local camera driver (based on OpenCV VideoCapture) is instantiated; otherwise, the remote camera driver is instantiated. After initialization, this interface exposes a unified `get_frame()` method for upper layers to call and obtain image or depth information.

[0055] The principle and effect of the above technical solution are as follows: by adopting configuration-driven and interface abstraction design, "software-defined hardware" is realized. Developers do not need to modify a single line of code, but only need to modify the YAML configuration file to seamlessly switch between different hardware environments (such as switching from local testing to remote deployment), which greatly reduces deployment and debugging costs.

[0056] In one embodiment, S20 includes:

[0057] S201, Rotational Object Detection Based on YOLOv11-OBB. When the task scenario is "item classification," the planning layer loads the YOLOv11-OBB model. The acquired images are input into the model for inference, and the model outputs a prediction tensor containing the pixel coordinates of the target's center point. Length and width dimensions Confidence level and rotation angle relative to the horizontal axis of the image coordinate system (Scope) ).

[0058] S202, based on semantic parsing of a multimodal large model. When the task scenario is "large model semantic capture", the planning layer starts the large model adapter (LLM Adapter). Step A: Receive the user's voice command (e.g., "Give me the long red block") and convert it to text. Step B: Send the current frame image and text command simultaneously to the multimodal large model API (e.g., Qwen-VL or GPT-4V). Step C: The large model returns the bounding box or center coordinates of the target object in the image through reference expression understanding (REC) capabilities.

[0059] The principle and effect of the above technical solution are as follows: it combines the high-precision rotation perception of YOLOv11-OBB with the strong semantic understanding capability of multimodal large models. The former solves the problem that traditional horizontal boxes cannot adapt to the grasping posture of irregular objects (such as slanted pens or screws), and realizes adaptive angle grasping; the latter breaks the limitation of predefined categories, enabling the robotic arm to understand natural language instructions and realize open-world interactive grasping.

[0060] In one embodiment, S30 includes:

[0061] S301, Perform hand-eye calibration data acquisition based on human-computer interaction. Step A (Feature point selection): Prepare a marker point with obvious visual features (such as a small screw or a specific crosshair) on the working plane of the robotic arm. The operator clicks on the location of the marker point using the mouse in the real-time camera view displayed by the system, and the system records the pixel coordinates of the point. Step B (Physical Point Alignment): The operator controls the movement of the robotic arm to precisely move the center point of the end effector (gripper) directly above the physical location of the aforementioned marked point. During this process, the static link of the gripper must remain perpendicular to the tabletop, ensuring that the X and Y coordinates of the gripper's root and end effector are consistent, eliminating horizontal projection errors caused by height differences. Step C (Data Recording): Once the robotic arm end effector is aligned, the operator triggers the recording command (pressing the spacebar). The system reads and records the current physical coordinates of the robotic arm end effector. Step D (Loop Acquisition): Change the position of the marker point on the desktop and repeat steps A to C. To ensure calibration accuracy, at least 4 sets (preferably 10 sets) of corresponding pixel coordinate and physical coordinate data pairs need to be acquired.

[0062]

[0063] Once the data collection is complete, trigger the end command (e.g., press the ESC key).

[0064] S302, Calculate and store the homography matrix. The system uses multiple data pairs acquired in step S301 and employs the least squares method to solve for the perspective transformation relationship between the image pixel coordinate system and the robot arm base coordinate system, obtaining a 3×3 homography matrix. (Homography Matrix)

[0065]

[0066] Where s is the homogeneous coordinate scaling factor, calculated The value is set to 1. This matrix describes the mapping relationship between the two-dimensional camera plane and the two-dimensional robotic arm base plane (with the Z-axis representing the constant height of the desktop), and the calculation results are automatically saved to the local configuration file.

[0067] S303, Runtime Space Mapping and Pose Generation. When the system executes the grasping task, the planning layer reads the homography matrix generated in step S302. When the vision algorithm outputs the pixel coordinates of the target At that time, the system performs matrix multiplication:

[0068]

[0069] Normalize the homogeneous coordinate scaling factor s to obtain the physical coordinates. :

[0070]

[0071]

[0072] At the same time, the system determines the grab height based on the configured z_offset (height offset). and the rotation angle recognized by vision Combined with length and width dimensions Calculate the orientation angle of the longer side of the object and define the coordinate system of the robotic arm's end effector, mapping it to the Yaw axis angle of the robotic arm's end effector. Define the coordinates of the two endpoints of the longer side of the object as... The specific formula is:

[0073]

[0074]

[0075] Finally, standardized pose data is generated. Send to the control layer.

[0076] The principle and effect of the above technical solution are as follows: For common desktop operation scenarios, the homography matrix is ​​used to replace the complex three-dimensional eye-in-hand calibration, which greatly simplifies the calibration process (only a few points need to be clicked), reduces the amount of calculation and operation complexity, and can effectively correct camera distortion and perspective deviation, establish the mapping relationship between the camera coordinate system and the robotic arm coordinate system, and realize the fast and accurate mapping from 2D image to 3D physical space.

[0077] In one embodiment, S40 includes:

[0078] S401, Construct a cost function based on weighted error. The system defines a cost function to measure the difference between the current state of the robotic arm and the target state. :

[0079]

[0080]

[0081] in Where n represents the configuration space of the robotic arm, and n represents the degrees of freedom excluding the gripper, in this embodiment... ; This is the set of feasible configurations. This represents the positive kinematic transformation of the robotic arm. This represents the reachable region in the robotic arm's task space. The function first performs forward kinematics calculations, taking the current guessed joint angles as input. The pose matrix of the current end effector is calculated using the forward kinematics algorithm of the robotic arm's kinematic chain. , and then With the target pose matrix Compare the two pose matrices. Calculate the Euclidean distance between the translation vectors of the two pose matrices.

[0082]

[0083] Then, the geodesic distance is obtained through the error quaternion.

[0084]

[0085] The magnitude of the rotation difference between the two pose matrices is calculated, and finally, based on the pre-specified position weights... and pose weights Calculate the total cost, i.e., the objective function. :

[0086]

[0087] S402 executes a numerical optimization solution based on SLSQP. Sequential Least Squares Programming (SLSQP) is used as the optimizer to obtain the initial guesses of the joint angles of the robotic arm. Set to the zero vector or the current joint angle. The optimizer uses... Using the objective function and the physical limits of the robotic arm joints as constraints, the joint angles are iteratively adjusted. Until the total cost is found Minimize the optimal solution ,Right now:

[0088]

[0089] After convergence This is output as the final target joint angle sequence.

[0090] S403, trajectory smoothing interpolation. This is used to obtain the target joint angle. Then, the system determines the current joint angle. Difference from the target angle and interpolation steps Linear interpolation S-shaped velocity programming is used to generate sequence points. And then distribute them to the underlying drivers in sequence.

[0091] The principle and effect of the above technical solution are as follows: This solution adopts a numerical optimization method, by introducing configurable weight parameters into the cost function. The system obtains a universal control interface: when In certain situations, the algorithm automatically prioritizes positional requirements by sacrificing posture accuracy (e.g., allowing gripper tilt during extreme distance grasping). This design enables the same inverse kinematics algorithm to adapt to grasping tasks with varying accuracy requirements, exhibiting good versatility and robustness. Simultaneously, smooth interpolation effectively suppresses mechanical shocks and jitters during movement, ensuring the continuity and stability of the grasping action, particularly effective when gripping delicate or fragile items.

[0092] Example 2

[0093] The present invention also provides a multi-scenario robotic arm perception and control system based on a layered decoupled architecture to implement the above method. The system includes a perception subsystem, a planning subsystem, and a control subsystem, which constructs a three-layer architecture of complete decoupling of "perception-planning-control". The layers communicate and interact with each other through standardized data interfaces.

[0094] The perception subsystem, as the underlying data acquisition unit of the system, shields the differences in underlying hardware through the camera abstraction interface. It is used to establish communication connections with external vision sensors, collect and output standardized image or video stream data in real time (accessing heterogeneous image input sources such as local cameras or network video streams, and encapsulating the collected raw images or depth information and other perception information into a standard perception data stream for upward transmission).

[0095] The planning subsystem, serving as the core decision-making and computation center of the system, integrates a configuration management module, a multi-scenario algorithm integration module, and a two-dimensional calibration and adaptation module. The configuration management module parses the global configuration file, extracts task scenario parameters and algorithm weight paths, and schedules corresponding algorithm resources. The multi-scenario algorithm integration module integrates a YOLO rotation detection unit, an LLM semantic parsing unit, and a game theory decision-making unit, activating specific units according to instructions from the configuration management module. The two-dimensional calibration and adaptation module stores a pre-calibrated homography matrix, used to perform linear transformations from the pixel coordinate system to the physical coordinate system of the robotic arm base.

[0096] The planning subsystem is used to receive the data output by the perception subsystem, load and run the multi-scene algorithm integration module according to the configuration file, and use the two-dimensional calibration adapter to map the pixel features output by the algorithm into standardized pose data in the physical coordinate system.

[0097] The control subsystem, as the execution unit of the system, receives the standardized pose data, uses a weighted inverse kinematics solver to adjust the weights of position and posture according to task requirements, calculates the target angles of each joint, and generates smooth motion commands through a trajectory smoothing interpolation module to drive the robotic arm to perform actions.

[0098] The trajectory smoothing interpolation module is configured to: receive discrete joint angles obtained from inverse kinematics solutions; use linear interpolation or S-shaped velocity planning algorithms to generate continuous intermediate interpolation points between the current angle and the target angle; and decrease the time interval between two intermediate interpolation points as the number of interpolation steps increases, then send these intermediate interpolation points to the underlying driver. This process ensures that the speed and acceleration curves of the joint motors remain continuous and smooth during the "movement-pressing-grasping-lifting" motion sequence switching of the robotic arm, while maintaining a relatively fixed time for each movement. This eliminates mechanical jitter caused by sudden changes in commands, extends hardware lifespan, and improves grasping stability.

[0099] The principle and effects of the above technical solution are as follows: This system achieves physical isolation between hardware drivers, business logic, and motion control through modular subsystem division. The perception subsystem ensures the diversity of input sources, the planning subsystem, as the core brain, provides compatibility with multi-scenario algorithms and accuracy in spatial mapping, and the control subsystem ensures the flexibility and stability of motion execution in unstructured environments. These three subsystems work together to form a general-purpose robot platform that can adapt to both industrial-grade precision operations and support cutting-edge AI interactive experiments.

[0100] Application Example 2

[0101] In one embodiment, the sensing subsystem includes:

[0102] The Camera Abstraction Interface Module (PAMT) has built-in various hardware driver adapters, configured to dynamically load specific driver implementation classes based on initialization parameters. If the input source is a local device, it calls the local capture interface (such as OpenCV VideoCapture); if the input source is a network address, it calls the streaming media decoding interface (such as FFmpeg or RTSP Client). It can also be extended to include other image sources.

[0103] The Data Stream Standardization Module receives raw data from the hardware driver, performs format verification, color space conversion (such as YUV to RGB), and size normalization, and finally outputs a standard image frame object with timestamps and metadata for downstream subsystems to subscribe to.

[0104] The principle and effect of the above technical solution are as follows: Through the camera abstraction interface module, the system achieves hardware compatibility of "write once, run anywhere". Whether it is a low-cost USB camera, a high-precision depth camera, or a remote monitoring network stream, they can all be directly connected to the system without modifying the core code, which greatly improves the deployment flexibility of the system.

[0105] In one embodiment, the planning subsystem includes:

[0106] Configuration Management Module: This module parses the global configuration file (config.yaml), extracts task scenario parameters, algorithm weight paths, and hardware preference settings, and is responsible for scheduling the corresponding algorithm resources.

[0107] Multi-Scenario Algorithm Integration Module: This module integrates a YOLO rotation detection unit, an LLM semantic parsing unit, and a game-theoretic decision-making unit. The system activates specific units based on instructions from the configuration management module: the YOLO rotation detection unit runs the YOLOv11-OBB model and outputs the rotation bounding box parameters of the target. The LLM semantic parsing unit calls the multimodal large model API to process natural language instructions and return the target location; the game decision unit calculates the pixel coordinates of the next move based on the state of the chessboard image.

[0108] 2D Calibration Adapter Module: This module stores a pre-calibrated homography matrix H, which is used to perform matrix operations, linearly transform the pixel coordinates output by the algorithm integration module into physical coordinates in the robot arm base coordinate system, and superimpose height compensation information to generate the final target pose.

[0109] The principle and effect of the above technical solution are as follows: This subsystem is the "brain" of the entire architecture. The design of the multi-scenario algorithm integration module enables a single system to be reused in completely different business scenarios, greatly improving code reusability; the two-dimensional calibration and adaptation module provides a spatial mapping scheme with low computational load and accuracy that meets the requirements of desktop operations, enabling the visual perception results to be directly understood and executed by the robotic arm.

[0110] In one embodiment, the control subsystem includes:

[0111] Weighted IK Solving Module: This module performs inverse kinematics solving based on a numerical solver using the SLSQP optimization algorithm. It receives the target pose matrix as input and reads the position weight and rotation weight parameters. It constructs a cost function containing position and rotation error terms and iteratively calculates the optimal joint angle sequence.

[0112] The Trajectory Smoothing Module receives the discrete joint angles obtained from the inverse kinematics solution and uses linear interpolation or S-shaped velocity planning algorithms to generate a series of smoothly transitioning intermediate control points between the current angle and the target angle to prevent sudden changes in commands.

[0113] The Driver Communication Module encapsulates the smoothed joint angle sequence into a communication protocol packet (such as Hex command) that can be recognized by the underlying hardware (such as serial servo motors or bus motors), and is responsible for monitoring the feedback status of the hardware.

[0114] The principle and effect of the above technical solution are as follows: the control subsystem solves the problem that the robotic arm is prone to singular points or no solution at extreme positions by using a weighted inverse solution strategy, and gives the system the ability to adapt its posture while ensuring that the position is reached; in conjunction with the trajectory smoothing interpolation module, it effectively eliminates the shaking during the movement of the robotic arm, protects the transmission mechanism and improves the stability of operation.

[0115] In one embodiment, the process of the system performing the item classification task is as follows, and the flowchart is shown in Figure 2:

[0116] After the robotic arm is powered on and a communication connection is established, the system reads the global configuration file and loads parameters including the desktop height, placement area coordinates, YOLO model weight path, and the robotic arm's zero-point position. The system then drives the robotic arm to move to the initial zero-point position, ensuring that the robotic arm is in a known joint state.

[0117] The system enters the visual perception phase. The perception layer captures the current camera image and calls the YOLOv11-OBB and other visual algorithm models loaded by the planning layer for inference. The model outputs the recognition result, and the planning layer determines whether there is a target object that meets the grasping requirements within the current field of view based on the result. If there is no target that meets the requirements (e.g., no object is detected or the object confidence is too low), the system tries to acquire the image again. To avoid the robotic arm itself obstructing the camera's field of view, the planning layer sends a command to drive the robotic arm to move to a preset side position and enter the next round of detection loop.

[0118] If a target meeting the requirements is detected, the system enters the pose calculation stage. The planning layer extracts the center pixel coordinates of the target and performs calculations based on the pre-stored homography matrix, mapping the pixel coordinates to physical coordinates in the robot arm base coordinate system. Simultaneously, the desktop height parameter in the configuration file is read as the Z-axis reference, and combined with the rotation angle output by the OBB model, the final target grasping pose is generated.

[0119] Then, the grasping action is performed. The control layer generates a motion trajectory based on the target grasping pose. The robotic arm first moves to directly above the target object (target-above pose), and then descends vertically to the grasping height. After reaching the target position, the control layer sends a command to close the gripper. At this time, the system reads the feedback angle value of the gripper for judgment: if the gripper angle is less than a preset threshold, it indicates that the gripper is completely closed (no object was grasped), and the grasping is judged as a failure. The system controls the robotic arm to reset and restart the visual perception process; if the gripper angle is greater than the preset threshold, it indicates that the gripper is opened by the object, and the grasping is judged as successful.

[0120] Finally, the handling and placement actions are performed. After successful grasping, the robotic arm lifts back to a safe height. The planning layer retrieves the corresponding placement coordinates from the configuration file based on the identified object category. The robotic arm moves directly above the placement position, descends vertically to the placement height, and opens its grippers to release the object. After placement, the robotic arm lifts back to a safe height and returns to its zero-point position, ready to begin the next grasping task.

[0121] The principles and effects of the above technical solution are as follows: The process design fully considers robustness and stability in actual operations. By setting a "side position," the obstruction of the camera's field of view by the robotic arm itself under the Eye-to-Hand architecture is effectively avoided, ensuring the integrity of visual recognition. The introduction of a "gripper angle threshold judgment" mechanism realizes a physical-level grasping closed-loop feedback, which can promptly detect missed grasps and automatically retry, avoiding invalid handling processes. In addition, the dynamic management of desktop height and classification placement points through configuration files allows the system to adapt to workbenches of different heights or change sorting logic without modifying the code, demonstrating high flexibility. At the same time, the adoption of a "gate-shaped" trajectory planning of "approaching from above - descending vertically - grasping - lifting vertically" effectively prevents the robotic arm from colliding with surrounding obstacles during horizontal movement.

[0122] The above execution process and control method were tested experimentally. A total of 50 item classification tasks were performed, with 15 blocks per task: 6 red blocks, 5 blue blocks, and 4 yellow blocks. After data collection and training, the model weight file was obtained, and the F1-Confidence curve is shown in Figure 3. The confidence threshold was set to 0.7, and the IOU (Intersection over Union) threshold was set to 0.45. Under this parameter configuration, all tasks were correctly classified in 49 attempts, with a success rate of 98%; 11 attempts failed and were retried, resulting in a success rate of [missing information]. .

[0123] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture, characterized in that, The method is based on a system architecture in which the perception layer, planning layer, and control layer are decoupled, and includes the following steps: S10, System Initialization and Establishment of Perception Data Stream: After the system starts, communication channels are established between each level. The perception layer initializes the visual sensor through the camera abstraction layer, collects environmental images or video stream data, and encapsulates it into a standardized perception data stream for transmission to the planning layer. S20, Multi-scenario algorithm reasoning and decision generation: The planning layer receives the perception data stream, calls the corresponding vision and decision algorithm modules according to the currently set task scenario, and obtains the feature information or decision results of the target on the image plane; the task scenario includes at least one of the following: item classification and grasping scenario, chess game scenario, and semantic interaction scenario. S30, Spatial mapping and standardized pose calculation: The planning layer inputs the target feature information or decision results obtained in step S20 into the two-dimensional calibration adapter, uses the pre-calibrated homography matrix to perform coordinate transformation, maps the pixel coordinate system of the image plane to the physical coordinate system of the robot arm's base, generates standardized pose data containing spatial position coordinates and end-effector posture angles, and sends the data to the control layer. S40, Weighted Inverse Kinematics Control and Action Execution: The control layer receives the standardized pose data and generates action commands; the weighted inverse kinematics solver assigns corresponding position weights and attitude weights according to the different requirements of the current task for position accuracy and attitude accuracy, solves the combination of robot arm joint angles, and drives the robot arm to execute actions after trajectory smoothing interpolation.

2. The multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture according to claim 1, characterized in that, In step S10, the perception layer has a camera abstraction interface; this interface encapsulates the underlying drivers of different image acquisition devices, dynamically binds specific driver implementation classes by reading fields in the configuration file, and provides a unified data acquisition method to enable switching between local camera input and network video stream input.

3. The multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture according to claim 1, characterized in that, In step S20, when in an item classification or grasping scenario, the visual algorithm module called by the planning layer is a rotating target detection model; The prediction results output by the rotating target detection model include the center pixel coordinates, length and width dimensions of the target, and the rotation angle relative to the image coordinate system. In step S30, the planning layer maps the rotation angle to the rotation attitude angle of the robotic arm end effector through coordinate system transformation, thereby achieving adaptive grasping of objects placed at any angle.

4. The multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture according to claim 1, characterized in that, The rotating target detection model uses the YOLOv11-OBB network or its variant network.

5. The multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture according to claim 1, characterized in that, In step S20, when in a semantic interaction scenario, the planning layer activates the large model adapter; The large model adapter receives the user's natural language instructions and calls a local or cloud-based multimodal large language model. It combines the perceptual data stream to perform reference expression understanding or open set object detection, and outputs the bounding box or center coordinates of the target object in the image.

6. The multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture according to claim 1, characterized in that, In step S30, the homography matrix is ​​a 3×3 matrix obtained by collecting the pixel coordinates of multiple pairs of feature points in the image and the corresponding point coordinates in physical space, and solving it using the least squares method. The specific calculation process of the coordinate transformation is as follows: the pixel coordinates output by the visual algorithm are linearly transformed into physical coordinates through matrix multiplication, and a preset height offset is superimposed in the Z-axis direction.

7. The multi-scenario robotic arm perception and control method based on a hierarchical decoupling architecture according to claim 1, characterized in that, In step S40, the weighted inverse kinematics solver employs a sequential least squares programming algorithm; The objective function constructed by the system includes position error terms and attitude error terms, and assigns weight coefficients to them respectively. When the target object is located at the edge of the workspace of the robotic arm or there is a risk of kinematic singularity, the solver calculates the joint angle that prioritizes ensuring that the end effector of the robotic arm reaches the target position by reducing the attitude weight and increasing the position weight.

8. A multi-scenario robotic arm perception and control system based on a hierarchical decoupling architecture, used to implement the method of any one of claims 1 to 7, characterized in that, The system includes: The perception subsystem is used to establish communication connections with external vision sensors, shielding the differences in underlying hardware through the camera abstraction interface, and acquiring and outputting standardized image or video stream data in real time. The planning subsystem is used to receive the data output by the perception subsystem, load and run the vision and decision algorithm module according to the configuration file, and use a two-dimensional calibration adapter to map the pixel features output by the algorithm into standardized pose data in the physical coordinate system; the vision and decision algorithm module includes a rotating target detection model, a large model adapter, and a game decision model. The control subsystem is used to receive the standardized pose data, calculate the joint angles using a weighted inverse kinematics solver, and generate smooth motion commands through a trajectory smoothing interpolation module to drive the robotic arm to perform actions.

9. The multi-scenario robotic arm perception and control system based on a hierarchical decoupling architecture according to claim 8, characterized in that, The planning subsystem includes a configuration management module, a multi-scenario algorithm integration module, and a two-dimensional calibration and adaptation module. The configuration management module is used to parse the global configuration file, extract task scenario parameters and algorithm weight paths, and schedule corresponding algorithm resources; The multi-scenario algorithm integration module integrates a YOLO rotation detection unit, an LLM semantic parsing unit, and a game decision-making unit, and activates specific units according to the instructions of the configuration management module. The two-dimensional calibration and adaptation module stores a pre-calibrated homography matrix, which is used to perform a linear transformation from the pixel coordinate system to the physical coordinate system of the robot arm base.

10. The multi-scenario robotic arm perception and control system based on a hierarchical decoupling architecture according to claim 8, characterized in that, The trajectory smoothing interpolation module in the control subsystem is configured to: receive discrete joint angles obtained from inverse kinematics, use linear interpolation or S-shaped velocity planning algorithms to generate a series of smooth transition intermediate control points between the current angle and the target angle, and send the intermediate control points to the underlying driver.

Images