Multi-mode self-adaptive clamping system for assembling key parts of humanoid robot and control method of multi-mode self-adaptive clamping system

Abstract

The invention discloses a multi-modal self-adaptive clamping system for assembling key components of a humanoid robot and a control method of the multi-modal self-adaptive clamping system, and the system comprises a multi-modal sensing module which is used for collecting assembling parameters; the edge calculation module is used for generating a control instruction by utilizing a multi-mode deep learning model according to the assembly parameters; the cloud service module is used for generating distillation data according to the assembly parameters; the self-adaptive clamping module is used for clamping and moving key components of the humanoid robot; and the programmable logic control module is used for controlling the self-adaptive clamping module to execute component assembling operation according to the control instruction and the distillation data. According to the invention, a multi-mode self-adaptive clamping system is realized, and the efficiency and the clamping stability are improved. The robot can be widely applied to the technical field of industrial robots.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial robots, and in particular to a multi-modal adaptive gripping system for assembling key components of a humanoid robot and a control method thereof. Background Art

[0002] With breakthroughs in bionic control, integrated electric drive and deceleration, and high-energy-density batteries, humanoid robots are rapidly moving toward mass production and multi-scenario applications. Key components such as joint modules, lightweight frames, end effectors, and multimodal sensing heads undergo frequent iterations and experience significant model variation. Assembly requires dual precision requirements: submillimeter coaxiality and millinewton-level compliant clamping. Furthermore, humanoid robots often utilize a hybrid structure of a carbon fiber-magnesium-aluminum alloy frame and a silicone outer layer. Excessive clamping pressure can cause permanent deformation or internal microcracks, further complicating force control in assembly fixtures. The traditional approach employs a "rigid, dedicated fixture" system with a PLC program and manual instruction. However, each replacement of a joint or peripheral requires the complete replacement of the fixture and line stoppage for leveling, resulting in lengthy changeover times. Furthermore, process data and PLC ladder diagrams require manual transcription or script conversion, requiring complete program reprogramming and repeated verification after design changes, leading to low efficiency. At the same time, the sensing method mainly relies on a single RGB camera or offline probe, and manual teaching is performed, which can easily lead to quality problems such as crushing, slippage, and cable kinking, and has low clamping stability.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The embodiments of the present invention provide a multi-modal adaptive gripping system and a control method thereof for assembling key components of a humanoid robot, which effectively improves efficiency and gripping stability.

[0005] In one aspect, an embodiment of the present invention provides a multimodal adaptive gripping system for assembling key components of a humanoid robot, comprising:

[0006] Multimodal sensing module for collecting assembly parameters;

[0007] an edge computing module for generating control instructions based on the assembly parameters using a multimodal deep learning model;

[0008] A cloud service module, configured to generate distillation data according to the assembly parameters;

[0009] Adaptive gripping module for gripping and moving key components of humanoid robots;

[0010] A programmable logic control module is used to control the adaptive gripping module to perform component assembly operations according to the control instructions and the distillation data.

[0011] In some embodiments, the assembly parameters include a color image, depth information, three-dimensional force, three-dimensional moment, position slip of the fixture silicone shell, and temperature change of the fixture silicone shell, and the multimodal sensing module includes:

[0012] Industrial-grade RGB-D camera for acquiring color images and depth information;

[0013] Six-dimensional force-displacement sensor, used to collect three-dimensional forces and three-dimensional moments;

[0014] A micro inertial measurement unit is used to collect the position slip of the silicone shell of the fixture;

[0015] Flexible temperature patch, used to collect temperature changes on the silicone shell of the fixture.

[0016] In some embodiments, the cloud service module includes:

[0017] A digital twin cluster is used to perform assembly simulation using a digital twin framework according to the assembly parameters to obtain simulation data;

[0018] The cloud server is used to perform model training based on the simulation data to generate the distillation data.

[0019] In some embodiments, the adaptive gripping module includes:

[0020] Adaptive grippers for gripping key components of humanoid robots;

[0021] A collaborative robotic arm is used to move the adaptive fixture to a predetermined position.

[0022] In some embodiments, the adaptive fixture comprises:

[0023] A mechanical fingertip subassembly, comprising a parallel three-finger structure and a micro ball screw, and configured to grip key components of the humanoid robot through width adjustment;

[0024] A vacuum adsorption sub-component, used for adsorbing key components of the humanoid robot through atmospheric pressure;

[0025] An electro-permanent magnetic sub-component used to magnetically attract key components of a humanoid robot with a metal shell.

[0026] The beneficial effects of the present invention are as follows:

[0027] An embodiment of the present invention provides a multimodal adaptive gripping system for assembling key components of a humanoid robot, comprising a multimodal sensing module, an edge computing module, a cloud service module, an adaptive gripping module, and a programmable logic control module. The multimodal sensing module is used to collect assembly parameters; the edge computing module is used to generate control instructions based on the assembly parameters using a multimodal deep learning model; the cloud service module is used to generate distillation data based on the assembly parameters; the adaptive gripping module is used to grip and move key components of the humanoid robot; and the programmable logic control module is used to control the adaptive gripping module to perform component assembly operations based on the control instructions and distillation data, thereby realizing a multimodal adaptive gripping system and improving efficiency and gripping stability.

[0028] On the other hand, an embodiment of the present invention provides a control method applied to the system, comprising the following steps:

[0029] Acquire humanoid robot data, the humanoid robot data including a humanoid robot joint CAD model and a bill of materials;

[0030] generating a design semantic vector and a virtual production line scenario based on the humanoid robot data;

[0031] In the virtual production line scenario, offline simulation and distillation are performed based on the design semantic vector to generate a fixture control model and a programmable logic controller ladder diagram;

[0032] Adaptive clamping control is performed according to the fixture control model and the programmable logic controller ladder diagram to obtain component assembly records, which include coaxiality, maximum force, temperature rise, and number of retries.

[0033] In some embodiments, generating a design semantic vector and a virtual production line scenario based on the humanoid robot data includes:

[0034] Analyzing the humanoid robot joint CAD model using a CAD parser to obtain basic elements and adjacency relationships, wherein the basic elements include faces, edges, and vertices;

[0035] generating a geometric topology graph according to the basic elements and the adjacency relationship;

[0036] Generating a geometric vector using a graph convolutional network according to the geometric topology graph;

[0037] generating a first semantic vector according to the bill of materials, the assembly sequence, and the torque curve;

[0038] Concatenating the geometric vector and the first semantic vector to obtain the design semantic vector;

[0039] The virtual production line scenario is generated according to the design semantic vector.

[0040] In some embodiments, performing offline simulation and distillation based on the design semantic vector to generate a fixture control model and a programmable logic controller ladder diagram includes:

[0041] According to the design semantic vector, incoming material tolerance, temperature and humidity, and assembly posture, reinforcement learning is used to perform model training to obtain the fixture control model;

[0042] Compressing the fixture control model using a contrast distillation strategy to obtain lightweight network weights;

[0043] The programmable logic controller ladder diagram is generated according to the lightweight network weight.

[0044] In some embodiments, performing adaptive gripping control according to the fixture control model and the programmable logic controller ladder diagram to obtain a component assembly record includes:

[0045] After calibrating the multimodal sensing module, collecting assembly parameters;

[0046] Using a cross-modal multi-head attention network to align and fuse the assembly parameters to obtain an assembly semantic vector;

[0047] Generate instruction parameters based on the assembly semantic vector using a graph neural network and the fixture control model, wherein the instruction parameters include posture adjustment amount, clamping force, insertion speed curve, and motion trajectory;

[0048] According to the instruction parameters and the programmable logic controller ladder diagram, the adaptive gripping module is controlled to perform component assembly operations to obtain the component assembly records.

[0049] In some embodiments, the method further comprises:

[0050] When performing online quality monitoring, fine-tuning parameters of the fixture control model is performed according to the component assembly record, and the fixture control model is updated;

[0051] When performing full-day data aggregation, parameters of the fixture control model are aggregated according to the component assembly records and extreme working condition simulation results, and the fixture control model is updated;

[0052] When a model change is performed, the fixture control model is updated according to the component assembly records and small-scale simulation results.

[0053] In another aspect, an embodiment of the present invention provides a computer device, comprising:

[0054] at least one processor;

[0055] at least one memory for storing at least one program;

[0056] When the at least one program is executed by the at least one processor, the at least one processor implements the method.

[0057] On the other hand, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0058] The beneficial effects of the present invention are as follows:

[0059] The embodiment of the present invention first obtains humanoid robot data, and then generates design semantic vectors and virtual production line scenarios based on the humanoid robot data. Then, based on the design semantic vectors, offline simulation and distillation are performed to generate a fixture control model and a programmable logic controller ladder diagram. Finally, based on the fixture control model and the programmable logic controller ladder diagram, adaptive clamping control is performed to obtain component assembly records, thereby realizing adaptive clamping control and improving efficiency and clamping stability.

[0060] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the structures particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0062] Figure 1 This is a schematic structural diagram of a multi-modal adaptive gripping system for assembling key components of a humanoid robot according to an embodiment of the present invention;

[0063] Figure 2 This is a flow chart of a control method applied to a multi-modal adaptive gripping system according to an embodiment of the present invention;

[0064] Figure 3 A schematic diagram of a fixture control operation process according to an embodiment of the present invention;

[0065] Figure 4 The figure is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0067] It will be understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0068] The terms "at least one", "plurality", "each", "any", etc. used in this application include "at least one", "two" or more, "plurality" or "each", "any" or "any one", "each" or "any one" in the context of the present invention, and "at least one" or "at least one" includes one, two or more, "plurality" or "any one" includes two or more, "each" or "each one" in the context of the present invention, and "any" or "any one

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0070] Before explaining the embodiments of the present application in detail, some of the nouns and terms involved in the embodiments of the present application are first explained. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0071] A Programmable Logic Controller (PLC) is a digital computing and operating electronic system designed specifically for industrial applications. It uses a programmable memory to store instructions for performing logic operations, sequence control, timing, counting, and arithmetic operations. It controls various types of machinery or production processes through digital or analog input and output.

[0072] With breakthroughs in bionic control, integrated electric drive and deceleration, and high-energy-density batteries, humanoid robots are rapidly moving toward mass production and multi-scenario applications. Core components such as joint modules, lightweight frames, end effectors, and multimodal sensing heads undergo frequent iterations and experience significant model variation. Assembly demands dual precision requirements: submillimeter coaxiality and millinewton-level compliant clamping. Furthermore, humanoid robots often utilize a hybrid structure of a carbon fiber-magnesium-aluminum alloy frame and a silicone outer layer. Even the slightest overpressure can cause permanent deformation or internal microcracks, further complicating force control in assembly fixtures. Existing production lines mostly rely on the traditional model of "rigid dedicated fixtures + manual teaching + PLC program curation." For one thing, every change in a joint or peripheral requires complete fixture replacement and line stoppage for leveling, resulting in changeover times of 30-60 minutes. Furthermore, sensing methods primarily rely on single RGB cameras or offline probes, lacking real-time integration with force and temperature data, leading to quality risks such as crushing, slippage, and cable kinking. Even though the industry has begun to introduce parallel / collaborative robots and offline simulation automatic programming, it is still constrained by the rigid dependence of fixtures, perception information islands and the cost of PLC program reprogramming, making it difficult to support the "high heterogeneity, small batch, and fast iteration" assembly rhythm and yield requirements of humanoid robots.

[0073] The problems with traditional methods include: (1) Reliance on rigid fixtures and high conversion costs. Traditional humanoid robot production lines use rigid fixtures and tooling customized for single parts. Whenever there is a slight change in joint specifications, shell materials, or wiring harness directions, the production line must replace the fixture as a whole or add temporary gaskets, and then manually level, calibrate, and recalibrate. The average line stoppage is 30-60 minutes, which not only delays delivery but also increases fixture inventory and maintenance costs exponentially. (2) Single-modal perception is fragmented and cannot be controlled flexibly. Existing solutions mostly rely on single RGB / 2D vision or offline laser probes to obtain posture information, and lack deep integration with real-time data such as force perception, temperature, and IMU. When the fixture grasps soft and hard hybrid parts such as carbon fiber-silicone, it is difficult to detect overpressure or slippage in time, resulting in quality hazards such as crushing, cable entanglement, and microcracks, affecting the gait and safety of the entire machine. (3) PLC programs are rigid and lack self-learning and hot swapping. Manual transcription or script conversion is still used between process data (CAD / BOM) and PLC ladder diagrams. Design changes require the entire program to be rewritten and repeatedly verified. At the same time, assembly strategies cannot be adaptively optimized using real-time sensor feedback, resulting in the need to stop the line and adjust parameters once incoming material tolerances and environmental drift exceed the boundaries.

[0074] Therefore, there is an urgent need for an adaptive gripping system that can integrate design priors (CAD / BOM) with real-time multimodal perception (vision, force, temperature, and IMU) within a unified algorithmic framework, and possess online self-learning and PLC hot-swap capabilities. This system should be able to "see clearly, feel accurately, and learn quickly" like a human hand, enabling rapid switching between different humanoid robot joints and materials without stopping the line, while balancing high-precision alignment with flexible force control, fundamentally improving production line flexibility, yield, and cost-effectiveness. To solve the following problems: How to achieve rapid changeover and alignment without stopping the line in the face of rapid iteration of humanoid robot size, material, and assembly interface without changing the fixture; How to unify CAD / BOM design priors and multi-modal real-time data such as vision, force perception, temperature, and IMU into the same algorithm framework, and complete attitude-force control dual closed-loop decision-making in milliseconds to meet sub-millimeter coaxiality and millinewton-level compliant clamping requirements; How to introduce online self-learning and PLC hot replacement mechanisms so that assembly strategies can continue to evolve with part batches, tolerance fluctuations, and environmental changes without the need for manual teaching or downtime to reprogram.

[0075] In light of this, this paper proposes a four-domain closed-loop, panoramic, multimodal adaptive gripping system for humanoid robot joint-actuator assembly, encompassing "design-perception-control-learning." Core to this system is a digital twin-driven, cross-modal Transformer algorithm stack that integrates CAD / BOM design priors, RGB-D vision, and real-time force-temperature-IMU data. Using a GNN-RL decision network, the system outputs gripper pose and clamping force commands within 20ms. A PLC hot-loading interface enables millisecond-level action execution and online fine-tuning. Through simulation-to-real-machine comparison distillation and edge fine-tuning, the assembly strategy evolves continuously as part model, tolerance, and environmental conditions change. This enables rapid, non-stop changeovers, submillimeter alignment, and millinewton-level compliant force control for multiple humanoid robot joint models on a single fixture platform. This solution simultaneously addresses high precision, flexibility, and cost-effectiveness, overcoming the technical bottlenecks of traditional rigid fixtures, single-modal perception, and rigid PLC programming. It provides a new, replicable, and sustainably optimized intelligent assembly path for the large-scale production of humanoid robots.

[0076] The following is a detailed explanation of the embodiments of the present application with reference to the accompanying drawings:

[0077] like Figure 1 As shown, an embodiment of the present invention provides a multi-modal adaptive gripping system for assembling key components of a humanoid robot, comprising:

[0078] Multimodal sensing module 101, used to collect assembly parameters;

[0079] Edge computing module 102, for generating control instructions based on assembly parameters using a multimodal deep learning model;

[0080] The cloud service module 103 is used to generate distillation data according to the assembly parameters;

[0081] Adaptive gripping module 104, used to grip and move key components of the humanoid robot;

[0082] The programmable logic control module 105 is used to control the adaptive gripping module to perform component assembly operations according to control instructions and distillation data.

[0083] In some embodiments, a multimodal adaptive gripping system for assembling key components of a humanoid robot provided in this embodiment includes a multimodal sensing module 101, an edge computing module 102, a cloud service module 103, an adaptive gripping module 104, and a programmable logic control module 105. Assembly parameters can be collected through the multimodal sensing module 101. For example, an industrial-grade binocular RGB-D camera (120 frames per second) can be used to collect color images and depth information, a six-dimensional force-displacement sensor (accuracy of 0.1 Newton) can be used to collect three-dimensional forces and three-dimensional moments, a micro-inertial measurement unit (1 kHz sampling) can be used to collect the position slip of the silicone shell of the fixture, and a flexible temperature patch can be used to collect the temperature change of the silicone shell of the fixture. All devices use PTP (Precision Time Protocol) for clock calibration to ensure that the data timestamps are consistent.

[0084] Based on assembly parameters, control instructions can be generated by the edge computing module 102 in conjunction with a multimodal deep learning model. The edge computing module has a built-in high-performance GPU that can be used to run multimodal deep learning models, supporting a 20-millisecond inference cycle and generating control instructions. For example, the edge computing module can be implemented as an edge computing box, which serves as the "brain" of the entire production line. It uses an embedded GPU card with an inference computing power of over 20 TOPS (tera operations per second), completing multimodal data fusion and policy calculation in approximately 20 milliseconds. To ensure industrial reliability, the motherboard features a wide-temperature design, a 24V DC power supply, and an integrated uninterruptible power supply module to prevent model parameter loss due to transient power outages. The edge computing box is pre-installed with real-time Linux and a container management system. Various versions of the deep learning model are distributed as Docker images, enabling hot-loading of new weights without interrupting device operation. The edge computing box communicates with the upload server via 10 Gigabit Ethernet and rapidly sends instructions to the on-site PLC via a dual-port TSN (Time-Sensitive Networking) network interface card, ensuring sub-millisecond end-to-end latency in the decision-making chain.

[0085] Based on assembly parameters, distillation data can be generated via the cloud service module 103. For example, digital twin simulations can be run through the cloud service module, using the Unity-ROS framework to simulate various humanoid robot assembly scenarios and generate training and distillation data in batches. Key components of the humanoid robot can be gripped and moved using the adaptive gripper module 104. For example, the adaptive gripper module synchronizes with the PLC in real time via the EtherCAT bus. An integrated fixture mounted at the end of the gripper enables both mechanical gripping with fingertips and switching between vacuum and electro-permanent magnetic gripping. Based on control instructions and distillation data, the adaptive gripper module can be controlled by the programmable logic control module 105 to perform component assembly operations. For example, the programmable logic control module is responsible for 1 kHz (i.e., once per millisecond) servo closed-loop control and provides a "ladder diagram" program hot-swap interface. Production line communication utilizes a two-tier network. The motion control layer utilizes EtherCAT, with a one-millisecond bus cycle, responsible for high-speed interaction between the robot arm joint drive, servo motor, and force sensor. The information fusion layer utilizes a Gigabit TSN network, handling large amounts of visual data streams, model inference instructions, and log uploads. Both network cards support PTP (Precision Time Protocol) hardware timestamping. All nodes are synchronized via a master clock, with maximum clock drift kept to less than one microsecond, ensuring strict timeline alignment of multimodal data. The network switch integrates dual power supplies and industrial-grade fans, supporting port-level power outage detection and ring redundancy. If packet loss or latency exceeds the specified limit on any link, the switch immediately switches to a backup path and sends an alarm to the edge box and PLC, ensuring that production lines are not shut down due to communication failures.

[0086] As you can understand, this embodiment follows a closed-loop process of "design-perception-control-learning." The top layer is the digital twin and process management layer: it receives CAD 3D drawings, bills of materials (BOMs), and tolerance requirements, and builds a virtual production line in the cloud that is synchronized with the real production line to verify and optimize the assembly process. The middle layer is the edge inference and motion control layer: a multimodal inference model runs on an industrial computer close to the equipment, integrating "design information" with "field sensor information" in real time to calculate the fixture's posture (position and angle) and clamping force, and transmits the results to the programmable logic controller (PLC). The bottom layer is the execution layer: consisting of a collaborative robotic arm, an adjustable gripper, a sensor array, and a PLC, it is responsible for the specific grasping, handling, and insertion actions. High-speed communication between the three layers is achieved via Gigabit Industrial Ethernet and Time-Sensitive Networking (TSN), ensuring a sub-millisecond closed-loop command and feedback loop. In terms of information flow, the process data flow is transmitted from the digital twin layer to the edge layer to guide the assembly strategy; the perception data flow (vision, force perception, temperature, inertia, etc.) is uploaded from the execution layer to the edge layer, and then aggregated to the cloud for historical analysis; and the strategy update flow is distilled from the cloud and sent to the edge layer, completing the hot replacement of the model without stopping the line, so that the assembly strategy can be continuously iterated.

[0087] In some embodiments, the assembly parameters include a color image, depth information, three-dimensional force, three-dimensional torque, position slip of the fixture silicone shell, and temperature change of the fixture silicone shell. The multimodal sensing module includes:

[0088] Industrial-grade RGB-D camera for acquiring color images and depth information;

[0089] Six-dimensional force-displacement sensor, used to collect three-dimensional forces and three-dimensional moments;

[0090] A micro inertial measurement unit is used to collect the position slip of the silicone shell of the fixture;

[0091] Flexible temperature patch, used to collect temperature changes on the silicone shell of the fixture.

[0092] In some embodiments, the multimodal sensing module includes an industrial-grade RGB-D camera, a six-dimensional force-displacement sensor, a micro-inertial measurement unit, and a flexible temperature patch. The industrial-grade RGB-D camera can capture color images and depth information. For example, a pair of industrial-grade RGB-D cameras with a resolution of 1,280 by 960 pixels and a frame rate of 120 frames per second can be used, enabling stable acquisition of color images and depth information in diverse lighting environments. The industrial-grade RGB-D camera can be secured to the end of the robotic arm and above the workstation using an all-aluminum alloy bracket, equipped with a four-point LED strip light to reduce measurement errors in reflective materials. The six-dimensional force-displacement sensor can capture three-dimensional forces and three-dimensional moments. For example, the six-dimensional force-displacement sensor can be mounted between the fixture and the flange, simultaneously outputting three-dimensional forces and three-dimensional moments. It has a nominal resolution of 0.1 Newton and withstands a full-scale overload of 150 parts per second without damage, meeting the requirements for flexible assembly of mixed soft and hard materials. The position slip of the silicone shell of the fixture can be collected through a micro inertial measurement unit, and the temperature change of the silicone shell of the fixture can be collected through a flexible temperature patch. For example, a micro inertial measurement unit (IMU) can be used to monitor small slippage, that is, the position slippage of the silicone shell of the fixture, and a flexible temperature patch can be used to monitor surface heat changes, that is, the temperature change of the silicone shell of the fixture. When the temperature or slip inertia deviates from the normal curve, the system is promptly reminded to adjust the assembly rhythm to avoid the silicone shell from being heated by friction.

[0093] In some embodiments, the cloud service module includes:

[0094] The digital twin cluster is used to perform assembly simulation based on assembly parameters using the digital twin framework to obtain simulation data;

[0095] The cloud server is used to generate distillation data through model training based on simulation data.

[0096] In some embodiments, the cloud service module includes a digital twin cluster and a cloud server. According to the assembly parameters, the digital twin cluster can be combined with the digital twin framework to perform assembly simulation to obtain simulation data. For example, the digital twin framework adopts the Unity-ROS dual engine, Unity is responsible for high-precision physical rendering and collision detection, and ROS is responsible for robot kinematics and message communication. The simulation data (including strategies and logs) obtained by simulation are first stored in the object storage, and then pushed to the edge computing module through the version control system. The cloud-edge is encrypted through a VPN tunnel to ensure the security and integrity of the simulation data and model parameters during transmission. Based on the simulation data, model training can be performed on the cloud server to generate distillation data. For example, the cloud service module consists of a group of high-performance servers, and each cloud server is equipped with multiple professional GPU cards for large-scale parallel simulation and model training to generate distillation data. The cloud servers are interconnected via Infiniband and can complete reinforcement learning iterations of thousands of virtual assembly tasks in a few hours. InfiniBand (literally, "Infinite Bandwidth" technology, abbreviated as IB) is a computer network communication standard used for high-performance computing. It offers extremely high throughput and low latency, and is used for data interconnection between computers. InfiniBand is also used as a direct or switched interconnection between servers and storage systems, as well as for interconnection between storage systems.

[0097] In some embodiments, the adaptive gripping module includes:

[0098] Adaptive grippers for gripping key components of humanoid robots;

[0099] A collaborative robotic arm that moves an adaptive gripper to a predetermined position.

[0100] In some embodiments, the adaptive gripping module includes an adaptive fixture and a collaborative robotic arm. The key components of the humanoid robot can be gripped by the adaptive fixture. For example, the adaptive fixture can quickly switch modes and combines three clamping methods: mechanical fingertips, vacuum adsorption, and electro-permanent magnets. The adaptive fixture is connected to the collaborative robotic arm through a quick-change interface, and it takes no more than one minute from disassembly to reassembly. The adaptive fixture can be moved to a predetermined position by the collaborative robotic arm. For example, the collaborative robotic arm has six degrees of freedom and is loaded with a safety torque sensor. The maximum rated torque of a single axis can cover the assembly needs of the humanoid robot joints; its joint module integrates a servo drive and an encoder, and the maximum repeatability accuracy reaches ±0.03 mm.

[0101] In some embodiments, the adaptive fixture includes:

[0102] The mechanical fingertip subassembly consists of a parallel three-finger structure and a micro ball screw, and is used to grip key components of the humanoid robot through width adjustment;

[0103] Vacuum adsorption sub-component, used to adsorb key components of humanoid robots through atmospheric pressure;

[0104] An electro-permanent magnetic sub-component used to magnetically attract key components of a humanoid robot with a metal shell.

[0105] In some embodiments, the adaptive clamp includes a mechanical fingertip sub-component, a vacuum adsorption sub-component, and an electro-permanent magnetic sub-component. The mechanical fingertip sub-component is composed of a parallel three-finger structure and a micro ball screw. The key components of the humanoid robot can be clamped by the mechanical fingertip sub-component in combination with width adjustment. The mechanical fingertip sub-component can automatically adjust the width within three seconds. The key components of the humanoid robot can be adsorbed by atmospheric pressure through the vacuum adsorption sub-component, and vacuum adsorption has a good bonding effect on carbon fiber or plastic parts. The key components of the humanoid robot with a metal casing can be adsorbed by magnetic force through the electro-permanent magnetic sub-component, which can specifically handle metal housings, and can be started and stopped quickly without continuous energy consumption. It is understandable that metal housing generally refers to the casing or protective cover of a device or machine.

[0106] The beneficial effects of implementing the embodiments of the present invention include: a multimodal adaptive gripping system for assembling key components of a humanoid robot provided by the embodiments of the present invention includes a multimodal sensing module, an edge computing module, a cloud service module, an adaptive gripping module, and a programmable logic control module. The multimodal sensing module is used to collect assembly parameters; the edge computing module is used to generate control instructions based on the assembly parameters using a multimodal deep learning model; the cloud service module is used to generate distillation data based on the assembly parameters; the adaptive gripping module is used to grip and move key components of the humanoid robot; and the programmable logic control module is used to control the adaptive gripping module to perform component assembly operations based on the control instructions and distillation data, thereby realizing a multimodal adaptive gripping system and improving efficiency and gripping stability.

[0107] Figure 2 This is an optional flowchart of a control method for a system provided in an embodiment of the present application and applied to this embodiment. Figure 2 The method may include but is not limited to steps S201 to S204.

[0108] Step S201: Acquire humanoid robot data, where the humanoid robot data includes a humanoid robot joint CAD model and a bill of materials;

[0109] Step S202: Generate a design semantic vector and a virtual production line scenario based on the humanoid robot data;

[0110] Step S203: In the virtual production line scenario, perform offline simulation and distillation based on the design semantic vector to generate a fixture control model and a programmable logic controller ladder diagram;

[0111] Step S204: Adaptive gripping control is performed according to the fixture control model and the programmable logic controller ladder diagram to obtain component assembly records, which include coaxiality, maximum force, temperature rise, and number of retries.

[0112] Steps S201 to S204 shown in the embodiment of the present application implement adaptive clamping control, thereby improving efficiency and clamping stability.

[0113] In some embodiments, in step S201, humanoid robot data may be obtained from a robot database. Humanoid robot data may also be obtained through other methods, without limitation. The humanoid robot data may be uploaded to a PLM platform, where PLM stands for Product Lifecycle Management (PLM).

[0114] In some embodiments, in step S202, generating a design semantic vector and a virtual production line scenario based on the humanoid robot data may include but is not limited to the following steps:

[0115] The CAD parser is used to parse the humanoid robot joint CAD model to obtain basic elements and adjacency relationships. The basic elements include faces, edges and vertices.

[0116] Generate geometric topology graph based on basic elements and adjacency relationships;

[0117] According to the geometric topology graph, a graph convolutional network is used to generate geometric vectors;

[0118] Generate a first semantic vector based on the bill of materials, assembly sequence, and torque curve;

[0119] Concatenate the geometric vector and the first semantic vector to obtain a design semantic vector;

[0120] Generate a virtual production line scenario based on the design semantic vector.

[0121] In some embodiments, in the design semantic parsing layer, a CAD parser can be used to parse the humanoid robot joint CAD model to obtain basic elements and adjacency relationships, wherein the basic elements include faces, edges, and vertices. Then, based on the basic elements and adjacency relationships, a geometric topology graph is generated, and the adjacency relationship can be saved using a graph structure. Based on the geometric topology graph, a geometric vector is generated using a graph convolutional network. For example, the geometric topology graph is mapped to a 1024-dimensional vector as a geometric vector through a set of lightweight graph convolutional networks (GCNs), and each sub-segment of the vector corresponds to interpretable features such as local curvature, hole diameter, or tolerance bandwidth. Then, based on the bill of materials, assembly sequence, and torque curve, a first semantic vector is generated. For example, the materials, heat treatment methods, and load limits in the BOM (Bill of Material, BOM) are converted into 256-dimensional semantic vectors by a pre-trained text encoder (BERT), and are combined with process data such as assembly sequence and torque curve to encode the first semantic vector. The geometric vector and the first semantic vector are spliced ​​together to obtain the design semantic vector. Through this "dual-track encoding", the system not only saves spatial information but also carries engineering constraints, achieving "readable, calculable, and traceable" design semantics. Furthermore, the assembly sequence and torque curve can be written into the YAML process file as a priori for subsequent reinforcement learning. This process file is consistent with the geometric topology. Figure 1 Generate a "virtual assembly condition" to ensure that the offline simulation corresponds to the real production line in terms of part posture and mechanical boundary conditions. It is understandable that YAML is a highly readable format for expressing data serialization. Finally, based on the design semantic vector, a virtual production line scenario is generated. It is possible to create a virtual production line scenario that corresponds one-to-one to the real production line, providing a high-fidelity environment for subsequent offline simulation and strategy verification. It is understandable that the design semantic parsing layer is responsible for decomposing the CAD model into a "geometric topology diagram", and then encoding it into a high-dimensional vector together with the BOM tolerance and material information. In this way, "what you want to do" can be clearly conveyed to the subsequent algorithm. The goal is to reliably project R&D intentions into the algorithmic world.

[0122] In some embodiments, in step S203, offline simulation and distillation are performed based on the design semantic vector to generate a fixture control model and a programmable logic controller ladder diagram, which may include but is not limited to the following steps:

[0123] Based on the design semantic vector, incoming material tolerance, temperature and humidity, and assembly posture, reinforcement learning is used to train the model and obtain the fixture control model.

[0124] The fixture control model is compressed using a contrastive distillation strategy to obtain lightweight network weights.

[0125] Generate a programmable logic controller ladder diagram based on the lightweight network weights.

[0126] In some embodiments, in a virtual production line scenario, reinforcement learning can be used to train the model based on the design semantic vector, incoming material tolerances, temperature and humidity, and assembly posture to obtain a highly robust fixture control model. After training, the fixture control model is compressed using a comparative distillation strategy to obtain lightweight network weights. Based on the lightweight network weights, a programmable logic controller ladder diagram (PLC ladder diagram) is generated. If verification passes, the fixture control model is pushed to the edge computing box and the on-site PLC with a version number.

[0127] In some embodiments, in step S204, adaptive gripping control is performed according to the fixture control model and the programmable logic controller ladder diagram to obtain a component assembly record, which may include but is not limited to the following steps:

[0128] After calibrating the multimodal sensing module, the assembly parameters are collected;

[0129] A cross-modal multi-head attention network is used to align and fuse assembly parameters to obtain an assembly semantic vector.

[0130] Based on the assembly semantic vector, the graph neural network and the fixture control model are used to generate instruction parameters. The instruction parameters include posture adjustment amount, clamping force, insertion speed curve and motion trajectory.

[0131] According to the instruction parameters and the ladder diagram of the programmable logic controller, the adaptive gripping module is controlled to execute the component assembly operation and obtain the component assembly record.

[0132] In some embodiments, assembly parameters can be collected after calibrating the multimodal sensing module. For example, before starting the production line, engineers perform no-load calibration: an industrial-grade binocular RGB-D camera completes coordinate calibration on the target plate, the six-dimensional force-displacement sensor is zeroed on a standard test block, and the micro inertial measurement unit and flexible temperature patch record the baseline value. The edge computing box runs a self-test script and prevents startup if the prediction-execution error, clock drift, or network latency exceeds the limit. The collaborative robotic arm then attempts to grasp several samples, and the model compares the theoretical and measured force curves. If the deviation exceeds the threshold, the clamping parameters are immediately fine-tuned online until the first piece is released. Assembly parameters can be collected using a multimodal perception layer, covering four signals: vision, force, temperature, and inertia. The visual encoder uses an improved Swin Transformer: a layer of depthwise separable convolution is added before the basic window attention structure to extract micro features such as edges and holes at a low resolution. The force and temperature signals enter the one-dimensional convolutional network + gated recurrent unit (GRU) to capture the dynamic evolution of the three stages of "micro-shake-compression-steady state" during the assembly process. After the inertial measurement unit (IMU) data passes through the Kalman filter, it is again stacked into the same GRU so that the model can perceive micron-level slippage. Ultimately, these heterogeneous features will be resampled to the same beat and spliced ​​into a "perception vector" of length 768. To prevent the perception noise from drifting over time, the system uses self-supervised comparative learning: fragment-level raw signals are collected every day, and "the same workstation and the same time period" are used as positive samples, and "cross-workstation and cross-time periods" are used as negative samples. The encoder weights are continuously updated to ensure the stability of the feature distribution.

[0133] A cross-modal multi-head attention network is then used to align and fuse assembly parameters to generate an assembly semantic vector. For example, a visual network can extract the object's shape and pose, a force network can analyze force changes during clamping, and a temperature and inertia network can monitor for signs of friction or slip. A cross-modal Transformer (multi-head attention network) aligns these features with the design vector to generate a unified assembly semantic vector. As can be seen, cross-modal fusion is the core of the entire algorithm stack. The system utilizes a 12-layer multi-head attention Transformer, with 16 heads per layer. Each attention head simultaneously receives both the "design vector" (dimensions, materials, tolerances) and the "perception vector" (shape, force-temperature, inertia). To align the time scales of the different modalities, a "dynamic delay compensation block" is inserted after each attention layer. This module self-learns a set of differentiable shift parameters based on the correlation coefficients of the previous attention layer, automatically aligning the visual frame and the force sampling in feature space. After fusion is complete, an assembly semantic vector of length 512 is output. This step does not directly give movement instructions, but compresses implicit information such as "how to grab, how much force to use, and where to insert" into a unified semantic space for consumption by downstream decision-making networks.

[0134] Based on the assembly semantic vector, a graph neural network and fixture control model are used to generate command parameters. These parameters include posture adjustment, clamping force, insertion speed curve, and motion trajectory. For example, a decision-making and motion planning layer can be used for calculations, with the decision-making layer considering both "geometric contact relationships" and "dynamic mechanical feedback." The system abstracts the fixture's mechanical fingertips, the part to be assembled, and the reference hole positions into nodes in a graph, with the edge weights between nodes initialized using the assembly semantic vector. Subsequently, a three-layer graph neural network (GNN) propagates information through this contact graph, outputting three continuous motion parameters: end-point posture compensation, clamping force setting, and insertion speed curve. End-point posture compensation refers to the x, y, and z displacements and posture angles that the robot arm needs to fine-tune. The clamping force setting refers to the target force values ​​for the fingertips, vacuum, and magnetic channels. The insertion speed curve is a speed-time table that follows an accumulation-uniform-decrease pattern. The motion parameters are fed into a reinforcement learning-based policy head that utilizes a proximal policy optimization (PPO) algorithm. The reward function comprehensively considers coaxiality error, maximum force, assembly time, and whether secondary alignment is required, ensuring that the model does not take excessive risks while pursuing rhythm. Periodically, the cloud will use digital twins to simulate rare extreme working conditions and perform "risk fallback training" on the strategy head, so that the model can learn to actively slow down or evacuate when anomalies occur. Ultimately, the graph neural network (GNN) is combined with the reinforcement learning strategy to output three sets of command parameters, including posture adjustment amount (position offset and angle offset), clamping force (number of Newtons, taking into account different materials), and motion trajectory (the path of the robot arm from the current point to the target point). It can also include the insertion speed curve.

[0135] Finally, according to the instruction parameters and the programmable logic controller ladder diagram, the adaptive gripping module is controlled to perform the component assembly operation and obtain the component assembly record. For example, the first assembly can be carried out, or after entering mass production, the camera captures the shape and posture of the part, and the edge computing box integrates vision, force perception, temperature and design semantics in about 20 milliseconds to output the posture adjustment amount, clamping force size and insertion speed curve. The PLC drives the robotic arm to execute at a frequency of 1 kHz, and the force-position-temperature-inertia signal is fed back in real time. If abnormal force or posture drift is detected, the model immediately calculates the retreat path and performs secondary alignment to ensure smooth and damage-free assembly.

[0136] In some embodiments, the method further comprises:

[0137] When performing online quality monitoring, fine-tune the parameters of the fixture control model according to the component assembly records and update the fixture control model;

[0138] When performing full-day data aggregation, the parameters of the fixture control model are aggregated and updated based on the component assembly records and extreme working condition simulation results;

[0139] When a model change occurs, the fixture control model is updated based on component assembly records and small-scale simulation results.

[0140] In some embodiments, an online self-learning layer can be used for model training. The cloud can leverage digital twins to batch simulate various assembly scenarios, distilling offline experience into lightweight model weights. The edge collects real-time success / failure labels and performs daily fine-tuning. Model weights are then sent to the PLC via a containerized hot update, eliminating the need for downtime or manual re-teaching.

[0141] During online quality monitoring, the fixture control model can be fine-tuned and updated based on component assembly records. For example, after each completed assembly, the system records metrics such as coaxiality, maximum force, temperature rise, and number of retries and stores them in the quality database. The edge computing box aggregates "success / failure" samples on an hourly basis and performs small-step gradient updates, fine-tuning only the decision layer and delay compensation blocks. This allows for rapid adaptation to batch errors without disrupting core features.

[0142] During daily data aggregation, the fixture control model parameters can be aggregated and updated based on component assembly records and extreme operating condition simulation results. For example, each night, the cloud compares the day's production logs with the digital twin scenario, supplements the extreme operating condition simulation results, and generates a new version of incremental weights. The new weights are first grayscaled and tested on a single robotic arm for three hours. If the pass rate improves or remains the same and no alarms are triggered, they are automatically rolled out to the entire production line. Otherwise, the system rolls back to the previous stable version and records the reason for the rollback. The entire release or rollback process lasts approximately three minutes and does not require production downtime.

[0143] When a model change occurs, the fixture control model can be updated based on component assembly records and small-scale simulation results. For example, when the R&D department uploads a new joint design file, the digital twin platform automatically parses and runs a small-scale simulation to generate small-scale simulation results. The edge computing box completes fine-tuning within the first ten trial runs. If the first pieces pass, mass production can proceed directly. This allows "design changes to be rolled out" on the same flexible production line without changing fixtures, manual teaching, or long downtime.

[0144] The closed-loop system in this embodiment achieves rapid, non-stop model changeover, submillimeter alignment accuracy, and millinewton-level compliant force control, providing a continuously evolving intelligent solution for multi-model, high-iteration assembly of humanoid robots. Furthermore, in real-time production, the model records the differences between "actual actions" and "planned actions" as replay clips, labeling them as "success / failure." The edge box aggregates the last 24 hours of data daily and performs a small-scale gradient update. This step only adjusts the decision head and delay compensation block, maintaining the stability of the encoder and Transformer backbone to prevent catastrophic forgetting. The updated weights are timestamped and assigned a workstation ID via the version control system before undergoing a phased release process: a small-scale pilot run is first conducted on a single robotic arm. If the pass rate increases instead of decreasing, the new model is gradually rolled out to the entire production line. If a rollback threshold is reached during the phased release (e.g., three consecutive assembly failures), the scheduler immediately reverts to the previous stable version and records the rollback reason for subsequent offline analysis. The entire software stack runs in a real-time Linux container, leveraging GPU virtualization technology to simultaneously serve multiple production lines. Each production line has its own independent "model sandbox," ensuring that fine-tuning on one line doesn't interfere with other workstations. The cloud acts as a "knowledge hub," continuously extracting the most valuable strategy increments from each sandbox and feeding them back into the next round of offline training. This two-way closed loop of "cloud integration and edge self-learning" allows the system to continuously evolve without downtime, meeting the long-term requirements of humanoid robot assembly for diverse models, high precision, and rapid iteration.

[0145] In some embodiments, to verify the feasibility of the method of this embodiment in the assembly scenario of a humanoid robot elbow joint module, an experiment can be conducted by setting up a single-station verification line in a pilot production workshop. The experimental environment is as follows.

[0146] (1) Workstation and equipment configuration. Collaborative robot arm: 1 × UR10e (6 degrees of freedom, working radius 1300mm, repeatability ±0.03mm), with a self-developed three-finger vacuum-electro-permanent magnet integrated clamp installed at the end; the maximum fingertip opening is 120mm, and the clamping force is continuously adjustable from 0 to 200N. Force-position sensor: ATIMini45-E (resolution 0.01N / 0.5mN·m, sampling 1kHz), installed between the clamp and the flange. Vision system: Binocular Intel Realsense D435 camera (1280×720@120fps), one fixed at the top of the workstation looking down, and one moving with the end of the robot arm. Industrial PLC: Beckhoff CX5140 embedded controller (Intel Atom E3845 1.9GHz, EtherCAT master), cycle time 1ms. Edge computing box: NVIDIA Jetson AGX Orin (2048 CUDA cores, 64 Tensor cores, 22 TOPS, Ubuntu 22.04 RT patch), connected to the EtherCAT clock domain via dual TSN network ports. Temperature & IMU patch: Flexible NTC array + Bosch BMI270 (1kHz sampling), adhered to the elbow joint's aluminum alloy housing and silicone protective layer.

[0147] (2) Network and time synchronization. Motion layer bus: EtherCAT 100Mb / s, PLC master cycle 1ms; the robotic arm and force sensor are slaves. Data layer network: Cisco IE-4000 TSN switch, 1Gb / s, uplink to the edge box; all nodes use IEEE1588 PTP, and clock drift is controlled within 0.8μs. Cloud connection: Independent VPN channel for the laboratory, uploading production logs and model weights; download bandwidth 200Mb / s.

[0148] (3) Software and algorithm stack.

[0149]

[0150] (4) Samples and evaluation indicators. Sample: 2025-A elbow joint module, weighing 1.2 kg, outer shell made of magnesium-aluminum alloy + partial silicone coating; the diameter of the reference hole at the matching shoulder end is 25 mm, with a tolerance of IT7. Batch: 500 pieces in total, with a random introduction of ±2 mm posture deviation and ±0.03 mm aperture deviation. Core indicators: △ coaxiality ≤ 0.04 mm (90% percentile); maximum clamping force peak ≤ 85 N (no indentation on the silicone surface); first insertion success rate ≥ 95%; workstation cycle 11 seconds (including alignment, insertion, clamping, and resetting).

[0151] (5) Environmental control and safety. The laboratory workshop maintains a constant temperature of 23 ± 2°C and a humidity of 45 ± 5% RH. The robotic arm's safety torque limit is 100 N·m. Fixture overload and force-position anomalies trigger a Level 2 EMS shutdown. All data is encrypted and stored in a local NAS and mirrored daily to cloud object storage.

[0152] During the experiment, the fixture control operation process is as follows Figure 3 As shown, the Unity-ROS virtual production line can be invoked in the cloud-based digital twin platform to perform 20,000 offline training cycles on the 2025-A elbow joint under conditions of ±2mm pose perturbation and ±0.03mm aperture tolerance. After reinforcement learning converges, the distilled 22MB lightweight weight file (version number cfs-elbow-v5.2) is packaged with the corresponding PLC ladder diagram into a "policy package." The edge computing box pulls this policy package via VPN, and the container hot-loads the new weights. During idle cycles, the PLC calls TwinCAT's "Online Change" function to complete the ladder diagram hot replacement, which takes approximately 45 seconds. After loading, the model's SHA-256 checksum and version number are recorded to ensure subsequent traceability.

[0153] Then, the no-load system calibration is performed, including: visual calibration: fix the camera to the cross target plate, run the ROScamera_calibration node to obtain the internal and external parameter matrix; the end camera is aligned with the workstation coordinate system through the four-point ARUCO code. Force-position zero point calibration: the robot arm moves to the "zero posture" and the fixture is not pressurized. ATIMini45 automatically collects 2000 frames of no-load data, calculates the average value and writes it to the zero bias compensation register. Temperature and inertial reference: the flexible NTC array and BMI270 respectively collect 30s environmental baselines; the edge box caches the baseline sequence in Redis, and subsequently judges temperature rise and vibration anomalies. After the calibration is completed, the system executes the self-test script: if the visual reprojection error is >0.15px, the force sensor zero drift is >1N, or the network delay is >0.5ms, it will be prevented from entering the next step.

[0154] Next, a first-article test assembly and model fine-tuning were performed. The collaborative robotic arm grasped 10 prototype parts and assembled them one by one. After each piece was assembled, the edge box aligned the "assembly semantic vector" with the actual force-position-temperature curve and calculated the coaxiality error and maximum clamping force. If the coaxiality error was ≤0.04mm and the maximum force was ≤85N, the result was marked as "successful." Otherwise, an online gradient update was triggered, the decision head learning rate was set to 1e-5, and the test assembly was repeated after 200 iterations. After three consecutive pieces were successfully marked, a First-Article-Pass report was generated, and the fine-tuned weights were saved as cfs-elbow-v5.2.1. The entire first-article phase took 18 minutes, with two online updates.

[0155] Batch assembly and real-time closed-loop processing are performed. The edge box switches to production mode, and the MES system issues a production task for 500 parts. When the part to be assembled is in place, the overhead camera sends a / part_ready trigger signal; the end-user camera simultaneously captures the local shape. The multimodal fusion model outputs the end-user pose compensation Δx, Δy, Δz, ΔRoll, ΔPitch, and ΔYaw within 22ms; the clamping force is set to 65N (for aluminum alloy surfaces) or 40N (for silicone surfaces); and the insertion speed curve is accelerated to 0.5m / s. 2 →Constant speed 0.2m / s→Deceleration 0.6m / s 2 The PLC updates the servo every 1ms. If the force-position sensor detects a peak force greater than 90N or a torque reversal, the "retreat path" is triggered, and the robot arm retreats 5mm and realigns. After assembly is complete, the clamp is released, and the collaborative robot arm returns to its home position. The system also records the assembly log entry and pushes it to Kafka.

[0156] Online quality monitoring is performed during the assembly process. The quality monitoring service subscribes to Kafka streams and calculates the following in real time: Coaxiality distribution: A box plot is generated for every 50 pieces. If the 90th percentile is greater than 0.04mm, a yellow light warning is triggered. Clamping force peak trend: A linear regression slope greater than 2N / 100 pieces in a rolling window of 100 pieces triggers a maintenance prompt. Temperature rise curve: If the local temperature exceeds the baseline by +3°C for 10 consecutive seconds, production is suspended and a prompt is issued to inspect the silicone coating. Edge boxes are fine-tuned using success / failure labels in a 60-minute cycle. Fine-tuning only updates the last layer of fully connected and delay compensation blocks, taking less than 45 seconds and not affecting the cycle time.

[0157] Finally, cloud-based distillation and grayscale release are performed. After production is complete, the edge box uploads complete logs and the latest weights, v5.2.1-edge. During a nighttime task, the cloud supplements the simulation environment with real-world data, retrains for two hours on extreme poses and temperature rise scenarios, and generates an incremental version, v5.3-candidate. The next day, the scheduler deploys the grayscale version of v5.3-candidate to this workstation: 25 parts are assembled at a 5% sampling rate. If the pass rate is ≥95% and the average coaxiality does not degrade, a 100% release is automatically achieved. If the pass rate does not meet the standards, a rollback to v5.2.1 is generated, and a rollback report is generated. In this experiment, v5.3-candidate successfully passed the grayscale test, with a full line switchover time of 2 minutes and 37 seconds, without any downtime.

[0158] In some embodiments, advantages of this embodiment include:

[0159] (1) Changeover time is almost zero. Traditional production lines require replacement of special fixtures, recalibration, and line downtime of 30-60 minutes. This system relies on multi-modal algorithms and PLC ladder diagram hot swapping. Only container loading and coordinate self-checking are required to switch to a new model. The entire process takes less than 3 minutes, and the impact on production line utilization is negligible.

[0160] (2) Excellent assembly accuracy and flexible force control. In a 500-piece batch test, the 90th percentile of coaxiality was maintained at 0.037mm, and the maximum clamping force peaked at 82N, both of which are superior to traditional rigid fixtures (typically 0.07-0.10mm, >100N). No indentations or cracks appeared on the silicone-coated surface, demonstrating that the multimodal closed loop can achieve millinewton-level force control and submillimeter-level positioning in a hard-soft hybrid material.

[0161] (3) The first-time assembly success rate has been significantly improved. Thanks to the immediate "retreat-second alignment" strategy, the first-time insertion success rate has increased from 86-88% on the manual teaching line to 96.8%, and the rework and repair rate has been reduced to less than 2%, saving approximately 15% in labor and material costs per month.

[0162] (4) Continuous self-learning, with yield increasing over time. Through the closed loop of "cloud distillation + edge fine-tuning + grayscale release," model weights can be iterated without stopping the production line; in the experiment, the yield increased by another 1.3 percentage points from v5.2 to v5.3, without any manual teaching or parameter adjustment. This capability enables the production line to automatically "evolve" with minor design changes and incoming material fluctuations, avoiding the traditional solution of repeatedly requiring engineers to visit the site for debugging.

[0163] (5) Fixture and tooling assets are halved. A single adaptive fixture combines mechanical fingertip, vacuum, and electro-permanent magnetic modes, covering a variety of materials and sizes. By sharing cameras, force sensors, and edge boxes on the same production line, fixture procurement and maintenance costs are reduced by approximately 40-50% compared to the "one-to-one model-to-fixture" model.

[0164] (6) Data traceability and rapid tracing. The multimodal data of the entire chain (design, vision, force-position, temperature, IMU) is archived at the component level. If a defect occurs, it can be located to a certain clamping or a certain track within seconds. Compared with the traditional method of relying solely on appearance inspection and intermittent sampling, the traceability efficiency is improved by an order of magnitude, laying the foundation for subsequent large-scale quality management and AI diagnosis.

[0165] (7) Versatility and scalability. The algorithm framework does not rely on a specific brand of robotic arm or PLC and can be ported to UR, ABB, KUKA, as well as EtherCAT and Profinet networks. Multimodal features and reward functions can be adapted to different assembly units such as shoulder joints, grippers, and battery packs by simply adjusting the weights, achieving "same technology stack, covering the entire machine."

[0166] In some embodiments, this embodiment implements a bidirectional design-perception semantic mapping mechanism. This mechanism parses 3D CAD models and BOM process files into geometric topology graphs based on faces, edges, vertices, and material properties. This is then refined into high-dimensional design semantic vectors using graph convolution and text encoders. Simultaneously, multimodal signals such as RGB-D, force-temperature, and inertia collected in real time are uniformly encoded into homogeneous perception vectors. Once these two signals are aligned in the same feature space, the algorithm can simultaneously incorporate both engineering priors and instantaneous states, providing an accurate and traceable "semantic coordinate system" for subsequent decision-making.

[0167] This embodiment implements a cross-modal Transformer fusion network and dynamic delay compensation. A multi-head attention Transformer fuses the design and perception vectors layer by layer. A learnable delay compensation module is inserted after each layer. This module automatically adjusts the temporal alignment of the different modalities based on the previous attention weight, eliminating misalignment caused by differences in visual and force sampling frequencies. This network outputs a unified assembly semantic vector within 20 milliseconds, achieving high-precision, low-latency information fusion.

[0168] This embodiment implements an adaptive assembly decision framework based on graph neural networks and proximal policy optimization. The algorithm abstracts the fixture fingertips, the part to be assembled, and the reference hole positions into nodes. The edge weights between the nodes are derived from semantic vectors, and the contact topology information is transmitted by a three-layer graph neural network. Proximal policy optimization (PPO) is then used to generate end-point posture compensation, clamping force settings, and insertion speed curves. The decision framework automatically adjusts the grip-insert-release strategy to achieve smooth and stable assembly when incoming material tolerances, material hardness, or assembly posture vary.

[0169] This embodiment implements a three-layer self-learning closed loop of cloud distillation, edge fine-tuning, and grayscale release. The system first conducts large-scale reinforcement learning in a digital twin environment, then distills the model into lightweight weights and distributes it to the production line; the edge end fine-tunes a small number of parameters on an hourly basis based on real success / failure samples; the updated model is published online in grayscale or quickly rolled back, without the need for line downtime or manual teaching, thus ensuring that the production line continues to ramp up without interruption.

[0170] This embodiment implements the online hot replacement and clock consistency strategy of the PLC ladder diagram. The edge AI inference results are converted into hot-loadable ladder diagram segments through safety constraints and seamlessly replaced by the TwinCAT online change interface. All nodes maintain microsecond-level synchronization through the TSN-PTP clock domain, which not only meets the industrial hard real-time requirements but also avoids the downtime risk of traditional version switching.

[0171] This embodiment implements a three-mode adaptive gripper with rapid switching between mechanical fingertip, vacuum, and permanent magnet modes. The gripper integrates an automatic ball screw width adjustment mechanism, a dual-circuit vacuum channel, and a matrix of permanent magnets. This allows for switching between gripping modes within a single command, and a quick-change interface allows for complete gripper replacement within one minute. An algorithm continuously adjusts the clamping force from 0 to 200 Newtons, balancing the rigidity of the metal housing with the compliant protection of the silicone sheath.

[0172] This embodiment implements multi-modal safety monitoring and abnormal self-recovery strategy. The system simultaneously monitors force-position, temperature rise and inertia signals, and adopts redundant threshold judgment and trend analysis for double insurance. If excessive force, slippage or abnormal temperature rise is detected, the decision-making layer generates a retreat path at the millisecond level and triggers secondary alignment to prevent crushing, biting and cable entanglement, thereby achieving active safety of the assembly process.

[0173] This embodiment implements a full-data traceability framework for part-level assembly. The system archives the design semantic vector, motion parameters, actual multimodal curves, and qualification determinations for each part, achieving a one-object-one-code data chain throughout the entire process. When defects occur, the corresponding clamping action or force peak anomaly can be located within seconds, providing high-value training data for engineering analysis, quality improvement, and subsequent AI diagnostics.

[0174] The beneficial effects of implementing the embodiments of the present invention include: the embodiments of the present invention first obtain humanoid robot data, then generate design semantic vectors and virtual production line scenarios based on the humanoid robot data, and then perform offline simulation and distillation based on the design semantic vectors to generate a fixture control model and a programmable logic controller ladder diagram. Finally, based on the fixture control model and the programmable logic controller ladder diagram, adaptive clamping control is performed to obtain component assembly records, thereby realizing adaptive clamping control and improving efficiency and clamping stability.

[0175] like Figure 4 As shown, an embodiment of the present invention further provides a computer device, including:

[0176] at least one processor 901;

[0177] At least one memory 902, configured to store at least one program;

[0178] When at least one program is executed by at least one processor, the at least one processor implements Figure 2 The method shown.

[0179] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] The embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, which is executed by a processor to implement Figure 2 The method shown.

[0181] The contents of the above method embodiments are all applicable to the present storage medium embodiment. The functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0182] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A multi-modal adaptive gripping system for assembling key components of a humanoid robot, characterized in that: include: Multimodal sensing module for collecting assembly parameters; an edge computing module for generating control instructions based on the assembly parameters using a multimodal deep learning model; A cloud service module, configured to generate distillation data according to the assembly parameters; Adaptive gripping module for gripping and moving key components of humanoid robots; A programmable logic control module is used to control the adaptive gripping module to perform component assembly operations according to the control instructions and the distillation data.

2. The system according to claim 1, wherein: The assembly parameters include color images, depth information, three-dimensional forces, three-dimensional moments, position slippage of the fixture silicone shell, and temperature change of the fixture silicone shell. The multimodal sensing module includes: Industrial-grade RGB-D camera for acquiring color images and depth information; Six-dimensional force-displacement sensor, used to collect three-dimensional forces and three-dimensional moments; A micro inertial measurement unit is used to collect the position slip of the silicone shell of the fixture; Flexible temperature patch, used to collect temperature changes on the silicone shell of the fixture.

3. The system according to claim 1, wherein: The cloud service module includes: A digital twin cluster is used to perform assembly simulation using a digital twin framework according to the assembly parameters to obtain simulation data; The cloud server is used to perform model training based on the simulation data to generate the distillation data.

4. The system according to claim 1, wherein: The adaptive gripping module includes: Adaptive grippers for gripping key components of humanoid robots; A collaborative robotic arm is used to move the adaptive fixture to a predetermined position.

5. The system according to claim 4, characterized in that The adaptive fixture comprises: A mechanical fingertip subassembly, comprising a parallel three-finger structure and a micro ball screw, and configured to grip key components of the humanoid robot through width adjustment; A vacuum adsorption sub-component, used for adsorbing key components of the humanoid robot through atmospheric pressure; An electro-permanent magnetic sub-component used to magnetically attract key components of a humanoid robot with a metal shell.

6. A control method applied to the system according to any one of claims 1 to 5, characterized in that: The following steps are involved: Acquire humanoid robot data, the humanoid robot data including a humanoid robot joint CAD model and a bill of materials; generating a design semantic vector and a virtual production line scenario based on the humanoid robot data; In the virtual production line scenario, offline simulation and distillation are performed based on the design semantic vector to generate a fixture control model and a programmable logic controller ladder diagram; Adaptive clamping control is performed according to the fixture control model and the programmable logic controller ladder diagram to obtain component assembly records, which include coaxiality, maximum force, temperature rise, and number of retries.

7. The method according to claim 6, characterized in that Generating a design semantic vector and a virtual production line scenario based on the humanoid robot data includes: Analyzing the humanoid robot joint CAD model using a CAD parser to obtain basic elements and adjacency relationships, wherein the basic elements include faces, edges, and vertices; generating a geometric topology graph according to the basic elements and the adjacency relationship; Generating a geometric vector using a graph convolutional network according to the geometric topology graph; generating a first semantic vector according to the bill of materials, the assembly sequence, and the torque curve; Concatenating the geometric vector and the first semantic vector to obtain the design semantic vector; The virtual production line scenario is generated according to the design semantic vector.

8. The method according to claim 6, characterized in that The method of performing offline simulation and distillation based on the design semantic vector to generate a fixture control model and a programmable logic controller ladder diagram includes: According to the design semantic vector, incoming material tolerance, temperature and humidity, and assembly posture, reinforcement learning is used to perform model training to obtain the fixture control model; The fixture control model is compressed using a contrast distillation strategy to obtain lightweight network weights; The programmable logic controller ladder diagram is generated according to the lightweight network weight.

9. The method according to claim 6, characterized in that The method of performing adaptive gripping control according to the fixture control model and the programmable logic controller ladder diagram to obtain a component assembly record includes: After calibrating the multimodal sensing module, collecting assembly parameters; Using a cross-modal multi-head attention network to align and fuse the assembly parameters to obtain an assembly semantic vector; Generate instruction parameters based on the assembly semantic vector using a graph neural network and the fixture control model, wherein the instruction parameters include posture adjustment amount, clamping force, insertion speed curve, and motion trajectory; According to the instruction parameters and the programmable logic controller ladder diagram, the adaptive gripping module is controlled to perform component assembly operations to obtain the component assembly records.

10. The method according to claim 6, characterized in that The method further comprises: When performing online quality monitoring, fine-tuning parameters of the fixture control model is performed according to the component assembly record, and the fixture control model is updated; When performing full-day data aggregation, parameters of the fixture control model are aggregated according to the component assembly records and extreme working condition simulation results, and the fixture control model is updated; When a model change is performed, the fixture control model is updated according to the component assembly records and small-scale simulation results.

Images