Robot cap screwing method and system based on multi-modal perception and machine learning

Abstract

This invention discloses a robotic capping method and system based on multimodal perception and machine learning, belonging to the field of embodied robot operation technology. The system includes a visual perception module, a robotic capping device, a data processing module, a pitch prediction module, and a control module. Key geometric parameters of the bottle cap are acquired non-contactly through visual perception; subsequently, the robotic capping device is controlled to perform small-angle reciprocating screwing motions, while multidimensional force data is collected in real time through tactile sensors; after preprocessing and feature engineering of the force data, it is input into a constructed and trained machine learning-based pitch prediction model to predict the bottle cap pitch in real time; by fusing visual geometric priors and predicted pitch, the model is decoupled and mapped to the axial and rotational coordinated motion of the robotic arm's end effector, achieving adaptive, high-precision capping or closing operations for bottles of different sizes. This invention can significantly improve the robustness, versatility, and intelligence level of robots performing fine force control tasks.

Description

Technical Field

[0001] This invention relates to the field of robotic operation technology, specifically to a robotic cap-screwing method and system based on multimodal perception and machine learning. Background Technology

[0002] With the rapid development of embodied intelligent robot technology, robot applications are gradually expanding from highly structured and repetitive industrial production scenarios to unstructured environments such as home services, medical assistance, laboratory automation, and flexible manufacturing. In these complex and varied application scenarios, capping (opening or closing) threaded containers (such as medicine bottles, reagent bottles, and beverage bottles) is an extremely common and critical task. Achieving highly reliable and highly adaptive automated capping can not only significantly reduce the burden of manual labor but also ensure operator safety in environments with toxic, hazardous, or strictly sterile requirements, and improve the automation level of precision operations. Currently, mainstream robotic capping technologies mainly fall into three representative categories: The first category is traditional industrial capping systems, which rely on customized mechanical structures and preset motion trajectories. Although they perform stably on large-scale, single-specification product production lines, they lack the necessary flexibility and are difficult to adapt to non-standard or multi-specification bottles, limiting their application scope. The second type is vision-guided capping systems, which rely primarily on visual sensors such as cameras for cap positioning. However, two-dimensional visual information cannot capture hidden thread geometry features such as pitch, and it completely lacks the crucial mechanical interaction perception during the tightening process, resulting in insufficient thread matching accuracy and a tendency for misalignment or jamming. The third type is force-controlled capping systems, which utilize compliant control methods such as impedance control or admittance control to adjust the relationship between force and position. However, these systems typically rely on human experience to preset control parameters, exhibiting poor adaptability when facing bottles with unknown pitches or variable thread specifications, easily leading to slippage, over-tightening, or cap jamming. Operational robustness and versatility urgently need improvement. In summary, existing robotic capping technologies still face significant technical bottlenecks when dealing with non-standard and variable thread bottles in complex, unstructured environments. These bottlenecks include a single perception dimension, a lack of effective fusion of spatiotemporal decoupling between visual and force information, and the inability to identify dynamic parameters in real time and perform online compensation, thus hindering further improvements in the level of robotic intelligent operation.

[0003] Therefore, existing technologies still need further development. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a robot capping method and system based on multimodal perception and machine learning to solve the problems existing in the prior art.

[0005] To achieve the above technical objectives, a robotic cap-screwing method based on multimodal perception and machine learning is provided, comprising the following steps:

[0006] S100. Obtain bottle cap geometric parameters through non-contact visual measurement;

[0007] Geometric parameters include cap diameter, cap height, bottle diameter, and bottle height; regional global statistical characteristics are used to locate the cap boundary, and the cap diameter and cap height are accurately extracted through a three-segment fitting algorithm and a dual-modal calibration mechanism (depth information and ArUco markers); the cap is accurately segmented through a three-segment fitting algorithm, and the cap diameter and cap height are further accurately extracted;

[0008] Non-contact visual measurement specifically employs a dual-modal calibration mechanism, combining depth information and ArUco markers (square QR codes) for adaptive size estimation; accurately acquiring the diameter and height of the bottle cap; where the bottle cap diameter serves as a key input for proxy torque calculation in tactile feature engineering, primarily used for pitch prediction, and the bottle cap height is mainly used to assist in calculating the number of thread turns, dynamically calculating the number of screw-on turns based on the predicted pitch; S200, acquiring tactile force data through small-angle rotation; preprocessing and feature engineering the tactile force data to generate a standardized feature tensor; the amount of tactile force data (number of samples), the time step of each data point, and the 22-dimensional features constructed based on the tactile force data jointly generate the standardized feature tensor; the standardized feature tensor is represented as... Where B is the number of samples, 320 is the time step, and 22 is the dimension of the feature vector constructed in feature engineering;

[0009] Feature standardization specifically employs Z-Score standardization to eliminate the significant magnitude difference between force values ​​(typically large) and progress features (ranging from 0 to 1). By subtracting the training set mean and dividing by the standard deviation, all features are mapped to a uniform distribution space, thereby eliminating the influence of scale, accelerating model convergence, and ensuring numerical stability. Finally, each preprocessed sample is represented as a feature tensor.

[0010] S300. Construct a machine learning-based pitch prediction model and use the processed standardized feature tensor and pitch prediction model to predict the bottle cap pitch.

[0011] S400, Perform intelligent capping operation based on the geometric parameters and predicted pitch.

[0012] Specifically, in the visual measurement step, a dual-modal calibration mechanism is used, combining depth information and ArUco markers (using square QR codes) for adaptive size estimation.

[0013] Specifically, in the step of collecting tactile force data, a reciprocating small-angle rotational motion is performed to obtain multimodal timing data containing thread characteristics, and the action phase is marked.

[0014] Specifically, in the pitch prediction step, a temporal deep learning model is used to process the standardized feature tensor, and an attention mechanism is used to focus on key mechanical frames.

[0015] Specifically, the temporal deep learning model is a bidirectional long short-term memory network combined with an attention layer, used for pitch classification prediction.

[0016] The present invention also provides a robotic capping system based on multimodal perception and machine learning, comprising:

[0017] The visual perception module is used to acquire geometric parameters of the target bottle, such as the bottle cap diameter and bottle cap height, in a non-contact manner.

[0018] The robotic capping device includes a robotic arm and a capping gripper. The capping gripper is equipped with a tactile sensor for collecting multi-dimensional force data during the capping process.

[0019] The data processing module is used to preprocess and construct features from the multidimensional force data to generate standardized feature tensors.

[0020] The pitch prediction module employs a machine learning model to predict the bottle cap pitch based on the feature tensor.

[0021] The control module is used to control the robot capping device to perform adaptive capping operations based on the cap height and the predicted screw pitch.

[0022] Specifically, the visual perception module includes a depth camera and an instance segmentation model, which dynamically calibrates the conversion relationship between pixels and physical size through depth information and image segmentation technology.

[0023] Specifically, the visual perception module uses regional global statistical characteristics to locate the bottle cap boundary, separates the bottle cap from the bottle body through a three-segment fitting algorithm, and accurately extracts the bottle cap diameter and bottle cap height.

[0024] Specifically, in feature engineering, bottle cap diameter and tactile force data are used as source data for features.

[0025] Specifically, the predicted screw pitch and cap height are combined to dynamically calculate the number of screw-on turns, guiding the robot's capping device to rotate a certain number of times.

[0026] Specifically, in the robot capping device, the capping gripper includes a flange, an electric push rod, a gimbal motor, and a gripper assembly. The gimbal motor drives the gripper to rotate, and the electric push rod controls the opening and closing of the gripper through a slider mechanism.

[0027] Specifically, the gripper assembly of the cap screw-on gripper includes grippers, gripper connecting rods, and a central fixing frame, forming a parallelogram mechanism. The electric push rod drives the grippers to move relative to each other through a slider and a slider connecting rod.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] The robot capping system and method based on multimodal perception and machine learning provided by this invention brings about many significant benefits by innovatively integrating visual and tactile information and introducing intelligent learning algorithms.

[0030] First, this invention effectively overcomes the limitations of a single perception mode by deeply fusing and complementing visual and tactile multimodal information. The visual perception module provides global geometric priors of the bottle (such as cap diameter, cap height, bottle diameter, and bottle height) and spatial information of the operating scene, providing precise guidance for the robotic arm's grasping and positioning. Specifically, the cap diameter and cap height are not only used to guide the robotic arm's capping gripper posture, but the former also serves as a key input for proxy torque calculation in tactile feature engineering, while the latter is combined with predicted pitch to dynamically calculate the number of turns to tighten the cap. Meanwhile, the tactile sensor captures subtle mechanical changes in real time during high-frequency interactions, accurately reflecting local physical interaction states such as thread engagement and rotational resistance. This fusion of global and local, geometric and mechanical information enables the system to comprehensively perceive the operating object and environmental state, laying a solid foundation for intelligent decision-making.

[0031] Secondly, this invention possesses powerful online adaptive and parameter identification capabilities. By designing small-angle reciprocating trial operations and collecting corresponding tactile force data, and utilizing machine learning models such as bidirectional long short-term memory networks based on attention mechanisms, the system can predict the key parameter of the unknown bottle cap's screw pitch in real time and with high accuracy from brief mechanical interactions. This capability breaks the constraint of traditional methods that require a preset fixed screw pitch, enabling the system to autonomously adapt to various non-standard and variable-specification threaded containers, significantly improving the technology's versatility and flexibility.

[0032] Furthermore, the robustness and operational safety of this invention are greatly enhanced. The visual perception module introduces a dual-modal dynamic calibration mechanism using depth information and ArUco markers, ensuring the stability of dimensional measurements under different lighting conditions and with different materials (including transparent and reflective bottles). The data processing flow includes rigorous signal denoising, stage alignment, and feature standardization steps, effectively suppressing noise interference. At the control level, a dynamic motion compensation strategy based on predicted pitch ensures precise coordination between rotational angular displacement and axial feed displacement, fundamentally avoiding gripper slippage, thread wear, or sensor overload caused by motion mismatch.

[0033] Finally, this invention achieves a complete intelligent closed loop from perception and decision-making to execution. The entire process requires no manual intervention or complex parameter adjustments, has a high degree of automation, and is not only suitable for structured industrial environments, but also capable of handling delicate operational tasks in unstructured scenarios such as homes and laboratories, thus significantly promoting the practical application of embodied intelligent robots in complex physical interaction tasks. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall structure of the cap screw-on gripper provided in a specific embodiment of the present invention;

[0035] Figure 2 This is an exploded view of the overall structure of the cap-screwing gripper provided in a specific embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the robot capping data acquisition device provided in a specific embodiment of the present invention;

[0037] Figure 4 This is a flowchart illustrating a robotic capping system based on multimodal perception and machine learning, provided in a specific embodiment of the present invention.

[0038] Figure 5 This is a schematic flowchart of the overall process of pitch prediction and opening / closing method of robot capping method based on multimodal perception and machine learning provided in a specific embodiment of the present invention.

[0039] The above figures contain the following reference numerals:

[0040] 11. Cap gripper; 12. Robotic arm; 111. Flange; 112. Electric push rod fixing component; 113. Electric push rod; 114. Gimbal motor; 115. Wifi control module; 116. Motor drive module; 117. Battery; 118. Gripper fixing block; 119. Slider; 1110. Slider connecting rod; 1111. Gripper; 1112. Gripper connecting rod; 1113. Central fixing frame; 1114. Tactile sensor. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in this application, other similar embodiments obtained by those skilled in the art without creative effort should all fall within the scope of protection of this application. Furthermore, directional terms mentioned in the following embodiments, such as "up," "down," "left," and "right," are only for reference to the directions in the accompanying drawings; therefore, the directional terms used are for illustrative purposes and not for limiting the invention.

[0042] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments.

[0043] Please see Figure 1-3 This invention provides a robotic cap-screwing system based on multimodal perception and machine learning, comprising:

[0044] The visual perception module is used to acquire the bottle cap diameter and bottle cap height of the target bottle in a non-contact manner.

[0045] The robotic capping device includes a robotic arm 12 and a capping gripper 11. The capping gripper 11 is equipped with a tactile sensor 1114 for collecting multi-dimensional force data during the capping process.

[0046] The data processing module is used to preprocess and construct features from the multidimensional force data to generate standardized feature tensors.

[0047] The pitch prediction module employs a machine learning model to predict the bottle cap pitch based on the standardized feature tensor.

[0048] The control module is used to control the robot capping device to perform adaptive capping operations based on the cap height and the predicted screw pitch.

[0049] For better understanding, please continue reading. Figure 1 , Figure 1 The overall three-dimensional mechanical structure of the robot capping system is shown. Figure 1 As can be seen, the main body of the system is an integrated box structure with supporting columns at the bottom to ensure stability. The core control and computing unit is integrated in the middle of the box. A disc mounting flange with screw holes is provided on the top for fixing the top cap screwing actuator. On the left side of the box structure, specifically corresponding to the integrated Wi-Fi control module 115 and motor drive module 116, the Wi-Fi control module 115 is responsible for wireless communication with the host computer or network, while the motor drive module 116 is used to drive the gimbal motor 114 and the electric push rod 113. The protruding block on the right side of the box corresponds to the battery 117 that provides power to the system. All components are tightly connected through internal circuits and mechanical interfaces, forming a complete mechatronic device. This figure clearly presents the main layout, module division, and spatial assembly relationship of the system hardware, which is the basis for those skilled in the art to understand the physical structure and electrical connections of the system.

[0050] For further information, please refer to [link / reference]. Figure 2 , Figure 2The exploded view details the components of the cap-screwing gripper 11 and their assembly relationships. The view shows the hierarchical structure of the gripper from bottom to top: the two symmetrically arranged elongated components at the bottom are grippers 1111, whose inner sides are connected to the central fixing frame 1113 below via gripper connecting rods 1112, together forming a parallelogram motion mechanism. Above the central fixing frame 1113 is a gripper fixing block 118. Electric push rod fixing parts 112 are fixed on both sides of the gripper fixing block 118 for mounting electric push rods 113. A slider 119 is connected to the end of the electric push rod 113 and is linked to the gripper 1111 via a slider connecting rod 1110. A gimbal motor 114 is mounted above the gripper fixing block 118, its rotor embedded in a groove in the gripper fixing block 118. The gimbal motor 114 is connected to the end of the robotic arm 12 via a flange 111. In addition, a tactile sensor 1114 is installed on the inner working surface of the gripper 1111 to sense contact force. This exploded view clearly presents all functional modules of the gripper, including clamping, rotation drive, and force sensing, illustrating the working principle of the electric push rod 113 extending and retracting to drive the slider 119, which in turn drives the gripper 1111 to open and close through the parallelogram mechanism, and the gimbal motor 114 driving the entire gripper assembly to rotate. The capping gripper 11 is equipped with two tactile sensors 1114, which are respectively installed on the inner side of the gripper. The two sensors collect three-dimensional force data in real time (including...). These components together constitute six channels of force data.

[0051] For further information, please refer to [link / reference]. Figure 3 , Figure 3 The connection between the end effector and the robotic arm in the robot capping device is clearly shown. Figure 3 In the diagram, component 11 is the cap-screwing gripper, serving as an end effector. Component 12 is the last joint or link of the robotic arm. As clearly seen in the figure, the cap-screwing gripper 11 is directly fixed to the end interface of the robotic arm 12 via its right-side flange 111 (not separately shown in the figure, but part of component 11). This rigid connection ensures that the robotic arm 12 can accurately position the cap-screwing gripper 11 above the target bottle cap, providing stable support and pose adjustment capabilities for subsequent capping operations. This figure emphasizes the overall mechanical integration of the robot system; the cap-screwing gripper 11, as a dedicated end tool, is mounted on a multi-degree-of-freedom robotic arm 12, thereby giving the system the ability to move flexibly in three-dimensional space and perform complex tasks. The robotic arm 12 can be adapted to industrial collaborative arms or humanoid robot arms, enhancing the system's versatility.

[0052] Furthermore, in the robotic capping device, one side of the flange 111 of the capping gripper 11 is fixed to the end of the robotic arm 12, and the other side is connected to the gimbal motor 114. The electric push rod 113 drives the gripper 1111 to move through the slider 119, and the tactile sensor 1114 collects 6-channel force data in real time. The data processing module uses an exponential moving average smoothing signal with a preferred smoothing coefficient of 0.25 to balance noise reduction and response speed. Feature engineering is extended to 22 dimensions, including resultant force, surrogate torque, etc. The pitch prediction module uses the LSTM Atn Classifier model constructed in this invention, and the control module is based on kinematic constraints. Adjust the speed, where v represents the axial feed speed, n represents the rotary joint speed, and P represents the predicted pitch.

[0053] Understandably, this system enhances the adaptive capability of the cap by multimodal fusion, provides global geometric priors through visual perception, captures local thread features through tactile perception, and enables online parameter recognition through machine learning models. This avoids the dependence of traditional methods on preset parameters and significantly enhances robustness and accuracy in unstructured environments.

[0054] Specifically, the visual perception module includes a depth camera and an instance segmentation model, which dynamically calibrates the conversion relationship between pixels and physical size through depth information and image segmentation technology.

[0055] It should be further noted that the depth camera uses an RGB-D sensor, such as Intel RealSense, and the intrinsic parameter matrix includes focal length and principal point coordinates. The preferred instance segmentation model is YOLOv11-seg. In dual-modal calibration, the depth mode takes precedence, and automatically switches to ArUco mode when the depth value is invalid for three consecutive frames or the variance is greater than 10mm; ArUco marks the side length. Set to 100mm. Pixel conversion factor. and Updated every frame, combined with temporal smoothing feedback, the sliding window size is set to 5 frames, the exponential smoothing factor is 0.3, and the physical size jump threshold is set to 3.5mm to filter out instantaneous false detections.

[0056] Understandably, the dynamic calibration mechanism overcomes the limitations of fixed working distance, adapts to changes in lighting and reflection environments, ensures mask purity through instance segmentation, improves the measurement accuracy of geometric parameters to the sub-millimeter level, and provides reliable priors for subsequent operations.

[0057] It should be further explained that the visual perception module acquires RGB and depth images through a depth camera at a frame rate of 30 frames per second. It then uses an instance segmentation model such as YOLOv11-seg for object detection and segmentation. After outputting a mask, morphological closing operations and maximum connected component processing are performed to eliminate noise. Depth information is combined with camera intrinsic parameters to dynamically calculate pixel / millimeter conversion coefficients.

[0058] Furthermore, when depth data is missing, the system switches to ArUco labeling mode. The bottle cap diameter and height are extracted using a three-segment fitting algorithm, with the residual sum of squares calculated using the following formula:

[0059] ,

[0060] in This represents the total sum of squares of the residuals. Indicates the first The width value of each pixel. This represents the average width of the bottle cap segment. This represents the average width of the bottleneck segment. This represents the average width of the bottle body segment. Indicates the dividing point between the bottle cap and the bottle neck. The value represents the dividing point between the bottle neck and the bottle body, and n represents the total number of pixel rows.

[0061] Specifically, the visual perception module uses regional global statistical characteristics to locate the bottle cap boundary, and accurately extracts the bottle cap diameter and height through a three-segment fitting algorithm and a dual-modal calibration mechanism (depth information and ArUco markers).

[0062] It should be further explained that the three-segment fitting algorithm extracts the mask width sequence along the central axis of the bottle. Preset feature span threshold 5 pixels to ensure average size of bottle cap segment Average value of bottle body All are greater than the average value of the bottleneck segment. and feature span threshold The sum. Exhaustive search for split points i1 and i2, such that Minimum, the formula for calculating bottle cap height is: ,in Indicates the pixel height of the bottle cap segment. Indicates the vertical transformation factor. This indicates the offset correction, with a preferred value of 2 pixels, to compensate for segmentation errors. The bottle diameter is extracted at 1 / 3 of the bottle height, and the bottle height is calculated based on the pixel distance between the top and bottom edges of the mask.

[0063] Understandably, this algorithm uses global statistics to replace local gradients, which makes it highly resistant to interference, accurately segments the bottle cap boundary, avoids the influence of reflection or noise, and has a bottle cap height measurement error of less than 0.5mm, providing a precise upper limit for the capping stroke.

[0064] Specifically, in the robot capping device, the capping gripper 11 includes a flange 111, an electric push rod 113, a gimbal motor 114, and a gripper assembly. The gimbal motor 114 drives the gripper to rotate, and the electric push rod 113 controls the opening and closing of the gripper through a slider mechanism.

[0065] It should be further noted that the flange 111 is made of aluminum alloy and is bolted to the end of the robotic arm 12. The stator of the gimbal motor 114 is embedded in the flange 111, and the rotor is connected to the gripper fixing block 118. The maximum torque is 2.7 N·m. The electric push rod 113 has a stroke of 50 mm and a thrust of 120 N. It is installed through the electric push rod fixing part 112. The slider 119 and the gripper 1111 adopt a concave-convex embedding design with a gap of less than 0.1 mm. The Wi-Fi control module 115 and the motor drive module 116 are powered by the battery 117 with a voltage of 24V, realizing wireless control.

[0066] Understandably, the structure is compact and reliable, with a gimbal motor providing precise rotation, an electric actuator enabling smooth opening and closing, and a wireless module enhancing flexibility, making it suitable for space-constrained scenarios such as laboratories or homes.

[0067] Specifically, the gripper assembly of the capping gripper 11 includes a gripper 1111, a gripper connecting rod 1112, and a central fixing frame 1113, forming a parallelogram mechanism. The electric push rod 113 drives the gripper 1111 to move relative to each other through the slider 119 and the slider connecting rod 1110.

[0068] It should be further noted that the gripper connecting rod 1112 is connected to the central fixing frame 1113 via a pin, with a tolerance controlled within 0.05mm. When the electric push rod 113 extends or retracts, the slider 119 moves along the guide rail, pushing the gripper 1111 to rotate around the central fixing frame 1113 via the slider connecting rod 1110, with an opening and closing range of 0-100mm. The tactile sensor 1114 is a six-dimensional force sensor with a flexible surface, a range of ±25N, a measurement accuracy of 0.01N, and is installed inside the gripper 1111, with a sampling rate of 250Hz.

[0069] Understandably, the parallelogram mechanism ensures symmetrical movement and avoids uneven loading, while the tactile sensor monitors force feedback in real time to prevent over-tightening or slippage, thus improving operational safety.

[0070] Please see Figures 4-5 The present invention provides another embodiment, which provides a robot capping method based on multimodal perception and machine learning. The robot capping method based on multimodal perception and machine learning includes:

[0071] S100. Obtain bottle cap geometric parameters through visual measurement.

[0072] It should be further noted that in the visual measurement step, the camera intrinsic parameters are obtained through calibration, the depth value is taken as the median of the target region, and the confidence threshold of the instance segmentation model is set to 0.7. S200: Tactile force data is acquired through small-angle rotation; the tactile force data is preprocessed and feature engineering is performed to generate a standardized feature tensor. Where B is the number of samples, 320 is the time step, and 22 is the dimension of the feature vector constructed in feature engineering.

[0073] It should be further noted that the small-angle rotation can be set to 12°, corresponding to the number of revolutions. ,in Indicates the rotation angle. n s This indicates the number of revolutions corresponding to small angles. Data preprocessing includes signal smoothing, stage-aware resampling to fix the total length at T=320 time steps, and feature engineering to construct a 22-dimensional vector, such as the resultant force. , where F xy1 F represents the resultant force in the x and y directions of sensor 1 (left tactile sensor). x1 This represents the force in the x-direction of sensor 1. This represents the force in the y-direction of sensor 1. (Proxy torque) ,in This indicates the surrogate torque of sensor 1. Indicates the bottle cap diameter; sensor 2 (right-side tactile sensor) is the same, the surrogate torque... The calculation directly relies on the bottle cap diameter obtained by the visual perception module. This demonstrates the deep integration of visual geometric priors and tactile force data, providing key physical interaction features for pitch prediction.

[0074] Furthermore, to more comprehensively describe the overall force applied by the grippers during the capping process, a feature based on dual-sensor force data fusion was introduced in the feature engineering. The vector sum of the forces exerted by the two tactile sensors in the plane (xy direction) is calculated and defined as the total resultant force. The calculation formula is as follows:

[0075]

[0076] in:

[0077] The force components (in Newtons, N) in the x and y directions are collected in real time by the tactile sensor (sensor 1) installed on the left side of the cap gripper.

[0078] The force components (unit: Newtons, N) in the x and y directions are collected in real time by the tactile sensor (sensor 2) installed on the right side of the cap gripper.

[0079] The magnitude of the vector sum of the forces measured by the two sensors in a plane (unit: Newtons, N). This physical quantity characterizes the magnitude of the total lateral force applied to the cap by the grippers at any given moment during capping, reflecting the overall force level and stability of the operation, and is a key indicator for evaluating the state of coordinated two-handed operation;

[0080] Ultimately, the feature engineering of this invention constructs a 22-dimensional feature vector, which is input into the subsequent machine learning model for pitch prediction. This feature set comprehensively covers local mechanical features from a single sensor (such as...) From ) to multi-sensor global fusion features (such as (Multi-layered information)

[0081] Furthermore, the design of the 22-dimensional vector specifically includes:

[0082] 1. The raw force signal (6D) is the raw three-dimensional force component directly acquired by the tactile sensor, reflecting the local mechanical state, specifically including:

[0083] : Force (tangential force) in the x-direction of sensor 1 (left side);

[0084] : Force in the y-direction (radial force) of sensor 1;

[0085] : Force in the z-direction (axial force) of sensor 1;

[0086] : Force (tangential force) in the x direction of sensor 2 (right side);

[0087] : Force in the y-direction (radial force) of sensor 2;

[0088] : The force (axial force) in the z direction of sensor 2.

[0089] 2. Planar resultant force characteristics (5-dimensional) are based on force calculation in the xy plane and describe the lateral force application state. It is a core supplement, embodying the vector fusion of dual-sensor data, specifically including:

[0090] : The magnitude of the resultant force in the xy plane of sensor 1;

[0091] : The magnitude of the resultant force in the xy plane of sensor 2;

[0092] : The scalar sum of the combined forces of the two sensors;

[0093] The absolute difference in the resultant force between the two sensors reflects the force imbalance.

[0094] The vector and amplitude of the force data from the two sensors characterize the total lateral force (net global external force) applied by the gripper.

[0095] 3. Axial force characteristics (1D) are the resultant force focused in the z-direction, related to the clamping force of the cap, specifically including:

[0096] The sum of the forces in the z-direction from the two sensors reflects the total axial pressure and the degree of force imbalance.

[0097] 4. Temporal difference features (2D) capture the trend of resultant force over time and are used to identify dynamic interactions. Specifically, they include:

[0098] : First-order difference of resultant force (rate of change);

[0099] : Second-order difference of resultant force (acceleration-based change).

[0100] 5. Proxy torque features (4-dimensional) fused with visual geometric priors (bottle cap diameter) ), calculating the equivalent torque is a key aspect of multimodal fusion, specifically including:

[0101] : The surrogate torque of sensor 1;

[0102] : The proxy torque of sensor 2;

[0103] General agent torque;

[0104] The ratio of torque to normal force reflects the efficiency relationship.

[0105] 6. Action phase features (4-dimensional) label the operation context, helping the model perceive the temporal phase, specifically including:

[0106] One-hot encoding (4-dimensional): identifies the current action phase (open, pause 1, return, pause 2), for example This indicates the unscrewing stage.

[0107] Specifically, in the step of collecting tactile force data, a reciprocating small-angle rotational motion is performed to obtain multimodal timing data containing thread characteristics, and the action phase is marked.

[0108] It should be further explained that the reciprocating motion includes four stages: opening, pause 1, returning, and pause 2, lasting a total of 2 seconds. Force data baseline correction is performed during the pause stage. A tactile sensor collects 6-channel force data. And record the one-hot encoding of the stage. ,in Indicates stage identifier, Indicates to unscrew. Indicates pause 1, means rotation, Indicates pause 2. Stage progress. ,in Indicates a schedule scalar. Indicates the current time. Indicates the start time of the phase. Indicates the end time of the phase.

[0109] Understandably, stage markers enable the model to perceive the operational context, capture the instantaneous features of thread engagement, and improve the accuracy of pitch prediction. Understandably, the amount of tactile force data (number of samples), the time step of each data point, and the 22-dimensional features constructed based on the tactile force data collectively generate a standardized feature tensor. Where B is the number of samples, 320 is the time step, and 22 is the dimension of the feature vector constructed in feature engineering.

[0110] S300 uses processed data and machine learning models to predict bottle cap pitch.

[0111] It should be further noted that the machine learning model is a pitch prediction model, and the input is a standardized tensor. Output pitch value.

[0112] It should be further explained that the pitch prediction module adopts a machine learning model. This invention constructs a deep neural network model called LSTM Attn Classifier, which is used to generate feature tensors and output pitch values. Its core components are a bidirectional long short-term memory network (Bi-LSTM) combined with an attention mechanism (Attention Layer), including a temporal feature extraction unit, an adaptive attention focusing unit, and a classification decision unit, used to predict the bottle cap pitch in real time from tactile force data.

[0113] Furthermore, the LSTM Atn Classifier model contains Bi-LSTM layers, with hidden layer dimensions... The model was trained using the AdamW optimizer with a learning rate of 0.001 and a dropout rate of 0.2.

[0114] Understandably, the attention mechanism adaptively weights key mechanical frames, suppresses noise, and improves the model's ability to identify thread features, achieving a pitch prediction accuracy of over 96%.

[0115] Specifically, the temporal deep learning model is a bidirectional long short-term memory network combined with an attention layer, used for pitch classification prediction.

[0116] It should be further explained that the Bi-LSTM has 128 units each in the forward and backward layers, outputting 256-dimensional features. The attention layer is an MLP hidden layer with 64 units, and the output layer uses softmax activation. The number of classes n corresponds to the pitch type, such as 2.0mm, 2.8mm, etc. The loss function is cross-entropy, the training epochs are 100, and the batch size is 32.

[0117] Understandably, this model captures mechanical time-series dependencies, has strong generalization ability, is applicable to various bottle cap specifications, and reduces the false positive rate.

[0118] S400 calculates the screw pitch and the number of opening turns, and the robotic arm performs intelligent capping operation.

[0119] It should be further explained that the number of turns required to open the smart screw cap is... ,in Indicates the height of the bottle cap. This represents the bottle cap conversion factor, with a preferred value of 3.0. This value is set based on statistical experience to cover common bottle cap height-to-diameter ratios. Here, the bottle cap height... The visual perception module determines the number of turns required to screw on the cap, ensuring that the operation is compatible with different cap sizes.

[0120] Understandably, this method reduces risk by probing at small angles, uses feature engineering and machine learning models to capture thread characteristics and achieve high-precision pitch recognition, dynamically compensates for motion trajectories, ensures adaptive opening and closing, and achieves a success rate of over 98%.

[0121] For easier understanding, please refer to Figure 5 , Figure 5The overall logic and execution steps of the robotic capping method are fully illustrated in flowchart form. The process begins with "sensor data acquisition," including the acquisition of visual and tactile force data. Subsequently, the raw tactile force data undergoes "data preprocessing" and "feature engineering" to extract effective feature representations. The processed historical data is then used to "train the pitch prediction model," completing the offline training phase. The trained model enters the "model deployment and online operation" phase for online inference. During online operation, firstly, the visual perception module performs "bottle size parameter estimation" to obtain dimensions such as the cap diameter and height; secondly, the "grippers clamp the cap," and a "small-angle cap opening attempt" is performed to obtain real-time tactile force data. Next, the obtained cap diameter and tactile force data are input into the model through the program interface to predict the screw pitch. Finally, the system estimates the number of screw threads using the obtained cap height and the predicted screw pitch. Finally, based on the predicted pitch and estimated number of rotations, the motion trajectory is planned (for each rotation of the capping gripper, the robotic arm needs to rise / fall by one pitch until it is opened / tightened), the "open / close cap operation" is executed, and "open / close cap detection" is continuously performed until the operation is completed. This flowchart clearly reveals the core two-stage idea of ​​this invention: "learn first, then apply." A general pitch prediction model is trained offline, and the pitch of the current cap is predicted in real time online through a small-angle trial motion, thereby achieving adaptive full-stroke capping. This perfectly embodies the closed-loop process of the method from data to decision.

[0122] Specifically, in the visual measurement step, a dual-modal calibration mechanism is used to perform adaptive size estimation by combining depth information and ArUco markers.

[0123] It should be further noted that depth information is prioritized, and depth anomalies are determined by a variance greater than 10mm for three consecutive frames. ArUco marks the side length. Pixel perimeter Calculated through contour detection. Scale factor. When used for depth defects, the bottle cap diameter is included in the size calculation. ,in This indicates the pixel width of the bottle cap. This represents the horizontal transformation coefficient. Data stabilization employs median filtering and exponential smoothing, with an anti-jump threshold of 3.5mm to ensure stable output.

[0124] Understandably, the dual-modal mechanism enhances robustness, adapts to transparent or reflective bottles, avoids measurement failures, and provides reliable input for subsequent steps.

[0125] This invention provides a robotic capping system and method based on visual-tactile fusion perception and machine learning. The system first uses a visual perception module, combined with depth space decoupling and instance segmentation techniques, to accurately acquire key geometric parameters of the target bottle, including the cap diameter, cap height, bottle diameter, and bottle height. This provides precise grasping posture guidance for the robotic arm's end effector and presets a theoretical upper limit for the capping stroke based on the cap height. Subsequently, the system collects six-channel force signals from the robot's end effector during the interaction process in real time, using a temporal feature deep learning model to achieve sub-millimeter-level accurate prediction of the bottle neck thread pitch. Finally, the system fuses the geometric priors from visual perception with the pitch parameters from visual-tactile fusion perception, decoupling and mapping them to a coordinated relationship between the axial feed displacement and angular displacement of the robotic arm's end effector. By dynamically compensating for the helical motion trajectory, adaptive opening / closing operations for bottles of different sizes are achieved. The specific design scheme of this invention includes:

[0126] 1. Vision-based adaptive estimation of bottle geometry parameters

[0127] This invention develops a non-contact visual perception module. First, the module performs non-contact geometric modeling of the target bottle. Then, using depth information and instance segmentation technology, it accurately obtains dimensional parameters such as the bottle cap diameter and height, providing precise grasping pose guidance for the robotic arm's end effector. A theoretical upper limit for the opening stroke based on the bottle cap height is preset. The core logic of the visual perception module includes deep learning-based semantic perception, dual-modal physical size calibration, structured geometric analysis, and temporal smoothing feedback, as detailed below:

[0128] (1) Dual-modal dynamic calibration and environmental adaptation mechanism: In order to overcome the dependence of traditional two-dimensional vision measurement on fixed focal length and constant working distance, this invention introduces a dual-modal calibration mechanism of depth feedback and ArUco QR code-assisted calibration:

[0129] The first is the main calibration mode (based on depth information): The system extracts the physical depth sampling value of the target center area in real time, combines it with the focal length parameters (including horizontal and vertical focal lengths) in the depth camera intrinsic parameter matrix, and dynamically calculates the pixel / millimeter conversion coefficient (px / mm) for each frame according to the pinhole imaging principle.

[0130] Second, the compensation mode (based on ArUco QR code calibration): In environments with missing depth data or high reflectivity, the system automatically switches to ArUco auxiliary mode. This is achieved by identifying known physical side lengths in the scene. The ArUco marker is used to calculate its pixel perimeter. Determine the scaling factor between pixels and actual size. .

[0131] (2) The instance segmentation and mask refinement system uses pre-trained instance segmentation models (such as YOLOv11-Seg, YOLOv8-Seg, SAM, etc.) to perform real-time detection and segmentation of the target bottle to obtain pixel-level masks. The internal holes are filled by morphological closing operations, and the background discrete noise is eliminated by the maximum connected component extraction algorithm, providing a clean topological input for subsequent geometric analysis.

[0132] (3) Structured geometry extraction strategy targets four key parameters of the bottle (bottle cap diameter) Bottle cap height Bottle diameter and bottle height Implement a differentiated extraction strategy. (Bottle cap diameter) and bottle cap height The three-segment fitting logic: To accurately extract the bottom boundary of the bottle cap for precise measurement of its height, this invention utilizes a three-segment constant fitting algorithm to divide the bottle body into a cap segment, a neck segment, and a body segment. First, the system extracts the width sequence of the top mask along the central axis of the bottle body. The algorithm constructs a spatial width profile curve. It presupposes that the bottle body consists of three geometric feature regions: the cap, the neck, and the body, with a width distribution conforming to a "wide-narrow-wide" topology. The system exhaustively searches for two optimal segmentation points, i1 (bottom boundary of the cap) and i2 (bottom boundary of the neck), to divide the width sequence into three sub-intervals. For each possible segmentation scheme, the algorithm calculates the sum of squared residuals between the pixel width within each segment and its corresponding mean:

[0133]

[0134] To ensure the rationality of the physical meaning, the algorithm introduces geometric constraints: these constraints must be satisfied. and (in (This is a preset feature span threshold). The system optimizes the total residual to achieve this. Minimize this value; the corresponding optimal dividing point i1 is then defined as the precise geometric boundary between the bottle cap and the bottle neck. The bottle cap diameter is... Bottle cap height ( (To correct the offset). Compared to traditional single-line edge detection or gradient operators, this algorithm utilizes the global statistical characteristics of the region, which can effectively resist the interference of mask edge jaggedness, reflective noise and local segmentation defects, and improve the measurement accuracy of bottle cap height to the sub-millimeter level.

[0135] Bottle diameter Extract the horizontal pixel width of the mask row at a height of 1 / 3 of the distance from the bottom of the bottle. Multiply by a coefficient get.

[0136] Bottle height Based on the vertical pixel distance between the top and bottom pixels of the mask. Multiply by a coefficient get.

[0137] (4) Dynamic smoothing stabilizer based on time-series feedback

[0138] A feedback mechanism integrating anomaly rejection and multi-level filtering is introduced: a physical size jump threshold (e.g., 3.5mm) is set to filter instantaneous false detection frames; median filtering is performed using a sliding window to eliminate impulse noise, and smoothed industrial-grade measurement data is output by combining the exponential moving average (EMA) algorithm.

[0139] To comprehensively evaluate the performance of the visual perception module, this invention measured bottle samples of various sizes and used the measurement results of a high-precision vernier caliper as the true value (GT). Figure 3 As shown. Performance verification employs various statistical metrics, including the coefficient of determination. Assess linear correlation, and evaluate accuracy and systematic bias using mean absolute error and mean error.

[0140] 2. Capping data acquisition

[0141] A robotic capping device is used to collect six-channel force data in real time during the capping process. The robotic capping device, such as... Figures 1-2 As shown, the device includes a cap-screwing gripper 11 and a robotic arm 12. The cap-screwing gripper 11 includes a flange 111, an electric push rod 113, an electric push rod fixing component 112, a gimbal motor 114, a Wi-Fi control module 115, a motor drive module 116, a battery 117, a gripper fixing block 118, a slider 119, a slider connecting rod 1110, a gripper 1111, a gripper connecting rod 1112, a central fixing frame 1113, and a tactile sensor 1114. One side of the flange is fixed to the end of the robotic arm, and the other side is fixed to the gimbal motor 114. The gimbal motor 114 includes a stator and a rotor. The stator is fixed to the flange 111, and the rotor is fixed in the groove of the gripper fixing block 118. The rotation of the rotor drives the gripper to rotate, thereby realizing the cap-screwing operation. The end of the electric push rod 113 is fixed in the slider 119. The slider 119 and the gripper 1111 are interlocked. The gripper 1111, the gripper connecting rod 1112 and the central fixing frame 1113 form a parallelogram mechanism. The extension and retraction of the motor push rod drives the push slider 119 to move up and down, thereby pushing the gripper to rotate around the central fixing frame 1113. The symmetrical grippers 1111 on both sides move relative to each other to clamp or release the bottle cap.

[0142] During data acquisition, the gripper clamps the bottle cap, the tactile sensor contacts the cap, and the gimbal motor drives the gripper to rotate counterclockwise by a small angle (e.g., 12°), then clockwise by the same small angle, to loosen and tighten the cap respectively. Through the loosening and tightening process of the gripper and the cap, multimodal time-series data samples containing thread characteristics are acquired, including 6-channel raw force data from tactile sensor 1 and tactile sensor 2, as well as markers for the four action stages of the capping process: unscrewing, pause 1, rewinding, and pause 2.

[0143] 3. Data Preprocessing and Feature Engineering Construction: In order to eliminate sensor noise, align spatiotemporal dimensions and enhance the physical expressiveness of features, this invention designs a rigorous data preprocessing pipeline to transform the variable-length raw tactile signals into standardized spatiotemporal tensors.

[0144] (1) Signal denoising and smoothing: The original torque signal is smoothed using the exponential moving average method. This is achieved by setting a smoothing coefficient. The model can effectively filter out transient interference while maintaining the responsiveness of mechanical features in the time dimension, providing a stable input benchmark for subsequent feature extraction.

[0145] (2) Stage-aware resampling and temporal alignment: For the four stages of “spin-pause 1-spin-pause 2”, linear interpolation sampling is performed independently for each stage, followed by sequence splicing, and the total length is uniformly fixed at T=320 time steps. This ensures that different samples have consistent physical semantics at the same index, achieving accurate alignment in the temporal domain.

[0146] (3) Construction of multi-dimensional feature engineering: In order to capture deeper physical interaction information, the input features were expanded from the original 6-channel force sensory signal to a 22-dimensional fused feature tensor, and a high-dimensional spatiotemporal feature matrix was constructed, covering physical force signals, proxy torque and action stages:

[0147] The original force signals from sensor 1 and sensor 2: The resultant force of sensor 1 and sensor 2 in the x and y directions and , the sum of the combined forces And the difference between the sum and the sum of forces The sum of the forces in the z-direction of sensor 1 and sensor 2. First-order difference of resultant force: Second-order difference of resultant force .

[0148] The surrogate torque of sensor 1 and sensor 2: , The ratio of torque to normal force: .

[0149] Action Phase: Integrating one-hot encoding (4-dimensional) of four action phases—"Spin Open - Pause 1 - Spin Back - Pause 2"—and the execution progress within each phase (0-1 scalar), the model is endowed with the ability to perceive the current operation state. The phase one-hot encoding is as follows: ,in This indicates whether the model is currently processing the spin-out, pause 1, spin-back, or pause 2 stage, enabling the model to have action semantic awareness. Stage progress This provides the model with its relative position in a specific action at the current moment, thereby capturing the nonlinear characteristics of the thread engagement moment more accurately.

[0150] (4) Feature Standardization: This invention employs Z-Score standardization to eliminate the significant magnitude difference between force values ​​(typically large) and progress features (range 0-1). By subtracting the training set mean and dividing by the standard deviation, all features are mapped to a uniform distribution space, thereby eliminating the influence of magnitude, accelerating model convergence, and ensuring numerical stability. Finally, each preprocessed sample is represented as a feature tensor. Where B is the number of samples, 320 is the aligned time step, and 22 is the feature vector dimension defined above.

[0151] 4. Pitch Prediction Model

[0152] Model Definition: This invention constructs a deep neural network model called LSTM Attn Classifier for processing feature tensors and outputting pitch values. The model mainly consists of the following three cascaded units:

[0153] (1) Bidirectional Temporal Feature Extraction Unit (Bi-LSTM Layer): This unit consists of two Bi-LSTM layers used to capture the sequential dependencies of mechanical signals during the cap-tightening process. The input layer receives a tensor of shape (Batch_size, T, N), where Batch_size, T, and N represent the sample size, sample data time step, and feature dimension, respectively. The output of this layer is a sequence feature containing deep physical semantics. The feature extraction unit contains a multi-layer bidirectional long short-term memory network (Bi-LSTM). The forward LSTM is responsible for learning the evolution trend of force from "contact" to "engagement"; the backward LSTM is responsible for using subsequent mechanical feedback to correct the understanding of the early state; finally, it outputs a 256-dimensional (128×2, in this embodiment the hidden layer dimension is set to 128) hidden layer feature vector for each time step.

[0154] (2) Adaptive Attention Layer: During the rotation process, not all moments contribute equally to pitch recognition. Key information is often contained in the brief moments of thread engagement or sudden changes in resistance. Therefore, this embodiment introduces a lightweight attention mechanism. First, an importance score for each time step is calculated using a multilayer perceptron (MLP) with a Tanh activation function; second, the score is normalized into attention weights using a Softmax function; finally, the variable-length sequence is aggregated into a fixed-length global context vector through weighted summation. This mechanism enables the model to adaptively focus on the most discriminative physical interaction frames while suppressing noise interference during idle or unstable phases.

[0155] (3) Classification Decision Unit: This unit is responsible for mapping abstract features to pitch category probabilities. The processing flow is as follows: Normalization layer: Normalizes features to eliminate magnitude differences between different samples and accelerates convergence; Random deactivation layer: Randomly discards 20% of neuron connections to enhance the model's generalization ability and prevent overfitting; Linear projection layer: Fully connects the prediction results to n output nodes (corresponding to n pitch categories), outputting the classification log odds (Logits) value. Model training: The model uses the cross-entropy loss function for end-to-end training and uses Adam W as the optimizer to minimize the difference between the predicted distribution and the true label, and saves the optimal model parameters / weights according to the loss function during training. The model establishes the mapping relationship between force and pitch by learning the force change law during screw rotation.

[0156] 5. Intelligent lid opening operation

[0157] (1) Real-time pitch prediction and model deployment

[0158] To achieve adaptive opening of non-standard bottle caps, the optimal weights of the trained LSTM Attn Classifier model are deployed in the robot's real-time control system. First, data triggering and acquisition are performed. The robotic arm drives the cap-screwing gripper to perform an initial detection action, and the gimbal motor drives the gripper to perform a reciprocating motion of "counterclockwise loosening-pause-clockwise tightening-pause" at a preset small angle (e.g., 12°). During this process, the tactile sensor 1114 collects 6 channels of raw force data in real time. Next, online inference is performed. The collected force data is transformed into standardized feature tensors through a preprocessing pipeline of "signal denoising, stage resampling, feature engineering construction, and standardization." This tensor is input into the online prediction model, which outputs the predicted pitch value P of the current bottle cap in real time.

[0159] (2) Adaptive motion compensation strategy:

[0160] ① During the capping process, the end joint motor of the robotic arm 12 is locked, and the rotation is driven only by the gimbal motor 114 of the capping gripper 11, achieving continuous multi-turn motion (≥1 turn). This design avoids the inefficient operation of releasing the gripper to return to its original position after each rotation in the traditional method. Through the continuous rotation mechanism of the gimbal motor, the movement limitations of the robotic arm joints are overcome, achieving seamless capping operation and improving the adaptability and efficiency of the invention.

[0161] ② The robotic arm control system dynamically adjusts the coordination between the end effector rotary joint and the axial movement axis based on the real-time acquired pitch prediction value P, achieving precise helical trajectory compensation. The rotational speed n (rad / s) and axial feed speed v (m / s) of the robotic arm end effector joint motor must strictly meet the kinematic constraints: This strategy ensures that the axial lifting height of the gripper is exactly equal to one pitch P for each rotation of the gripper, effectively avoiding damage to the tactile sensor, wear of the bottle neck threads, or slippage of the gripper due to speed mismatch.

[0162] (3) Dual determination mechanism for completion of opening

[0163] The system combines visual feedback and tactile perception to establish a multimodal fusion mechanism for determining the opening state, ensuring operational robustness. This invention proposes two determination strategies:

[0164] Strategy 1 (Vision-based determination): Utilizing the estimated bottle cap height from the vision system. The theoretical number of rotations required to open the cap is calculated as m = H / 3P (or other preset proportional coefficients). When the gimbal motor rotates to the target number of rotations m, it is determined that the cap has disengaged from the thread.

[0165] Strategy 2 (based on tactile torque determination): During the execution of helical motion, the system continuously monitors the proxy torque value fed back by the tactile sensor. The torque value drops significantly the instant the bottle cap is completely disengaged from the bottle neck thread; when... When the temperature remains below the preset low threshold, the opening task is considered successfully completed, and the system then instructs the robotic arm to stop rotating and lift.

[0166] 6. Intelligent lid closing and tightening detection

[0167] (1) Adaptive cap closing: When performing the cap closing task, the gripper precisely aligns the cap with the bottle opening, calls the predicted pitch value P, and performs a spiral pressing motion that is opposite to the opening direction and matches the parameters.

[0168] (2) Tightening safety monitoring: The system judges the degree of tightening by monitoring the force of the tactile sensor in real time. When the force feedback reaches the preset tightening threshold, the rotation will stop automatically to prevent the bottle cap from being damaged or the motor from being overloaded due to over-tightening, thus realizing the intelligence and flexibility of the entire opening-closing process.

[0169] Furthermore, during the cap-closing operation, the system first calculates the number of turns *n* to open the cap based on the cap height and predicted screw pitch obtained from the visual perception module, and then performs the same number of turns in the opposite direction to initially tighten the cap. This is followed by a slow tightening phase (rotation speed reduced to 20%-30% of the initial speed to enhance sensitivity). During this phase, the system uses tactile sensors on the cap-closing jaws to collect tangential force in real time. Components (including the left side) and the right side The data was preprocessed at a sampling rate of 250Hz, including exponential moving average smoothing (smoothing coefficient a=0.25) to suppress noise, and calculations were performed. instantaneous first-order difference (i.e., the rate of change of force between the current frame and the previous frame) to capture abrupt changes; when When more than 3 consecutive sampling points are collected and the average value exceeds the preset threshold (preferred value is 1.5N / s, which is determined based on the tactile sensor range of ±25N, measurement accuracy of 0.01N, and calibration based on a large amount of experimental data to ensure that it is triggered when the force value suddenly increases by 5%-10%), the system determines that the tightening is complete and immediately stops the operation of the gimbal motor to prevent over-tightening or damage to the bottle cap.

[0170] The following specific example illustrates the size estimation process of this invention:

[0171] This example provides a method for estimating bottle size parameters based on a depth camera and a YOLOv11-seg segmentation model, which provides initial prior parameters for subsequent capping operations. The system acquires RGB and depth images using a depth camera, identifies the bottle and extracts its contour using the YOLOv11-seg model, and converts pixel coordinates into physical dimensions by combining depth information and camera intrinsic parameters, outputting bottle cap height, bottle cap width, bottle diameter, and bottle height parameters.

[0172] S1. Image Acquisition and Alignment: Simultaneously acquire RGB color and depth images using a depth camera at a frame rate of 30 frames per second. Align the two data streams, mapping the depth image to the RGB image coordinate system to establish pixel coordinates. A one-to-one mapping relationship with the depth value d. Reading camera intrinsic parameters includes focal length. and principal point coordinates Used for pixel-to-physical coordinate transformation.

[0173] S2. Auxiliary Calibration: Place an ArUco QR code marker (100mm×100mm) of known size in the scene. The system automatically detects the marker and calculates the pixel / millimeter ratio (scale_ppm). When the depth data quality is good, depth information combined with intrinsic parameters is used to calculate the size; when the depth data is abnormal, the system automatically switches to the ArUco calibration scheme to ensure measurement robustness.

[0174] S3. Object Detection and Segmentation: Input the RGB image into the instance segmentation model (such as YOLOvl1-seg) for real-time inference, and output the detection box coordinates, segmentation mask, and confidence score. Perform morphological preprocessing on the segmentation mask: binarization, hole filling by closing operation, and noise removal by connected component analysis, and output a clear and continuous bottle outline mask.

[0175] S4. Depth Extraction and Contour Construction: Region sampling (e.g., radius 4 pixels) is performed around the target centroid or measurement point, and median filtering is used to filter depth noise. The mask is scanned line by line, extracting the start and end pixel coordinates of each line. Combining the depth and focal length of the midpoint of that line, a spatial physical width sequence is constructed using the principle of similar triangles. A sliding window moving half-average filter is applied to this sequence to eliminate discrete spikes caused by segmentation fluctuations. ArUco scaling is used when depth is unavailable.

[0176] S5. Bottle Cap Boundary Localization: A dual-strategy approach is employed. The primary strategy is a three-segment fitting algorithm, dividing the bottle into three segments: the cap, the neck, and the body. Based on the characteristic that the neck segment has the smallest width, the algorithm iterates through the combination of segmentation points and calculates the fitting error, selecting the optimal segmentation scheme to output the cap-neck boundary line. The backup strategy is width abrupt change detection, calculating the width change rate and combining it with edge strength to construct a score, selecting the position with the highest score as the boundary line.

[0177] S6. Size Parameter Calculation: The system extracts the physical depth sample value z of the target's central region in real time, and combines it with the focal length parameter in the depth camera's intrinsic parameter matrix. (including horizontal focal length) With vertical focal length Based on the principle of pinhole imaging, the pixel / millimeter conversion factor for each frame is dynamically calculated. The system calculates the size by taking the mask pixels of each part in advance and combining them with the depth value.

[0178] Bottle cap diameter: ;

[0179] Bottle cap height: ( (To correct the offset)

[0180] Bottle diameter: ;

[0181] Bottle height: .

[0182] S7. Data Stabilization: Employing multiple stabilization strategies, specifically including:

[0183] Median filtering: Maintain the queue of the most recent 5 frames and take the median.

[0184] Exponential smoothing: stable_value = 0.3 × median + 0.7 × stable_value_prev.

[0185] Anti-jump: Discard data when the difference is >3.5mm, and reset after 15 consecutive invalid frames.

[0186] Furthermore, stable values ​​are output for practical applications.

[0187] S8. Result Output: Mark the detection box, measurement lines, and dimension values ​​on the RGB image and display them in real time at 30 frames per second. End-to-end latency <50ms.

[0188] The following specific example illustrates the process of tactile force data acquisition, processing, and model training in this invention:

[0189] This example provides a data acquisition, processing, and model training method for the bottle cap screwing process. By acquiring bilateral three-dimensional force signals and performing consistent preprocessing and feature construction, a pitch recognition model and its supporting standardized files are trained. The training is completed in an offline training mode, and the preprocessing and feature construction logic in the training stage is consistent with that in online inference, thereby ensuring that the input distribution is consistent with the training when the model is deployed.

[0190] S1. Data Acquisition: During the twisting process, tactile sensors installed on both sides of the gripper synchronously acquire timing data. The raw signals include at least the three-dimensional force on the right side. With the three-dimensional force on the left Simultaneously, the bottle cap diameter estimated by the visual perception module is recorded. The collected data is stored in CSV format and associated with the sample labels for subsequent offline training processing.

[0191] S2. Data Preprocessing: The original sequence is standardized to reduce the impact of noise and outliers and ensure sample comparability. This includes truncating or aligning the sequence to form a fixed-length time-series input (total length of 320 time steps); detecting outlier sampling points and repairing them using interpolation; smoothing and filtering the repaired signal to suppress high-frequency noise and retain trend information; and standardizing field names to ensure consistency between feature fields in the training and inference stages.

[0192] S3. Feature Construction and Standardization: Using Six-Channel Original Force Signals and As a fundamental feature, resultant force features and moment features are constructed, including:

[0193] The resultant force of sensor 1 and sensor 2 in the x and y directions , the sum of the combined forces And the difference between the sum and the sum of forces ;

[0194] The sum of the forces of sensor 1 and sensor in the z-direction ;

[0195] In addition, differential features are constructed to describe the temporal variation trend, including the first-order difference of the resultant force and the second-order difference of the resultant force. ;

[0196] The surrogate torque of sensor 1 and sensor 2 and the combined surrogate torque: The ratio of combined torque to normal pressure: .

[0197] In addition, the features also include action phase information: a one-hot encoding (4-dimensional) integrating four action phases "rotate-pause 1-rotate-pause 2" and the execution progress within each phase (0-1 scalar), giving the model the ability to perceive the current operation state. The phase one-hot encoding is as follows: ,in This indicates whether the model is currently processing the open, pause 1, return, or pause 2 stage, enabling the model to have the ability to perceive action semantics.

[0198] Phase progress This provides the model with its relative position within a specific action at the current moment. The above features are combined to form a feature matrix and then standardized. A Z-Score normalizer is used to perform a zero-mean, unit variance transformation on each feature dimension, and the normalizer is saved for reuse in the inference stage.

[0199] S4. Label and Dataset Construction: Establish a label mapping relationship of "category index - pitch" based on the preset pitch standard set, and divide the samples into training set, validation set and test set to support model training and performance evaluation; a hierarchical method is adopted when dividing to keep the distribution of samples in each category consistent.

[0200] S5. Model Training and Output: Model training and output; a time-series classification model is used to train pitch category prediction on a fixed-length sequence, preferably using a Bi-LSTM network; the model input is the feature sequence constructed and standardized according to S3, and the output is the prediction results for each pitch category; after training, the model weight file and the corresponding normalizer and feature column definition file are saved to ensure that the data processing and feature construction process of S2 and S3 can be reproduced in the deployment and inference stage.

[0201] The following specific example illustrates the process of the intelligent opening / closing lid operation of the present invention:

[0202] This example provides an intelligent lid opening / closing method based on visual measurement and pitch prediction models. In online application mode, the system executes the following steps sequentially: visual measurement, small-angle acquisition, data processing, pitch prediction, and lid opening / closing control, forming an automated closed-loop process.

[0203] S1. Visual Inspection and Geometric Parameter Initialization: The system first obtains key geometric quantities of the bottle body and cap through the camera and object detection network, including the cap diameter. Bottle cap height Bottle diameter and bottle height The detection results are written into the parameter cache of subsequent control steps, used to calculate the gripper's descent height and motion planning, and can be displayed and saved in the interface.

[0204] S2. Small Angle Acquisition (for online pitch identification): To reduce trial costs and avoid the risks associated with direct long-stroke cap tightening, the system performs small-angle rotation acquisition: First, the entire system is lowered to near the bottle opening and the cap is clamped. Then, it performs four actions at a preset angle θ (12°): "unscrew → stationary → retract → stationary," continuously recording tactile force signals throughout the process. The four data segments are written to the same CSV file for subsequent rapid processing and inference. The number of rotations corresponding to the small angle is:

[0205]

[0206] S3. Online Data Processing: For the original CSV generated in step 2, the system calls the data processing module to automatically complete: truncation (preserving the key front part), outlier interpolation, Savitzky-Golay smoothing, and column name standardization, and outputs the processed CSV file to ensure that the online inference input is consistent with the training phase.

[0207] S4. Pitch Prediction and Confidence Output: The system loads the trained LSTM Attn Classifier model and its corresponding Standard Scaler, constructs features from the processed sequence, completes one forward inference, obtains the discrete pitch category, and maps it to continuous pitch values. The system also outputs the confidence score (conf). The prediction results are saved to a CSV file for the control module to read and use for decision-making.

[0208] S5. Cap opening / closing execution based on predicted screw pitch (closed-loop control): The control module reads the cap height saved in step 1. The pitch output in step 4 Calculate the number of rotations *n* required and execute the complete cap-opening action chain: "gripper descends, clamps the cap, rotates *n* times, releases, and rises back to its original position." Closing the cap involves executing the action chain: "gripper descends, aligns the cap with the bottle opening, rotates *n* times, releases, and rises back to its original position." The number of rotations is calculated as (where...). (α is the conversion factor / calibration factor for the mechanism; in this embodiment, based on the statistical relationship between cap height and number of turns, α is taken as 3.0). And print key intermediate values ​​in real time during execution. (Parameters such as...) are used for interpretable diagnoses.

[0209] In a preferred embodiment, this application also provides an electronic device, the electronic device comprising:

[0210] The system includes a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, implement the described robotic cap-screwing system based on multimodal perception and machine learning. This computer device can be broadly categorized as a server, terminal, or any other electronic device with the necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, memory, network interface, communication interface, etc., connected via a system bus. The processor of the computer device can be used to provide the necessary computing, processing, and / or control capabilities. The memory of the computer device may include non-volatile storage media and internal memory. The non-volatile storage media may store an operating system, computer programs, etc. The internal memory can provide an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface and communication interface of the computer device can be used to connect and communicate with external devices via a network. When the computer program is executed by the processor, it performs the steps of the method of the present invention.

[0211] This invention can be implemented as a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the steps of the methods of embodiments of the invention to be performed. In one embodiment, the computer program is distributed across multiple network-coupled computer devices or processors, such that the computer program is stored, accessed, and executed in a distributed manner by one or more computer devices or processors. A single method step / operation, or two or more method steps / operations, may be executed by a single computer device or processor or by two or more computer devices or processors. One or more method steps / operations may be executed by one or more computer devices or processors, and one or more other method steps / operations may be executed by one or more other computer devices or processors. One or more computer devices or processors may execute a single method step / operation, or execute two or more method steps / operations.

[0212] Those skilled in the art will understand that the method steps of this invention can be performed by a computer program instructing related hardware, such as a computer device or processor, to perform the steps of this invention when executed. Depending on the context, any references herein to memory, storage, databases, or other media may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0213] The technical features described above can be combined arbitrarily. Although not all possible combinations of these technical features are described, any combination of these technical features should be considered to be covered by this specification, provided that such combination does not contain contradictions.

[0214] The purpose of disclosing the specific embodiments of the present invention described above is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A robotic cap-screwing method based on multimodal perception and machine learning, characterized in that, Includes the following steps: S100. Obtain the bottle cap's geometric parameters through visual measurement; the geometric parameters include the bottle cap diameter, bottle cap height, bottle diameter, and bottle height. S200: Collect tactile force data by rotating the bottle cap through a small-angle reciprocating motion; preprocess and feature engineering the tactile force data to generate a standardized feature tensor; S300. Construct a machine learning-based pitch prediction model and use the pitch prediction model to predict the bottle cap pitch based on the processed standardized feature tensor data. The constructed pitch prediction model includes a temporal feature extraction unit, an adaptive attention focusing unit, and a classification decision unit. Based on the temporal deep learning model, it processes the standardized feature tensor and focuses on key mechanical frames through an attention mechanism to perform pitch classification prediction. Based on the tactile force data obtained from mechanical interaction, it accurately predicts the pitch of the bottle cap in real time. S400, Estimate the number of thread turns based on the cap height and predicted pitch; Based on geometric parameters and predicted pitch, the robot plans its motion trajectory and performs intelligent capping operations through capping grippers and robotic arms.

2. The robot capping method based on multimodal perception and machine learning according to claim 1, characterized in that, In visual measurement to obtain bottle cap geometric parameters, the bottle cap is accurately segmented using a three-segment fitting algorithm, and the bottle cap diameter and height are further accurately extracted. A dual-modal calibration mechanism is used to combine depth information and ArUco markers for environmentally adaptive size estimation. The process of the three-segment fitting algorithm includes: The bottle body is divided into a cap section, a neck section, and a body section; the width sequence of the top mask is extracted along the central axis of the bottle body. Construct the spatial width profile curve; By exhaustively searching for two optimal split points, the bottom boundary of the bottle cap and the bottom boundary of the bottle neck, the width sequence is divided into three sub-intervals; For each partitioning scheme, calculate the sum of squared residuals between the pixel width and the corresponding mean in each segment, expressed as: in, This represents the total sum of squares of the residuals. Indicates the first The width value of each pixel. This represents the average width of the bottle cap segment. This represents the average width of the bottleneck segment. This represents the average width of the bottle body segment. Indicates the dividing point between the bottle cap and the bottle neck. The value represents the dividing point between the bottle neck and the bottle body, and n represents the total number of pixel rows.

3. The robot capping method based on multimodal perception and machine learning according to claim 1, characterized in that, Tactile force data is collected by performing small-angle reciprocating rotational motion to obtain multimodal time-series data including the characteristics of bottle cap threads, and then constructing multidimensional feature engineering. The normalized feature tensor is represented as Where B is the number of samples, 320 is the time step, and 22 is the dimension of the feature vector constructed in feature engineering.

4. The robot capping method based on multimodal perception and machine learning according to claim 1, characterized in that, The constructed pitch prediction model establishes a mapping relationship between force and pitch by learning the force changes during thread rotation; in the pitch prediction model: The bidirectional temporal feature extraction unit (Bi-LSTM Layer) consists of two Bi-LSTM layers used to capture the sequential dependencies of mechanical signals during the cap-tightening process. The input layer receives a tensor of shape (Batch_size, T, N), where Batch_size, T, and N represent the sample size, time step size, and feature dimension, respectively. The output is a sequence of features containing deep physical semantics. The extraction unit contains a multi-layer bidirectional long short-term memory network Bi-LSTM. The forward LSTM is used to learn the evolution trend of force from contact to engagement; the backward LSTM uses subsequent mechanical feedback to correct the understanding of the early state; and finally outputs a 256-dimensional hidden layer feature vector for each time step. The adaptive attention layer employs a lightweight attention mechanism; firstly, it calculates the importance score for each time step using a multilayer perceptron (MLP) with a Tanh activation function; secondly, it normalizes the score into attention weights using a Softmax function. The variable-length sequences are then aggregated into a fixed-length global context vector through weighted summation. The classification decision unit is used to map abstract features to pitch class probabilities, including: a normalization layer: normalizes the features; a random deactivation layer: randomly discards some neuron connections to prevent overfitting; and a linear projection layer: the prediction results are fully mapped to n output nodes, corresponding to n pitch classes, and outputting the classification log odds value.

5. The robot capping method based on multimodal perception and machine learning according to claim 1, characterized in that, The temporal deep learning model is a bidirectional long short-term memory learning network model based on an attention mechanism, used for pitch classification prediction.

6. A robotic capping system based on multimodal perception and machine learning implemented using the method of claim 1, characterized in that, include: The visual perception module is used to acquire the bottle cap diameter and bottle cap height of the target bottle in a non-contact manner. The robotic capping device includes a robotic arm and a capping gripper. The capping gripper is equipped with a tactile sensor for collecting multi-dimensional force data during the capping process. The data processing module is used to preprocess and construct features from the multidimensional force data to generate standardized feature tensors. The pitch prediction module employs a machine learning model to predict the bottle cap pitch based on the feature tensor. The control module is used to control the robot capping device to perform adaptive capping operations based on the cap height and the predicted screw pitch.

7. The robotic capping system based on multimodal perception and machine learning according to claim 6, characterized in that, The visual perception module includes a depth camera and an instance segmentation model, which dynamically calibrates the conversion relationship between pixels and physical size through depth information and image segmentation technology.

8. The robotic capping system based on multimodal perception and machine learning according to claim 7, characterized in that, The visual perception module uses regional global statistical characteristics to locate the bottle cap boundary and accurately extracts the bottle cap diameter and height through a three-segment fitting algorithm and a dual-modal calibration mechanism.

9. The robotic capping system based on multimodal perception and machine learning according to claim 6, characterized in that, In the robot capping device, the capping gripper includes a flange, an electric push rod, a gimbal motor, and a gripper assembly. The gimbal motor drives the gripper to rotate, and the electric push rod controls the opening and closing of the gripper through a slider mechanism. The gripper assembly of the capping gripper includes a gripper, a gripper connecting rod, and a central fixing frame, forming a parallelogram mechanism. The electric push rod drives the gripper to move relative to each other through a slider and a slider connecting rod.

10. An electronic device, characterized in that, The electronic device includes: a memory and a processor; The memory stores computer-readable instructions, which, when executed by the processor, implement the robot capping system based on multimodal perception and machine learning as described in claim 6.

Images