Robot autonomous disassembling method and system based on multi-source visual perception

Abstract

The invention discloses a robot autonomous disassembly method and system based on multi-source visual perception, and relates to the technical field of biological pharmacy, and the method comprises the following steps: obtaining initial image data of a target bagged product, including a color image collected by an RGB camera and a depth image collected by a depth camera, and respectively marking collection timestamps; synchronous alignment of image data is carried out based on timestamps, and spatial position information of different source images at the same moment is matched by constructing a unified time coordinate system. Through integration of technologies such as multi-source visual image synchronization, three-dimensional modeling and track control, residue detection and the like, high-precision autonomous disassembly of the biopharmaceutical bagged product by the robot is realized, the problem of misoperation caused by image time sequence asynchronization is solved, the system stability and the cleanliness control capability are improved, and good application value is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of biopharmaceuticals, and particularly to a robot autonomous disassembly method and system based on multi-source visual perception. Background Art

[0002] The "robot autonomous disassembly method based on multi-source visual perception" refers to a method in the field of biopharmaceuticals that uses a variety of visual sensing technologies (such as RGB cameras, depth cameras, infrared thermal imaging, etc.) to collect information such as the shape, position, and material of bagged products, and through artificial intelligence algorithms for identification and judgment, enabling the robot to have the capabilities of environmental perception, dynamic decision-making, and precise operation, so as to achieve automatic positioning, grasping, unsealing, and classification of sealed bagged items (such as sterile packages, medicine bags). This method can be widely applied to the feeding link before sterilization or sorting in the pharmaceutical production line, solving problems such as low efficiency of manual bag opening, high pollution risk, and inconsistent operation, and significantly improving the production cleanliness, automation level, and production efficiency.

[0003] The existing technology has the following deficiencies: In the existing robot autonomous disassembly process based on multi-source visual perception, the asynchronous timing of multi-source visual information is an easily overlooked but seriously affecting problem. Taking the disassembly of multi-layer sealed bags in the field of biopharmaceuticals as an example, if the data collected by the RGB camera and the depth camera are not aligned in real time, the system may misjudge that the bag body has been stably positioned, but in fact it is still in a micro-vibration state. At this time, if the robot performs tearing or cutting operations, it is very easy to cause uneven damage to the bag body, and even accidentally damage the sterile layer of the internal medicine, thus leading to serious consequences such as pollution, product scrapping, and shutdown of the entire production line.

[0004] The above information disclosed in the background art section is only used to enhance the understanding of the background of the present disclosure, and therefore it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] The purpose of the present invention is to provide a robot autonomous disassembly method and system based on multi-source visual perception. Through the integration of technologies such as multi-source visual image synchronization, three-dimensional modeling and trajectory control, and residual detection, the present invention realizes the high-precision autonomous disassembly of biopharmaceutical bagged products by the robot, solves the problem of misoperation caused by asynchronous image timing, improves the system stability and cleanliness control ability, and has good application value to solve the problems in the above background art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions: A robot autonomous disassembly method and system based on multi-source visual perception, including the following steps:

[0007] Obtain the initial image data of the target bagged product, including the color image collected by the RGB camera and the depth image collected by the depth camera, and mark the acquisition timestamps respectively;

[0008] Synchronize and align the image data based on the timestamps. By constructing a unified time coordinate system, match the spatial position information of images from different sources at the same moment;

[0009] Perform fusion processing on the synchronized image data, extract the three-dimensional contour information and dynamic state features of the bag body, and determine whether the target is in a stable static state;

[0010] On the premise of detecting that the target bag body is stable and static, perform the three-dimensional model positioning operation based on the fused image to determine the initial starting point and ending point of the tearing path;

[0011] Control the robot to perform the disassembly action along a fixed trajectory, perform cutting with a constant pressure and a preset speed, and maintain the path stability during the operation through feedback control;

[0012] After the disassembly is completed, call the vision system again to perform an integrity check on the disassembly area. After confirming that the packaging bag body is completely removed and there is no pollution risk inside, transfer the product to the next process.

[0013] Preferably, the acquisition frequencies of the RGB image and the depth image are set to 60 frames per second and 30 frames per second respectively. Interpolate the time points of the depth image to the time points of the RGB image through the linear interpolation algorithm to ensure that there are matching data points for both at each moment during the fusion processing stage;

[0014] To ensure the synchronization accuracy, perform a timestamp difference threshold judgment on the image data before it enters the processing module. If the time difference between any pair of images exceeds the set threshold of 20 milliseconds, discard the pair of image data and re-acquire it to ensure the consistency of the image input and the processing accuracy;

[0015] In addition, set a sliding window mechanism in the image frame data buffer area to perform alignment and matching in the buffer queue, improve the image synchronization success rate, and ultimately ensure the consistency of the subsequent three-dimensional modeling data.

[0016] Preferably, in the step of synchronizing and aligning the image data based on the timestamps, construct a unified time coordinate system, and use the master clock module to uniformly schedule the sampling behaviors of all vision sensors;

[0017] The master clock sends a global sampling trigger signal with millisecond-level precision. After all image sensors respond to the trigger signal, they immediately perform image acquisition and embed the current timestamp mark in the acquired data to ensure the logical time consistency of each data source;

[0018] The collected color images and depth images are sorted in time series respectively, and the sliding window matching algorithm is adopted to find the depth image frame with the closest timestamp for each frame of color image. If the time difference is within the set threshold range, the image is included in the fusion processing queue; otherwise, it is marked as invalid data and discarded.

[0019] To further improve the spatial alignment accuracy, after each pair of images is matched, spatial correction of the image resolution and field of view angle is also performed. The image is remapped through the pre-established internal and external camera parameter matrices, and finally, multi-source visual fusion input data is constructed under the unified time axis and unified spatial reference to ensure the consistency and accuracy basis for subsequent image analysis and processing.

[0020] Preferably, the extraction of three-dimensional contour information adopts a deep learning algorithm based on point cloud fusion, and the steps are as follows;

[0021] The RGB image and depth map data are respectively used to extract key region features through the feature extraction network to construct a joint feature vector, and then the spatial coordinate transformation model is used to restore each pixel point in the depth map to a three-dimensional spatial coordinate point to generate a dense point cloud model;

[0022] A convolutional neural network is used to judge the pose of the point cloud model, and it is judged whether the stable state is reached based on the stability confidence threshold;

[0023] When the confidence level exceeds 0.92, it is determined that the bag body enters the operable state; otherwise, wait for the static state to be detected again.

[0024] Preferably, the judgment of the stable static state of the bag body is based on the dynamic image frame difference analysis algorithm. By calculating the frame-by-frame difference of the continuously collected RGB image sequence, the movement trend of the bagged product within a short time window is quantified. Let the continuous N frames of RGB images be {I1, I2,..., I N}, each frame contains M pixel points, and the pixel difference between every two frames is normalized to calculate the frame difference value, which is used to measure the stability of the current bag body surface texture and edge position. The frame difference value calculation expression is as follows:

[0025]

[0026] , where D i is the flat frame difference value between the i-th frame and the i + 1-th frame, I i (f) is the gray value of the f-th pixel in the i-th frame image, and i i+1 (f) is the gray value of the f-th pixel in the i + 1-th frame image;

[0027] Construct a residual sequence for all frame difference values and calculate the standard deviation, which is used to identify the consistency degree of the inter-frame image changes. The standard deviation calculation formula is as follows:

[0028]

[0029] Wherein, is the average value of the frame difference sequence, and σ D is the standard deviation of the frame difference sequence, reflecting the stability of the inter-frame change;

[0030] If σ D < θ1, and then it is determined that the bag body is in a stable state, meeting the prerequisite conditions for performing the cutting operation. Among them, θ1 is the judgment threshold of the frame difference standard deviation, and θ2 is the judgment threshold of the average frame difference value.

[0031] Preferably, the determination of the cutting path adopts a composite algorithm based on edge detection and structural texture analysis, and the specific steps are as follows:

[0032] Use the Canny algorithm to perform preliminary edge extraction on the fused image;

[0033] Remove noise and pseudo-edges through morphological closing operation to obtain the stable bag body contour line;

[0034] Judge the sealing position of the bag body according to the image texture gradient direction, and input the sealing line as the starting point into the path planning module;

[0035] The path planning module calculates the optimal cutting trajectory using the principle of minimum curvature, ensuring that the trajectory extends along the sealing direction and avoiding cutting areas outside the packaging bag.

[0036] Preferably, the robot control system performs trajectory planning based on a vision guidance model and uses an adaptive trajectory calibration mechanism;

[0037] During the trajectory calibration process, a feedback controller is introduced to compare the deviation value between the actual tool head path and the predetermined path. If the deviation value exceeds 1.5 mm, the cutting direction is adjusted in real time, and the constant pressure value is adjusted synchronously;

[0038] Use the PID control algorithm to continuously adjust the angles of the three-axis robotic arm to ensure that the trajectory cutting is stable within the set error threshold range, and the maximum allowable error is 2 mm;

[0039] When the robot performs the disassembly operation, it uses a force control feedback system to dynamically adjust the pressure parameters, and the specific steps are as follows:

[0040] Collect the data of the tool head pressure sensor every 5 milliseconds during the cutting process, and judge the deviation between the current contact force and the preset value in real time;

[0041] If the current value is too high, reduce the supply voltage of the driving motor to reduce the tool head contact force; if it is too low, increase the voltage to maintain a constant cutting depth;

[0042] Meanwhile, an anti-interference filtering mechanism is introduced to filter out the noise data caused by micro-vibrations, ensuring the stability and effectiveness of the feedback results.

[0043] Preferably, for three-dimensional model positioning and path control, a high-order path smoothing function is adopted in combination with an error feedback control mechanism to ensure that the robot executes along a predetermined trajectory with high precision when disassembling bagged products. The specific deficiencies are as follows:

[0044] Extract the three-dimensional path point set P, P = {p j} = {p1, p2,..., p n}, where p j is the j-th discrete path point in the three-dimensional path, n is the total number of discrete path points, and each discrete path point is defined as p j = (x j , y j , z j ), where x j , y j , and z j respectively represent the coordinates of the j-th discrete path point in the x-axis, y-axis, and z-axis directions;

[0045] To achieve smooth movement of the robot in three-dimensional space, a spline interpolation path function is constructed based on the path point set to provide a continuous and differentiable path curve for the robot end effector to track. The formula of the spline interpolation path function is as follows:

[0046]

[0047] , where B j (t) is the B-spline basis function corresponding to the j-th discrete path point, a j is the interpolation weight coefficient corresponding to the j-th discrete path point, and S(t) is the spline interpolation path function;

[0048] During the execution of the robot's movement, the actual spatial position coordinates (x r , y r , z r ) of the robot end effector are obtained in real time, and the ideal trajectory point (x(t), y(t), z(t)) corresponding to the moment t is solved from the path function. The current execution error is calculated through the Euclidean distance formula, and the calculation expression is as follows:

[0049]

[0050] , where x r , y r , z rIt is the actual three-dimensional coordinate position of the robot end effector fed back in real time through encoders and position sensor devices. x(t), y(t), and z(t) represent the three-dimensional theoretical coordinate positions corresponding to the spline interpolation path function S(t) at time point t, which are used to compare with the current position of the robot. Δ(t) is the Euclidean distance error between the actual position and the theoretical path position of the robot under the current motion state;

[0051] When Δ(t) exceeds the set threshold δ, it is determined that there is a path deviation and feedback correction needs to be performed;

[0052] In the error offset state, the direction angle correction logic is started. According to the local geometric change trend between consecutive path points, the attitude adjustment angles in three dimensions are calculated, and the direction angle vector θ = (θ x 、θ y 、θ z ) is constructed. The calculation formula for each direction angle is as follows:

[0053]

[0054] In the formula, is the coordinate value of the discrete path point p j in the k-th dimension, is the coordinate value of the discrete path point p j+1 in the k-th dimension, d is the average Euclidean distance between consecutive path points, and θ k represents the angle change amount in the k-axis direction, which is used to correct the motion direction, and k ∈ {x, y, z};

[0055] The angle vector is transmitted to the motion solution module in the robot controller to drive it to adjust the current joint angles, so that the attitude of the end effector is realigned with the predetermined path direction again, realizing the precise tracking of the cutting path.

[0056] Preferably, the final integrity verification is jointly completed through visual inspection and residual quality control. The specific steps are as follows:

[0057] In the visual inspection part, the template matching algorithm is used to compare the pixel differences of the bag body boundaries before and after disassembly. If the boundary afterimage exceeds the set threshold, the "incomplete disassembly" state is prompted;

[0058] In the quality control part, the infrared imaging is used to detect whether there are residual particles or liquid reflection features on the workbench surface, and the infrared reflection intensity is converted into a binary image for residual recognition;

[0059] After combining the results of the two parts, the weighted confidence algorithm is used to comprehensively judge whether the disassembly is successful. Only when both the visual and quality control results are qualified, the product transfer process will be started.

[0060] Preferably, a robot autonomous disassembly system based on multi-source visual perception includes an image acquisition module, an image synchronization module, an image fusion and state determination module, a three-dimensional positioning and path planning module, a trajectory execution and force control feedback module, and a result detection and qualification determination module:

[0061] The image acquisition module obtains the initial image data of the target bagged product, including the color image collected by the RGB camera and the depth image collected by the depth camera, and respectively marks the acquisition timestamps.

[0062] The image synchronization module synchronizes and aligns the image data based on the timestamps, and matches the spatial position information of the images from different sources at the same moment by constructing a unified time coordinate system.

[0063] The image fusion and state determination module performs fusion processing on the synchronized image data, extracts the three-dimensional contour information and dynamic state features of the bag body, and determines whether the target is in a stable static state.

[0064] The three-dimensional positioning and path planning module, on the premise of detecting that the target bag body is stable and static, performs three-dimensional model positioning operations based on the fused image, and determines the initial starting point and ending point of the tearing path.

[0065] The trajectory execution and force control feedback module controls the robot to perform disassembly actions along a fixed trajectory, cuts with a constant pressure and a preset speed, and maintains the path stability during the operation through feedback control.

[0066] The result detection and qualification determination module, after the disassembly is completed, calls the vision system again to perform integrity verification on the disassembly area. After confirming that the packaging bag body is completely removed and there is no pollution risk inside, the product is transferred to the next process.

[0067] In the above technical solution, the technical effects and advantages provided by the present invention:

[0068] The present invention realizes high-precision, safe and automated disassembly operations of bagged products by robots in the biopharmaceutical field through constructing an image synchronization mechanism based on multi-source visual perception, integrating three-dimensional modeling and dynamic trajectory planning control, and combining full-process intelligent disassembly methods of residual detection and quality confirmation. It effectively solves the misjudgment problem caused by the asynchronous time sequence of multi-source images, fundamentally reduces the product pollution risk and the probability of operation errors, significantly improves the stability and controllability of the disassembly process, and has good industrial application prospects and promotion value. Description of the Drawings

[0069] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required in the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other accompanying drawings can also be obtained based on these drawings.

[0070] Figure 1 This is the method flow chart of the robot autonomous disassembly method based on multi-source visual perception of the present invention.

[0071] Figure 2 This is the module schematic diagram of the robot autonomous disassembly system based on multi-source visual perception of the present invention. Detailed implementation manners

[0072] Now, the exemplary embodiments will be described more comprehensively with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these exemplary embodiments are provided so that the present disclosure will be more thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art.

[0073] The present invention provides a Figure 1 robot autonomous disassembly method based on multi-source visual perception as shown in the following, including the following steps:

[0074] Obtain the initial image data of the target bagged product, including the color image collected by the RGB camera and the depth image collected by the depth camera, and mark the acquisition timestamps respectively;

[0075] The acquisition frequencies of the RGB image and the depth image are set to 60 frames per second and 30 frames per second respectively. The time points of the depth image are interpolated to the time points of the RGB image through the linear interpolation algorithm to ensure that there are matching data points at each moment during the fusion processing stage;

[0076] To ensure the synchronization accuracy, the time stamp difference threshold judgment is performed before the image data enters the processing module. If the time difference of any pair of images exceeds the set threshold of 20 milliseconds, the pair of image data is discarded and re-acquired to ensure the consistency and processing accuracy of the image input;

[0077] In addition, a sliding window mechanism is set in the image frame data buffer area for alignment and matching in the buffer queue to improve the image synchronization success rate and ultimately ensure the consistency of the subsequent three-dimensional modeling data.

[0078] Synchronize and align the image data based on the timestamps, and match the spatial position information of the images from different sources at the same moment by constructing a unified time coordinate system;

[0079] Steps for synchronizing and aligning image data based on timestamps, constructing a unified time coordinate system, and using a master clock module to uniformly schedule the sampling behaviors of all vision sensors;

[0080] The master clock sends a global sampling trigger signal with millisecond-level precision. After all image sensors respond to the trigger signal, they immediately perform image acquisition and embed the current timestamp mark in the acquired data to ensure that the data sources are logically time-consistent;

[0081] The collected color images and depth images are respectively sorted by time sequence, and the sliding window matching algorithm is used to find the depth image frame with the closest timestamp for each frame of color image. If the time difference is within the set threshold range (for example, less than 10 milliseconds), the image is included in the fusion processing queue; otherwise, it is marked as invalid data and discarded;

[0082] To further improve the spatial alignment accuracy, after each pair of images is matched, spatial correction of the image resolution and field of view angle is also performed. The image is remapped through the pre-established internal and external camera parameter matrices, and finally, multi-source visual fusion input data is constructed under a unified time axis and a unified spatial reference to ensure the consistency and accuracy basis for subsequent image analysis and processing.

[0083] Perform fusion processing on the synchronized image data, extract the three-dimensional contour information and dynamic state characteristics of the bag body, and judge whether the target is in a stable static state;

[0084] The extraction of three-dimensional contour information uses a deep learning algorithm based on point cloud fusion, and the steps are as follows;

[0085] Extract the key region features of the RGB image and depth map data through the feature extraction network respectively, construct a joint feature vector, and then use the spatial coordinate transformation model to restore each pixel point in the depth map to a three-dimensional space coordinate point to generate a dense point cloud model;

[0086] Use a convolutional neural network (CNN) to judge the pose of the point cloud model and judge whether it reaches a stable state based on the stability confidence threshold;

[0087] When the confidence level exceeds 0.92, it is determined that the bag body enters the operable state; otherwise, wait for the static state to be detected again.

[0088] Improve the accuracy and safety of the disassembly action through the above-mentioned depth point cloud fusion mechanism.

[0089] The judgment of the stable static state of the bag body is based on the dynamic image frame difference analysis algorithm, specifically including constructing a combined judgment mechanism of the dynamic segment difference matrix and the average frame difference index. By calculating the frame-by-frame difference of the continuously collected RGB image sequence, the movement trend of the bagged product within a short time window is quantified. Let N consecutive frames of RGB images be {I1, I2, …, I N}, each frame contains M pixel points. The pixel difference between every two frames is normalized to obtain the frame difference value, which is used to measure the stability of the current bag body surface texture and edge position. The frame difference calculation expression is as follows:

[0090]

[0091] In the formula, D i is the average frame difference between the i-th frame and the (i + 1)-th frame, I i (f) is the gray value of the f-th pixel in the i-th frame image, representing the light intensity at a certain frame and a certain pixel point of the image, and is used for the basic judgment of the inter-frame image change. I i+1 (f) is the gray value of the f-th pixel in the (i + 1)-th frame image;

[0092] Construct a residual sequence for all frame differences and calculate the standard deviation, which is used to identify the consistency degree of the inter-frame image change. The standard deviation calculation formula is as follows:

[0093]

[0094] In the formula, is the average value of the frame difference sequence, that is, the arithmetic average of all D i , σ D is the standard deviation of the frame difference sequence, reflecting the stability of the inter-frame change. After summing the squared deviations of each frame difference D i from the average frame difference and then taking the square root, if the standard deviation is very small, it means that the target change tends to be stable and the movement amplitude is low;

[0095] If σ D < θ1, and then it is determined that the bag body is in a stable state, meeting the prerequisite conditions for performing the cutting operation. Among them, θ1 is the judgment threshold of the frame difference standard deviation, and θ2 is the judgment threshold of the average frame difference value.

[0096] For example, in a typical application, θ1 = 0.08 and θ2 = 2.5 are set to ensure that the misjudgment probability is less than 0.5%. This stability judgment mechanism can effectively filter out false triggering behaviors caused by bag body shaking, jitter or environmental interference, and improve the decision-making reliability and production safety of the disassembly system.

[0097] Under the premise that the target bag is stable and stationary, a three-dimensional model positioning operation based on the fused image is performed to determine the initial starting point and end point of the tearing path;

[0098] The cutting path is determined by a composite algorithm based on edge detection and structural texture analysis. The specific steps are as follows:

[0099] Use the Canny algorithm to perform preliminary edge extraction on the fused image;

[0100] The noise and false edges are removed by morphological closing operation to obtain a stable bag contour line;

[0101] Determine the sealing position of the bag according to the direction of the image texture gradient, and input the sealing line as the starting point into the path planning module;

[0102] The path planning module uses the curvature minimization principle to calculate the optimal cutting trajectory, ensuring that the trajectory extends along the sealing direction to avoid cutting areas outside the packaging bag.

[0103] Through the above image structure analysis and path optimal control algorithm, automatic, accurate and lossless bag unloading path identification can be achieved.

[0104] Control the robot to perform disassembly actions along a fixed trajectory, use constant pressure and preset speed for cutting, and maintain path stability during operation through feedback control;

[0105] The robot control system performs trajectory planning based on a vision-guided model and uses an adaptive trajectory calibration mechanism;

[0106] During the trajectory calibration process, a feedback controller is introduced to compare the deviation between the actual tool head path and the predetermined path. If the deviation exceeds 1.5 mm, the cutting direction is adjusted in real time and the constant pressure is adjusted synchronously.

[0107] The PID control algorithm is used to continuously adjust the angle of the three-axis robot arm to ensure that the trajectory cutting is stable within the set error threshold range, with a maximum allowable error of 2 mm.

[0108] When executing the path, the robot will also dynamically update its end point coordinates based on the current visual feedback to prevent cutting errors caused by slight displacement of the bag.

[0109] When the robot performs the disassembly operation, it uses the force control feedback system to dynamically adjust the pressure parameters. The specific steps are as follows:

[0110] During the cutting process, the pressure sensor data of the cutter head is collected every 5 milliseconds to determine the deviation between the current contact force and the preset value in real time;

[0111] If the current value is too high, reduce the supply voltage of the driving motor to lower the cutting force; if it is too low, increase the voltage to maintain a constant cutting depth.

[0112] Meanwhile, an anti-interference filtering mechanism is introduced to filter out the noise data caused by micro-vibrations, ensuring the stability and effectiveness of the feedback results.

[0113] This force control mechanism can effectively reduce the interference risk of the cutting process to internal products and improve the disassembly accuracy.

[0114] Among them, for three-dimensional model positioning and path control, a high-order path smoothing function is adopted and combined with an error feedback control mechanism to ensure that the robot executes along the predetermined trajectory with high precision when disassembling bagged products. The specific deficiencies are as follows:

[0115] Extract the three-dimensional path point set P, P = {p j} = {p1, p2,..., p n}, where p j is the j-th discrete path point in the three-dimensional path, n is the total number of discrete path points, which is derived from the extraction of the structural edge coordinates after fusing visual image processing. The set is automatically selected from the point cloud data generated by fusing the RGB image and the depth map. Each discrete path point is defined as p j = (x j , y j , z j ), and x j , y j and z j respectively represent the coordinates of the j-th discrete path point in the x-axis, y-axis, and z-axis directions;

[0116] The three-dimensional path point set covers the edge sealing of the bag body as the target area, and the starting and ending positions of the cutting are identified through the structure boundary enhancement model, thereby constructing a complete cutting path;

[0117] To achieve smooth movement of the robot in three-dimensional space, a spline interpolation path function is constructed based on the path point set to provide a continuous and differentiable path curve for the robot end effector to track. The formula of the spline interpolation path function is as follows:

[0118]

[0119] In the formula, B j (t) is the B-spline basis function corresponding to the j-th discrete path point. The B-spline basis function is a piecewise-defined polynomial function used to construct a smooth curve within its definition interval. a jis the interpolation weight coefficient corresponding to the j-th discrete path point, which plays a role in controlling the path shape during the spline interpolation process. S(t) is the spline interpolation path function. As t varies from 0 to 1, it represents the spatial trajectory that the robot end effector should move along. t ∈ [0, 1] is the normalized time parameter of the path, which is used for continuous interpolation control from the path start point (t = 0) to the end point (t = 1).

[0120] The spline interpolation path function usually uses the B-spline function (B-spline) or the cubic spline (CubicSpline) in the spline curve as the mathematical expression form. In practical applications, especially in robot path planning, the cubic B-spline (CubicB-spline) is the most common form of the interpolation path function. Its characteristics are: each curve segment has continuous first-order and second-order derivatives at the start and end points, ensuring smooth curve transition and being suitable for dynamic motion control. In addition, variants such as natural spline, Bezier spline, or Catmull-Rom spline can also be used, but the B-spline has more obvious advantages in flexibility and local controllability, so it is widely used in precision control fields such as industrial robots, advanced modeling, and medical image reconstruction.

[0121] The B-spline basis function (B-spline Basis Function) is the mathematical basis for constructing the B-spline curve. It is a set of piecewise polynomial functions with local support. Each basis function B j (t) corresponds to a control point, and its non-zero interval is only on the fixed "knot interval" (knots), so it has the characteristic of locality: a single basis function only affects a small segment of the curve. This property enables adjusting a single control point without affecting the entire path, thus achieving local optimization and real-time correction, which is very suitable for robot path generation in a dynamic environment.

[0122] During the robot motion execution process, the actual spatial position coordinates (x r , y r , z r ) of the current position of the robot end effector are obtained in real time, and the ideal trajectory points (x(t), y(t), z(t)) corresponding to the moment t are solved from the path function. The current execution error is calculated through the Euclidean distance formula, and the calculation expression is as follows:

[0123]

[0124] In the formula, x r , y r , z rIt is the actual three-dimensional coordinate position that the robot end effector (such as the tool tip) feeds back in real time through the encoder and position sensor device. x(t), u(t), and z(t) represent the three-dimensional theoretical coordinate positions corresponding to the spline interpolation path function S(t) at time point t, which are used to compare with the current position of the robot. Δ(t) is the Euclidean distance error between the actual position and the theoretical path position in the current motion state of the robot;

[0125] When Δ(t) exceeds the set threshold δ (such as 2 mm), it is determined that there is a path deviation and feedback correction needs to be performed;

[0126] This error control strategy ensures that the trajectory executed by the robot strictly fits the path planning curve and avoids cutting deviation, damaging the edge of the bag or the contents due to the accumulation of small displacements.

[0127] In the error deviation state, the direction angle correction logic is started. According to the local geometric change trend between consecutive path points, the attitude adjustment angle in the three-dimensional direction is calculated, and the direction angle vector θ = (θ x 、θ y 、θ z ) is constructed. The calculation formula for each direction angle is as follows:

[0128]

[0129] is the coordinate value of the discrete path point p j in the k-th dimension (i.e., the x, y, and z directions), is the coordinate value of the discrete path point p j+1 in the k-th dimension (i.e., the x, y, and z directions), j ∈ {1, 2,..., n - 1}, d is the average Euclidean distance between consecutive path points, which is used for normalizing the angle calculation to avoid the problem of infinite angle magnification in the case of a small path spacing. θ k represents the angle change amount in the k-axis direction (x, y, z), which is used to correct the motion direction, k ∈ {x, y, z};

[0130] The angle vector is transmitted to the motion solution module in the robot controller to drive it to adjust the current joint angles so that the attitude of the end effector is realigned with the predetermined path direction again, realizing the precise tracking of the cutting path.

[0131] Through the above path interpolation function modeling, real-time error feedback calculation, and angle fine-tuning mechanism, the present invention significantly improves the spatial accuracy and dynamic robustness of the disassembly operation, effectively avoiding the problem of disassembly failure caused by the accumulation of path errors, and is particularly suitable for the automatic unpacking scenario in the biopharmaceutical field with extremely high requirements for cleanliness and product integrity. While ensuring the disassembly efficiency, this mechanism also improves the system's adaptability to complex bag shapes and the success rate of task execution.

[0132] After the disassembly is completed, the vision system is called again to perform an integrity check on the disassembly area. After confirming that the packaging bag is completely removed and there is no pollution risk inside, the product is transferred to the next process.

[0133] The final integrity check is jointly completed by visual inspection and residual quality control. The specific steps are as follows:

[0134] In the visual inspection part, the template matching algorithm is used to compare the pixel differences of the bag boundary before and after disassembly. If the boundary afterimage exceeds the set threshold, a "disassembly incomplete" status is prompted.

[0135] In the quality control part, infrared imaging is used to detect whether there are residual particles or liquid reflection features on the surface of the workbench, and the infrared reflection intensity is converted into a binary image for residual recognition.

[0136] After combining the results of the two parts, the weighted confidence algorithm is used to comprehensively judge whether the disassembly is successful. Only when both the visual and quality control results are qualified will the product transfer process be initiated.

[0137] Embodiment 1: In this embodiment, the system designs a stability judgment system based on a multi-source visual data synchronization mechanism for the actual application requirements of automatic disassembly operations on bagged sterile materials in the biopharmaceutical industry. The system mainly consists of a high-frame-rate RGB industrial camera, a structured light depth camera, a unified timestamp module, a data caching and alignment algorithm module, a three-dimensional reconstruction module, and a stable state judgment module. Its core objective is to accurately judge whether the bagged target object is in a stable static state before the robot performs the disassembly action, so as to avoid serious problems such as misjudging the cutting path and cutting the outer packaging of the medicine due to the movement, shaking, or tilting of the bag.

[0138] First, during the image acquisition stage, an RGB camera and a depth camera are respectively deployed. The RGB camera is used to capture high-resolution color images, and the depth camera is used to obtain the spatial depth data of the target object. Since the two types of cameras have different sampling frequencies and working mechanisms, there is a problem of temporal offset in the directly acquired data, which may cause the images to be out of sync and correspondence, thereby affecting the accuracy of 3D reconstruction and the accuracy of subsequent decision-making. To solve this problem, the system integrates a unified clock signal source in the image acquisition module, and all image acquisition devices are triggered to take pictures strictly according to this unified clock. Each frame of image is marked with a high-precision timestamp when it is acquired.

[0139] After the image acquisition is completed, the image data first enters the cache sliding window mechanism. This mechanism uses the timestamp as the primary key to automatically pair the image frames within a short period, ensuring that each RGB image frame can find a depth image frame with a time difference less than 20 milliseconds. If the time difference exceeds the threshold, this pair of images will be excluded, and the system waits for the next frame to be acquired. After being screened by the sliding window mechanism, it enters the image fusion stage.

[0140] The image fusion processing module uses the feature matching algorithm to pair the texture features in the RGB image with the spatial points in the depth image, and maps the two-dimensional image information into the 3D point cloud space, thereby generating a dense 3D model with spatial structure and surface texture information. This 3D model completely restores the actual spatial posture, edge structure, and surface morphology of the current bag.

[0141] After the 3D model is generated, the stable state judgment module continuously receives multiple frames of image data, and judges whether the object is in a stable state by comparing the degree of change in the point cloud position of consecutive frames. If the system detects that the spatial change of consecutive frames is small and the fluctuation trend tends to converge (for example, the bag no longer shakes, rotates, or tilts), it is judged that it is in an operable state; otherwise, it waits for the bag to stabilize before entering the cutting process. This mechanism ensures that the robot only operates at the safest moment, greatly reducing the probability of misoperation and potential pollution risk during the disassembly process.

[0142] Through the above mechanism, this implementation effectively improves the temporal consistency of visual perception, the accuracy of 3D model construction, and the ability to judge the timing of operation, providing a prerequisite judgment support with high reliability for the entire set of autonomous disassembly systems.

[0143] Embodiment 2: Facing the key problems such as inaccurate path recognition, large operation error and high aseptic requirements during the disassembly process of bagged materials, a high-precision cutting path planning and execution mechanism based on multi-source image fusion is proposed. Through multi-stage recognition and path optimization means, this mechanism can achieve functions such as highly matching the cutting path with the target sealing structure, stable and smooth disassembly actions, and dynamic adaptive control of cutting pressure and speed during the operation process, ensuring that the bag body is opened completely, safely and efficiently, and avoiding drug contamination or material waste.

[0144] First of all, the image processing part extracts edge features from the RGB image and performs image preprocessing. The system uses the Canny edge detection algorithm to preliminarily process the image, extracts clear contour lines of the bag body, and then eliminates noise and virtual edges caused by background interference, uneven packaging texture or light change through morphological operations such as dilation and erosion. Subsequently, the accurate position of the bag body seal is identified through image gradient direction analysis and structure pattern matching methods. This seal line, as the starting structure for subsequent path planning, is input into the path planning module.

[0145] The path planning module generates multiple path candidate solutions based on the seal line recognized by structure matching, and selects the optimal path through the curvature change minimization algorithm to ensure that the cutting path is smooth, continuous and highly accurate. Subsequently, this path is discretized into multiple position points and input into the trajectory control module for secondary interpolation and transition section smoothing processing, and finally a three-dimensional motion trajectory suitable for industrial robots is generated.

[0146] In the trajectory control stage, the system converts this three-dimensional trajectory into actual execution actions through the robot control unit. During the execution process, the system turns on the real-time position detection and pressure feedback mechanism to ensure that the robot moves stably along the set trajectory. Once the system detects a deviation between the displacement of the tool head and the set path, the controller immediately issues an adjustment instruction for fine-tuning. At the same time, a force control sensor is integrated at the front end of the tool head to monitor the cutting pressure in real time. If it is found that the pressure exceeds the set threshold, the cutting depth and speed are controlled by adjusting the drive voltage to avoid excessive penetration of the tool head and damage to the drug outer packaging. The entire operation process is carried out in a closed clean operation area to ensure that no external contamination is introduced during the system operation process.

[0147] The technical advantage of this embodiment is that it can realize the linkage of high-precision seal recognition and dynamic path planning, has the ability of adaptive adjustment, and greatly improves the execution reliability and operation safety of the robot in uncertain scenarios.

[0148] Embodiment 3: This embodiment focuses on the problems of residual detection and product integrity verification after the disassembly of bagged materials. Especially in the high-cleanliness production environment of biopharmaceuticals, it is about how to ensure that the working area after disassembly is free of residual debris, liquid or particulate contamination, and how to accurately determine whether the disassembly is successful, whether the drug is exposed or contaminated without manual intervention. For this purpose, the system designs a residual detection mechanism and a quality confirmation process based on the fusion of visual detection and infrared sensing to comprehensively improve the safety and controllability of the disassembly process.

[0149] First, after the robot completes the cutting and disassembly action, the visual detection module is immediately activated. This module calls the reference image of the target bag taken before the task starts, and compares the edge structure, lighting characteristics, color distribution and other information of the current area image with the image before disassembly through the image template matching algorithm, so as to judge whether there are abnormal residues, such as incomplete peeling of the bag corners, cracks at the incision or external materials left on the platform surface. To improve the detection accuracy, the system adopts a multi-angle and multi-light source supplementary lighting structure to eliminate the recognition errors caused by shadows and uneven lighting.

[0150] After visual detection, the system enables the infrared thermal imaging detection module. This module scans the thermal radiation characteristics of the disassembly area to identify whether there are suspicious residual liquids, oil stains or highly reflective particles on the surface. Even if such substances are not easily detectable in visible light images, they usually appear as temperature difference areas or abnormal reflection areas in thermal imaging diagrams. The system binarizes the detected high-temperature points or non-background hot areas and compares them with the preset area template. If there are irregular reflection patterns larger than the threshold in the detection area, it is determined that there is a pollution risk in this area.

[0151] Finally, the system uniformly inputs the visual detection results and the infrared detection results into the weighted confidence decision module. This module sets weights according to each detection index and comprehensively calculates whether the current disassembly meets the standards of "complete", "safe" and "residue-free". If both subsystems give a positive judgment, the control system issues a product release signal; if any detection fails, the system triggers the automatic cleaning mechanism or alarms to prompt the operator to intervene and handle.

[0152] Through the construction of the above dual detection mechanism, this embodiment can efficiently complete the quality confirmation process after disassembly without manual intervention, effectively prevent drug contamination and production anomalies, and improve the safety closed-loop level of automated operations. It has significant practical application value for high-end drug packaging, high-cleanliness environments and unmanned pharmaceutical production lines.

[0153] The present invention realizes high-precision, safe and automated disassembly operations of bagged products by robots in the field of biopharmaceuticals through the construction of an image synchronization mechanism based on multi-source visual perception, the integration of three-dimensional modeling and dynamic trajectory planning control, and the combination of residue detection and quality confirmation in the whole process intelligent disassembly method. It effectively solves the misjudgment problem caused by the asynchronous time series of multi-source images, fundamentally reduces the product pollution risk and the probability of operation errors, significantly improves the stability and controllability of the disassembly process, and has good industrial application prospects and promotion value.

[0154] The present invention provides a Figure 2 robot autonomous disassembly system based on multi-source visual perception as shown, including an image acquisition module, an image synchronization module, an image fusion and state determination module, a three-dimensional positioning and path planning module, a trajectory execution and force control feedback module, and a result detection and qualification determination module:

[0155] The image acquisition module obtains the initial image data of the target bagged product, including the color image collected by the RGB camera and the depth image collected by the depth camera, and respectively marks the acquisition timestamps;

[0156] The image synchronization module synchronizes and aligns the image data based on the timestamps, and matches the spatial position information of different source images at the same moment by constructing a unified time coordinate system;

[0157] The image fusion and state determination module performs fusion processing on the synchronized image data, extracts the three-dimensional contour information and dynamic state characteristics of the bag body, and determines whether the target is in a stable static state;

[0158] The three-dimensional positioning and path planning module, on the premise of detecting that the target bag body is stable and static, performs three-dimensional model positioning operations based on the fused image to determine the initial starting point and ending point of the tearing path;

[0159] The trajectory execution and force control feedback module controls the robot to perform disassembly actions along a fixed trajectory, cuts with a constant pressure and a preset speed, and maintains the path stability during the operation through feedback control;

[0160] The result detection and qualification determination module, after the disassembly is completed, calls the vision system again to perform integrity verification on the disassembly area. After confirming that the packaging bag body is completely removed and there is no pollution risk inside, the product is transferred to the next process.

[0161] The robot autonomous disassembly method based on multi-source visual perception provided by the embodiment of the present invention is realized through the above-mentioned robot autonomous disassembly system based on multi-source visual perception. The specific methods and processes of the robot autonomous disassembly system based on multi-source visual perception are detailed in the embodiments of the above-mentioned robot autonomous disassembly method based on multi-source visual perception, and will not be elaborated here.

[0162] The above formulas are all dimensionless and take their numerical values for calculation. The formulas are obtained by collecting a large amount of data for software simulation to get a formula closest to the actual situation. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0163] Only some exemplary embodiments of the present invention have been described above by way of illustration. Undoubtedly, for those of ordinary skill in the art, without departing from the spirit and scope of the present invention, the described embodiments can be modified in various different ways. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0164] It should be noted that in this text, if there are relational terms such as first and second, they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of another identical element in the process, method, article or device comprising the element.

[0165] It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution is prior or subsequent. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0166] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this text can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0167] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0168] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed over multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in various embodiments of the present application may be integrated in a processing unit, may exist separately as individual physical units, or two or more units may be integrated in one unit.

[0170] As mentioned above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0171] Only some exemplary embodiments of the present invention have been described above by way of illustration. Undoubtedly, for those of ordinary skill in the art, the described embodiments can be modified in various different ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the protection scope of the claims of the present invention.

Claims

1. A robot autonomous disassembly method based on multi-source visual perception, characterized in that, It includes the following steps: Obtain the initial image data of the target bagged product, including the color image collected by the RGB camera and the depth image collected by the depth camera, and mark the acquisition timestamps respectively; Based on the timestamps, synchronize and align the image data. By constructing a unified time coordinate system, match the spatial position information of images from different sources at the same moment; Perform fusion processing on the synchronized image data, extract the three-dimensional contour information and dynamic state features of the bag body, and judge whether the target is in a stable static state; On the premise of detecting that the target bag is stable and static, perform the three-dimensional model positioning operation based on the fused image to determine the initial starting point and ending point of the tearing path; Control the robot to perform the disassembly action along a fixed trajectory, cut with a constant pressure and a preset speed, and maintain the path stability during the operation through feedback control; After the disassembly is completed, call the vision system again to check the integrity of the disassembly area. After confirming that the packaging bag body is completely removed and there is no pollution risk inside, transfer the product to the next process.

2. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, wherein The acquisition frequencies of the RGB image and the depth image are set to 60 frames per second and 30 frames per second respectively. The time points of the depth image are interpolated to the time points of the RGB image through the linear interpolation algorithm to ensure that there are matching data points at each moment during the fusion processing stage; To ensure the synchronization accuracy, before the image data enters the processing module, judge the timestamp difference threshold. If the time difference between any pair of images exceeds the set threshold of 20 milliseconds, discard the pair of image data and re-acquire it to ensure the consistency and processing accuracy of the image input; In addition, a sliding window mechanism is set in the image frame data buffer area to perform alignment matching in the buffer queue, improve the image synchronization success rate, and finally ensure the consistency of the subsequent three-dimensional modeling data.

3. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, characterized in that In the step of synchronizing and aligning the image data based on the timestamps, construct a unified time coordinate system, and use the master clock module to uniformly schedule the sampling behaviors of all vision sensors; The master clock sends a global sampling trigger signal with millisecond-level accuracy. After all image sensors respond to the trigger signal, they immediately perform image acquisition and embed the current timestamp mark in the acquired data to ensure the time consistency of each data source logically; The collected color images and depth images are sorted in time sequence respectively, and the sliding window matching algorithm is used to find the depth image frame with the closest timestamp for each frame of color image. If the time difference is within the set threshold range, the image is included in the fusion processing queue, otherwise it is marked as invalid data and discarded; To further improve the spatial alignment accuracy, after each pair of images is matched, spatial correction of the image resolution and field of view angle is also performed. The images are remapped through the pre-established internal and external camera parameter matrices, and finally multi-source visual fusion input data is constructed under the unified time axis and unified spatial reference to ensure the consistency and accuracy basis for subsequent image analysis and processing.

4. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, characterized in that, The extraction of the three-dimensional contour information adopts a deep learning algorithm based on point cloud fusion, and the steps are as follows; The key area features of the RGB image and depth map data are extracted through the feature extraction network, and a joint feature vector is constructed. Then, each pixel in the depth map is restored to a three-dimensional spatial coordinate point using the spatial coordinate transformation model to generate a dense point cloud model. A convolutional neural network is used to judge the posture of the point cloud model, and the stability confidence threshold is used as the basis to determine whether a stable state has been reached; When the confidence level exceeds 0.92, the bag is judged to be in an operable state, otherwise it waits for the stationary state to be detected again.

5. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, wherein The judgment of the stable static state of the bag body is based on the dynamic image frame difference analysis algorithm. By performing frame-by-frame difference calculation on the continuously acquired RGB image sequence, the movement trend of the bagged product within a short time window is quantified. Let N consecutive frames of RGB images be {I1, I2, …, I N}, each frame contains M pixel points. The pixel difference between every two frames is normalized to calculate the frame difference value, which is used to measure the stability of the current bag body surface texture and edge position. The calculation expression of the frame difference value is as follows: where D i is the flat frame difference between the i-th frame and the (i + 1)-th frame, and I i (f) is the gray value of the f-th pixel in the i-th frame image, and I i+1 (f) is the gray value of the f-th pixel in the (i + 1)-th frame image; The residual sequence is constructed for all frame differences and the standard deviation is calculated to identify the consistency of image changes between frames. The standard deviation calculation formula is as follows: wherein, is the average value of the frame difference sequence, and σ D is the standard deviation of the frame difference sequence, reflecting the stability of the change between frames; If σ D < θ1, and then it is determined that the bag body is in a stable state, meeting the prerequisite conditions for performing the cutting operation, where θ1 is the judgment threshold of the standard deviation of frame difference and θ2 is the judgment threshold of the average frame difference value.

6. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, wherein The cutting path is determined by a composite algorithm based on edge detection and structural texture analysis. The specific steps are as follows: Use the Canny algorithm to perform preliminary edge extraction on the fused image; The noise and false edges are removed by morphological closing operation to obtain a stable bag contour line; Determine the sealing position of the bag according to the direction of the image texture gradient, and input the sealing line as the starting point into the path planning module; The path planning module uses the curvature minimization principle to calculate the optimal cutting trajectory, ensuring that the trajectory extends along the sealing direction and avoiding cutting areas outside the packaging bag.

7. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, wherein The robot control system performs trajectory planning based on a vision-guided model and uses an adaptive trajectory calibration mechanism; During the trajectory calibration process, a feedback controller is introduced to compare the deviation between the actual tool head path and the predetermined path. If the deviation exceeds 1.5 mm, the cutting direction is adjusted in real time and the constant pressure is adjusted synchronously. Use PID control algorithm to continuously adjust the angle of the three-axis robot arm to ensure that the trajectory cutting is stable within the set error threshold range, with a maximum allowable error of 2 mm; When the robot performs the disassembly operation, it uses the force control feedback system to dynamically adjust the pressure parameters. The specific steps are as follows: During the cutting process, the blade pressure sensor data is collected every 5 milliseconds to determine the deviation between the current contact force and the preset value in real time; If the current value is too high, the power supply voltage of the actuator motor is reduced to reduce the contact force of the cutter head; if it is too low, the voltage is increased to maintain a constant cutting depth; At the same time, an anti-interference filtering mechanism is introduced to filter out noise data caused by micro-vibration to ensure that the feedback results are stable and effective.

8. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, wherein The three-dimensional model positioning and path control use a high-order path smoothing function combined with an error feedback control mechanism to ensure that the robot executes along the predetermined trajectory with high precision when disassembling bagged products. The specific deficiencies are as follows: Extract the three-dimensional path point set P, P = {p j} = {p1, p2,..., p n}, where p j is the j-th discrete path point in the three-dimensional path, n is the total number of discrete path points, and each discrete path point is defined as p j = (x j , y j , z j ), where x j , y j and z j represent the coordinates of the j-th discrete path point in the x-axis, y-axis, and z-axis directions, respectively; In order to achieve smooth motion of the robot in three-dimensional space, a spline interpolation path function is constructed based on a set of path points to provide a continuous and differentiable path curve for the robot end effector to track. The formula of the spline interpolation path function is as follows: Where B j (t) is the B-spline basis function corresponding to the j-th discrete path point, and a j is the interpolation weight coefficient corresponding to the j-th discrete path point, and S(t) is the spline interpolation path function; During the execution of the robot's motion, the actual spatial position coordinates (x r , y r , z r ) where the robot's end effector is currently located are obtained in real time, and the ideal trajectory point (x(t), y(t), z(t)) corresponding to the moment t is solved from the path function. The current execution error is calculated using the Euclidean distance formula, and the calculation expression is as follows: where x r , y r , z r are the actual three-dimensional coordinate positions of the robot's end effector in real-time feedback through encoders and position sensor devices, x(t), y(t), z(t) represent the three-dimensional theoretical coordinate positions corresponding to the spline interpolation path function S(t) at time point t, which are used to compare with the current position of the robot, and Δ(t) is the Euclidean distance error between the actual position and the theoretical path position under the current motion state of the robot; When Δ(t) exceeds the set threshold δ, it is determined that there is a path deviation and feedback correction needs to be performed; In the error offset state, the direction angle correction logic is started. According to the local geometric change trend between consecutive path points, the attitude adjustment angles in three dimensions are calculated, and a direction angle vector θ = (θ x , θ y , θ z ) is constructed. The calculation formula for each direction angle is as follows: In the formula, is the coordinate value of the discrete path point p j on the k-th dimension, is the coordinate value of the discrete path point p j+1 on the k-th dimension, d is the average Euclidean distance between consecutive path points, and θ k represents the amount of angular change in the k-axis direction, which is used to correct the movement direction, where k ∈ {x, y, z}; The angle vector is transmitted to the motion solver in the robot controller, which drives it to adjust the current joint angle so that the end effector posture is realigned with the predetermined path direction, thus achieving accurate tracking of the cutting path.

9. The robot autonomous disassembly method based on multi-source visual perception according to claim 1, characterized in that, The final integrity check is completed by visual inspection and residual quality control. The specific steps are as follows: The visual inspection part uses the template matching algorithm to compare the pixel differences of the bag body boundary before and after disassembly. If the boundary afterimage exceeds the set threshold, it will prompt the "incomplete disassembly" state; The quality control part detects whether there are residual particles or liquid reflection features on the surface of the workbench through infrared imaging, and converts the infrared reflection intensity into a binary image for residue recognition; After combining the results of the two parts, the weighted confidence algorithm is used to comprehensively judge whether the disassembly is successful. Only when both the visual and quality control results are qualified, the product transfer process will be started.

10. A robot autonomous disassembly system based on multi-source visual perception, which is used to implement the robot autonomous disassembly method based on multi-source visual perception according to any one of the above claims 1-9, characterized in that, It includes an image acquisition module, an image synchronization module, an image fusion and state determination module, a three-dimensional positioning and path planning module, a trajectory execution and force control feedback module, and a result detection and qualification determination module: The image acquisition module obtains the initial image data of the target bagged product, including the color image collected by the RGB camera and the depth image collected by the depth camera, and marks the acquisition timestamp respectively; The image synchronization module synchronizes and aligns the image data based on the timestamp. By constructing a unified time coordinate system, it matches the spatial position information of images from different sources at the same moment; The image fusion and state determination module performs fusion processing on the synchronized image data, extracts the three-dimensional contour information and dynamic state features of the bag body, and judges whether the target is in a stable static state; The three-dimensional positioning and path planning module, on the premise that the target bag body is detected to be stable and static, performs the three-dimensional model positioning operation based on the fused image, and determines the initial starting point and ending point of the tearing path; The trajectory execution and force control feedback module controls the robot to perform the disassembly action of a fixed trajectory, cuts with a constant pressure and a preset speed, and maintains the path stability during the operation through feedback control; The result detection and qualification determination module, after the disassembly is completed, calls the visual system again to check the integrity of the disassembly area. After confirming that the packaging bag body is completely removed and there is no pollution risk inside, the product is transferred to the next process.

Images