Intelligent positioning assembly method and system based on machine vision

Abstract

The invention relates to the field of artificial intelligence, discloses an intelligent positioning assembly method and system based on machine vision, and aims to solve the problem of precision reduction caused by static calibration failure, error accumulation and lack of closed-loop correction in the existing assembly process. The method comprises the following steps: constructing a digital twin model under a unified coordinate system; the six-degree-of-freedom poses of the mounted and to-be-mounted parts are recognized in real time through an industrial camera; calculating actual and ideal pose deviations, and dynamically correcting the state of the digital twinborn body to block error transmission; re-planning the track of the mechanical arm based on the correction reference and executing assembly; re-checking and comparing after assembling, and triggering rollback when abnormity occurs; and periodically optimizing hand-eye calibration parameters on line by using historical data. According to the invention, high-precision, self-adaptive and closed-loop control intelligent assembly is realized, the accumulated positioning precision is remarkably improved, and the rework rate is reduced.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to an intelligent positioning and assembly method and system based on machine vision. Background Technology

[0002] In modern industrial assembly, machine vision-based guided positioning technology has been widely applied to the automated assembly processes of products such as automotive electronics and precision instruments. Typical methods rely on pre-calibrated hand-eye relationships between a camera and a robotic arm, visually recognizing the workpiece's pose and converting it into commands in the robotic arm's coordinate system to achieve gripping and assembly. However, this approach still has significant shortcomings in long-term practical operation.

[0003] First, existing systems are mostly based on the static calibration assumption, assuming that the hand-eye relationship remains constant throughout the production cycle. However, factors such as temperature changes and mechanical vibrations in industrial settings can cause calibration parameters to drift. Systems lack the ability to detect and compensate for such drifts online, leading to systematic positioning errors. Second, in multi-station continuous assembly, errors from preceding stations propagate and accumulate, affecting the final assembly accuracy, especially causing a decrease in yield in precision assembly applications with high tolerance requirements. Furthermore, most existing methods employ an open-loop control architecture, providing visual positioning only once before assembly. After assembly, there is a lack of real-time verification mechanisms, making it impossible to proactively detect and correct anomalies such as misassembly or misalignment. Although digital twin technology is used in simulation and monitoring, it often fails to deeply integrate with real-time visual data, making it difficult to achieve dynamic correction based on the actual assembly state.

[0004] Therefore, existing technologies suffer from problems such as easy drift of calibration parameters, accumulation of error propagation, lack of closed-loop verification, and ineffective use of digital twins for real-time correction, which restrict the accuracy and reliability of assembly systems during long-term operation. It is necessary to propose a closed-loop assembly method that can achieve dynamic calibration compensation, block error propagation, and integrate digital twins and real-time vision to improve the adaptability and accuracy of multi-station assembly. Summary of the Invention

[0005] This invention provides a machine vision-based intelligent positioning and assembly method and system, aiming to solve the technical problems of decreased positioning accuracy and increased assembly failure rate in existing industrial automated assembly processes caused by static calibration failure, progressive error accumulation, lack of closed-loop correction mechanism, and ineffective use of digital twin models. This invention achieves high-precision, adaptive continuous assembly operations by constructing a closed-loop intelligent assembly architecture that integrates real-time visual perception, dynamic benchmark update, multi-station error blocking, and digital twin feedback correction.

[0006] Firstly, a machine vision-based intelligent positioning and assembly method includes the following steps: S1, Construct a digital twin model that corresponds to the physical assembly system and can be dynamically updated; S2, before each assembly station is executed, the target area image is acquired by an industrial camera, and the six-degree-of-freedom pose information of the assembled parts and the parts to be assembled in the camera coordinate system is identified and extracted. S3, calculate the spatial deviation between the extracted actual pose of the assembled component and the corresponding ideal pose in the digital twin model to obtain the local error vector; S4. Based on the local error vector and the actual pose, the digital twin model is dynamically corrected, including replacing the ideal pose of the corresponding component with the actual pose, and updating the expected pose of the associated downstream component in a cascade manner according to the predefined assembly topology relationship, and using the corrected model state as the basis for subsequent assembly planning. S5, based on the assembly datum modified by S4, plans and generates the robotic arm motion trajectory instructions for the parts to be assembled. S6, after the robotic arm executes the motion trajectory to complete the assembly, the camera is triggered again to perform a re-inspection imaging and extract the final pose after the assembly is completed; S7. The final pose obtained from the re-inspection is compared with the expected state in the digital twin model. The assembly result is judged according to the preset threshold. If it is qualified, the data is archived. If it is abnormal, the error correction mechanism is triggered. S8, based on historical archived effective assembly data, periodically optimizes and updates the hand-eye transformation matrix between the camera and the robotic arm online.

[0007] Preferably, constructing the digital twin model in step S1 includes: The base coordinate system of the robotic arm is determined to be the global reference system; A three-dimensional geometric model integrating the kinematic parameters of the robotic arm, camera parameters, tooling fixtures, and workpiece; Define the assembly topology relationships and motion constraints between the entities; The data is organized into a hierarchical structure that can be accessed in real time and stored in high-speed memory.

[0008] Preferably, step S2 specifically includes: The industrial camera is triggered to capture images synchronously after the robotic arm stops moving. The acquired images are subjected to filtering and noise reduction, as well as adaptive threshold segmentation. Based on geometric features, candidate regions for workpieces are initially screened out from the processed image; The candidate regions are matched with an offline-built visual feature template library to determine the workpiece category and rough pose. Based on the 3D model of the successfully matched workpiece, the six-degree-of-freedom pose of the workpiece in the camera coordinate system is calculated using the perspective n-point algorithm.

[0009] Preferably, in step S3, calculating the spatial deviation specifically includes: Transform the actual pose in the camera coordinate system to the robot arm base coordinate system; Read the ideal pose of the corresponding component in the digital twin in the base coordinate system; The root mean square difference of the translation vector is calculated as the translation deviation, and the minimum angle between the rotation matrix or the unit quaternion is calculated as the rotation deviation. The translational and rotational deviations are combined to form a local assembly error vector.

[0010] Preferably, in step S4, the step of updating the expected pose of the associated downstream components in a cascade manner according to the predefined assembly topology specifically includes: Based on the predefined relative pose relationships between components, and combined with the updated actual pose of the upstream component, the new expected absolute pose of the downstream component in the global coordinate system is recalculated, and its record in the digital twin is updated.

[0011] Preferably, in step S5, replanning the robotic arm's motion trajectory includes: Based on the corrected digital twin state, the target installation pose of the part to be assembled in the robot arm base coordinate system is calculated; The target pose is converted into the target joint angle through inverse kinematics; A smooth joint space trajectory is generated between the initial joint angle and the target joint angle using polynomial interpolation, and the velocity of the final segment of the trajectory is limited.

[0012] Preferably, in step S6, the re-inspection imaging and pose extraction steps adopt the same image processing and pose calculation process as the pose identification and extraction steps in step S2, and ensure that the part to be re-inspected after assembly is visible to the camera.

[0013] Preferably, in step S7, the triggering error correction mechanism includes performing the following operations in sequence: Control the robotic arm to retract along its original trajectory to a safe position; Reset system task status flags; Control the end effector to release the workpiece; The control flow jumps to step S2 to re-execute the assembly at the current workstation.

[0014] Preferably, in step S8, the periodic online optimization update specifically includes: Optimization is triggered when the number of successful assembly reaches a set threshold or the standard deviation of historical pose deviation exceeds a set threshold. The initial recognition pose and the final re-inspection pose of multiple successful assemblies are extracted from the historical database as observation samples. A time-weighted sliding window averaging algorithm is used to estimate the rotation and translation components of the hand-eye transformation matrix, respectively. After offline verification of the new parameters, a double-buffering mechanism is used to safely switch to the main system.

[0015] Secondly, a machine vision-based intelligent positioning and assembly system includes: The digital twin modeling unit is used to build and maintain dynamically updated digital twin models. The real-time vision perception unit is used to acquire images before and after assembly and to identify and calculate the six-degree-of-freedom pose of the workpiece. The local error calculation unit is used to calculate the spatial deviation between the actual pose and the ideal pose of the digital twin; A digital twin dynamic correction unit is used to dynamically update the digital twin model based on the deviation and the actual pose, and to cascade update the pose of related components; An assembly trajectory planning unit is used to replan the robotic arm's motion trajectory based on the updated digital twin state; The assembly execution and re-inspection unit is used to control the robotic arm to perform assembly and trigger visual re-inspection after assembly. The consistency verification and anomaly handling unit is used to verify the assembly results and perform qualified data archiving or anomaly correction processes. An online calibration and optimization unit is used to periodically optimize the hand-eye transformation matrix based on historical assembly data; Each unit works collaboratively through a communication interface to form a closed-loop intelligent assembly system.

[0016] In summary, this application includes at least one of the following beneficial technical effects: 1. This invention constructs a dynamically updated digital twin as an assembly benchmark, and compares and calculates the deviation between the actual pose of the visually recognized component and the ideal pose of the digital twin in real time at each workstation. Once a significant error is detected, the digital twin is immediately updated with the actual pose, and the expected poses of all downstream components are cascaded and corrected according to the assembly topology. This fundamentally cuts off the transmission path of upstream workstation errors to downstream components, effectively overcoming the problem of decreased assembly accuracy caused by static calibration parameter drift and error accumulation, and significantly improving the overall accuracy and stability of multi-workstation, long-cycle assembly systems.

[0017] 2. This invention introduces an automatic visual re-inspection step after assembly execution, comparing and verifying the actual pose of the assembly result with the expected state updated by the digital twin. This design breaks through the limitation of traditional open-loop control relying solely on single positioning, achieving a complete closed loop of "perception-decision-execution-verification". When an assembly anomaly (such as misassembly or offset) is detected, the system can automatically trigger error correction mechanisms including safety reversal, workpiece release, and process reset, enabling real-time detection and autonomous recovery of anomalies during the process, significantly reducing reliance on external manual intervention and the defect rate of the final product.

[0018] 3. This invention continuously archives key pose data from the qualified assembly process and, based on this historical data, uses algorithms such as sliding window weighted averaging to periodically estimate and update the hand-eye transformation matrix between the camera and the robotic arm online. This self-optimization mechanism enables the system to automatically compensate for long-term parameter drift caused by factors such as mechanical wear and temperature changes, maintaining and optimizing the accuracy of calibration parameters. Its design, independent of the main control loop, ensures deterministic real-time control while endowing the system with self-learning and adaptive capabilities during long-term operation, improving the system's intelligence level and maintenance efficiency. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the intelligent positioning and assembly method based on machine vision according to the present invention. Figure 2 This is a flowchart illustrating the intelligent positioning and assembly system based on machine vision according to the present invention. Detailed Implementation

[0020] This invention provides a machine vision-based intelligent positioning and assembly method and system. Its core lies in constructing a closed-loop intelligent assembly architecture that integrates real-time visual perception, dynamic benchmark updates, multi-station error prevention, and digital twin feedback correction. Before each assembly station, this architecture acquires the actual six-degree-of-freedom pose of the assembled components in the current physical space and calculates the deviation between this pose and the ideal pose of the corresponding component in the digital twin. This dynamically corrects the state of the digital twin, preventing errors from propagating downstream. Simultaneously, after assembly, automatic re-inspection is performed to verify the consistency between the actual assembly result and the expected state. Effective assembly data is used for periodic online optimization of hand-eye calibration parameters, thereby achieving high-precision, adaptive continuous assembly operations.

[0021] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on specific implementation methods of the present invention.

[0022] Example 1 A machine vision-based intelligent positioning and assembly method includes the following steps: S1. Before the assembly line starts, a digital twin model is constructed that precisely corresponds to the physical assembly system and can be dynamically updated during runtime. This model is not only a static mapping of the physical world, but also a dynamic benchmark for real-time assembly decisions and error corrections in subsequent steps. The construction process is as follows: S101: Determine a unified coordinate system and establish the root node.

[0023] Typically, the base coordinate system of the robotic arm is chosen as the global reference system (i.e., the root node of the digital twin model) to provide a unified coordinate framework for all spatial data.

[0024] S102: Digital representation of integrated physical entities.

[0025] Within the aforementioned coordinate system, the known model data of each key component is integrated. This includes: the DH kinematic parameters of the robotic arm, the intrinsic parameters of the camera and the extrinsic parameters obtained through hand-eye calibration (hand-eye transformation matrix), and the three-dimensional geometric models (such as CAD models) of the tooling fixtures and the workpieces to be assembled. This data can be imported via robot modeling files (such as URDF) or simulation software interfaces.

[0026] S103: Define the structural and assembly relationships between entities.

[0027] Each entity is assigned a unique identifier, and its initial pose, motion constraints, and assembly topology relationships with other entities are recorded (e.g., "workpiece A is installed on a locating pin of fixture B"). These relationships are organized in a traversable data structure (e.g., a figure or a tree).

[0028] S104: The organization is a hierarchical structure that can be accessed in real time.

[0029] All entities are organized into a hierarchical tree structure (scene graph) according to their hierarchical relationships. Each node stores its transformation matrix relative to its parent node. Finally, the complete digital twin model is placed in a high-speed memory area (such as shared memory) to ensure that the real-time requirements of millisecond-level read and write and low-latency update of model data in subsequent processes can be met.

[0030] S105: Added status version management.

[0031] Version management mechanisms can be added to the model, such as saving a snapshot each time the state changes, which helps to trace the history of state changes during the assembly process for subsequent analysis and optimization.

[0032] In summary, the construction of a digital twin model begins with establishing a global reference system, gradually integrating precise models of each component, defining their interrelationships, and ultimately forming a digital mirror system that can be accessed in real time and traced back to its history.

[0033] S2, before each assembly station, uses an industrial camera deployed at the end of the robotic arm or above the worktable to acquire images of the target area of ​​the current station, and identifies and extracts the six-degree-of-freedom pose information of the assembled parts and the parts to be assembled. This process is a chain of transformation from raw images to three-dimensional spatial data, specifically executed according to the following steps: S201: Synchronous triggering and image acquisition.

[0034] When the assembly process enters a certain workstation, the main controller synchronously triggers the industrial camera to take a single shot via a hard-wired signal.

[0035] The camera uses a global shutter CMOS sensor with high resolution (e.g., 2448×2048 pixels) and frame rate (typically ≥30fps), and optical calibration has been completed during the system initialization phase (acquiring intrinsic parameters and extrinsic parameters relative to the robotic arm base or end effector).

[0036] The triggering time is strictly locked after the robotic arm has completely stopped moving, in order to eliminate motion blur and ensure image clarity.

[0037] S202: Image preprocessing.

[0038] The acquired raw RGB or grayscale image is first subjected to Gaussian filtering for noise reduction, usually using a 5×5 convolution kernel with a standard deviation of about 1.2 to smooth the noise.

[0039] Next, an adaptive threshold segmentation algorithm is used to binarize the image. For each pixel, the algorithm calculates the gray mean of its local neighborhood (e.g., a 15×15 window) and then subtracts a predefined offset (e.g., 10) as the binarization threshold for that pixel, thereby effectively overcoming the influence of uneven illumination and highlighting the outline of the workpiece.

[0040] S203: Initial screening of candidate regions based on geometric features.

[0041] Eight-connected-domain analysis is performed on the binary image to extract all independent closed contours. For each contour, its minimum bounding rectangle and fitted ellipse are calculated to obtain basic geometric features such as area, aspect ratio, and roundness. These features are compared with a preset workpiece geometric feature library, which is derived from the CAD model or physical measurement data of the workpiece to be assembled. The range of geometric features that various workpieces may present in the image is defined (for example, the aspect ratio of a rectangular workpiece should be between 1.8 and 2.2).

[0042] This comparison quickly identifies several candidate regions that may be the target workpiece, significantly reducing the search range for subsequent fine processing.

[0043] S204: Accurate recognition and matching based on visual feature templates.

[0044] For each candidate region, the system matches it against an offline-built visual feature template library. This template library is constructed as follows: Images of standard workpieces are captured from multiple perspectives (usually no fewer than 20) under various lighting conditions. Robust features are extracted, such as SIFT (Scale Invariant Feature Transform) descriptors (e.g., 128-dimensional vectors), ORB (Oriented Fast and Rotated BRIEF) feature points, or edge contour features. All template features are indexed using efficient data structures such as KD-Tree. During online matching, the similarity (e.g., Euclidean distance) between the feature descriptors extracted from candidate regions and features in the template library is calculated. The template with the most similarity and a similarity higher than the confidence threshold is selected to determine the specific category and rough orientation of the workpiece.

[0045] S205: Six-DOF pose calculation.

[0046] For a successfully matched workpiece, the perspective n-point (PnP) algorithm is used, combined with the workpiece's accurate 3D CAD model (the 3D coordinates of its key feature points are known), to solve the complete 6-DoF pose of the workpiece in the camera coordinate system. The solution is usually expressed as a rotation matrix R (3×3) and a translation vector t (3×1).

[0047] To facilitate subsequent robot kinematics calculations and data interpolation, the pose is often converted into a three-dimensional translation vector and a unit quaternion.

[0048] In summary, steps S201 to S205 constitute a complete visual perception pipeline. It begins with a clear image acquired at a precise moment, undergoes progressive processing (denoising, segmentation, coarse screening, and fine matching), and finally outputs the precise position and orientation of the workpiece in three-dimensional space. This orientation information serves as the most direct "perceptual feedback" at the physical assembly site and is the absolute basis for comparing with the "ideal reference" of the digital twin, calculating errors, and initiating a dynamic correction closed loop in subsequent steps (S3).

[0049] S3, based on the actual pose of the assembled component obtained in step S2, calculate its spatial deviation from the corresponding ideal pose in the digital twin, and obtain the local error vector that quantifies the assembly accuracy of the current workstation. The specific steps are as follows: First, the actual pose in the camera coordinate system obtained in step S2 is... Through the currently effective hand-eye transformation matrix A 4×4 homogeneous transformation matrix is ​​used to transform the system into its unified reference frame, i.e., the robot arm's base coordinate system. The transformation is achieved through matrix multiplication, as shown in the formula: ; in, That is, the actual pose represented in the base coordinate system.

[0050] S302: Read the ideal pose in the digital twin.

[0051] Simultaneously, the theoretical (ideal) pose of the corresponding component in the base coordinate system is read from the digital twin model. This pose indicates the spatial position and orientation of the component under perfect assembly conditions.

[0052] S303: Calculate the translation deviation (Δt).

[0053] extract and The translation vectors in the equation are denoted as follows: and .

[0054] Translational deviation The overall positional offset is characterized by calculating the root mean square value of the difference between these two vectors. The calculation formula is as follows: .

[0055] S304: Calculate rotational deviation (Δr).

[0056] Will and The rotated part (3×3 rotation matrix) is converted into unit quaternion form, denoted as follows: and .

[0057] Rotational deviation The attitude difference is characterized by calculating the minimum angle (in radians) between these two unit quaternions. The formula for calculation is as follows: ; in, It represents the absolute value of the dot product of two quaternions.

[0058] S305: Construct the local assembly error vector.

[0059] The calculated translation deviation (Scalar, usually in millimeters) and rotational deviation (Scalars, usually in radians) are combined to form a two-dimensional local assembly error vector. This vector is a quantitative expression of the assembly accuracy at the current workstation.

[0060] In summary, steps S301 to S305 complete a precise "comparison between reality and ideal". The coordinate system (S301, S302) ensures the consistency of the comparison benchmark. Mathematical calculations (S303, S304) quantitatively assess the deviation of position and posture, and finally output a structured error vector (S305). This error vector is not only a "quantitative report" of the current workstation assembly quality, but also the core decision-making basis for the subsequent step S4 to perform dynamic correction of the digital twin and to determine whether to trigger the error blocking closed loop.

[0061] S4, based on the local assembly error vector calculated in step S3 and the actual pose data, dynamically correct the digital twin, and use the corrected model state as the sole benchmark for subsequent assembly planning at all workstations, thereby preventing the error from propagating downstream. This process specifically includes the following steps: S401: Decision Triggering and Data Preparation.

[0062] The system receives the local assembly error vector from step S3. First, it determines whether the error is significant (e.g., whether the translation or rotation deviation exceeds a set small threshold, such as 0.01 mm or 0.1 degrees, to filter out measurement noise). If the error is significant, it determines that the digital twin needs to be updated, and prepares the core data for the update—that is, the actual pose of the assembled parts in the base coordinate system. ).

[0063] S402: Perform core pose replacement.

[0064] The system locates the record of the corresponding component in the digital twin database and stores it in its ideal pose. ) Directly overwrite and update to the actual pose confirmed in step S401 ( This operation is the starting point for correction, bringing the state of that component in the digital twin in line with the physical world.

[0065] S403: Cascade update of the expected pose of associated components.

[0066] Because there are definite geometric constraints between the components of an assembly (such as fitting, screwing, and nesting), a change in the pose of one component will directly affect the expected position of other related components in the global coordinate system.

[0067] Implementation method: Based on the assembly topology established in stage S1, the system maintains a component dependency graph, which uses components as nodes and assembly constraints as edges. When the pose of a component (source node) is updated in S402, the system automatically traverses this dependency graph to find all downstream components (target nodes) that are directly or indirectly dependent on the source node.

[0068] Pose recalculation: For each affected downstream component, the system recalculates the new expected absolute pose of the downstream component in the base coordinate system based on its predefined relative pose relationship with the source component (determined during model building, e.g., "the bracket is installed at the center of the upper surface of the base, offset by 10 mm in the positive Z-axis direction"), combined with the updated actual pose of the source component, and updates its record in the digital twin.

[0069] Example: Suppose the actual position of the currently assembled base is offset by +0.1 mm in the X direction relative to the ideal position. In the digital twin, not only is the pose of the base updated, but the expected installation pose of the "lens bracket" originally planned to be installed on the base is also recalculated. The new pose will also be compensated by +0.1 mm in the X direction based on the ideal pose, thereby ensuring that the assembly relationship between the "lens bracket" and the "base" in the digital twin strictly corresponds to their actual relative relationship in the physical world.

[0070] S404: Switch and lock the new global assembly reference.

[0071] After completing the above pose replacement and cascading update, the system immediately activates the latest state of the entire digital twin model as a globally valid assembly reference. Thereafter, when any subsequent workstation (step S5) plans assembly actions, the calculation of its target pose will be based entirely on this updated model state that reflects the current actual assembly progress, and no longer any outdated ideal data will be used.

[0072] In summary, steps S401 to S404 together constitute the core dynamic benchmark update and error blocking mechanism of this invention. S401 to S403 ensure that the digital twin can evolve in real time and consistently following the actual state of the physical assembly line; S404 then establishes this latest evolved state as the absolute basis for subsequent operations. The innovation of this mechanism lies in the fact that it does not perform local remedies after an error occurs, but rather dynamically resets the "reference coordinate system" (i.e., the digital twin benchmark) of the entire system, making downstream stations "unable to perceive" the existence of upstream errors, thereby fundamentally cutting off the propagation path of errors between stations, which is the key to achieving high-precision multi-station continuous assembly.

[0073] S5, based on the latest assembly datum established after dynamic correction in step S4, replans and generates the robotic arm motion trajectory instructions for the parts to be assembled. This step transforms the "corrected digital datum" into "executable physical actions." The specific process is as follows: S501: Load the latest assembly reference.

[0074] The system loads the complete state data updated in step S4 from the digital twin. This data includes the measured poses of all assembled parts, as well as the assembly interface definitions between parts based on the current actual geometric relationship (such as screw hole positions, mating surface orientation, etc.).

[0075] S502: Calculate the target mounting pose for adaptability.

[0076] Based on the latest reference loaded in S501, the system determines the target mounting pose of the component to be assembled in the robot arm's base coordinate system (denoted as ) through geometric calculation, according to the assembly interface definition between the component to be assembled and the relevant assembled components (whose poses already reflect the actual deviations). ).

[0077] The key to this calculation is that it is no longer based on an initial ideal model, but rather entirely on the latest digital twin state, which includes the actual deviations from upstream, thereby naturally generating a target pose that can adapt to the current physical assembly state.

[0078] S503: Plans smooth and precise joint space trajectories.

[0079] Obtain the target pose Then, it needs to be converted into a motion sequence of each joint of the robotic arm.

[0080] Kinematics calculation and input preparation: First, through inverse kinematics... The planner converts the angle to the target joint angle and simultaneously obtains the current starting joint angle of the robotic arm (at rest). It also sets the velocity and acceleration at the starting and target points to zero to ensure smooth start and stop.

[0081] Trajectory generation: A fifth-order polynomial interpolation method is used to interpolate between the above conditions to generate a smooth joint space trajectory that continuously varies in position, velocity, and acceleration. The trajectory is discretized at a high sampling rate (e.g., 200Hz) to generate dense path points (no fewer than 100).

[0082] End-efficiency fine motion control: To ensure assembly accuracy during the final contact stage, a speed limit (e.g., linear speed ≤ 5 mm / s) is applied to the last segment of the trajectory (e.g., the last 10 millimeters from the target) to suppress impact and vibration.

[0083] S504: Encapsulation and instruction delivery.

[0084] Once the planning is complete, the system encapsulates the timing sequence of joint angles into a standard format motion control instruction package and sends it to the robotic arm controller through a defined real-time communication interface (such as EtherCAT or PROFINET) to drive it to begin executing the assembly action.

[0085] S6, after the robotic arm executes the motion trajectory command to complete the assembly action, the camera is triggered again to perform a re-inspection imaging of the assembly result, extracting the final pose after the actual assembly is completed. This re-inspection process aims to verify the assembly quality, and its successful implementation depends on ensuring that the assembled part to be tested is visible to the camera during the system design phase. Specifically, it includes the following steps: S601: Re-inspection trigger and image acquisition.

[0086] Once the robotic arm stops moving and the assembly process is confirmed to be complete, the main controller synchronously triggers the industrial camera to take a picture via a hardwired signal. The camera parameters and initialization settings are consistent with those in step S2 to ensure consistent imaging conditions. The triggering timing is strictly controlled to occur only after the robotic arm has come to a complete stop to eliminate motion blur and ensure image clarity.

[0087] It should be noted that the effectiveness of this step and subsequent image processing relies on thorough consideration during the system layout design. Specifically, the installation position, viewing angle, and lighting scheme of the industrial camera must ensure that, after assembly, the key features of the workpiece requiring re-inspection (such as locating surfaces and assembly reference holes) remain within the camera's field of view, and that the images are clear enough for reliable identification by subsequent algorithms. For complex assemblies, this requirement can be met through a multi-camera layout or by selecting cameras with specific viewing angles.

[0088] S602: Image preprocessing for re-inspection.

[0089] The acquired re-inspection images are first subjected to Gaussian filtering for noise reduction (e.g., using a 5×5 convolution kernel with a standard deviation of 1.2), and then binarized using an adaptive threshold segmentation algorithm to suppress the influence of uneven illumination and highlight the contour features of the assembled workpiece.

[0090] S603: Regional preliminary screening based on geometric features.

[0091] Eight-connected region contour extraction is performed on the binarized image, and the minimum bounding rectangle, fitted ellipse, and basic geometric features such as area, aspect ratio, and roundness of each contour are calculated. These features are compared with a pre-defined workpiece geometric feature library, which contains the range of geometric features that the assembled target component may exhibit. Through comparison, several candidate regions that may correspond to the assembled target component are quickly selected, thereby narrowing down the search range for subsequent fine processing.

[0092] S604: Visual feature template matching.

[0093] For each candidate region, robust visual features are extracted, such as SIFT descriptors, ORB feature points, or significant edge contour features. These features are then matched against an offline-built visual feature template library. During matching, the similarity (e.g., Euclidean distance) between the candidate region features and features in the template library is calculated. The template with the highest similarity and a similarity higher than a preset confidence threshold is selected, thereby determining the specific category of the workpiece to be re-inspected and its approximate pose in the image.

[0094] S605: Six-DOF pose calculation.

[0095] For a successfully matched workpiece, the Perspective n-Point (PnP) algorithm is used, combined with the workpiece's accurate 3D CAD model (the 3D coordinates of its key feature points are known in the model), to solve for the workpiece's complete 6-DOF pose in the camera coordinate system. The solution is usually expressed as a rotation matrix R (3×3) and a translation vector t (3×1). For ease of subsequent processing, this pose can also be converted into a 3D translation vector combined with a unit quaternion.

[0096] S606: Position is unified to the reference coordinate system.

[0097] The pose obtained through S605 in the camera coordinate system is transformed into a unified robotic arm base coordinate system using the "hand-eye transformation matrix" (i.e., the coordinate transformation relationship between the camera and the robotic arm base) that is currently effective in the system.

[0098] After this step, the final pose of the actual assembly is obtained in the unified reference frame of the system. This pose data truly reflects the assembly result in the physical world and will be used in the next step (S7) to compare the consistency with the expected state of the digital twin.

[0099] S7 automatically verifies the actual assembled results and determines the process flow based on the verification results. If the results are satisfactory, the data is archived for future optimization; if abnormalities are found, an error correction mechanism is triggered. Specifically, the following sub-steps are executed: S701: Set the consistency judgment threshold.

[0100] During the system deployment or process design phase, a comparison threshold for quality judgment is preset based on the specific precision requirements of the products to be assembled.

[0101] Typically, the threshold for translational deviation is set to 0.05 mm, and the threshold for rotational deviation is set to 0.5 degrees. These two thresholds are key parameters for distinguishing between acceptable assembly deviations and unacceptable assembly anomalies, and can be adjusted according to the tolerance requirements of different products.

[0102] S702: Calculate the deviation between the final pose and the expected state.

[0103] The conformance verification module reads the final assembly pose (denoted as the actual pose) from step S6. At the same time, the theoretical expected state (denoted as the expected pose) after the current assembly task is completed is obtained from the digital twin model. Both are expressed in the robot arm's base coordinate system.

[0104] The consistency verification module then calculates the translational and rotational deviations between the actual pose and the expected pose. The calculation method is the same as described in step S3, that is, the root mean square difference of the translation vector and the minimum angle between the rotation matrix (or quaternion) are calculated respectively.

[0105] S703: Judgment and Process Diversion.

[0106] The system will calculate the translation deviation from S702. and rotational deviation Each of these is compared with the corresponding threshold set in S701. If or If any of the conditions in the above conditions are met, the assembly result is determined to be "abnormal"; otherwise, it is determined to be "qualified".

[0107] S704: Execute the exception handling procedure (if an exception is determined).

[0108] If an assembly anomaly is detected, the system immediately triggers a pre-defined anomaly handling procedure designed to safely interrupt the current operation, restore the initial state, and create conditions for retry. This procedure performs the following operations in sequence: Safe retraction: Control the robotic arm to retract along the opposite motion trajectory to the assembly process until the end effector reaches a preset safe height. This safe height is determined during the system layout design and must ensure that there is no risk of collision between the robotic arm and any fixed components or workpieces within the workstation.

[0109] Status Reset: Clears the "task status flag" used by the system to mark the current assembly task being executed, thus resetting the system logic.

[0110] Release workpiece: Control the end effector (such as a gripper or vacuum chuck) to perform a release action to release the currently assembled workpiece (or an assembled workpiece with incorrect orientation) it is holding.

[0111] Process Reset: After completing the above operations, the system control process jumps back and starts executing from step S2 (visual recognition) again, entering a new round of "recognition-planning-execution-re-inspection" cycle, and attempts to complete the assembly of this workstation again.

[0112] S705: Archive valid assembly data (if deemed acceptable).

[0113] If the assembly is deemed successful, the system packages the complete data records generated during the assembly process and stores them in a specially maintained "historical assembly database." This structured historical data forms the data basis for the subsequent online calibration and optimization step S8. The archived data package typically contains the following: Original image hash value: Using cryptographic hash algorithms such as SHA-256, the hash value of the original image file of the key assembly steps (such as the identification image in step S2 and the re-inspection image in step S6) is calculated for data uniqueness identification and integrity verification.

[0114] Key pose data: including the initial recognition pose output in step S2 and the final re-examination pose output in step S6.

[0115] Process data: the sequence of trajectory points followed by the robotic arm during assembly, and the local error vector calculated in step S3.

[0116] Context information: The timestamp when the data was generated.

[0117] To efficiently manage and query this time-related data, the historical assembly database is organized using a time-series data structure. Simultaneously, the system implements a data lifecycle management strategy, for example: frequently accessed data from the last 7 days (or other periods set according to storage capacity and business needs) is retained in memory or high-speed solid-state storage (high-speed access area); data exceeding this period is automatically migrated to lower-cost, higher-capacity disks or archive storage systems (long-term storage area).

[0118] S8, based on the cumulative deviation data in the historical assembly database, uses a sliding window weighted average algorithm to periodically update the hand-eye transformation matrix between the camera and the robotic arm online. The update cycle is triggered every fifty assembly cycles or when the cumulative pose deviation standard deviation exceeds 0.02 mm. Specifically, it includes the following steps: S801: Set optimized trigger conditions.

[0119] The system presets two conditions to trigger calibration updates; the optimization process will start when either condition is met: Triggered by count: The cumulative number of successful assembly cycles completed since the last calibration update has reached 50.

[0120] Accuracy warning trigger: Calculate the final pose deviation of the most recent 100 successful assembly records in the historical assembly database (i.e., the translation deviation recorded in step S7). If the standard deviation exceeds 0.02 mm, the current calibration parameters may be unstable, triggering optimization.

[0121] S802: Prepare to optimize data.

[0122] When the triggering condition is met, the system extracts the dataset for this optimization from the "historical assembly database". This dataset consists of the most recent N (e.g., N=100) successful assembly records. Each record contains the initial recognition pose obtained in step S2 (in the camera coordinate system) and the final re-inspection pose obtained in step S6 (in the robot arm base coordinate system). These N pairs of pose data constitute the observation samples for this parameter estimation.

[0123] S803: Calculate candidate parameters based on sliding window weighted average.

[0124] The system uses a sliding window weighted average algorithm to process the above N pairs of observation data, estimating the rotation and translation components of the hand-eye transformation matrix respectively. The specific operation is as follows: 1. Weighting: Assign a weight to each record in the dataset. The weights decrease exponentially in reverse chronological order (i.e., the newest record has the highest weight). Assuming the index of the latest record is n, the weight of the i-th record (i≤n) is... The calculation formula is: This means that newer assembly data carries a higher weight in parameter estimation, enabling optimization to respond more quickly to recent system changes.

[0125] 2. Translation Component Average: The weighted arithmetic mean of the translation vector observations in N records is used to directly calculate the optimized estimate of the translation component.

[0126] 3. Rotational Partial Average: Observations of the rotation matrix from N records (usually first converted to unit quaternion form). A weighted average is then performed. Since unit quaternions lie on a hypersphere, their weighted average cannot be directly calculated using a linear average. This step employs a weighted spherical average method for unit quaternions. A common implementation is to obtain a weighted averaged unit quaternion by iteratively applying spherical linear interpolation (Slerp) or by solving for the principal eigenvectors of a weighted covariance matrix. Ultimately, Convert to a 3x3 rotation matrix .

[0127] S804: Synthesis and normalization of the novice-eye transformation matrix.

[0128] The optimized rotation matrix calculated by S803 Translation vector Combined, they form a 4x4 homogeneous transformation matrix, which can be used as a candidate hand-eye transformation matrix. .

[0129] To ensure its mathematical correctness (the rotation matrices must be orthogonal and have a determinant of 1), it is necessary to... Orthogonalization is performed. Singular value decomposition (SVD) is typically used: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Perform SVD decomposition to obtain Then, all singular values ​​in its singular value matrix Σ are forced to be 1, forming a new singular value matrix I (identity matrix). Finally, the matrix is ​​recombined to obtain the normalized rotation matrix. This operation effectively eliminates numerical errors that may be introduced during calculation, ensuring... Satisfy strict rotation matrix constraints ( and ).

[0130] S805: Background verification and security switching.

[0131] To ensure the reliability of the new parameters and prevent erroneous updates from causing immediate system failure, the following verification and switching mechanisms are adopted: Independent generation: The calculation process of S802 to S804 above is completed in an independent real-time computing node or thread, ensuring that the real-time performance of the main assembly control loop is not affected.

[0132] Offline verification: using the newly generated candidate hand-eye transformation matrix The pose calculation process in step S2 is re-executed on the original recognition images of the 10 most recent assembly attempts in the historical database to obtain a new set of "back-calculated poses". The average deviation between this set of back-calculated poses and the corresponding, known, correct final re-inspection poses is calculated.

[0133] Performance comparison: Compare the average deviation under the new parameters with the deviation using the current old parameters. The average deviations calculated using the same data are compared. The new parameter is considered superior and activated only if the average deviation under the new parameter is significantly smaller than the average deviation under the old parameter. To ensure robustness, a small improvement threshold can also be set (e.g., the reduction in deviation must exceed the measurement noise level).

[0134] Double-buffered switching: Parameter switching is performed using a double-buffered mechanism. Verified new parameters are written to a backup buffer, and then an atomic operation switches the pointer referenced by the main system from the old parameter buffer to the new parameter buffer. This method enables jitter-free, non-blocking real-time parameter updates, and the main assembly process remains unaffected during the switching process.

[0135] Example 2 Based on the method described in Embodiment 1, this invention also provides a highly integrated intelligent positioning and assembly system. This system consists of multiple collaborative functional units that jointly realize the closed-loop intelligent assembly process. The specific configuration and functions of each unit are as follows: The digital twin modeling unit operates during the production line deployment and initialization phase, responsible for constructing and initializing a digital twin model that precisely corresponds to the physical assembly system. Its main functions include: importing known model data (such as DH parameters, camera intrinsics, hand-eye extrinsic parameters, and CAD geometric models) of the robotic arm, camera, tooling fixtures, and workpiece; defining the assembly topology and motion constraints between entities within a selected unified coordinate system (usually the robotic arm's base coordinate system); and finally organizing the model into a hierarchical structure accessible in real-time (such as a scene graph), residing in high-speed shared memory to meet the real-time requirements of millisecond-level read / write and update.

[0136] The real-time vision perception unit is responsible for image acquisition and pose calculation during the assembly process. Its hardware core includes an industrial camera and an image acquisition card. The camera typically uses a high-resolution global shutter CMOS sensor and completes optical calibration during the initialization phase. The image acquisition card supports high-speed interfaces such as Camera Link, with a transmission bandwidth of no less than 600MB / s to ensure real-time transmission of image data. The software part of this unit includes a feature matching module and a pose calculation module. The feature matching module uses an offline-built visual feature template library to quickly identify and match the acquired images. Its single matching time is usually no more than 80 milliseconds. The pose calculation module is based on algorithms such as PnP and combines the workpiece CAD model to output the six-degree-of-freedom pose of the workpiece in the camera coordinate system.

[0137] The local error calculation unit receives pose data from the real-time visual perception unit and the corresponding ideal pose in the digital twin model. Its core function is to perform coordinate system transformation (through the current hand-eye transformation matrix) and deviation calculation. According to the method described in step S3 of embodiment 1, it quantitatively calculates translational deviation and rotational deviation, and outputs a structured local assembly error vector.

[0138] The digital twin dynamic correction unit performs dynamic updates to the digital twin model based on the error vector output by the local error calculation unit and the measured pose. Its core logic corresponds to step S4 in Embodiment 1: after determining that the error is significant, the ideal pose of the corresponding component in the model is replaced with the actual pose, and the expected poses of all related downstream components are updated in a cascade manner according to the assembly topology. This unit achieves efficient data exchange with the assembly trajectory planning unit through shared memory, ensuring that the end-to-end latency of model state updates is less than 5 milliseconds.

[0139] The assembly trajectory planning unit, based on the latest model state provided by the digital twin dynamic correction unit (i.e., the benchmark reflecting the actual assembly progress), recalculates the target installation pose of the component to be assembled and plans the motion trajectory of the robotic arm. The planning process, as described in step S5 of Example 1, includes inverse kinematics calculation, generating a smooth joint space trajectory using fifth-order polynomial interpolation, and performing fine-grained speed control on the final segment of the trajectory to ensure assembly accuracy and stability.

[0140] The assembly execution and re-inspection unit is responsible for coordinating the execution of the robotic arm's movements and the visual re-inspection after assembly. It achieves high-precision synchronization between the robotic arm controller and the industrial camera through hard-wired signals, ensuring that the camera only triggers capture after the robotic arm's movement has completely stopped, thus eliminating motion blur. This unit's time synchronization accuracy is better than 0.5 milliseconds. It drives the robotic arm to complete the assembly actions and controls the camera to perform re-inspection imaging, thereby providing the final assembly pose data for consistency verification.

[0141] The consistency verification and anomaly handling unit, corresponding to step S7 in embodiment 1, functions to automate quality judgment and process control. It compares the final pose obtained from the re-inspection with the expected state in the digital twin, and determines whether the assembly result is "qualified" or "abnormal" based on preset thresholds (such as translation of 0.05mm or rotation of 0.5 degrees). If qualified, the control data is archived; if abnormal, a complete anomaly handling process, including safety rollback, state reset, workpiece release, and process reset, is triggered.

[0142] The online calibration and optimization unit operates independently of the main real-time control loop and is responsible for the long-term self-optimization of the system. It periodically (or based on accuracy warnings) extracts valid data from the historical assembly database and uses algorithms such as the sliding window weighted average described in step S8 of Example 1 to estimate and update the hand-eye transformation matrix between the camera and the robotic arm online. Its independent operation design ensures that the calibration parameter optimization process does not interfere with the real-time performance and determinism of the main assembly process.

[0143] The eight functional units mentioned above are tightly integrated through defined software interfaces and communication protocols (such as shared memory, real-time Ethernet, and hardwired signals) to form an organic and collaborative whole. The dynamic model established by the digital twin modeling unit is the core data hub of the entire system; the real-time visual perception unit and the assembly execution and re-inspection unit constitute the system's "perception-execution" layer; the local error calculation unit, the digital twin dynamic correction unit, the assembly trajectory planning unit, and the consistency verification and anomaly handling unit constitute the "decision-control-verification" layer; and the online calibration and optimization unit constitutes the "self-learning" layer. From hardware configuration and software modules to the collaborative mechanism between units, this system fully supports the entire process method of "dynamic benchmark update, error blocking, closed-loop verification, and online optimization" described in Example 1.

[0144] Therefore, this system and method together constitute a complete, high-precision, and adaptive intelligent positioning and assembly system, effectively solving a series of technical problems pointed out in the background technology, such as static calibration failure, error accumulation, open-loop control, and in-depth utilization of digital twin models.

[0145] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0146] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A machine vision-based intelligent positioning and assembly method, characterized in that, Includes the following steps: S1, Construct a digital twin model that corresponds to the physical assembly system and can be dynamically updated; S2, before each assembly station is executed, the target area image is acquired by an industrial camera, and the six-degree-of-freedom pose information of the assembled parts and the parts to be assembled in the camera coordinate system is identified and extracted. S3, calculate the spatial deviation between the extracted actual pose of the assembled component and the corresponding ideal pose in the digital twin model to obtain the local error vector; S4. Based on the local error vector and the actual pose, the digital twin model is dynamically corrected, including replacing the ideal pose of the corresponding component with the actual pose, and updating the expected pose of the associated downstream component in a cascade manner according to the predefined assembly topology relationship, and using the corrected model state as the basis for subsequent assembly planning. S5, based on the assembly datum modified by S4, plans and generates the robotic arm motion trajectory instructions for the parts to be assembled. S6, after the robotic arm executes the motion trajectory to complete the assembly, the camera is triggered again to perform a re-inspection imaging and extract the final pose after the assembly is completed; S7. The final pose obtained from the re-inspection is compared with the expected state in the digital twin model. The assembly result is judged according to the preset threshold. If it is qualified, the data is archived. If it is abnormal, the error correction mechanism is triggered. S8, based on historical archived effective assembly data, periodically optimizes and updates the hand-eye transformation matrix between the camera and the robotic arm online.

2. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, Step S1, which involves constructing the digital twin model, includes: The base coordinate system of the robotic arm is determined to be the global reference system; A three-dimensional geometric model integrating the kinematic parameters of the robotic arm, camera parameters, tooling fixtures, and workpiece; Define the assembly topology relationships and motion constraints between the entities; The data is organized into a hierarchical structure that can be accessed in real time and stored in high-speed memory.

3. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, Step S2 specifically includes: The industrial camera is triggered to capture images synchronously after the robotic arm stops moving. The acquired images are subjected to filtering and noise reduction, as well as adaptive threshold segmentation. Based on geometric features, candidate regions for workpieces are initially screened out from the processed image; The candidate regions are matched with an offline-built visual feature template library to determine the workpiece category and rough pose. Based on the 3D model of the successfully matched workpiece, the six-degree-of-freedom pose of the workpiece in the camera coordinate system is calculated using the perspective n-point algorithm.

4. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, Step S3, calculating the spatial deviation specifically includes: Transform the actual pose in the camera coordinate system to the robot arm base coordinate system; Read the ideal pose of the corresponding component in the digital twin in the base coordinate system; The root mean square difference of the translation vector is calculated as the translation deviation, and the minimum angle between the rotation matrix or the unit quaternion is calculated as the rotation deviation. The translational and rotational deviations are combined to form a local assembly error vector.

5. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, In step S4, the step of updating the expected pose of the associated downstream components in a cascade manner according to the predefined assembly topology specifically includes: Based on the predefined relative pose relationships between components, and combined with the updated actual pose of the upstream component, the new expected absolute pose of the downstream component in the global coordinate system is recalculated, and its record in the digital twin is updated.

6. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, In step S5, replanning the robotic arm's motion trajectory includes: Based on the corrected digital twin state, the target installation pose of the part to be assembled in the robot arm base coordinate system is calculated; The target pose is converted into the target joint angle through inverse kinematics; A smooth joint space trajectory is generated between the initial joint angle and the target joint angle using polynomial interpolation, and the velocity of the final segment of the trajectory is limited.

7. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, In step S6, the re-inspection imaging and pose extraction steps use the same image processing and pose calculation process as the pose identification and extraction steps in step S2, and ensure that the part to be re-inspected after assembly is visible to the camera.

8. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, In step S7, the triggering error correction mechanism includes performing the following operations in sequence: Control the robotic arm to retract along its original trajectory to a safe position; Reset system task status flags; Control the end effector to release the workpiece; The control flow jumps to step S2 to re-execute the assembly at the current workstation.

9. The intelligent positioning and assembly method based on machine vision according to claim 1, characterized in that, In step S8, the periodic online optimization update specifically includes: Optimization is triggered when the number of successful assembly reaches a set threshold or the standard deviation of historical pose deviation exceeds a set threshold. The initial recognition pose and the final re-inspection pose of multiple successful assemblies are extracted from the historical database as observation samples. A time-weighted sliding window averaging algorithm is used to estimate the rotation and translation components of the hand-eye transformation matrix, respectively. After offline verification of the new parameters, a double-buffering mechanism is used to safely switch to the main system.

10. A machine vision-based intelligent positioning and assembly system, characterized in that, include: The digital twin modeling unit is used to build and maintain dynamically updated digital twin models. The real-time vision perception unit is used to acquire images before and after assembly and to identify and calculate the six-degree-of-freedom pose of the workpiece. The local error calculation unit is used to calculate the spatial deviation between the actual pose and the ideal pose of the digital twin; A digital twin dynamic correction unit is used to dynamically update the digital twin model based on the deviation and the actual pose, and to cascade update the pose of related components; An assembly trajectory planning unit is used to replan the robotic arm's motion trajectory based on the updated digital twin state; The assembly execution and re-inspection unit is used to control the robotic arm to perform assembly and trigger visual re-inspection after assembly. The consistency verification and anomaly handling unit is used to verify the assembly results and perform qualified data archiving or anomaly correction processes. An online calibration and optimization unit is used to periodically optimize the hand-eye transformation matrix based on historical assembly data; Each unit works collaboratively through a communication interface to form a closed-loop intelligent assembly system.

Images