A digital twin modeling method and system based on binocular vision and target dynamic tracking

Abstract

The application discloses a digital twin modeling method and system based on binocular vision and target dynamic tracking, and the method comprises the following steps: constructing a binocular camera system, completing internal parameter calibration, distortion calibration and external parameter calibration, and establishing a coordinate system mapping relationship; synchronously collecting target images and correcting the target images, extracting feature point coordinates, and performing abnormal elimination and multi-target area division; calculating feature point parallax, obtaining three-dimensional coordinates through triangulation, and generating an optimized target point cloud; solving the spatial pose of the target through a PNP algorithm, realizing single / multi-target dynamic tracking in combination with time sequence difference; performing abnormal detection and smoothing processing on the pose sequence, constructing a weighted error model, and optimizing feature recognition parameters in a closed loop; transmitting the pose data to realize virtual-real synchronous visualization of workpieces; monitoring the spatial relationship of the workpieces and collision risks, and alarming; and storing data for backtracking analysis. The method and system realize high-precision dynamic measurement, attitude tracking and digital and visual assembly process of a complex workpiece assembly process.

Description

Technical Field

[0001] This invention belongs to the field of industrial digitalization and intelligent manufacturing technology, and in particular relates to a digital twin modeling method and system based on binocular vision and target dynamic tracking. Background Technology

[0002] With the increasing demands for assembly precision in industries such as aerospace, shipbuilding, and energy equipment, traditional single-point measurement and manual visual inspection methods can no longer meet the assembly monitoring needs of complex curved surfaces or large workpieces. While existing binocular vision systems can achieve certain spatial measurement functions, they have the following shortcomings in dynamic scenarios:

[0003] Unstable pose calculation: Feature matching is easily lost when the target moves, leading to the accumulation of pose calculation errors;

[0004] Measurement results are isolated: binocular measurement data failed to be integrated with the 3D model in real time, and digital twin feedback was lacking;

[0005] Errors cannot be self-optimized: System calibration errors and image recognition deviations are difficult to correct automatically during operation;

[0006] The process is not traceable: there is a lack of historical pose data recording and assembly process playback mechanism.

[0007] Therefore, there is an urgent need for a binocular vision digital twin system that can achieve real-time tracking, error self-correction, and visual feedback to support high-precision assembly and process control. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention proposes a digital twin modeling method and system based on binocular vision and target dynamic tracking. By integrating binocular vision measurement, error self-optimization, and three-dimensional digital twin rendering, it aims to achieve high-precision dynamic measurement, attitude tracking, and digital visualization of the assembly process for complex workpieces.

[0009] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution:

[0010] A digital twin modeling method based on binocular vision and target dynamic tracking, specifically including:

[0011] S1. System Startup and Binocular Calibration: Initialize the workpiece assembly process monitoring environment, combine two monocular industrial cameras into a binocular camera system through preset angles, and perform real-time visual monitoring of the workpiece assembly process monitoring environment; perform parameter calibration of intrinsic, distortion, and extrinsic parameters of the binocular camera system in sequence, and establish the mapping relationship between the world coordinate system and the binocular coordinate system.

[0012] S2. Image Acquisition and Feature Recognition: Use a binocular camera to acquire target images with feature patterns in real time and synchronously. Correct the acquired target images using distortion parameters and extrinsic parameters. After correction, extract the coordinates of target feature points and perform outlier removal and differentiation management in the case of multiple targets.

[0013] S3. Parallax Calculation and Depth Reconstruction: Calculate the parallax between corresponding feature points in the target images obtained by two monocular cameras, combine the dual-target positioning parameters, and calculate the three-dimensional spatial coordinates of the feature points through a triangulation model; combine the three-dimensional spatial coordinates of multiple feature points of the same target to generate a real-time three-dimensional point cloud model of the target, and perform noise reduction and optimization processing.

[0014] S4. Pose Determination and Dynamic Tracking: Select the target's frontal feature point set as the initial reference template, and make the target's frontal normal direction parallel to the optical axis of the left eye camera; match the noise-reduced and optimized real-time 3D point cloud with the initial reference template one-to-one, and solve the target's spatial pose using the PNP algorithm; continuously acquire multiple frames of pose of a single target per unit time to obtain the target's pose sequence; achieve dynamic tracking of a single target through temporal differential pose sequence; for multi-target environments, distinguish different measured objects by the target's unique identifier ID, and perform multi-target dynamic tracking in parallel;

[0015] S5. Error Modeling and Self-Optimization: Perform multi-level anomaly detection and error feedback on the target pose sequence, as well as spatiotemporal consistency smoothing.

[0016] S6. Data Communication and 3D Visualization: The smoothed target pose sequence is transmitted to the 3D visualization software through a real-time communication protocol; based on the rigid binding relationship between the target and the 3D model of the workpiece, the virtual model of the workpiece is rendered in real time, and the position and attitude of the target are synchronized to achieve virtual-real synchronous visualization.

[0017] S7. Alarm and Process Retrospective: Real-time monitoring of the spatial relationship and collision risk of each workpiece and its corresponding target during the assembly process. An alarm is triggered when the risk exceeds the preset risk threshold. All relevant data of the entire process is stored for playback and traceability analysis of the assembly process.

[0018] Furthermore, step S1 specifically includes:

[0019] Initialize the hardware and software involved in the workpiece assembly process monitoring environment, and establish the communication connection between the binocular camera system and the 3D visualization software;

[0020] Based on the workpiece digital model and the monitoring site layout, a global 3D environment model including the measurement area, workpiece geometric features and obstacles is constructed in 3D visualization software to assist in the calibration of the binocular camera system and subsequent coordinate mapping.

[0021] A binocular camera system is formed by setting up a left-eye camera and a right-eye camera. The two monocular cameras are not parallel, but their lenses are oriented to form a preset angle. First, the intrinsic parameters and distortion of the two monocular cameras are calibrated separately. Then, the extrinsic parameters of the binocular camera system are calibrated by using a checkerboard or dot array target to obtain extrinsic parameters including binocular baseline length, rotation matrix, and translation matrix.

[0022] After calibration, the mapping relationship between the binocular camera coordinate system and the world coordinate system is obtained based on the intrinsic parameters, distortion parameters, and extrinsic parameters. At the same time, the calibration parameters are automatically stored, ready to enter real-time measurement.

[0023] Furthermore, the specific steps of extracting target feature point coordinates after correction and performing outlier removal and multi-target differentiation management are as follows:

[0024] Image preprocessing is performed using adaptive threshold segmentation with Gaussian weighted mean; target point extraction is performed using a sub-pixel feature extraction algorithm with gradient optimization and iterative least squares; after identifying target feature points, the center coordinates of the positioning points are calculated; based on the target geometric prior model, consistency verification and anomaly removal are performed on the identification results; in the case of multiple targets, the system uses coded features or identifier IDs to achieve unique identification and numbering management of multiple targets.

[0025] Furthermore, the noise reduction and optimization process described in step S3 includes: removing outliers by statistical outlier removal and removing outliers by radius outlier removal from the obtained real-time 3D point cloud.

[0026] Furthermore, the multi-level anomaly detection and error feedback described in step S5 specifically includes:

[0027] Perform joint identification of abnormal frames based on three criteria, including:

[0028] Pose jump threshold criterion: Calculate the pose change of the target between adjacent frames, expressed by the formula:

[0029] , ;

[0030] like or If so, it is marked as an abnormal pose;

[0031] in, , These are the rotation matrices of the target in the k-th frame and the (k-1)-th frame, respectively. , These are the translation vectors of the target in the k-th frame and the (k-1)-th frame, respectively; The change in the rotation matrix. The change in the translation vector. The time difference between two consecutive frames; It is an inverse cosine function. A function for finding the trace of a matrix; , These are the preset maximum linear velocity and maximum angular velocity, determined by the workpiece assembly process constraints.

[0032] Binocular reconstruction confidence criterion: Based on the disparity and reprojection error between corresponding feature points in the target images obtained by the two monocular cameras in the k-th frame, the confidence weight of the target pose is defined. The formula is expressed as:

[0033] ;

[0034] in, The average reprojection error of all feature points in this frame. For parallax standard deviation, β are adjustment coefficients; if , If the set confidence weight threshold is not met, it is considered a low-confidence frame;

[0035] Temporal trajectory deviation criterion: Select the pose of the target for N consecutive frames, and use the pose of the first N-1 frames to obtain the target fitted motion model; use the fitted motion model to predict the target pose for the Nth frame. If the actual pose of the Nth frame The error between the predicted pose and the actual pose exceeds a preset error threshold. If so, it is considered abnormal;

[0036] If any one of the above three criteria is met, the current frame is determined to be an abnormal frame; the abnormal frame is removed, and the pose estimation error between the abnormal frame pose and the reliable reference pose is stored. ;

[0037] Construct a weighted error model and dynamically adjust the feature recognition parameters based on the error:

[0038] Online cumulative statistical pose estimation error And calculate the values ​​of each degree of freedom according to the pose degree of freedom. If significant fluctuations occur in the mean and variance, the feature recognition parameters are adjusted in the pose degree of freedom direction, including: adjusting the minimum visible point threshold for feature extraction, dynamically modifying the matching search range, and enabling multi-frame feature fusion strategy in frequently occluded areas.

[0039] Furthermore, step S7 specifically includes:

[0040] The system calculates the relative pose relationship between the target of the workpiece to be assembled and the target of the workpiece being assembled in real time, and monitors the assembly gap and posture error. When the assembly offset or collision risk exceeds the threshold, the alarm mechanism is triggered and the operator is prompted to make adjustments. At the same time, the timestamp, original image frame, pose data and alarm events of the entire assembly process are recorded and stored in the database for full process playback and traceability analysis.

[0041] After assembly is completed, an assembly accuracy report is automatically generated. The assembly accuracy report includes: the maximum / average assembly gap and posture error at each key stage of the assembly process, statistics and time-series curves of the out-of-tolerance period of the assembly gap, target tracking confidence heat map, conformity judgment with process tolerance zone, and a link to the 3D animation playback of the assembly process.

[0042] Furthermore, this application also discloses a digital twin modeling system based on binocular vision and target dynamic tracking, which specifically includes:

[0043] The binocular calibration module is used to calibrate the distortion parameters of the binocular camera inside and outside, construct a global three-dimensional environment model containing the measurement area and workpiece features, and enter the real-time measurement mode after storing the parameters.

[0044] The image acquisition and feature recognition module is used to simultaneously acquire binocular images, correct the images using calibration parameters, and identify target feature points;

[0045] The disparity calculation and depth reconstruction module is used to calculate the disparity of feature points obtained from the left and right cameras, combine the calibration parameters to solve the three-dimensional coordinates of the target, and combine multiple feature points of the same target to generate a three-dimensional point cloud of the target.

[0046] The pose solving and dynamic tracking module is used to solve the target pose based on the target's 3D point cloud, generate a temporal difference pose sequence, and realize single / multiple target dynamic tracking through the target ID.

[0047] The error modeling and self-optimization module is used to smooth the pose sequence, remove outliers, establish an error measurement model, and feed the error back to the image acquisition and feature recognition module to adjust the parameters, forming a closed-loop optimization.

[0048] The data communication and 3D visualization module is used to transmit real-time pose to 3D software and achieve virtual-real synchronous mapping and dynamic rendering through the rigid binding relationship between the target and the workpiece model.

[0049] The alarm and process traceability module is used to monitor assembly gap / posture deviations and trigger an alarm when the deviation exceeds the threshold; at the same time, it records the entire assembly process data for playback and traceability, and outputs an accuracy report after assembly.

[0050] This application also discloses an electronic device comprising a memory and a processor, wherein:

[0051] Memory is used to store computer programs that can run on a processor;

[0052] A processor is configured to execute, while running the computer program, a digital twin modeling method based on binocular vision and dynamic target tracking as described above.

[0053] A computer-readable storage medium is also disclosed, which stores computer instructions for causing a processor to execute a digital twin modeling method based on binocular vision and target dynamic tracking as described above.

[0054] Based on the above technical solution, the present invention has at least the following beneficial effects:

[0055] By integrating binocular vision, target feature recognition, and PNP pose solving, and introducing a triple criterion to jointly identify abnormal frames, and accumulating abnormal frames to adjust the feature recognition parameters, the pose jitter caused by occlusion, lighting changes, or motion blur is effectively suppressed. In typical industrial environments, the repeatability accuracy of 3D tracking is stabilized within ±0.22 mm, which is significantly better than the traditional fixed parameter filtering method, and achieves high-precision and high-robust real-time monitoring of dynamic assembly processes.

[0056] A closed-loop self-optimizing measurement system architecture was constructed; error modeling, feature recognition parameter adjustment, and external parameter drift compensation were linked to form a closed-loop mechanism of "perception-evaluation-feedback-correction"; when the system detects a decline in measurement performance (such as increased reprojection error or fixed target drift), it can automatically trigger parameter fine-tuning or lightweight online calibration without manual intervention, which greatly improves the long-term operational stability of the system.

[0057] It supports intelligent risk warning and process traceability under multiple working conditions; by accurately calculating the relative pose, assembly gap and attitude deviation between the workpiece to be assembled and the workpiece being assembled, and combining the bounding box collision prediction model, it can realize sub-millimeter level assembly anomaly warning; the whole process data (including timestamp, pose and alarm event) is stored in a structured manner, supporting second-level backtracking and three-dimensional visualization review, providing a data foundation for quality analysis and process optimization.

[0058] Significantly improves the fidelity and real-time performance of digital twin virtual-real synchronization; by rigidly binding the target and the workpiece model, it can drive the rendering of the virtual scene based on the smoothed target pose sequence, obtain the 3D animation playback of the assembly process, realize the millisecond-level synchronization between physical assembly actions and digital models, and meet the needs of high real-time application scenarios such as human-machine collaborative assembly and remote guidance.

[0059] It possesses strong engineering adaptability and scalability; the entire system adopts a modular design, supports single / multi-target parallel tracking, different assembly process tolerance configurations, and multiple communication protocols (such as OPC UA, MQTT). It has been verified in production lines such as automotive electronic housing press-fitting and aerospace fastener alignment. It has a short deployment cycle, low maintenance costs, and good prospects for industrialization and promotion. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the method proposed in this invention;

[0061] Figure 2 This is a schematic diagram of the bi-target calibration and coordinate system transformation process in the method proposed in this invention;

[0062] Figure 3 This is a data flow diagram of pose solving and dynamic tracking in the method proposed in this invention;

[0063] Figure 4 This is a schematic diagram of the three-dimensional digital twin visualization process in the method proposed in this invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the following is combined with... Figures 1-4 The present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0065] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0066] like Figure 1 As shown, this invention proposes a digital twin modeling method based on binocular vision and dynamic target tracking, which specifically includes:

[0067] S1. System Startup and Binocular Calibration: Initialize the workpiece assembly process monitoring environment, combine two monocular industrial cameras into a binocular camera system through preset angles, and perform real-time visual monitoring of the workpiece assembly process monitoring environment; perform parameter calibration of intrinsic, distortion, and extrinsic parameters of the binocular camera system in sequence, and establish the mapping relationship between the world coordinate system and the binocular coordinate system.

[0068] As a preferred embodiment, such as Figure 2 As shown, step S1 specifically involves:

[0069] Initialize the hardware (two monocular cameras) and software (calibration program) involved in the workpiece assembly process monitoring environment, and establish the communication connection between the binocular camera system and the 3D visualization software;

[0070] Based on the workpiece digital model and the monitoring site layout, a global 3D environment model including the measurement area, workpiece geometric features and obstacles is constructed in 3D visualization software to assist in the calibration of the binocular camera system and subsequent coordinate mapping.

[0071] A binocular camera system is constructed by deploying a left-eye camera and a right-eye camera. The two monocular cameras are not parallel; instead, their lenses are oriented at a preset angle. This preset angle is the angle between the optical axes of the lenses of the left-eye and right-eye cameras. The preset angle value is determined based on the actual requirements of the workpiece assembly site (such as measurement field of view requirements, binocular baseline length requirements, measurement accuracy requirements, etc.), and is usually an acute angle. In this embodiment, the OpenCV checkerboard binocular camera calibration method is used. First, the intrinsic parameters and distortion of the two monocular cameras are calibrated separately. Then, the extrinsic parameters of the binocular camera system are calibrated using a checkerboard or dot array target to obtain extrinsic parameters including the binocular baseline length, rotation matrix, and translation matrix.

[0072] After calibration, the mapping relationship between the binocular camera coordinate system and the world coordinate system is obtained based on the intrinsic parameters, distortion parameters, and extrinsic parameters. At the same time, the calibration parameters are automatically stored, ready to enter real-time measurement.

[0073] S2. Image Acquisition and Feature Recognition: Use a binocular camera to acquire target images with feature patterns in real time and synchronously. Correct the acquired target images using distortion parameters and extrinsic parameters. After correction, extract the coordinates of target feature points and perform outlier removal and differentiation management in the case of multiple targets.

[0074] In a preferred embodiment, the step of extracting the target feature point coordinates after correction and performing outlier removal and multi-target differentiation management specifically involves:

[0075] Image preprocessing is performed using adaptive threshold segmentation with Gaussian weighted mean; target point extraction is performed using a sub-pixel feature extraction algorithm with gradient optimization and iterative least squares; after identifying target feature points, the center coordinates of the positioning points are calculated; based on the target geometric prior model, consistency verification and anomaly removal are performed on the identification results; in the case of multiple targets, the system uses coded features or identifier IDs to achieve unique identification and numbering management of multiple targets.

[0076] S3. Disparity Calculation and Depth Reconstruction: Calculate the disparity between corresponding feature points in the target images obtained by two monocular cameras, and calculate the three-dimensional spatial coordinates of the feature points using a triangulation model in combination with the dual-target positioning parameters; combine the three-dimensional spatial coordinates of multiple feature points of the same target to generate a real-time three-dimensional point cloud model of the target, and perform noise reduction and optimization processing; in this embodiment, noise reduction and optimization processing includes statistical outlier removal and radius outlier removal.

[0077] S4. Pose Determination and Dynamic Tracking: (e.g.) Figure 3 As shown, the target's frontal feature point set is selected as the initial reference template, and the target's frontal normal direction is made parallel to the optical axis of the left eye camera. The real-time 3D point cloud after noise reduction and optimization is mapped one-to-one with the initial reference template, and the spatial pose of the target is solved by the PNP algorithm. The pose of a single target is continuously acquired in multiple frames per unit time to obtain the pose sequence of the target. Dynamic tracking of a single target is achieved by using the temporal differential pose sequence. For multi-target environments, different targets are distinguished by the unique identifier ID of the target, and multi-target dynamic tracking is performed in parallel.

[0078] S5. Error Modeling and Self-Optimization: Perform multi-level anomaly detection and error feedback on the target pose sequence, as well as spatiotemporal consistency smoothing.

[0079] In this embodiment, abnormal frames that do not meet the standards are screened out through multi-level anomaly monitoring and error feedback, and the pose estimation error of the abnormal frames is synchronously fed back to the image acquisition and feature recognition stage, thereby dynamically adjusting the feature recognition parameters and forming a closed-loop optimization mechanism for the algorithm; while the spatiotemporal consistency smoothing processing is to perform confidence-weighted Kalman filtering on the current frame and the previous few frames to make the data smoother and easier for users to visualize.

[0080] In a preferred embodiment, the multi-level anomaly detection and error feedback in step S5 specifically refers to:

[0081] Perform joint identification of abnormal frames based on three criteria, including:

[0082] Pose jump threshold criterion: Calculate the pose change of the target between adjacent frames, expressed by the formula:

[0083] , ;

[0084] like or If so, it is marked as an abnormal pose;

[0085] in, , These are the rotation matrices of the target in the k-th frame and the (k-1)-th frame, respectively. , These are the translation vectors of the target in the k-th frame and the (k-1)-th frame, respectively; The change in the rotation matrix. The change in the translation vector. The time difference between two consecutive frames; It is an inverse cosine function. A function for finding the trace of a matrix; , These are the preset maximum linear velocity and maximum angular velocity, determined by the workpiece assembly process constraints.

[0086] Binocular reconstruction confidence criterion: Based on the disparity and reprojection error between corresponding feature points in the target images obtained by the two monocular cameras in the k-th frame, the confidence weight of the target pose is defined. The formula is expressed as:

[0087] ;

[0088] in, The average reprojection error of all feature points in this frame. For parallax standard deviation, β are adjustment coefficients; if , If the set confidence weight threshold is not met, it is considered a low-confidence frame;

[0089] Temporal trajectory deviation criterion: Select the pose of the target for N consecutive frames, and use the pose of the first N-1 frames to obtain the target fitted motion model; use the fitted motion model to predict the target pose for the Nth frame. If the actual pose of the Nth frame The error between the predicted pose and the actual pose exceeds a preset error threshold. If so, it is considered abnormal;

[0090] If any one of the above three criteria is met, the current frame is determined to be an abnormal frame; the abnormal frame is removed, and the pose estimation error between the abnormal frame pose and the reliable reference pose is stored. ;

[0091] Construct a weighted error model and dynamically adjust the feature recognition parameters based on the error:

[0092] Online cumulative statistical pose estimation error And calculate the values ​​of each degree of freedom according to the pose degree of freedom. The mean and variance fluctuations, such as significant fluctuations (e.g., the cumulative error after 30 pose estimations). Calculate these 30 If the mean and variance of the model change significantly compared to the previous calculation (e.g., the mean doubles), then the feature recognition parameters are adjusted in the pose degrees of freedom direction, including: adjusting the minimum visible point threshold for feature extraction, dynamically modifying the matching search range, and enabling multi-frame feature fusion strategies in frequently occluded areas.

[0093] S6. Data Communication and 3D Visualization: such as Figure 4 As shown, the smoothed target pose sequence is transmitted to the 3D visualization software through a real-time communication protocol (such as MQTT or WebSocket communication protocol); based on the rigid binding relationship between the target and the 3D model of the workpiece being tested, the virtual model of the workpiece is rendered in real time (in this embodiment, Unity3D or Qt 3DEngine is used for real-time rendering), and the position and orientation of the corresponding target are synchronized to achieve virtual-real synchronous visualization.

[0094] S7. Alarm and Process Retrospective: Real-time monitoring of the spatial relationship and collision risk of each workpiece and its corresponding target during the assembly process. An alarm is triggered when the risk exceeds the preset risk threshold. All relevant data of the entire process is stored for playback and traceability analysis of the assembly process.

[0095] In a preferred embodiment, step S7 specifically comprises:

[0096] The system calculates the relative pose relationship between the target of the workpiece to be assembled and the target of the workpiece being assembled in real time, and monitors the assembly gap and posture error.

[0097] In this embodiment, the binocular camera system tracks the pose of the target installed on the workpiece to be assembled in real time, while the pose of the target on the workpiece to be assembled (such as a base, housing or fixing fixture) has been determined by high-precision calibration before the assembly begins and is regarded as a static reference.

[0098] Real-time calculation of the relative pose transformation matrix between the two targets ,in, The orientation of the workpiece to be assembled. Given the pose of the workpiece to be assembled; calculate the assembly gap based on the obtained relative pose transformation matrix;

[0099] The assembly clearance is denoted as g: it is obtained by taking the distance of the nearest point in the normal direction of the critical mating surface, that is:

[0100] For all ;

[0101] Where P and Q are the sets of mating surface points defined in the CAD model for the two workpieces, respectively. , Points in P and Q are respectively; n_ref is the unit normal vector of the mating surface of the assembled workpiece;

[0102] When an assembly misalignment or collision risk exceeds a threshold, an alarm mechanism is triggered, prompting the operator to make adjustments; in this embodiment, when g < g min (i.e., there is a risk of over-profit) or g > g max (i.e., excessive assembly clearance), g min g max These are the preset lower and upper limits of the assembly clearance; or ( When the assembly posture angle is out of tolerance (θ_rot is the preset assembly posture angle out-of-tolerance threshold), a multi-level alarm mechanism is triggered (including audible and visual prompts + software pop-ups + pausing the automatic assembly process). Alternatively, the alarm mechanism can be triggered by detecting whether there is a collision in the predicted trajectory through AABB or OBB bounding box detection.

[0103] Simultaneously, the timestamp of the entire assembly process (set to 10s in this embodiment), original image frames, pose data (including the pose of the target relative to the camera after target smoothing, assembly gap / posture error) and alarm events are recorded and stored in a database (such as InfluxDB or TimescaleDB) for full process playback and traceable analysis.

[0104] After assembly is completed, an assembly accuracy report is automatically generated. The assembly accuracy report includes: the maximum / average assembly gap and posture error at each key stage of the assembly process, statistics and time series curves of the period when the assembly gap exceeds the tolerance, target tracking confidence heat map, and conformity judgment with the process tolerance zone (i.e., judging whether the assembly accuracy meets the process tolerance range); and a link to a 3D animation playback of the assembly process (containing an overlay view of the virtual theoretical model and the real assembly data).

[0105] Furthermore, this application also discloses a digital twin modeling system based on binocular vision and target dynamic tracking, which specifically includes:

[0106] The binocular calibration module is used to calibrate the distortion parameters of the binocular camera inside and outside, construct a global three-dimensional environment model containing the measurement area and workpiece features, and enter the real-time measurement mode after storing the parameters.

[0107] The image acquisition and feature recognition module is used to simultaneously acquire binocular images, correct the images using calibration parameters, and identify target feature points;

[0108] The disparity calculation and depth reconstruction module is used to calculate the disparity of feature points obtained from the left and right cameras, combine the calibration parameters to solve the three-dimensional coordinates of the target, and combine multiple feature points of the same target to generate a three-dimensional point cloud of the target.

[0109] The pose solving and dynamic tracking module is used to solve the target pose based on the target's 3D point cloud, generate a temporal difference pose sequence, and realize single / multiple target dynamic tracking through the target ID.

[0110] The error modeling and self-optimization module is used to smooth the pose sequence, remove outliers, establish an error measurement model, and feed the error back to the image acquisition and feature recognition module to adjust the parameters, forming a closed-loop optimization.

[0111] The data communication and 3D visualization module is used to transmit real-time pose to 3D software and achieve virtual-real synchronous mapping and dynamic rendering through the rigid binding relationship between the target and the workpiece model.

[0112] The alarm and process traceability module is used to monitor assembly gap / posture deviations and trigger an alarm when the deviation exceeds the threshold; at the same time, it records the entire assembly process data for playback and traceability, and outputs an accuracy report after assembly.

[0113] An electronic device is also disclosed, comprising a memory and a processor, wherein:

[0114] Memory is used to store computer programs that can run on a processor;

[0115] A processor is configured to execute, while running the computer program, a digital twin modeling method based on binocular vision and dynamic target tracking as described above.

[0116] A computer-readable storage medium is also disclosed, which stores computer instructions for causing a processor to execute a digital twin modeling method based on binocular vision and target dynamic tracking as described above.

[0117] In summary, the method proposed in this invention continuously optimizes the accuracy and stability of 3D measurement through error modeling, self-calibration algorithms, and extrinsic parameter compensation mechanisms. The digital twin modeling system based on binocular vision and dynamic target tracking is compact, versatile, and can be integrated with industrial robots, measuring instruments, and production execution systems to achieve high-precision measurement, assembly monitoring, and process optimization of complex structures or large-sized components. It is particularly suitable for fields such as aerospace, shipbuilding, energy equipment, and intelligent manufacturing, providing a high-precision, real-time technical solution for digital assembly and virtual-physical integrated production.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0119] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0120] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A digital twin modeling method based on binocular vision and dynamic target tracking, characterized in that, Specifically, the following steps are included: S1. System Startup and Binocular Calibration: Initialize the workpiece assembly process monitoring environment, combine two monocular industrial cameras into a binocular camera system through preset angles, and perform real-time visual monitoring of the workpiece assembly process monitoring environment; perform parameter calibration of intrinsic, distortion, and extrinsic parameters of the binocular camera system in sequence, and establish the mapping relationship between the world coordinate system and the binocular coordinate system. S2. Image Acquisition and Feature Recognition: Use a binocular camera to acquire target images with feature patterns in real time and synchronously. Correct the acquired target images using distortion parameters and extrinsic parameters. After correction, extract the coordinates of target feature points and perform outlier removal and differentiation management in the case of multiple targets. S3. Parallax Calculation and Depth Reconstruction: Calculate the parallax between corresponding feature points in the target images obtained by two monocular cameras, combine the dual-target positioning parameters, and calculate the three-dimensional spatial coordinates of the feature points through a triangulation model; combine the three-dimensional spatial coordinates of multiple feature points of the same target to generate a real-time three-dimensional point cloud model of the target, and perform noise reduction and optimization processing. S4. Pose Determination and Dynamic Tracking: Select the target front feature point set as the initial reference template, and make the target front normal direction parallel to the optical axis of the left eye camera; match the noise-reduced and optimized real-time 3D point cloud with the initial reference template one by one, and solve the target's spatial pose using the PNP algorithm; The pose of a single target is continuously acquired in multiple frames within a unit of time to obtain the pose sequence of the target; dynamic tracking of a single target is achieved by using the temporal differential pose sequence; for multi-target environments, different targets are distinguished by the unique identifier ID of the target, and dynamic tracking of multiple targets is performed in parallel. S5. Error Modeling and Self-Optimization: Perform multi-level anomaly detection and error feedback on the target pose sequence, as well as spatiotemporal consistency smoothing. S6. Data Communication and 3D Visualization: The smoothed target pose sequence is transmitted to the 3D visualization software through a real-time communication protocol; based on the rigid binding relationship between the target and the 3D model of the workpiece, the virtual model of the workpiece is rendered in real time, and the position and attitude of the target are synchronized to achieve virtual-real synchronous visualization. S7. Alarm and Process Retrospective: Real-time monitoring of the spatial relationship and collision risk of each workpiece and its corresponding target during the assembly process, triggering an alarm when the preset risk threshold is exceeded; Store all relevant data throughout the process for playback and traceability analysis of the assembly process.

2. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, characterized in that, Step S1 is as follows: Initialize the hardware and software involved in the workpiece assembly process monitoring environment, and establish the communication connection between the binocular camera system and the 3D visualization software; Based on the workpiece digital model and the monitoring site layout, a global 3D environment model including the measurement area, workpiece geometric features and obstacles is constructed in 3D visualization software to assist in the calibration of the binocular camera system and subsequent coordinate mapping. A binocular camera system is formed by setting up a left-eye camera and a right-eye camera. The two monocular cameras are not parallel, but their lenses are oriented to form a preset angle. First, the intrinsic parameters and distortion of the two monocular cameras are calibrated separately. Then, the extrinsic parameters of the binocular camera system are calibrated by using a checkerboard or dot array target to obtain extrinsic parameters including binocular baseline length, rotation matrix, and translation matrix. After calibration, the mapping relationship between the binocular camera coordinate system and the world coordinate system is obtained based on the intrinsic parameters, distortion parameters, and extrinsic parameters. At the same time, the calibration parameters are automatically stored, ready to enter real-time measurement.

3. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, characterized in that, The process of extracting target feature point coordinates after correction, and performing outlier removal and multi-target differentiation management specifically involves: Image preprocessing is performed using adaptive threshold segmentation with Gaussian weighted mean; target point extraction is performed using a sub-pixel feature extraction algorithm with gradient optimization and iterative least squares; after identifying target feature points, the center coordinates of the positioning points are calculated; based on the target geometric prior model, consistency verification and anomaly removal are performed on the identification results; in the case of multiple targets, the system uses coded features or identifier IDs to achieve unique identification and numbering management of multiple targets.

4. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, characterized in that, The noise reduction and optimization process described in step S3 includes: performing statistical outlier denoising and radius outlier denoising on the obtained real-time 3D point cloud.

5. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, characterized in that, The multi-level anomaly detection and error feedback mentioned in step S5 specifically refers to: Perform joint identification of abnormal frames based on three criteria, including: Pose jump threshold criterion: Calculate the pose change of the target between adjacent frames, expressed by the formula: ， ； like or If so, it is marked as an abnormal pose; in, , These are the rotation matrices of the target in the k-th frame and the (k-1)-th frame, respectively. , These are the translation vectors of the target in the k-th frame and the (k-1)-th frame, respectively; The change in the rotation matrix. The change in the translation vector. The time difference between two consecutive frames; It is an inverse cosine function. A function for finding the trace of a matrix; , These are the preset maximum linear velocity and maximum angular velocity, determined by the workpiece assembly process constraints. Binocular reconstruction confidence criterion: Based on the disparity and reprojection error between corresponding feature points in the target images obtained by the two monocular cameras in the k-th frame, the confidence weight of the target pose is defined. The formula is expressed as: ； in, The average reprojection error of all feature points in this frame. For parallax standard deviation, β are adjustment coefficients; if , If the set confidence weight threshold is not met, it is considered a low-confidence frame; Temporal trajectory deviation criterion: Select the pose of the target for N consecutive frames, and use the pose of the first N-1 frames to obtain the target fitted motion model; use the fitted motion model to predict the target pose for the Nth frame. If the actual pose of the Nth frame The error between the predicted pose and the actual pose exceeds a preset error threshold. If so, it is considered abnormal; If any one of the above three criteria is met, the current frame is determined to be an abnormal frame; the abnormal frame is removed, and the pose estimation error between the abnormal frame pose and the reliable reference pose is stored. ; Construct a weighted error model and dynamically adjust the feature recognition parameters based on the error: Online cumulative statistical pose estimation error And calculate the values ​​of each degree of freedom according to the pose degree of freedom. If significant fluctuations occur in the mean and variance, the feature recognition parameters are adjusted in the pose degree of freedom direction, including adjusting the minimum visible point threshold for feature extraction, dynamically modifying the matching search range, and enabling multi-frame feature fusion strategy in frequently occluded areas.

6. The digital twin modeling method based on binocular vision and target dynamic tracking according to claim 1, characterized in that, Step S7 is as follows: The system calculates the relative pose relationship between the target of the workpiece to be assembled and the target of the workpiece to be assembled in real time, and monitors the assembly gap and posture error; when the assembly offset or collision risk exceeds the threshold, the alarm mechanism is triggered and the operator is prompted to make adjustments. Simultaneously, timestamps, original image frames, pose data, and alarm events of the entire assembly process are recorded and stored in the database for full process playback and traceability analysis. After assembly is completed, an assembly accuracy report is automatically generated. The assembly accuracy report includes: the maximum / average assembly gap and posture error at each key stage of the assembly process, statistics and time-series curves of the out-of-tolerance period of the assembly gap, target tracking confidence heat map, conformity judgment with process tolerance zone, and a link to the 3D animation playback of the assembly process.

7. A digital twin modeling system based on binocular vision and dynamic target tracking, characterized in that, Specifically, it includes: The binocular calibration module is used to calibrate the distortion parameters of the binocular camera inside and outside, construct a global three-dimensional environment model containing the measurement area and workpiece features, and enter the real-time measurement mode after storing the parameters. The image acquisition and feature recognition module is used to simultaneously acquire binocular images, correct the images using calibration parameters, and identify target feature points; The disparity calculation and depth reconstruction module is used to calculate the disparity of feature points obtained from the left and right cameras, combine the calibration parameters to solve the three-dimensional coordinates of the target, and combine multiple feature points of the same target to generate a three-dimensional point cloud of the target. The pose solving and dynamic tracking module is used to solve the target pose based on the target's 3D point cloud, generate a temporal difference pose sequence, and realize single / multiple target dynamic tracking through the target ID. The error modeling and self-optimization module is used to smooth the pose sequence, remove outliers, establish an error measurement model, and feed the error back to the image acquisition and feature recognition module to adjust the parameters, forming a closed-loop optimization. The data communication and 3D visualization module is used to transmit real-time pose to 3D software and achieve virtual-real synchronous mapping and dynamic rendering through the rigid binding relationship between the target and the workpiece model. The alarm and process traceability module is used to monitor assembly gap / posture deviations and trigger an alarm when the deviation exceeds the threshold; at the same time, it records the entire assembly process data for playback and traceability, and outputs an accuracy report after assembly.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein: Memory is used to store computer programs that can run on a processor; A processor, configured to execute, while running the computer program, a digital twin modeling method based on binocular vision and target dynamic tracking as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a digital twin modeling method based on binocular vision and target dynamic tracking as described in any one of claims 1-6.

Images