Robot three-dimensional scanning method and device based on digital twin calibration

Abstract

The scheme provides a robot three-dimensional scanning method and device based on digital twin calibration, acquires a three-dimensional scanning device and a calibration device, models the three-dimensional scanning device and the calibration device into a digital twin space, calibrates parameters of the digital twin space based on an error between a scanning point cloud in the digital twin space and a real scene scanning point cloud, plans an optimal scanning path corresponding to a to-be-scanned workpiece in the calibrated digital twin space, and in a real environment, the three-dimensional scanning device scans the to-be-scanned workpiece based on the optimal scanning path at each path point in a corresponding first scanning pose. The scheme calibrates digital twin parameters through registration error of virtual and real scanning point clouds, realizes high matching of a digital model and physical device parameters, ensures that a virtually planned scanning pose can be accurately reproduced on a physical robot, and avoids scanning errors caused by virtual-real pose conversion deviation.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a robot 3D scanning method and apparatus based on digital twin calibration. Background Technology

[0002] In the current field of precision manufacturing, industrial robots equipped with 3D scanners such as laser line scanners and structured light scanners have become the core solution for automated 3D shape inspection of complex workpieces, and are widely used in non-contact dimensional measurement in industries such as automotive, aerospace, and 3C electronics. However, with the continuous improvement of the demand for unmanned production lines and high-precision inspection, the shortcomings of traditional 3D scanning calibration technology have become increasingly prominent, and it can no longer meet the cycle time and accuracy requirements of modern automated production lines.

[0003] Current 3D scanning calibration relies entirely on manual labor, which is not only cumbersome and inefficient, but also requires highly skilled operators. The manual placement of the calibration target ball results in poor repeatability, making it difficult to standardize the physical reference for each calibration. Furthermore, the inconsistent quality of standard parts directly leads to poor stability of calibration results, a lack of comparability between different batches of data, and an inability to reliably guarantee scanning accuracy.

[0004] Meanwhile, traditional calibration methods lack precise scanning path planning, making it difficult to control scanning distance and angle. The point cloud coverage on the standard sphere surface is insufficient, severely affecting the accuracy of sphere center fitting. More importantly, existing technologies have low digitization levels and lack digital model support for real-time interaction with the physical system. This makes it impossible to trace and dynamically compensate for system errors such as robot positioning, hand-eye calibration, and scanner measurement. Furthermore, calibration data cannot form a closed-loop feedback, making it difficult to support the long-term accuracy maintenance of the scanning system. Summary of the Invention

[0005] This application provides a robot 3D scanning method and apparatus based on digital twin calibration. By calibrating the digital twin parameters through the registration error between virtual and real scan point clouds, a high degree of matching between the digital model and physical device parameters is achieved, ensuring that the scan pose planned virtually can be accurately reproduced on the physical robot, and avoiding scanning errors caused by virtual-real pose conversion deviation.

[0006] In a first aspect, embodiments of this application provide a robot 3D scanning method based on digital twin calibration, the method comprising:

[0007] A 3D scanning device and a calibration device are acquired, and the 3D scanning device and the calibration device are modeled in a digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. In the digital twin space, the optimal simulated scanning pose of the 3D scanning device for each preset posture standard baseball bat is simulated, and the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat is obtained; in the real environment, the 3D scanning device performs real scanning on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The registration error between each virtual scan point cloud and the corresponding real scan point cloud is obtained as the twin scan error. Based on the twin scan error of the standard baseball bat with different preset postures, the parameters of the digital twin space are calibrated. The workpiece to be scanned is obtained, and the workpiece to be scanned is modeled in a digital twin space. Multiple path points of the workpiece to be scanned and the first scan pose of each path point are obtained in the twin space. The optimal scan path passing through each path point is generated based on the path planning algorithm. In a real-world environment, the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose, based on the optimal scanning path.

[0008] Secondly, embodiments of this application provide a robot 3D scanning device based on digital twin calibration, comprising: The acquisition module is used to acquire the 3D scanning device and the calibration device, and to model the 3D scanning device and the calibration device into the digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. The virtual scanning module is used to simulate the optimal simulated scanning pose of a 3D scanning device for each preset posture standard baseball bat in the digital twin space, and to obtain the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat; in the real environment, the 3D scanning device performs a real scan on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The calibration module is used to obtain the registration error between each virtual scan point cloud and the corresponding real scan point cloud as the twin scan error, and to calibrate the parameters of the digital twin space based on the twin scan error of the standard baseball bat with different preset postures. The path planning module is used to acquire the workpiece to be scanned, model the workpiece to be scanned in a digital twin space, acquire multiple path points of the workpiece to be scanned and the first scan pose of each path point in the twin space, and generate the optimal scan path passing through each path point based on the path planning algorithm. The real scanning module is used in a real environment, where the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose based on the optimal scanning path.

[0009] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run a robot 3D scanning method based on digital twin calibration.

[0010] The main contributions and innovations of this invention are as follows: This application embodiment combines the measurement point normal vector to construct three types of line-of-sight constraints: tilt angle, depth of field, and field width. Simultaneously, based on the point cloud fitting residual, coverage integrity, working distance deviation, and normal deviation, a comprehensive evaluation function is designed to iteratively generate the optimal simulated scanning pose, strictly constraining the scanning angle, distance, and field of view to avoid scanning reflection, measurement blind spots, and point cloud loss. This application embodiment calculates the registration error between the virtual and real scanned point clouds, calculates the sensitivity of each error term using the Jacobian matrix, calibrates the system parameters of the digital twin space, and accurately traces and compensates for system errors across all dimensions, including robot motion, hand-eye calibration, sensor measurement, equipment installation, and environment, achieving a high degree of matching between the digital model and physical equipment parameters. Based on the comprehensive error of the twin scanning error prediction system during the calibration stage, this application embodiment performs feedforward correction on the scanning pose before executing the real scan, compensating for position and angle errors in the real scan in advance, achieving feedforward precision compensation, further improving the detection accuracy of the workpiece's three-dimensional shape scanning, and meeting the precision inspection needs of the automotive, aerospace, and other fields.

[0011] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0012] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating a robot 3D scanning method based on digital twin calibration according to an embodiment of this application; Figure 2 This is a schematic diagram of a calibration device according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the construction of a line-of-sight constraint according to an embodiment of this application; Figure 4 This is a structural block diagram of a robot 3D scanning device based on digital twin calibration according to an embodiment of this application; Figure 5This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0014] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0015] Example 1 This application provides a robot 3D scanning method based on digital twin calibration. It calibrates digital twin parameters by adjusting the registration error between virtual and real scan point clouds, achieving a high degree of matching between the digital model and physical device parameters. This ensures that the virtually planned scanning pose can be accurately reproduced on the physical robot, avoiding scanning errors caused by virtual-to-real pose conversion deviations. Specifically, refer to... Figure 1 The method includes: A 3D scanning device and a calibration device are acquired, and the 3D scanning device and the calibration device are modeled in a digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. In the digital twin space, the optimal simulated scanning pose of the 3D scanning device for each preset posture standard baseball bat is simulated, and the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat is obtained; in the real environment, the 3D scanning device performs real scanning on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The registration error between each virtual scan point cloud and the corresponding real scan point cloud is obtained as the twin scan error. Based on the twin scan error of the standard baseball bat with different preset postures, the parameters of the digital twin space are calibrated. The workpiece to be scanned is obtained, and the workpiece to be scanned is modeled in a digital twin space. Multiple path points of the workpiece to be scanned and the first scan pose of each path point are obtained in the twin space. The optimal scan path passing through each path point is generated based on the path planning algorithm. In a real-world environment, the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose, based on the optimal scanning path.

[0016] In the current embodiment, a 3D scanning device is mounted on the end effector of the robot as a 3D scanning apparatus. A schematic diagram of the calibration device is shown below. Figure 2 As shown, a standard baseball bat is fixed on a posture generating bracket, which can automatically adjust the posture to place the standard baseball bat in different preset postures.

[0017] Specifically, the standard bat is a standard component with balls connected to both ends of a connecting rod, and has a calibrated standard ball center distance and standard ball size.

[0018] In the current embodiment, the 3D scanning device and calibration device are modeled in a 1:1 manner within the digital twin space, thereby achieving real-time virtual-real synchronization between the digital twin model and the physical system.

[0019] In the current embodiment, the preset posture standard bat includes a horizontally placed standard bat, a vertically placed standard bat, and a standard bat placed at a 45° angle.

[0020] Specifically, in real-world scenarios, the parameters of the attitude generation support are automatically adjusted to adjust the standard baseball bat to each preset attitude to obtain the preset attitude standard baseball bat. In the digital twin space, the parameters of the attitude generation support in real-world scenarios are acquired in real time to synchronize the preset attitude of the standard baseball bat.

[0021] In the current embodiment, a set of bat coordinate points for each preset posture standard bat in the world coordinate system is obtained in the digital twin space. The set of bat coordinate points includes the coordinates of the two centers of the corresponding preset posture standard bat and the coordinates of any measurement point on the surface of the bat. The normal vector of each measurement point on the preset posture standard bat is obtained based on the set of bat coordinates. A line-of-sight constraint is constructed based on the line-of-sight parameter of the 3D scanning device. Based on the normal vector of each measurement point on each preset posture standard bat and the line-of-sight constraint, and with the virtual scan point cloud quality meeting a set threshold as the optimization condition, the optimal simulated scan pose corresponding to each preset posture standard bat is generated.

[0022] Specifically, the world coordinate system is a global coordinate system in the real scene. The coordinates of each preset posture standard bat in the digital twin space are mapped to the world coordinate system, corresponding to the real scene. When the standard bat and the 3D scanning device are adjusted in the digital twin space, the operation in the digital twin space can be restored in the real scene based on the mapping of the same world coordinate system.

[0023] Furthermore, based on the first attitude transformation matrix and the second attitude transformation matrix, the pose matrix of each preset attitude standard bat in the world coordinate system is obtained, and based on the pose matrix of each preset attitude standard bat in the world coordinate system, the set of bat coordinate points corresponding to each preset attitude standard bat is obtained. Here, the first attitude transformation matrix is ​​the pose transformation matrix of the attitude generating bracket relative to the three-dimensional scanning device, and the second attitude transformation matrix is ​​the pose transformation matrix of the standard bat based on the attitude generating bracket.

[0024] Specifically, the formula for obtaining the pose matrix of each preset pose standard bat in the world coordinate system is expressed as:

[0025] in, For different preset standard bat positions, =1 indicates a standard baseball bat in a horizontal position. =2 indicates a standard baseball bat in a vertical orientation. =3 indicates a standard baseball bat tilted at 45°. This is the pose transformation matrix of a standard baseball bat based on a pose generation support. This is the pose transformation matrix of the attitude generation support relative to the 3D scanning device. This is the pose matrix of a pre-set standard baseball bat in the world coordinate system.

[0026] Furthermore, the measurement points on the surface of a standard baseball can be obtained through random sampling and discrete methods. The formula for calculating the normal vector of any measurement point on the surface of the baseball is:

[0027] in, The normal vector of the measurement point. The coordinates of the corresponding measurement point in the set of baseball bat coordinate points. These are the coordinates of the sphere's center corresponding to the measurement point.

[0028] Furthermore, the line-of-sight constraint includes tilt angle constraint, depth of field constraint, and field width constraint. The tilt angle constraint is used to ensure that the angle between the normal vector of the measurement point and the angle bisector of the incident laser of the 3D scanning device is not greater than the tilt angle constraint threshold. The depth of field constraint is used to ensure that the distance from the 3D scanning device to the measurement point is not greater than the optimal depth of field range. The field width constraint is used to ensure that the position of the measurement point is within the effective length of the laser stripe of the 3D scanning device.

[0029] Specifically, a schematic diagram of constructing the line-of-sight constraint is shown below. Figure 3 As shown, tilt angle constraints can prevent reflections or blind spots. The formula for tilt angle constraints is expressed as:

[0030] in, Let i be the measurement point. For measurement points The normal vector, where L is the optical center position of the 3D scanning device. To constrain the threshold, Point the 3D scanning device to the measurement point The angle between the bisectors of the incident laser angle.

[0031] Specifically, depth-of-field constraints are used to ensure that the 3D scanning device measures the measurement points within the optimal depth-of-field range, and the formula is expressed as:

[0032] in, The optimal depth of field range for 3D scanning equipment. Let i be the measurement point, and L be the optical center position of the 3D scanning device. The nearest effective measurement distance for 3D scanning equipment. This is the farthest effective measurement distance for a 3D scanning device.

[0033] Specifically, the field width constraint is used to ensure that the measured point is always within the effective field of view of the 3D scanning device and does not exceed the laser stripe, thus avoiding point cloud loss. The formula is expressed as:

[0034] in, The angle bisector of the boundary line of the laser beam emitted by the 3D scanning equipment. The field of view of the 3D scanning device is the maximum effective angle of the laser stripes. Point the 3D scanning device to the measurement point The angle between the bisectors of the incident laser angle.

[0035] Furthermore, based on the quality calculation of the virtual scanned point cloud, a comprehensive evaluation function is used, with the value of the comprehensive evaluation function being less than a preset threshold as the optimization condition for obtaining the optimal simulated scan pose. The formula for the comprehensive evaluation function is as follows:

[0036] in, To evaluate the value of the comprehensive evaluation function, , , , These are the weighting coefficients. The fitting residuals for fitting the virtual scanned point cloud to a sphere are used. To ensure the coverage completeness of the virtual scanned point cloud, For working distance deviation, This is a deviation from the normal direction.

[0037] Specifically, the virtual scan point cloud obtained from the standard baseball bat in the current preset posture is fitted, and the fitting residual is generated based on the fitting result; the coverage of the virtual scan point cloud over the standard baseball bat is calculated as the coverage integrity of the virtual scan point cloud; the working distance deviation is the deviation between the actual measurement distance from the 3D scanning device to the measurement point and the optimal measurement distance; the normal deviation is the degree to which the angle between the incident direction of the 3D scanning device and the normal of the measurement point deviates from the optimal incident direction.

[0038] Specifically, The calculation formula is:

[0039] in, Let i be the measurement point. Let r be the coordinates of the center of the sphere corresponding to the measurement point, r be the radius of the sphere, and N be the total number of measurement points.

[0040] Specifically, The calculation formula is:

[0041] in, Point the 3D scanning device to the measurement point The angle between the incident laser angle bisectors, where N is the total number of measurement points. This is an index for the total number of measurement points. This is the optimal measurement distance for 3D scanning equipment.

[0042] Specifically, The calculation formula is:

[0043] in, This represents the difference between the angle between the incident direction of the 3D scanning device and the normal of the measurement point, and the optimal incident direction. N is the total number of measurement points. This is the index for the total number of measurement points.

[0044] Furthermore, the optimal simulated scanning pose is generated by using the coverage of the virtual scanned point cloud over the standard baseball bat being greater than a first threshold as a constraint. In this scheme, the first threshold is 75%, which means that the optimal scanning pose must scan 75% of the standard baseball bat corresponding to the preset pose.

[0045] In the current embodiment, the hand-eye calibration relationship between the robot and the 3D scanning device is obtained. Based on the hand-eye calibration relationship, the optimal simulated scanning pose in the digital twin space is converted into the robot pose in the real scene. Based on the robot pose, motion parameters are calculated. Based on the motion parameters, the robot in the 3D scanning device adjusts the 3D scanning device to the optimal simulated scanning pose.

[0046] Specifically, the optimal simulated scanning pose is converted into the pose of the 3D scanning device, and then the 3D scanning device pose is converted into the robot pose based on the hand-eye calibration relationship. The formula for the 3D scanning device pose is expressed as:

[0047] in , W represents the world coordinate system, and S represents the coordinate system of the 3D scanning device. The target rotation matrix of the 3D scanning device. The target position of the optical center of the 3D scanning device. The optimal measurement distance for 3D scanning equipment. Let x, y, and z be the x, y, and z axis vectors of the target rotation matrix. Let these be the coordinates of the measured point in the world coordinate system. The pose of the 3D scanning device.

[0048] Furthermore, based on the hand-eye calibration relationship between the robot and the 3D scanning equipment in the 3D scanning device... To obtain robot pose The formula is expressed as:

[0049] Based on robot location The system calculates the control commands that the robot can execute, and based on these commands, moves the 3D scanning device to the optimal simulated scanning pose. The pose of the 3D scanning device.

[0050] In other words, this scheme finds the optimal simulated scanning pose for each preset standard baseball bat through an iterative process.

[0051] In the current embodiment, the three-dimensional scanning device in the three-dimensional scanning apparatus acquires multiple real scan point clouds under the corresponding optimal simulated scanning pose.

[0052] In the current embodiment, the twin scanning error is the spatial topography error between each measurement point in the virtual scan point cloud and the corresponding measurement point in the real scan point cloud. The digital twin spatial scanning error, which consists of multiple error terms, is defined as a parameter of the digital twin space. The Jacobian matrix of each measurement point in the spatial topography error is calculated in a first-order linearization manner. The Jacobian matrix contains the error sensitivity of different error terms in the digital twin spatial scanning error corresponding to each measurement point. The scanning error increment is calculated based on the Jacobian matrix of each measurement point, and the digital twin spatial scanning error is calibrated using the scanning error increment.

[0053] Specifically, the virtual scanned point cloud is registered with the corresponding real scanned point cloud in the world coordinate system. The spatial position residual between each measurement point in the virtual scanned point cloud and the corresponding measurement point in the real scanned point cloud is calculated. Based on the spatial position residual of each measurement point in the virtual scanned point cloud and the normal vector, the spatial shape error of the measurement point is calculated. The formula is expressed as:

[0054]

[0055]

[0056] in, The spatial position residual of measurement point i in the virtual scanned point cloud. To measure the spatial position of point i in the virtual scanned point cloud, This represents the spatial location of the corresponding measurement point in the actual scanned point cloud. Let i be the normal vector of the measurement point i in the virtual scanned point cloud. for The transpose of the matrix, The spatial topography error of measurement point i in the virtual scanned point cloud. This is a spatial topography error model used to obtain the spatial topography error of each measurement point within the digital twin space, where N is the total number of measurement points. This is the index for the total number of measurement points.

[0057] Specifically, since the digital twin space simulates the virtual scanned point cloud that can be obtained under optimal conditions, while significant errors may occur in real-world scenarios due to installation and environmental issues, this solution defines the digital twin space scanning error, which consists of multiple error terms, as a parameter of the digital twin space. The digital twin space scanning error is expressed as:

[0058] in, Due to twin scanning error, For robot motion error, To account for hand-eye calibration error, This is due to the measurement error of the 3D scanning equipment. Due to installation error, Let T be the environmental error and T be the transpose.

[0059] The formula for calculating the scan error increment is expressed as:

[0060]

[0061]

[0062] in, The spatial topography error of measurement point i in the virtual scanned point cloud. For digital twin spatial scanning error, Let i be the Jacobian matrix of the measurement point i in the virtual scanned point cloud. For partial derivatives, This is the increment of the scanning error. This is the weight matrix, which is used to comprehensively consider the measurement quality, incident angle conditions, and local point cloud reliability at different measurement points. The Jacobian matrix for all measurement points in the virtual scanned point cloud. for The transpose of .

[0063] Finally, the formula for calibrating the scanning error of the digital twin space using the scanning error increment is expressed as follows:

[0064] in, To correct the spatial scanning error of the digital twin, This is the increment of the scanning error. This represents the original digital twin spatial scanning error.

[0065] In other words, this solution calibrates the parameters of the digital twin space by using twin scanning errors of standard baseball bats with different preset postures, so that the 3D scanning device in the digital twin space is exactly the same as the 3D scanning device in the real scene, thereby better simulating the 3D scanning device in the real scene.

[0066] In the current embodiment, the outer surface of the workpiece to be scanned is discretized into multiple discrete points in the digital twin space, and the normal vector of each discrete point is obtained. Based on the position coordinates, normal vector, and measurement tilt angle constraints of the three-dimensional scanning device of each discrete point, the workpiece to be scanned is divided into multiple sub-regions, and path points and corresponding first scanning poses are generated for each sub-region. Based on the path planning algorithm, the optimal scanning path passing through each path point is generated with the shortest path as the optimization objective. Among them, all discrete points in the same sub-region can be obtained by scanning the corresponding first scanning pose in one scan.

[0067] Specifically, an adaptive triangular mesh sampling method is used to discretize the outer surface of the workpiece to be scanned, thereby ensuring that there are more discrete points in areas with complex surfaces and fewer discrete points in areas with smooth surfaces, in order to obtain better scanning results. The set of discrete points obtained by discretizing the outer surface of the workpiece to be scanned is represented as follows: ,in, These are discrete points in the discrete point set.

[0068] Specifically, the normal vector of each discrete point is obtained by the surface normal vector calculation method. In this scheme, the current discrete point and its multiple neighboring discrete points are formed into a tetrahedron, and the vector between the center of the circumscribed sphere of the tetrahedron and the current discrete point is used as the normal vector of the current discrete point.

[0069] Specifically, dividing the workpiece to be scanned into multiple sub-regions by measuring the tilt angle constraint ensures that discrete points within the same sub-region can be acquired in a single scan under the same scanning pose. That is, discrete points within the same sub-region have the same path points and the same first scanning pose, as expressed by the formula:

[0070] in, For discrete points spatial coordinates, For discrete points The normal vector.

[0071] Specifically, the path planning algorithm prioritizes connecting path points with high similarity to the first scan pose to form the optimal scan path.

[0072] In the current embodiment, the system comprehensive error of each first scanning pose is predicted in the digital twin model, and the corresponding first scanning pose is corrected with the system comprehensive error to obtain the second scanning pose. The three-dimensional scanning device scans the workpiece to be scanned at each path point with the corresponding second scanning pose based on the optimal scanning path.

[0073] Specifically, the system comprehensive error for each first scanning pose is predicted based on the twin scanning error obtained during the calibration phase. In other words, when a robot in a real-world scenario scans with the first scanning pose, there will be certain errors in the execution of standard instructions obtained within the digital twin space. This solution predicts the system comprehensive error in advance based on the twin scanning error, thereby performing feedforward error compensation before execution and improving the scanning accuracy of the workpiece. The formula is expressed as:

[0074] in, For the overall error of the system, This is the first scan pose. This is the second scan pose. =[ ], ,

[0075] in, For positional error, , , These represent the positional errors along the x, y, and z axes, respectively. For angular error, , , These are the angular errors under the scanning tilt angle, elevation angle, and azimuth angle, respectively.

[0076] In the current embodiment, the point cloud obtained by the 3D scanning device in each first scanning pose in the real scene is fitted to obtain the measured point cloud of the workpiece to be scanned. The measured point cloud of the workpiece to be scanned is registered and compared with the corresponding standard 3D model to generate a model processing error analysis report and realize digital quality evaluation.

[0077] Example 2 Based on the same concept, referencing Figure 4 This application also proposes a robot 3D scanning device based on digital twin calibration, comprising: The acquisition module is used to acquire the 3D scanning device and the calibration device, and to model the 3D scanning device and the calibration device into the digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. The virtual scanning module is used to simulate the optimal simulated scanning pose of a 3D scanning device for each preset posture standard baseball bat in the digital twin space, and to obtain the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat; in the real environment, the 3D scanning device performs a real scan on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The calibration module is used to obtain the registration error between each virtual scan point cloud and the corresponding real scan point cloud as the twin scan error, and to calibrate the parameters of the digital twin space based on the twin scan error of the standard baseball bat with different preset postures. The path planning module is used to acquire the workpiece to be scanned, model the workpiece to be scanned in a digital twin space, acquire multiple path points of the workpiece to be scanned and the first scan pose of each path point in the twin space, and generate the optimal scan path passing through each path point based on the path planning algorithm. The real scanning module is used in a real environment, where the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose based on the optimal scanning path.

[0078] Example 3 This embodiment also provides an electronic device, see reference. Figure 5 It includes a memory 404 and a processor 402, the memory 404 storing a computer program and the processor 402 being configured to run the computer program to perform the steps in any of the above method embodiments.

[0079] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0080] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0081] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0082] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the robot 3D scanning methods based on digital twin calibration in the above embodiments.

[0083] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0084] The transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0085] The input / output device 408 is used to input or output information. In this embodiment, the input information may be the optimal simulated scanning pose, the optimal scanning path, etc., and the output information may be the scanning result of the workpiece to be scanned, etc.

[0086] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program: A 3D scanning device and a calibration device are acquired, and the 3D scanning device and the calibration device are modeled in a digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. In the digital twin space, the optimal simulated scanning pose of the 3D scanning device for each preset posture standard baseball bat is simulated, and the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat is obtained; in the real environment, the 3D scanning device performs real scanning on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The registration error between each virtual scan point cloud and the corresponding real scan point cloud is obtained as the twin scan error. Based on the twin scan error of the standard baseball bat with different preset postures, the parameters of the digital twin space are calibrated. The workpiece to be scanned is obtained, and the workpiece to be scanned is modeled in a digital twin space. Multiple path points of the workpiece to be scanned and the first scan pose of each path point are obtained in the twin space. The optimal scan path passing through each path point is generated based on the path planning algorithm. In a real-world environment, the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose, based on the optimal scanning path.

[0087] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0088] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0089] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 5 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0090] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot 3D scanning method based on digital twin calibration, characterized in that, Includes the following steps: A 3D scanning device and a calibration device are acquired, and the 3D scanning device and the calibration device are modeled in a digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. In the digital twin space, the optimal simulated scanning pose of the 3D scanning device for each preset posture standard baseball bat is simulated, and the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat is obtained; in the real environment, the 3D scanning device performs real scanning on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The registration error between each virtual scan point cloud and the corresponding real scan point cloud is obtained as the twin scan error. Based on the twin scan error of the standard baseball bat with different preset postures, the parameters of the digital twin space are calibrated. The workpiece to be scanned is obtained, and the workpiece to be scanned is modeled in a digital twin space. Multiple path points of the workpiece to be scanned and the first scan pose of each path point are obtained in the twin space. The optimal scan path passing through each path point is generated based on the path planning algorithm. In a real-world environment, the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose, based on the optimal scanning path.

2. The robot 3D scanning method based on digital twin calibration according to claim 1, characterized in that, In the digital twin space, a set of baseball coordinate points for each preset posture standard baseball bat in the world coordinate system is obtained. The set of baseball coordinate points includes the coordinates of the two centers of the corresponding preset posture standard baseball bat and the coordinates of any measurement point on the surface of the bat. Based on the set of baseball coordinate points, the normal vector of each measurement point on the preset posture standard baseball bat is obtained. Based on the line-of-sight parameters of the 3D scanning device, a line-of-sight constraint is constructed. Based on the normal vector of each measurement point on each preset posture standard baseball bat and the line-of-sight constraint, and with the virtual scan point cloud quality meeting a set threshold as the optimization condition, the optimal simulated scan pose corresponding to each preset posture standard baseball bat is generated.

3. The robot 3D scanning method based on digital twin calibration according to claim 2, characterized in that, Based on the first attitude transformation matrix and the second attitude transformation matrix, the pose matrix of each preset attitude standard baseball bat in the world coordinate system is obtained. Based on the pose matrix of each preset attitude standard baseball bat in the world coordinate system, the set of baseball bat coordinate points corresponding to each preset attitude standard baseball bat is obtained. The first attitude transformation matrix is ​​the pose transformation matrix of the attitude generating bracket relative to the three-dimensional scanning device, and the second attitude transformation matrix is ​​the pose transformation matrix of the standard baseball bat based on the attitude generating bracket.

4. The robot 3D scanning method based on digital twin calibration according to claim 2, characterized in that, The line-of-sight constraints include tilt angle constraints, depth of field constraints, and field width constraints. The tilt angle constraint is used to ensure that the angle between the normal vector of the measurement point and the angle bisector of the incident laser of the 3D scanning device is not greater than the tilt angle constraint threshold. The depth of field constraint is used to ensure that the distance from the 3D scanning device to the measurement point is not greater than the optimal depth of field range. The field width constraint is used to ensure that the position of the measurement point is within the effective length of the laser stripe of the 3D scanning device.

5. A robot 3D scanning method based on digital twin calibration according to claim 2, characterized in that, Based on the comprehensive evaluation function for virtual scanned point cloud quality calculation, the optimal simulated scan pose is obtained when the value of the comprehensive evaluation function is less than a preset threshold. The formula for the comprehensive evaluation function is as follows: in, To evaluate the value of the comprehensive evaluation function, , , , These are the weighting coefficients. The fitting residuals for fitting the virtual scanned point cloud to a sphere are used. To ensure the coverage completeness of the virtual scanned point cloud, For working distance deviation, This is a deviation from the normal direction.

6. The robot 3D scanning method based on digital twin calibration according to claim 1, characterized in that, The hand-eye calibration relationship between the robot and the 3D scanning device is obtained. Based on the hand-eye calibration relationship, the optimal simulated scanning pose in the digital twin space is converted into the robot pose in the real scene. Based on the robot pose, motion parameters are calculated. Based on the motion parameters, the robot in the 3D scanning device adjusts the 3D scanning device to the optimal simulated scanning pose.

7. The robot 3D scanning method based on digital twin calibration according to claim 1, characterized in that, The twin scanning error is the spatial topography error between each measurement point in the virtual scan point cloud and the corresponding measurement point in the real scan point cloud. The digital twin spatial scanning error, which consists of multiple error terms, is defined as a parameter of the digital twin space. The Jacobian matrix of each measurement point in the spatial topography error is calculated in a first-order linearization manner. The Jacobian matrix contains the error sensitivity of different error terms in the digital twin spatial scanning error corresponding to each measurement point. The scanning error increment is calculated based on the Jacobian matrix of each measurement point, and the digital twin spatial scanning error is calibrated using the scanning error increment.

8. The robot 3D scanning method based on digital twin calibration according to claim 1, characterized in that, In the digital twin space, the outer surface of the workpiece to be scanned is discretized into multiple discrete points, and the normal vector of each discrete point is obtained. Based on the position coordinates, normal vector, and measurement tilt angle constraints of the 3D scanning device, the workpiece to be scanned is divided into multiple sub-regions, and path points and corresponding first scanning poses are generated for each sub-region. Based on the path planning algorithm, the optimal scanning path passing through each path point is generated with the shortest path as the optimization objective. Among them, all discrete points in the same sub-region can be obtained by scanning the corresponding first scanning pose in one scan.

9. A robot 3D scanning device based on digital twin calibration, characterized in that, include: The acquisition module is used to acquire the 3D scanning device and the calibration device, and to model the 3D scanning device and the calibration device into the digital twin space. The 3D scanning device is a robot equipped with a 3D scanning device, and the calibration device includes a posture generating bracket that holds a standard baseball bat and is used to adjust the standard baseball bat to a preset posture. The standard baseball bat is a standard component with a ball connected to both ends of a connecting rod. The virtual scanning module is used to simulate the optimal simulated scanning pose of a 3D scanning device for each preset posture standard baseball bat in the digital twin space, and to obtain the virtual scanning point cloud of the 3D scanning device under the optimal simulated scanning pose corresponding to each preset posture standard baseball bat; in the real environment, the 3D scanning device performs a real scan on each preset posture standard baseball bat under the corresponding optimal simulated scanning pose to obtain the corresponding real scanning point cloud. The calibration module is used to obtain the registration error between each virtual scan point cloud and the corresponding real scan point cloud as the twin scan error, and to calibrate the parameters of the digital twin space based on the twin scan error of the standard baseball bat with different preset postures. The path planning module is used to acquire the workpiece to be scanned, model the workpiece to be scanned in a digital twin space, acquire multiple path points of the workpiece to be scanned and the first scan pose of each path point in the twin space, and generate the optimal scan path passing through each path point based on the path planning algorithm. The real scanning module is used in a real environment, where the 3D scanning device scans the workpiece to be scanned at each path point with the corresponding first scanning pose based on the optimal scanning path.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform a robot 3D scanning method based on digital twin calibration as described in any one of claims 1-8.

Images