Non-rigid registration method, device and electronic equipment

Abstract

The present application provides a non-rigid registration method, apparatus, and electronic device. The method comprises: performing global matching on the non-rigid object to be registered based on scene data and model data of the object to be registered, obtaining an initial pose transformation relationship between the scene data and the model data of the object to be registered; determining a first type error and a second type error corresponding to any valid module among the modules obtained by performing finite element partitioning on the scene data of the object to be registered; optimizing the pose transformation relationship corresponding to each valid module based on the first type error and second type error obtained by performing finite element partitioning on the scene data of the object to be registered, and determining a registration result for the object to be registered. This method can improve the accuracy of non-rigid object matching.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a non-rigid registration method, device, and electronic device. Background Art

[0002] A non-rigid object (also called a deformable object) refers to an object that can deform locally, such as a rubber pad, a rubber ring, or a thin PCB (Printed Circuit Board) board.

[0003] Since non-rigid objects may be deformed, if the registration method for rigid objects is completely adopted when registering non-rigid objects, there will be large errors. Summary of the Invention

[0004] In view of this, the present application provides a non-rigid registration method, device, and electronic device.

[0005] According to a first aspect of an embodiment of the present application, a non-rigid registration method is provided, comprising:

[0006] Performing global matching on the non-rigid object to be registered based on the scene data and the model data of the object to be registered to obtain an initial pose transformation relationship between the scene data of the object to be registered and the model data;

[0007] For any valid module among the modules obtained by performing finite element division on the scene data of the object to be registered, determining the first type error and the second type error corresponding to the valid module; wherein the valid module is a module including the point cloud of the object to be registered, the first type loss is used to characterize the registration constraints between the points in the valid module and the corresponding points in the model data, the points in the model data corresponding to the points in the valid module are determined based on the posture transformation relationship corresponding to the valid module, the initial value of the posture transformation relationship corresponding to the valid module is the initial posture transformation relationship, and the second type loss is used to characterize the relative posture constraint between the valid module and the neighboring valid modules;

[0008] Based on the finite element division of the scene data of the object to be registered, the first type error and the second type error of each valid module are obtained, the posture transformation relationship corresponding to each valid module is optimized, and the registration result of the object to be registered is determined.

[0009] According to a second aspect of an embodiment of the present application, a non-rigid registration device is provided, comprising:

[0010] A global matching unit is used to perform global matching on the non-rigid object to be registered based on the scene data and model data of the object to be registered, so as to obtain an initial pose transformation relationship between the scene data of the object to be registered and the model data;

[0011] an error determination unit, configured to determine, for any valid module among the modules obtained by performing finite element division on the scene data of the object to be registered, a first type of error and a second type of error corresponding to the valid module; wherein the valid module is a module including a point cloud of the object to be registered, the first type of loss is used to characterize the registration constraints between points in the valid module and corresponding points in the model data, the points in the model data corresponding to the points in the valid module are determined based on a pose transformation relationship corresponding to the valid module, the initial value of the pose transformation relationship corresponding to the valid module is the initial pose transformation relationship, and the second type of loss is used to characterize the relative pose constraint between the valid module and a neighboring valid module;

[0012] The registration unit is used to obtain the first type error and the second type error of each valid module based on the finite element division of the scene data of the object to be registered, optimize the posture transformation relationship corresponding to each valid module, and determine the registration result of the object to be registered.

[0013] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a processor and a memory, wherein:

[0014] Memory for storing computer programs;

[0015] The processor is used to implement the method provided in the first aspect when executing the program stored in the memory.

[0016] The non-rigid registration method of the embodiment of the present application performs global matching on the non-rigid object to be registered based on the scene data and model data of the object to be registered, and obtains the initial posture transformation relationship between the scene data and model data of the object to be registered; for any valid module in the modules obtained by finite element division of the scene data of the object to be registered, determines the first type error and the second type error corresponding to the valid module; further, based on the first type error and the second type error of each valid module obtained by finite element division of the scene data of the object to be registered, optimizes the posture transformation relationship corresponding to each valid module, determines the registration result of the object to be registered, and improves the degree of freedom of non-rigid object registration by performing finite element division on the scene data of the object to be registered; in addition, for the valid modules in the modules obtained by finite element division, on the one hand, matching is performed through point cloud registration, and on the other hand, deformation restriction is performed through the constraints of adjacent modules, thereby improving the accuracy of non-rigid object matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a non-rigid registration method provided in an embodiment of the present application;

[0018] Figure 2A This is a schematic diagram of finite element partitioning of 2D scene data provided in an embodiment of the present application;

[0019] Figure 2B This is a schematic diagram of finite element partitioning of 3D scene data provided in an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the result of image finite element segmentation performed when scene data is deformed relative to model data, provided in an embodiment of the present application;

[0021] Figure 4 This is a schematic diagram of a defect detection process provided by an embodiment of the present application;

[0022] Figure 5 is a structural diagram of a non-rigid registration device provided in an embodiment of the present application;

[0023] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0026] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0027] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0028] See Figure 1 , is a flow chart of a non-rigid registration method provided in an embodiment of the present application, such as Figure 1 As shown, the non-rigid registration method may include the following steps:

[0029] Step S100 : performing global matching on the non-rigid object to be registered based on the scene data and the model data of the object to be registered, and obtaining an initial pose transformation relationship between the scene data and the model data of the object to be registered.

[0030] Exemplarily, the above-mentioned scene data may include but is not limited to a scene grayscale image, a scene depth image or a scene point cloud.

[0031] Similarly, the above-mentioned model data may include but is not limited to a model grayscale image, a model depth image or a model point cloud.

[0032] For example, scenario data is usually collected from actual application scenarios and usually has certain noise and uncertainty; while model data is processed and prepared data, which is cleaner and more accurate and is usually made for specific tasks.

[0033] It should be noted that point cloud data can be directly collected or converted from depth maps or grayscale images.

[0034] For example, module matching technology may be used for overall matching, or PPF (Point Pair Feature) feature matching and point cloud ICP (Iterative Closest Point) technology may be used for global matching.

[0035] Step S110: for any valid module in the modules obtained by performing finite element division on the scene data of the object to be registered, determine the first type error and the second type error corresponding to the valid module; wherein the valid module is a module including the point cloud of the object to be registered, the first type loss is used to characterize the registration constraints between the points in the valid module and the corresponding points in the model data, the points in the valid module corresponding to the points in the model data are determined based on the posture transformation relationship corresponding to the valid module, the initial value of the posture transformation relationship corresponding to the valid module is the initial posture transformation relationship, and the second type loss is used to characterize the relative posture constraint between the valid module and the neighboring valid modules.

[0036] In the embodiment of the present application, considering that non-rigid objects may be deformed, the registration error by global matching will be relatively large, that is, the initial posture transformation relationship obtained in step S100 may have a large error.

[0037] In view of the characteristics of non-rigid objects, in order to improve the accuracy of non-rigid object registration, the idea of ​​finite partitioning can be used to divide the global non-rigid object into a finite number of local rigid objects, and the degree of freedom of non-rigid object matching can be increased by dividing the finite element module.

[0038] Among them, finite element partitioning refers to dividing an image or point cloud into finite modules (which can be called primitives) according to certain rules.

[0039] Accordingly, the scene data of the object to be registered may be divided into finite element groups to obtain a plurality of finite element modules (which may be referred to as modules for short).

[0040] For example, for two-dimensional data, such as a grayscale image or a depth image, the finite element division method for the scene data of the object to be registered may include but is not limited to rectangular division in the XY direction, triangular division or other division methods.

[0041] For three-dimensional data, such as 3D point clouds, the finite element division method for scene data of the object to be registered may include but is not limited to voxel grid division or other cubic graphic division methods.

[0042] For example, for multiple modules obtained by finite element division of the scene data of the object to be registered, it can be determined which points in the point cloud data of the object to be registered belong to which module. The module in which the point cloud of the object to be registered does not exist can be determined as an invalid module and does not require matching processing; the module in which the point cloud of the object to be registered exists can be called a valid module and requires matching processing.

[0043] In the embodiment of the present application, considering that there may be multiple other valid modules in the surrounding neighborhood of each valid module, when the object does not deform, the constraints between these valid modules are the constraints of the unit matrix, that is, there is no relative pose transformation. However, for non-rigid objects, there may be relative pose transformations between the valid modules and the valid modules in the neighborhood. Therefore, in the process of aligning non-rigid objects, it is necessary to consider not only the point cloud registration constraints, but also the relative pose constraints (also called deformation constraints) between different valid modules and the valid modules in the neighborhood.

[0044] For example, Figure 2A As shown, taking a two-dimensional image divided into rectangles as an example, the neighborhood modules of a module may include 8 modules (the number of neighborhood modules of modules at the edge / corner position is reduced accordingly); Figure 2B As shown, taking a three-dimensional image divided into voxel grids as an example, the neighborhood modules of a module may include 26 neighborhood modules (the number of neighborhood modules of modules at the surface / edge / corner position is reduced accordingly).

[0045] Accordingly, for any valid module, the first type error and the second type error corresponding to the valid module can be determined respectively.

[0046] The first type of error can be used to characterize the registration constraints between the points in the valid module and the corresponding points in the model image. The points in the model image data corresponding to the points in the valid module are determined based on the pose transformation relationship corresponding to the valid module, and the initial value of the pose transformation relationship corresponding to the valid module is the initial pose transformation relationship determined in step S100.

[0047] The second type of error can be used to characterize the relative pose constraint between the valid module and the neighboring valid modules.

[0048] For example, the first type of error can be determined by the following formula:

[0049]

[0050] Among them, P s is a point in the scene data, P m The model data and P s The corresponding point, n is the normal vector of the point, T i is the transformation relationship from the model point cloud of the i-th finite element module to the scene point cloud, Ω is the information matrix, and the target weight is adjusted.

[0051] The second type of error can be determined by the following formula:

[0052]

[0053] Where V represents the operation of Lie group logarithmic mapping. i is the transformation relationship from the model point cloud to the scene point cloud of the i-th finite element module, T j is the transformation relationship from the model point cloud of the j-th finite element module to the scene point cloud.

[0054] Step S120: Perform finite element division on the scene data of the object to be registered to obtain the first type error and the second type error of each valid module, optimize the posture transformation relationship corresponding to each valid module, and determine the registration result of the object to be registered.

[0055] In an embodiment of the present application, when the first type error and the second type error of each valid module are determined in the manner described in step S110, the first type error and the second type error of each valid module can be obtained based on finite element division of the scene data of the object to be registered, and the posture transformation relationship corresponding to each valid module is optimized to determine the registration result of the object to be registered.

[0056] It can be seen that in Figure 1In the method flow shown, the object to be registered is globally matched based on the scene data and model data of the non-rigid object to be registered, and the initial posture transformation relationship between the scene data and model data of the object to be registered is obtained; for any valid module in the modules obtained by finite element division of the scene data of the object to be registered, the first type error and the second type error corresponding to the valid module are determined; then, based on the first type error and the second type error of each valid module obtained by finite element division of the scene data of the object to be registered, the posture transformation relationship corresponding to each valid module is optimized to determine the registration result of the object to be registered, and the degree of freedom of non-rigid object registration is improved by finite element division of the scene data of the object to be registered; in addition, for the valid modules in the modules obtained by finite element division, on the one hand, matching is performed through point cloud registration, and on the other hand, deformation restriction is performed through the constraints of adjacent modules, thereby improving the accuracy of non-rigid object matching.

[0057] In some embodiments, the first type error and the second type error of each valid module obtained by performing finite element division on the scene data of the object to be registered, and optimizing the posture transformation relationship corresponding to each valid module may include:

[0058] Taking minimizing the fusion value of the first type error and the second type error of each valid module as the optimization goal, the posture transformation relationship corresponding to each valid module is optimized.

[0059] For example, in order to maximize the accuracy of non-rigid object registration, the posture transformation relationship corresponding to each valid module can be optimized based on the fusion value of the first type error and the second type error of each valid module, with minimizing the fusion value as the optimization goal.

[0060] Exemplarily, the fusion value of the first type error and the second type error may include the sum of the first type error and the second type error, or the weighted sum of the first type error and the second type error.

[0061] For any valid module, the initial value of the posture transformation relationship corresponding to the valid module is the initial posture transformation relationship determined in step S100.

[0062] Exemplarily, in the process of optimizing the posture transformation relationship corresponding to each valid module, it can be determined that the optimization end condition is met when the number of optimizations reaches a preset maximum number, the optimization step is less than a preset step threshold, or the fusion value of the above error is less than a preset error threshold (which can be called a second error threshold).

[0063] It should be noted that in the embodiment of the present application, the optimization goal is not limited to minimizing the fusion value of the above-mentioned errors. The optimization goal can also be to minimize the sum of the first type errors and the sum of the second type errors respectively. That is, when the number of optimizations reaches the preset maximum number, the optimization step is less than the preset step threshold, or the sum of the first type errors of each valid module is less than the preset error threshold (which can be called the third error threshold), and the sum of the second type errors of each valid module is less than the preset error threshold (which can be called the fourth error threshold), it can be determined that the optimization end condition has been reached.

[0064] In one example, the above-mentioned finite element division of the image data of the object to be registered to obtain the first type error and the second type error of each valid module, optimizing the posture transformation relationship corresponding to each valid module, and determining the registration result of the object to be registered may include:

[0065] When the optimization end conditions are met, if the fusion value of the first type error and the second type error of each valid module exceeds the preset error value, it is determined that the object to be registered and the model data correspond to different types of objects (that is, the model data is not the model data of the object to be configured, but belongs to the model data of other objects).

[0066] Exemplarily, when the optimization is completed, if the fusion value of the first type error and the second type error of each valid module exceeds a preset error threshold (which can be called the first error threshold), it can be determined that the object to be registered and the model data correspond to different types of objects, that is, it can be determined that the registration of the object to be registered has failed.

[0067] If the fusion value of the first type error and the second type error of each valid module does not exceed the first error threshold, it can be determined that the object to be registered and the model data correspond to the same type of object, that is, it can be determined that the object to be registered is successfully registered.

[0068] By the above method, the counting of the designated non-rigid objects can be realized. When it is determined that the registration of the object to be registered is successful, the count of the corresponding non-rigid object is accumulated, for example, added by 1.

[0069] In one example, the solution provided by the embodiment of the present application may further include:

[0070] When the optimization end condition is met and the fusion value of the first type error and the second type error of each valid module does not exceed the preset error threshold, the coordinate system of each valid module is transformed according to the final pose transformation relationship corresponding to each valid module to obtain the data of the object to be registered in the model coordinate system;

[0071] A difference comparison is performed based on the data of the object to be registered in the model coordinate system and the model data, and defect detection is performed on the object to be registered based on the difference comparison result.

[0072] Exemplarily, when the optimization end conditions are met and the fusion value of the first type error and the second type error of each valid module does not exceed the preset error threshold, that is, when the object to be registered is successfully registered, the coordinate system transformation of each valid module can be performed separately based on the final posture transformation relationship corresponding to each valid module (that is, the posture transformation relationship corresponding to each valid module when the optimization end conditions are met) to obtain the data of the object to be registered in the model coordinate system.

[0073] For example, a difference comparison may be performed based on the data of the object to be registered in the model coordinate system and the model data, and defect detection may be performed on the object to be registered based on the difference comparison result.

[0074] For example, the data of the object to be registered in the model coordinate system and the model data can be differentiated to obtain the difference result between the data of the object to be registered in the model coordinate system and the model data, and the area where the absolute value of the difference result exceeds a preset threshold is determined as a defect area.

[0075] In one example, before performing the difference comparison between the data of the object to be registered in the model coordinate system and the model data, the following steps may also be included:

[0076] When the intervals between valid modules in the data of the object to be registered in the model coordinate system are greater than a preset distance, the intervals between the valid modules of the object to be registered in the model coordinate system are filled according to the data in the valid modules of the object to be registered in the model coordinate system.

[0077] For example, considering that when the scene data of the object to be registered is divided into finite elements, if the module division is too small, the computing resources may not be able to meet the demand due to the excessive number of modules. Therefore, in actual scenarios, there may be a situation where small module division cannot be performed due to limited computing resources, resulting in large gaps between different modules in the data of the object to be registered in the model coordinate system obtained after coordinate conversion (hereinafter referred to as projection data).

[0078] Accordingly, for the above situation, when the projection data of the object to be registered is obtained in the manner described in the above embodiment, it can be determined whether there is a situation in which the interval between the projection areas of the valid modules in the projection data (that is, between the valid modules of the object to be registered in the model coordinate system) is greater than a preset distance (such as 1 pixel).

[0079] In the case where the interval between the projection areas of the valid modules is greater than a preset distance (the interval may be referred to as a crack), the interval between the projection areas of the valid modules is filled based on the point cloud data in the projection areas of the valid modules.

[0080] For example, the crack can be filled by linear interpolation based on the points in the projection area on both sides of the crack.

[0081] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0082] In this embodiment, the idea of ​​finite element division is used to divide the global non-rigid object into a finite number of locally rigid objects, and the degree of freedom of non-rigid workpiece matching is increased by dividing the finite element modules. The matching result of each module is obtained through the rigid matching of the modules and the deformation constraints between the modules, so that the scene image can be converted to the model coordinate system in sequence according to the matching results of the modules. In addition, the cracks caused by the limited deformation between the modules can be filled by interpolation to obtain a more complete matching and correction result. After the non-rigid image and point cloud are converted to the model, the difference between the model and the model image and point cloud can be obtained by differential, so as to perform related applications such as defect detection. The implementation process is as follows: Figure 4 shown.

[0083] Some details of the above process are described below.

[0084] 1. Global matching

[0085] Use conventional template matching technology; or, use PPF feature matching and point cloud ICP technology for global matching to obtain a globally consistent initial matching value for the entire grayscale image / depth image / point cloud (obtaining the above-mentioned initial pose transformation relationship), that is, the transformation from model point (point in model data) to scene point (i.e., the midpoint of the scene data of the object to be registered).

[0086] For example, in the case of template matching, matching is usually performed through point cloud registration technology. For example, grayscale image matching may be achieved through the registration of edge points (2D points). The depth map can be converted into a grayscale image using 2D point cloud registration, or it can be converted into a 3D point cloud for 3D point cloud registration to achieve matching.

[0087] 2. Finite element division

[0088] Perform finite element segmentation on grayscale images / depth maps / point clouds.

[0089] Exemplarily, the division method may be rectangular division in the XY direction, or triangular division or other forms of division, so as to divide the entire grayscale image / depth image / point cloud into N finite element modules.

[0090] For example, for any finite element module, it is possible to determine which points in the point cloud data of the object to be registered belong to which finite element module. If a finite element module does not contain the point cloud of the object to be registered, the finite element module is determined to be an invalid module; a finite element module that contains the point cloud of the object to be registered is a valid module.

[0091] Each valid module has a pose transformation relationship to be solved from the model point cloud to the scene point cloud. The initial value of the pose transformation relationship is the matching initial value determined by the above global matching.

[0092] For example, the grayscale image / depth image segmentation can be performed by dividing the image into rectangles according to a certain image width and height.

[0093] For example, the image is divided into a limited number of modules such as 8*8 / 10*10 / 25*25. Each module has 8 neighboring modules (also called neighboring modules), and its schematic diagram can be shown as follows: Figure 2A As shown, each module can be projected onto the model by matching the initial value.

[0094] Among them, such as Figure 2A As shown, for module 5, modules 1 to 4, and modules 6 to 8 are its neighborhood modules.

[0095] Figure 3 ] shows the result of image finite element segmentation when the scene data is deformed relative to the model data.

[0096] For example, when the shapes of the scene data and the model data are inconsistent, it can be considered that the scene data is deformed relative to the model data, for example, the scene data is warped.

[0097] The 3D point cloud voxel grid division process can be as follows Figure 2B As shown in the figure, the grid is defined according to a certain size, and each grid contains a number of points. Each voxel grid (such as the dotted grid in the figure) and the surrounding 26 grids constitute the constraints of the adjacent finite element modules, and the points in each finite element module (voxel grid) and the points in the model constitute ICP constraints.

[0098] 3. Constructing point cloud registration constraints

[0099] The grayscale image / depth image / point cloud in each finite element module has a corresponding matching area in the model point cloud. The pose transformation relationship of the module needs to satisfy the transformation of the model point cloud into the scene point cloud through this pose relationship. Constraints can be used using point-to-surface distance or point-to-point distance. This part of the constraint is the commonly used point cloud ICP method, which is to transform the scene points and model points into coordinate systems for registration.

[0100] For example, for any valid module, the error representing the point cloud registration constraint (i.e., the first type of error mentioned above) can be determined according to the following formula:

[0101]

[0102] 4. Constructing neighborhood constraints

[0103] Each finite element module has M (M≥3, M=8 for non-2D non-edge / corner modules, M=26 for non-3D surface / edge / corner modules) other finite element modules (i.e., neighborhood modules) in its surrounding neighborhood.

[0104] For example, when the depth map / point cloud is unchanged, the constraints between modules and their neighbors are all unit matrix constraints, meaning there is no relative pose transformation. However, for non-rigid objects, the pose relationships between modules differ relative to the model. Therefore, deformation constraints between modules are required to avoid excessive offset during ICP operations.

[0105] For example, for any valid module, the error representing the neighborhood constraint (i.e., the second type of error described above) can be determined according to the following formula:

[0106]

[0107] 5. Joint Optimization

[0108] The above two constraints are optimized to solve the posture transformation relationship of each module relative to the model.

[0109] For example, the pose transformation relationship is determined by optimizing the sum of errors through the following formula:

[0110]

[0111] Where m is the total number of valid modules.

[0112] 6. Data Correction

[0113] Using the pose transformation relationship of each valid module, the valid modules of the scene image are transformed. Each point is transformed according to the pose transformation relationship of the module to which it belongs, and projected to the new position to obtain a new grayscale image / depth image / point cloud. Once all points in all modules are transformed, a new complete grayscale image / depth image / point cloud (i.e., the above-mentioned projection data) is obtained.

[0114] 7. Crack filling

[0115] Considering that in actual scenarios, computing resources may be limited and small module division cannot be performed, when the divided modules are large, cracks will appear between modules (that is, the interval between adjacent modules exceeds the preset distance). These tiny cracks can be filled by linear interpolation of points of adjacent modules.

[0116] 8. Applications based on rectified grayscale images / depth images / point clouds

[0117] For the grayscale image / depth map / point cloud obtained through the above processing, by comparing the difference with the model (such as differential), the differential result between the model and the grayscale image / depth map / point cloud can be obtained. For example, after the depth map is corrected, it can be differentiated with the corresponding points of the model image. The area with large difference can be judged as a defect, thereby realizing defect detection.

[0118] The above describes the method provided by this application. The following describes the device provided by this application:

[0119] See Figure 5 , is a structural diagram of a non-rigid registration device provided in an embodiment of the present application, such as Figure 5 As shown, the non-rigid registration device may include:

[0120] A global matching unit 510 is configured to perform global matching on the non-rigid object to be registered based on the scene data and the model data of the object to be registered, and obtain an initial pose transformation relationship between the scene data of the object to be registered and the model data;

[0121] An error determination unit 520 is configured to determine, for any valid module among the modules obtained by performing finite element partitioning on the scene data of the object to be registered, a first type of error and a second type of error corresponding to the valid module; wherein the valid module is a module including a point cloud of the object to be registered, the first type of loss is used to characterize the registration constraints between points in the valid module and corresponding points in the model data, the points in the model data corresponding to the points in the valid module are determined based on the pose transformation relationship corresponding to the valid module, the initial value of the pose transformation relationship corresponding to the valid module is the initial pose transformation relationship, and the second type of loss is used to characterize the relative pose constraint between the valid module and neighboring valid modules;

[0122] The registration unit 530 is used to obtain the first type error and the second type error of each valid module based on the finite element division of the scene data of the object to be registered, optimize the posture transformation relationship corresponding to each valid module, and determine the registration result of the object to be registered.

[0123] In some embodiments, the registration unit 530 performs finite element division on the scene data of the object to be registered to obtain the first type error and the second type error of each valid module, and optimizes the posture transformation relationship corresponding to each valid module, including:

[0124] Taking minimizing the fusion value of the first type error and the second type error of each valid module as the optimization goal, the posture transformation relationship corresponding to each valid module is optimized.

[0125] In some embodiments, the registration unit 530 performs finite element division on the scene data of the object to be registered to obtain the first type error and the second type error of each valid module, optimizes the posture transformation relationship corresponding to each valid module, and determines the registration result of the object to be registered, including:

[0126] When the optimization end condition is met, if the fusion value of the first type error and the second type error of each valid module exceeds the preset error threshold, it is determined that the object to be registered and the model data correspond to different types of objects;

[0127] When the optimization end conditions are met and the fusion value of the first type error and the second type error of each valid module does not exceed the preset error threshold, the coordinate system of each valid module is transformed according to the final posture transformation relationship corresponding to each valid module to obtain the data of the object to be registered in the model coordinate system; a difference comparison is performed based on the data of the object to be registered in the model coordinate system and the model data, and defect detection is performed on the object to be registered based on the difference comparison result.

[0128] In some embodiments, the registration unit 530 is also used to fill the gaps between the valid modules of the object to be registered in the model coordinate system according to the data in the valid modules of the object to be registered in the model coordinate system when the gaps between the valid modules in the data of the object to be registered in the model coordinate system are greater than a preset distance.

[0129] An embodiment of the present application further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program; and the processor is used to implement the non-rigid registration method described above when executing the program stored in the memory.

[0130] See Figure 6, is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 may communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions corresponding to the non-rigid registration logic in the memory 602, the processor 601 may perform the non-rigid registration method described above.

[0131] The memory 602 mentioned herein can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as a hard disk drive), solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0132] In some embodiments, a machine-readable storage medium is also provided. Figure 6 The memory 602 in the machine-readable storage medium stores machine-executable instructions. When executed by the processor, the machine-executable instructions implement the non-rigid registration method described above. For example, the machine-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0133] An embodiment of the present application further provides a computer program product, which stores a computer program and, when a processor executes the computer program, prompts the processor to execute the non-rigid registration method described above.

[0134] It should be noted that, in this document, relational terms such as target and objective are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0135] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A non-rigid registration method, characterized in that: include: Performing global matching on the non-rigid object to be registered based on the scene data and the model data of the object to be registered to obtain an initial pose transformation relationship between the scene data of the object to be registered and the model data; For any valid module among the modules obtained by performing finite element division on the scene data of the object to be registered, determining the first type error and the second type error corresponding to the valid module; wherein the valid module is a module including the point cloud of the object to be registered, the first type loss is used to characterize the registration constraints between the points in the valid module and the corresponding points in the model data, the points in the model data corresponding to the points in the valid module are determined based on the posture transformation relationship corresponding to the valid module, the initial value of the posture transformation relationship corresponding to the valid module is the initial posture transformation relationship, and the second type loss is used to characterize the relative posture constraint between the valid module and the neighboring valid modules; Based on the finite element division of the scene data of the object to be registered, the first type error and the second type error of each valid module are obtained, the posture transformation relationship corresponding to each valid module is optimized, and the registration result of the object to be registered is determined.

2. The method according to claim 1, characterized in that The method of performing finite element division on the scene data of the object to be registered to obtain the first type error and the second type error of each valid module and optimizing the posture transformation relationship corresponding to each valid module includes: Taking minimizing the fusion value of the first type error and the second type error of each valid module as the optimization goal, the posture transformation relationship corresponding to each valid module is optimized.

3. The method according to claim 2, characterized in that The method of performing finite element division on the scene data of the object to be registered to obtain a first type error and a second type error of each valid module, optimizing the posture transformation relationship corresponding to each valid module, and determining the registration result of the object to be registered includes: When the optimization end condition is met, if the fusion value of the first type error and the second type error of each valid module exceeds the preset error threshold, it is determined that the object to be registered and the model data correspond to different types of objects.

4. The method according to claim 2, characterized in that The method further comprises: When the optimization end condition is met and the fusion value of the first type error and the second type error of each valid module does not exceed the preset error threshold, the coordinate system of each valid module is transformed according to the final posture transformation relationship corresponding to each valid module to obtain the data of the object to be registered in the model coordinate system; A difference comparison is performed based on the data of the object to be registered in the model coordinate system and the model data, and defect detection is performed on the object to be registered based on the difference comparison result.

5. The method according to claim 4, characterized in that Before performing a difference comparison based on the data of the object to be registered in the model coordinate system and the model data, the method further includes: When the intervals between valid modules in the data of the object to be registered in the model coordinate system are greater than a preset distance, the intervals between the valid modules of the object to be registered in the model coordinate system are filled according to the data in the valid modules of the object to be registered in the model coordinate system.

6. A non-rigid registration device, characterized in that: include: A global matching unit is used to perform global matching on the non-rigid object to be registered based on the scene data and model data of the object to be registered, so as to obtain an initial pose transformation relationship between the scene data of the object to be registered and the model data; an error determination unit, configured to determine, for any valid module among the modules obtained by performing finite element division on the scene data of the object to be registered, a first type of error and a second type of error corresponding to the valid module; wherein the valid module is a module including a point cloud of the object to be registered, the first type of loss is used to characterize the registration constraints between points in the valid module and corresponding points in the model data, the points in the model data corresponding to the points in the valid module are determined based on a pose transformation relationship corresponding to the valid module, the initial value of the pose transformation relationship corresponding to the valid module is the initial pose transformation relationship, and the second type of loss is used to characterize the relative pose constraint between the valid module and a neighboring valid module; The registration unit is used to obtain the first type error and the second type error of each valid module based on the finite element division of the scene data of the object to be registered, optimize the posture transformation relationship corresponding to each valid module, and determine the registration result of the object to be registered.

7. The device according to claim 6, characterized in that The registration unit performs finite element division on the scene data of the object to be registered to obtain the first type error and the second type error of each valid module, and optimizes the posture transformation relationship corresponding to each valid module, including: Taking minimizing the fusion value of the first type error and the second type error of each valid module as the optimization goal, the posture transformation relationship corresponding to each valid module is optimized.

8. The device according to claim 7, characterized in that The registration unit obtains a first type error and a second type error of each valid module based on finite element division of the scene data of the object to be registered, optimizes the posture transformation relationship corresponding to each valid module, and determines the registration result of the object to be registered, including: When the optimization end condition is met, if the fusion value of the first type error and the second type error of each valid module exceeds the preset error threshold, it is determined that the object to be registered and the model data correspond to different types of objects; When the optimization end conditions are met and the fusion value of the first type error and the second type error of each valid module does not exceed the preset error threshold, the coordinate system of each valid module is transformed according to the final posture transformation relationship corresponding to each valid module to obtain the data of the object to be registered in the model coordinate system; a difference comparison is performed based on the data of the object to be registered in the model coordinate system and the model data, and defect detection is performed on the object to be registered based on the difference comparison result.

9. The device according to claim 8, characterized in that The registration unit is further configured to fill the gaps between the valid modules of the object to be registered in the model coordinate system according to the data in the valid modules of the object to be registered in the model coordinate system when the gaps between the valid modules in the data of the object to be registered in the model coordinate system are greater than a preset distance.

10. An electronic device, characterized in that: comprising a processor and a memory, wherein Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 5 when executing a program stored in a memory.

Images