Three-dimensional registration method and device, computer storage medium and terminal

Abstract

The invention discloses a three-dimensional registration method and device, a computer storage medium and a terminal. According to the embodiment of the invention, for a scene comprising a plurality of objects to be processed and images to be processed collected from more than two viewing angles of AR, residual errors still existing in the pose of a registered target object after rotation pose deviation correction are corrected, and the accuracy of the registration is improved. Performing contour projection on the rotation attitude correction pose on the to-be-processed image, searching to obtain candidate edge points, determining a residual error and a Jacobian matrix of the candidate edge points according to an energy function, expanding contour search to a cross-view angle and a constraint object, and feeding back an observation error on the constraint object to correct the rotation attitude correction pose. The amplification of the rotation error is inhibited; by introducing a normalization equation, solving of pose correction and consistency correction of pose residual errors of all objects to be processed are achieved; and performing pose correction on the registered target object according to the pose correction amount reaching the iteration termination condition, and realizing three-dimensional registration of the scene based on the registered target object without the pose residual error.

Description

Technical Field

[0001] This article relates to augmented reality technology, and more particularly to a method, apparatus, computer storage medium, and terminal for three-dimensional registration. Background Technology

[0002] Augmented reality (AR) technology has significant application value in complex assembly scenarios such as aerospace and automobile manufacturing by overlaying virtual information onto the real world. Compared with traditional paper process documents, AR headsets can intuitively display assembly information to workers, thereby reducing cognitive burden and improving operational efficiency.

[0003] In AR assembly systems, precise spatial alignment between virtual objects and physical entities is crucial to ensuring system effectiveness; this process is called 3D registration (AR head-mounted display registration). In large-scale scenarios such as aircraft or spacecraft modules, local registration of a single component's virtual-to-physical model is often insufficient. Typically, it is necessary to precisely match the entire virtual scene with the physical space; this is known as scene-level registration. Unlike local registration, scene-level registration requires maintaining the registration relationship of multiple objects (target objects) simultaneously over a larger spatial range, thus facing higher challenges in terms of accuracy and robustness.

[0004] 3D registration typically relies on monocular RGB images and is mainly divided into two categories: marker-based and markerless. Marker-based registration involves placing artificial markers such as QR codes in the environment, identifying and matching feature points, and solving for the pose to achieve registration. To apply this to scene-level registration, some researchers have proposed strategies using multiple markers to control the magnitude of error amplification. Given the limitations of marker-based registration methods, markerless registration, due to its flexibility and the fact that it does not require additional marker placement, has greater application potential in complex industrial environments. Its basic principle is to directly utilize the natural or geometric features of objects in the scene to perform 6D pose estimation for the camera. Current mainstream pose estimation methods can be mainly divided into two categories: one is direct pose regression, which uses deep neural networks to learn and output the six-degree-of-freedom pose of objects end-to-end; the other is a 2D-3D point-pair matching method, which first establishes a correspondence between the image and the 3D model, and then obtains the final pose through calculation. In practice, adding a pose optimization module after the above methods can often further improve the accuracy of pose estimation.

[0005] Marker-based registration faces three major challenges in actual industrial settings: First, deviations in marker placement from the designed location introduce new errors; second, some precision equipment or cabins prohibit the introduction of foreign objects; and third, in some confined work environments, operators' field of vision is limited, making it difficult to observe markers. These limitations restrict the applicability of this method. While markerless registration, with the development of computer vision technology, has achieved high registration accuracy in local registration of single objects, a fundamental problem arises when these methods are applied to scene-level registration: scene-level registration typically requires selecting an object as a global reference. If the pose is directly estimated for this reference object, the small residual rotational error in the result will be significantly amplified with increasing distance from the reference. For example, if the rotational attitude estimation result of reference object A has a 0.5° error, the position of object B, 5 meters away with its coordinate system as a reference, may have a spatial error exceeding 40 millimeters compared to its actual position. This spatial amplification effect of rotational attitude error leads to a significant discrepancy between distant objects and their true positions, directly affecting workers' assembly judgments and potentially causing assembly errors.

[0006] In summary, when performing scene-level registration in the field of augmented reality-assisted assembly, related technologies face a dilemma: marker-based registration methods are limited and cannot be used in some cases, while existing markerless registration, although performing well in local accuracy, fails to solve the core problem of attitude error amplification with space, resulting in the inability to guarantee the spatial consistency of registration accuracy and making it difficult to meet the needs of high-precision industrial assembly. Summary of the Invention

[0007] This application provides a method for 3D registration, applied to a scene containing three or more objects to be processed, the objects to be processed including registration target objects and constraint objects, including: The rotational pose correction pose of the registered target object and the constraint object is projected onto the image to be processed, and multiple candidate edge points of the registered target object and the constraint object are obtained by contour search. The rotational pose correction pose is the pose after the rotational pose deviation is corrected. The image to be processed includes images containing one or more objects to be processed, which are collected from two or more views of augmented reality (AR). The residual function corresponding to each candidate edge point is determined according to the predetermined energy function, where the energy function is the sum of the weighted log-likelihood loss generated by all candidate edge points, and the residual function is the error term generated by the candidate edge points; Based on each determined residual function, the corresponding Jacobian matrix with respect to the pose correction is determined, wherein the Jacobian matrix contains the pre-determined relative pose information of the constrained object relative to the registered target object. Based on the residual function, Jacobian matrix and pre-determined weight of each residual corresponding to each candidate edge point, a normalization equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. Solve the constructed normalized equation to obtain the pose correction amount; If the pose correction amount of the registered target object is determined to be less than the preset iteration termination condition, the rotational attitude correction pose of the registered target object and the constraint object is updated according to the pose correction amount, and the pose correction amount is obtained again based on the updated rotational attitude correction pose; if the iteration termination condition is determined to be met, the pose of the registered target object is corrected according to the pose correction amount of the registered target object. Based on the registered target object whose pose correction has been completed, perform 3D registration of the scene.

[0008] On the other hand, embodiments of this application also provide a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described three-dimensional registration method.

[0009] Furthermore, embodiments of this application also provide a terminal, including: a memory and a processor, wherein the memory stores a computer program; wherein, The processor is configured to execute computer programs in memory; When the computer program is executed by the processor, it implements the three-dimensional registration method described above.

[0010] Furthermore, embodiments of this application also provide a 3D registration apparatus applied to a scene containing three or more objects to be processed, the objects to be processed including registration target objects and constraint objects, comprising: an edge search unit, a residual determination unit, a Jacobian matrix determination unit, an equation construction unit, a solution unit, a judgment and correction unit, and a registration unit; wherein, The edge search unit is set to project the rotational attitude correction pose of the registered target object and the constraint object onto the image to be processed, and obtain multiple candidate edge points of the registered target object and the constraint object through contour search. The rotational attitude correction pose is the pose after the rotational attitude deviation is corrected. The image to be processed includes images containing one or more objects to be processed, which are collected from two or more views of augmented reality (AR). The residual unit is determined as follows: the residual function corresponding to each candidate edge point is determined according to the predetermined energy function, where the energy function is the sum of the weighted log-likelihood loss generated by all candidate edge points, and the residual function is the error term generated by the candidate edge points; The Jacobian matrix unit is set as follows: for each determined residual function, the corresponding Jacobian matrix with respect to the pose correction is determined respectively, wherein the Jacobian matrix contains the pre-determined relative pose information of the constrained object relative to the registered target object; The equation building unit is set as follows: based on the residual function, Jacobian matrix and pre-determined weight of each residual corresponding to each candidate edge point, a normalized equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. The solution element is set to solve the constructed normalized equations to obtain the pose correction. The determination and correction unit is set as follows: when the pose correction amount of the registered target object is determined to be less than the preset iteration termination condition, the rotational attitude correction pose of the registered target object and the constraint object is updated according to the pose correction amount, and the pose correction amount is obtained again based on the updated rotational attitude correction pose; when the iteration termination condition is determined to be met, the pose of the registered target object is corrected according to the pose correction amount of the registered target object. The registration unit is set to perform 3D registration of the scene based on the target object whose pose correction has been completed.

[0011] This embodiment addresses a scene containing three or more objects to be processed, where the images are acquired from two or more viewpoints in augmented reality (AR). For residual errors remaining in the rotational pose correction pose of the registered target object after rotational pose deviation correction, the rotational pose correction pose is projected onto the image to be processed using a contour search. Candidate edge points are obtained through contour search, and a residual function is determined for each candidate edge point based on a pre-determined energy function. A Jacobian matrix for pose correction is then determined based on the determined residual function. The Jacobian matrix contains pre-determined relative pose information of constrained objects relative to the registered target object. This extends the contour search across viewpoints and constrained objects, directly feeding back observation errors on any constrained object to correct the pose of the registered target object. By utilizing the geometric constraints of objects in the scene, the amplification effect of the rotation error of the registered target object in space is suppressed. Based on the residual function, Jacobian matrix, and pre-determined weight of each residual for each candidate edge point, a normalization equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. Through the introduced normalization equation, the pose correction amount in the iteration process is solved, achieving consistent correction of the pose residual error of all objects to be processed. The obtained pose correction amount determines whether the pre-set iteration termination condition has been met. When the obtained pose correction amount determines that the predetermined iteration termination condition has been met, the pose of the registered target object is corrected according to the pose correction amount. Based on the elimination of pose residual error of all objects to be processed, the 3D registration of the scene is achieved.

[0012] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0014] Figure 1 This is a flowchart of a three-dimensional registration method according to an embodiment of the present disclosure; Figure 2 This is a structural block diagram of the three-dimensional registration device according to an embodiment of the present disclosure; Figure 3 This is a schematic diagram of the system distribution for three-dimensional registration according to an embodiment of this disclosure; Figure 4 A schematic diagram for pre-calibration of rotational attitude; Figure 5 This is a schematic diagram of the process for correcting the pose after rotational attitude deviation correction in an embodiment of this disclosure. Detailed Implementation

[0015] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0016] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application can also be combined with any conventional features or elements to form unique inventive solutions. Any feature or element of any embodiment can also be combined with features or elements from other inventive solutions to form another unique inventive solution. Therefore, it should be understood that any feature shown and / or discussed in this application can be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes can be made within the scope of the appended claims.

[0017] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0018] Figure 1 This is a flowchart of a three-dimensional registration method according to an embodiment of the present disclosure, applied to a scene containing three or more objects to be processed. The objects to be processed include a registration target object and constraint objects, such as... Figure 1 As shown, the methods for 3D registration include: Step 101: Project the rotational pose correction pose of the registered target object and the constraint object onto the image to be processed, and obtain multiple candidate edge points of the registered target object and the constraint object through contour search. The rotational pose correction pose is the pose after the rotational pose deviation is corrected. The image to be processed includes images containing one or more objects to be processed, which are collected from two or more perspectives of augmented reality (AR). Step 102: Determine the residual function corresponding to each candidate edge point according to the predetermined energy function, where the energy function is the sum of the weighted log-likelihood losses generated by all candidate edge points, and the residual function is the error term generated by the candidate edge points; Step 103: Based on each determined residual function, determine the corresponding Jacobian matrix for the pose correction amount, wherein the Jacobian matrix contains the pre-determined relative pose information of the constrained object relative to the registered target object. Step 104: Based on the residual function, Jacobian matrix and pre-determined weight of each residual corresponding to each candidate edge point, construct a normalization equation based on the negative values ​​of the Hessian approximation matrix and gradient vector. Step 105: Solve the constructed normalized equation to obtain the pose correction amount; Step 106: If the pose correction amount of the registered target object is determined to be less than the preset iteration termination condition, update the rotational attitude correction pose of the registered target object and the constraint object according to the pose correction amount, and repeat steps 101-105 to obtain the pose correction amount based on the updated rotational attitude correction pose; if the iteration termination condition is determined to be met, perform pose correction on the registered target object according to the pose correction amount of the registered target object. Step 107: Perform 3D registration of the scene based on the registered target object whose pose correction has been completed.

[0019] This embodiment addresses a scene containing three or more objects to be processed, where the images are acquired from two or more viewpoints in augmented reality (AR). For residual errors remaining in the rotational pose correction pose of the registered target object after rotational pose deviation correction, the rotational pose correction pose is projected onto the image to be processed using a contour search. Candidate edge points are obtained through contour search, and a residual function is determined for each candidate edge point based on a pre-determined energy function. A Jacobian matrix for pose correction is then determined based on the determined residual function. The Jacobian matrix contains pre-determined relative pose information of constrained objects relative to the registered target object. This extends the contour search across viewpoints and constrained objects, directly feeding back observation errors on any constrained object to correct the pose of the registered target object. By utilizing the geometric constraints of objects in the scene, the amplification effect of the rotation error of the registered target object in space is suppressed. Based on the residual function, Jacobian matrix, and pre-determined weight of each residual for each candidate edge point, a normalization equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. Through the introduced normalization equation, the pose correction amount in the iteration process is solved, achieving consistent correction of the pose residual error of all objects to be processed. The obtained pose correction amount determines whether the pre-set iteration termination condition has been met. When the obtained pose correction amount determines that the predetermined iteration termination condition has been met, the pose of the registered target object is corrected according to the pose correction amount. Based on the elimination of pose residual error of all objects to be processed, the 3D registration of the scene is achieved.

[0020] In step 101 of this embodiment, the rotational attitude correction pose of the registered target object is its final pose after attitude correction; the rotational attitude correction pose of the constrained object is obtained by multiplying the pose by its pose transformation relative to the registered target object.

[0021] The residual errors described in the embodiments of this disclosure include: residual rotational error and residual positional error.

[0022] In steps 102-105 of this embodiment, the residual function, Jacobian matrix, normalization equation, and pose correction amount are determined for the candidate edge points of the registered target object and the constraint object.

[0023] The essence of the above processing in this embodiment is to align the virtual and real coordinate systems. When you want to know the pose or other information of an object in the real scene, you only need to read it from the virtual scene.

[0024] The scenario of this disclosure is that not all objects to be processed can be seen from one AR viewpoint. In other words, two or more viewpoints are required to see more than three objects to be processed. The captured images to be processed are RGB images, each RGB image contains more than one object to be processed, and at least one RGB image contains two objects to be processed.

[0025] In one exemplary instance, the energy function of this disclosure embodiment is the sum of weighted log-likelihood losses generated by all candidate edge points, which can be determined by an expert in the field.

[0026] In one exemplary instance, the expression for the energy function of this disclosure embodiment is: ; In the formula, The rotational pose correction pose (camera pose parameters to be optimized) is represented by a six-dimensional vector in the form of Lie algebra se(3); i represents all contour points (anchor points mi) projected onto the image to be processed by the 3D model; j represents the candidate edge points found in the normal search direction of each contour point (anchor point mi). ;w Indicates candidate edge points The weights; It is a smooth step function used to smoothly transition the foreground and background probabilities inside and outside the projected contour; Indicates candidate edge points The symbolic distance to the projected contour; Represents candidate edge points calculated based on a pre-determined appearance model. The posterior probability of belonging to the foreground; Represents candidate edge points calculated based on the appearance model. The posterior probabilities belonging to the foreground and background.

[0027] In one exemplary instance, the process of minimizing the energy function in this disclosure is equivalent to maximizing the posterior probability of the model projection matching the true edge.

[0028] In one exemplary instance, the residual function of this disclosure is the error term generated by the candidate edge points, which can be determined based on the log-likelihood term in the energy function; the residual of this disclosure is... The expression is: .

[0029] In this embodiment of the disclosure, the smaller the residual value, the more likely it is that the candidate edge point is... The more it conforms to the appearance model , and geometric model Features of actual edge points in the image to be processed, defined together.

[0030] In one exemplary instance, according to an embodiment of this disclosure, the Jacobian matrix is ​​the partial derivative of the residual function with respect to the pose correction.

[0031] To differentiate the reliability of different candidate edge points, embodiments of this disclosure assign a corresponding weight w to each residual term. .

[0032] In one exemplary instance, the weights of this disclosure embodiment are set based on the geometric and / or appearance attributes of the candidate edge points.

[0033] In one exemplary instance, when the weights of the present disclosure embodiments are set based on the geometric and appearance attributes of the candidate edge points, the expression is as follows: ; In the formula, and P( is a pre-defined hyperparameter used to balance the influence of geometric and appearance properties; |C) indicates the anchor point The probability corresponding to the actual edge point reflects the appearance confidence of the candidate edge point; Indicates candidate edge points Its anchor point The normalized distance between them reflects the geometric confidence (reprojection error) of the candidate edge point.

[0034] In one exemplary instance, embodiments of this disclosure can sample local pixel color information on both the inner and outer sides of the projected contour of the registered target object based on the segmentation mask obtained by the segmentation-guided contour search method, and construct multiple local foreground color histograms and local background color histograms respectively. Referring to related technologies, the constructed local foreground color histograms and local background color histograms constitute an anti-occlusion appearance model.

[0035] In one exemplary instance, the above iteration of this disclosure embodiment can be achieved by minimizing the weighted reprojection error energy function. To achieve this, pose updates can be implemented using Lie algebras. Perform parameterization.

[0036] In one exemplary instance, embodiments of this disclosure project the rotational pose correction pose of the target object onto the image to be processed, and obtain candidate edge points through contour search, including: The rotational orientation correction pose of the target object is projected onto the image to be processed and the outline is rendered. Construct a set of search lines for the target object Each search line is referenced by a projection point of the contour projection and extends along its normal vector. Two-way extension; On the image to be processed after contour rendering, candidate edge points are collected based on the constructed search lines to obtain a set of candidate edge points.

[0037] In one exemplary instance, embodiments of this disclosure perform candidate edge point search on each image to be processed. (at the pyramid level) Using segmentation masks, a corresponding appearance model composed of histograms is constructed to define the target object. A set of search lines for the object Each search line is formed by the projection point of a contour (coordinates are...). Using as a reference, and extending it along its normal vector Two-way extension.

[0038] In one exemplary instance, embodiments of this disclosure construct a normalized equation based on the negative values ​​of the Hessian approximation matrix and gradient vectors, including: The residual of each candidate edge point, the weight of each residual, and the Jacobian matrix are accumulated to the negative value of the pre-established Hessian approximation matrix and gradient vector. A normalization equation is constructed based on the accumulated Hessian approximation matrix and the negative values ​​of the gradient vector; The expression for the normalization equation is: In the formula, H represents the accumulated Hessian approximation matrix, and b represents the negative value of the accumulated gradient vector. Represents the pose correction amount.

[0039] In one exemplary instance, the candidate edge point set of this disclosure embodiment is: For each candidate edge point The calculated residual is and weights ; The summation of the Hessian approximation matrix H is: ; The accumulation of the negative values ​​b of the gradient vector is: .

[0040] In one exemplary instance, embodiments of this disclosure may collect candidate edge points based on appearance models and probability maps.

[0041] In one exemplary instance, the image to be processed in this disclosure includes: a pre-stored original image containing a registered target object.

[0042] In one exemplary instance, the image to be processed according to embodiments of this disclosure includes an image obtained by downsampling the original image.

[0043] This embodiment of the disclosure refers to related technologies to perform downsampling processing to obtain the corresponding image to be processed for each layer; for the image to be processed obtained by downsampling processing, the image pyramid method is used to process the pose correction amount in steps 101-105.

[0044] This embodiment of the disclosure iterates through an image pyramid on the downsampled image, from coarse to fine layer by layer, to obtain the pose correction amount for each layer, until the pose update amount is lower than a threshold or the maximum number of iterations is reached. If it is at the finest layer and the increment is small enough, it stops early, improving robustness and convergence speed. Taking a three-layer pyramid as an example, the stopping condition may include: if and If the two loops are broken (convergence), the observation error on any constrained object is directly fed back to correct the pose. This allows the geometric constraints of multiple objects in the scene to suppress the amplification effect of rotation error in space, ultimately resulting in a globally consistent and high-precision registration result.

[0045] In one exemplary instance, the pose correction amount of this disclosure embodiment can be obtained by solving a weighted least squares problem using the Gauss-Newton method.

[0046] In one exemplary instance, embodiments of this disclosure correct the pose of a target object based on a pose correction amount, including: Based on the pose correction, the pose of the target object is updated using the Lie algebra exponential mapping, expressed as follows: During initialization, The pose of the target object in the camera coordinate system is calculated based on a pre-built model.

[0047] In one exemplary instance, the optimization variable of this disclosure embodiment is the six-degree-of-freedom pose of the target object.

[0048] In one exemplary instance, before step 101 projects the rotational pose correction poses of the registered target object and constraint object onto the image to be processed and obtains multiple candidate edge points through contour search, the method of this disclosure embodiment further includes a pre-calibration process to obtain the rotational pose correction pose through the following processing: Based on the pose optimization results of a single object, the relative position of each constrained object with respect to the registered target object is obtained and denoted as the observation vector. The corresponding pre-stored design true value is read from the pre-created database and denoted as the reference vector. By using rotation transformation, the rotational attitude of the registered target object is iteratively corrected, so that the error angle between the observation vector and the reference vector gradually decreases until it is lower than the preset threshold or the maximum number of iterations is reached, thus obtaining the rotational attitude correction pose after the rotational attitude deviation correction is completed.

[0049] The embodiments disclosed herein are pre-calibrated, effectively eliminating large rotational deviations of the reference object.

[0050] In one exemplary instance, when the maximum number of iterations is reached and the pose correction amount is determined to be less than the preset iteration termination condition, feedback can be provided to re-obtain the image to be processed so as to re-obtain the pose correction amount through steps 101-105.

[0051] In one exemplary instance, before performing pre-calibration processing to obtain the rotational attitude correction pose, the method of this disclosure embodiment further includes pose optimization processing for each object to be processed by the following steps: Contour projection is performed on the original image according to the initial pose of each object to be processed. Based on segmentation masks, a contour search is performed on the contour projection using segmentation guidance, and the initial pose of the object is optimized based on the results of the contour search.

[0052] In one exemplary instance, embodiments of this disclosure utilize a segmentation mask to construct a foreground / background color histogram, then establish search lines around the projected contour, and use the statistical information of the histogram on each line to search for edge candidate points, with the goal of minimizing the reprojection error of each edge candidate point and the projected contour point.

[0053] In one exemplary instance, before processing the pose of a single object to be processed, this embodiment of the disclosure refers to related technologies to obtain an initial pose, which may include: randomly selecting a shooting object as a reference object from a monocular RGB image of an AR headset; processing the received monocular RGB image using a pre-trained deep learning model; and outputting a segmentation mask and a coarse initial pose for each object. This embodiment of the disclosure can construct a virtual-real registration system in a client-server collaborative working mode. When the system starts, the operator wears an AR headset and acquires images containing multiple objects to be processed from multiple different perspectives in the experimental scene. The AR headset simultaneously records the camera pose provided by its Simultaneous Localization and Mapping (SLAM) system. The images, object numbers, and camera poses are transmitted to the server via a wireless network. The server, relying on a pre-trained deep learning model, randomly selects a shooting object as a reference object, processes the received data, and outputs a segmentation mask and a coarse initial pose for each object.

[0054] This disclosure addresses the error amplification problem in large-scale AR scene 3D registration in related technologies by proposing a markerless 3D registration method and system based on coarse-to-fine rotational attitude correction pose processing of multiple objects. It abandons the single-step global solution approach of related technologies and designs a hierarchical approach from "single-object local pose optimization" to "multi-object rotational relationship pre-calibration" and then to "global multi-object joint rotational attitude correction pose correction." Through step-by-step optimization, local accuracy is ensured first, then major errors are calibrated, and finally global consistency is achieved, effectively improving the convergence and final accuracy of pose solving in complex scenes.

[0055] This disclosure also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described three-dimensional registration method.

[0056] This disclosure also provides a terminal, including: a memory and a processor, wherein the memory stores a computer program; wherein, The processor is configured to execute computer programs in memory; The computer program implements the three-dimensional registration method described above when executed by a processor.

[0057] Figure 2 This is a structural block diagram of a three-dimensional registration device according to an embodiment of the present disclosure. It is applied to a scene containing three or more objects to be processed, including a registration target object and constraint objects, such as... Figure 2 As shown, it includes: an edge search unit, a residual determination unit, a Jacobian matrix determination unit, an equation construction unit, a solution unit, a judgment and correction unit, and a registration unit; wherein, The edge search unit is set to project the rotational attitude correction pose of the registered target object and the constraint object onto the image to be processed, and obtain multiple candidate edge points of the registered target object and the constraint object through contour search. The rotational attitude correction pose is the pose after the rotational attitude deviation is corrected. The image to be processed includes images containing one or more objects to be processed, which are collected from two or more views of augmented reality (AR). The residual unit is determined as follows: the residual function corresponding to each candidate edge point is determined according to the predetermined energy function, where the energy function is the sum of the weighted log-likelihood loss generated by all candidate edge points, and the residual function is the error term generated by the candidate edge points; The Jacobian matrix unit is set as follows: for each determined residual function, the corresponding Jacobian matrix with respect to the pose correction is determined respectively, wherein the Jacobian matrix contains the pre-determined relative pose information of the constrained object relative to the registered target object; The equation building unit is set as follows: based on the residual function, Jacobian matrix and pre-determined weight of each residual corresponding to each candidate edge point, a normalized equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. The solution element is set to solve the constructed normalized equations to obtain the pose correction. The determination and correction unit is set as follows: when the pose correction amount of the registered target object is determined to be less than the preset iteration termination condition, the rotational attitude correction pose of the registered target object and the constraint object is updated according to the pose correction amount, and the pose correction amount is obtained again based on the updated rotational attitude correction pose; when the iteration termination condition is determined to be met, the pose of the registered target object is corrected according to the pose correction amount of the registered target object. The registration unit is set to perform 3D registration of the scene based on the target object whose pose correction has been completed.

[0058] In one exemplary instance, the expression for the energy function in this disclosure embodiment is: ; In the formula, The parameters represent the pose to be corrected for rotational orientation, where i represents all anchor points mi projected onto the image to be processed from the 3D model; j represents the candidate edge points found along the normal search direction of each anchor point mi. ;w Indicates candidate edge points The weights; It is a smooth step function used to smoothly transition the foreground and background probabilities inside and outside the projected contour; Indicates candidate edge points The symbolic distance to the projected contour; Represents candidate edge points calculated based on a pre-determined appearance model. The posterior probability of belonging to the foreground; Represents candidate edge points calculated based on the appearance model. The posterior probabilities belonging to the foreground and background.

[0059] In one exemplary instance, the residual in this disclosure embodiment is the error term generated by the candidate edge points, and the residual function is: The expression is: .

[0060] In one exemplary instance, the Jacobian matrix of this disclosure is the partial derivative of the residual function with respect to the pose correction.

[0061] In one exemplary instance, the weights of this disclosure embodiment are set based on the geometric and / or appearance attributes of the candidate edge points.

[0062] In one exemplary instance, when the weights of the present disclosure embodiments are set based on the geometric and appearance attributes of the candidate edge points, the expression is as follows: ; In the formula, and P( is a pre-defined hyperparameter used to balance the influence of geometric and appearance properties; |C) indicates the anchor point The probability corresponding to the actual edge point reflects the appearance confidence of the candidate edge point; Indicates candidate edge points Its anchor point The normalized distance between them reflects the geometric confidence (reprojection error) of the candidate edge point.

[0063] In one exemplary instance, the equation-building unit of this disclosure is configured as follows: The residual of each candidate edge point, the weight of each residual, and the Jacobian matrix are accumulated to the negative value of the pre-established Hessian approximation matrix and gradient vector. A normalization equation is constructed based on the accumulated Hessian approximation matrix and the negative values ​​of the gradient vector; The expression for the normalization equation is: In the formula, H represents the accumulated Hessian approximation matrix, and b represents the negative value of the accumulated gradient vector. Represents the pose correction amount.

[0064] In one exemplary instance, the image to be processed in this disclosure includes an image obtained by downsampling an original image containing a registered target object.

[0065] In one exemplary instance, the apparatus of this disclosure embodiment further includes a pre-calibration unit, configured as follows: Before projecting the rotational attitude correction poses of the registered target object and constraint object onto the image to be processed and obtaining multiple candidate edge points through contour search, the method of this disclosure embodiment further includes a pre-calibration process to obtain the rotational attitude correction pose: Based on the pose correction results of each object to be processed, the relative position of each constrained object with respect to the registered target object is obtained and denoted as the observation vector. The design true value of the corresponding object to be processed, which is stored in the pre-created database, is read and denoted as the reference vector. By using rotation transformation, the rotational attitude of the registered target object is iteratively corrected, so that the error angle between the observation vector and the reference vector gradually decreases until it is below the preset threshold or reaches the maximum number of iterations, thus obtaining the rotational attitude correction pose.

[0066] In one exemplary instance, the apparatus of this disclosure embodiment further includes a preprocessing unit, configured to... The pose of each object to be processed is corrected through the following steps: Contour projection is performed on the original image according to the initial pose of each object to be processed. Based on a pre-determined segmentation mask, a contour search is performed on the contour projection using segmentation guidance. The initial pose of the object to be processed is corrected based on the results of contour search.

[0067] The following application examples briefly illustrate the embodiments of this disclosure. These application examples are only used to illustrate the embodiments of this disclosure and are not intended to limit the scope of protection of the embodiments of this disclosure.

[0068] Application Examples Figure 3 This is a schematic diagram of the system distribution for three-dimensional registration according to an embodiment of this disclosure, such as... Figure 3 As shown, it includes a client AR headset 1, a server 2, the camera coordinate system of the AR headset 3, the coordinate system of the registered target object model 4, the virtual scene world coordinate system 5, the registered target object 6, and the constraint object 7.

[0069] This application example constructs a rocket section simulation experimental environment, which is a cylindrical structure with an inner diameter of 2.5 meters. The chamber is equipped with various devices, cables, and pipes to reproduce a real aerospace assembly scenario. Nine different physical components are selected from the chamber as target objects for registration. A database is established on the server side, containing the 3D CAD models of each object and the precise relative pose relationships between them. Pre-training is performed on the server side, and the trained deep learning model is stored on the server for subsequent instance segmentation and coarse initial pose estimation. An AR head-mounted display (HoloLens2) and a PC terminal are prepared as the server. The entire system platform can be developed based on Unity3D to achieve collaboration between the client and the server. The program of the 3D registration method described in this embodiment is deployed on the server side, and the client and server can communicate via WIFI.

[0070] The three-dimensional registration in this disclosure includes: Step 1: Data Acquisition and Initial Pose Estimation: The operator, wearing a HoloLens 2, takes photos of the scene from multiple different advantageous angles in front of the rocket module experimental platform. To ensure registration quality, the operator will select an angle containing at least three or more objects to be processed for each photo. HoloLens 2 automatically records the camera's pose relative to the world coordinate system at that moment, provided by its internal SLAM system. The data packet, containing the image, the object identifier of the object to be processed, and the corresponding camera pose, is packaged and sent to the server via a wireless network. For each visible object to be processed in each received image, the server uses a pre-trained deep learning network to obtain its segmentation mask and a coarse initial pose (initial 6D pose). ).

[0071] Step 2: High-precision pose optimization for a single object to be processed: For the coarse initial pose output by the deep learning network, this embodiment of the present disclosure uses a segmentation-guided contour search method to perform high-precision pose optimization for each object to be processed, so that its positional accuracy can meet the requirements of subsequent joint correction of rotational pose by multiple objects. The processing includes: (1) Constructing an appearance model: Using the segmentation mask obtained by the above processing, sample local pixel color information on both the inner and outer sides of the object's projected outline, and construct multiple local foreground color histograms and local background color histograms respectively; the set of these histograms constitutes an occlusion-resistant appearance model. (2) Iterative optimization: Starting from a coarse initial pose, an iterative processing loop is entered; this iterative processing is achieved by minimizing the weighted reprojection error energy function, where the pose update uses Lie algebra. Perform parameterization; in These are the weights of the candidate edge points. It is a smooth step function. and These are the foreground and background probabilities calculated from the appearance model. The pose update can be obtained by solving the weighted least squares problem using the Gauss-Newton method.

[0072] (3) Repeat the iteration until convergence, and output the high-precision local pose of each object to be processed. .

[0073] Step 3, Rotational Attitude Pre-calibration: Step 2 ensures high accuracy of the local pose of a single object to be processed, but due to the spatial amplification effect of rotational errors, this accuracy cannot meet the accuracy requirements of scene-level registration; to ensure high accuracy of the final assembly scene, this embodiment performs rotational attitude pre-calibration. Figure 4 This is a schematic diagram of rotational attitude pre-calibration. It corrects large initial rotational errors in the registered datum by aligning the observed position vector with the CAD reference vector. This prevents the projected profile of the constrained object from falling outside the effective search range, which could lead to processing failure. The process includes: (1) Among all the objects to be processed through step 2, designate one as the target registration object (registration benchmark). The rest are constrained objects. ; (2) Based on the high-precision local pose of each object to be processed obtained in step 2, calculate the constrained object. Relative to the registration benchmark Observation position vector ; (3) Read the standard CAD reference vectors between these objects to be processed from the database. ; (4) For any pair of observation vectors and reference vector Its corrected rotation matrix Calculated using the Rodrigues transform: ; In the formula, It is the identity matrix. It is the unit rotation axis vector antisymmetric matrix, These are rotation angles, defined by the following formula: ; (5) Using an iterative alignment method, each Rotate and align to the corresponding The calculated corrected rotation is then applied to the rotational pose of the registered target. superior: ; This iterative process ensures that the angle error monotonically decreases until all vectors reach a coplanar state, at which point the system converges. Since the angle between two vectors in 3D space is greater than or equal to the angle between their orthogonal projections on the reference plane, the observed vector continuously converges to its reference vector through iteration until the two vectors are coplanar and a stable solution is obtained.

[0074] The accuracy of the rotational attitude pre-calibration in this embodiment depends on the result of step 2. To ensure the accuracy of the pre-calibration, adaptive verification is performed after calibration. After rotational correction, the number of searchable edge points for each constrained object is evaluated. If the count is below a predefined threshold (e.g., 80% of the searchable edge points found during single-object optimization), the search line length of the object to be processed is extended to 20% of its initial length. This process is repeated up to 3 times. If the threshold is still not met after 3 extensions, the calibration is considered to have failed, proving that the pose optimization has not achieved sufficient accuracy.

[0075] Step 4: Correct the residual errors in the pose that still exist after the rotation attitude deviation correction: Although the pre-calibration in Step 3 effectively calibrated the large rotation error, small residual errors still exist. Figure 5This is a schematic diagram illustrating the process of correcting the pose after rotational attitude deviation correction according to an embodiment of this disclosure. Figure 5 As shown, this embodiment of the present disclosure is based on a global optimization algorithm for contour search that is generalized. This method extends contour search to a cross-view, multi-object consistency optimization task based on the principle of contour energy minimization, so as to eliminate residual errors and achieve globally consistent registration.

[0076] This disclosure defines a series of objects to be processed as... For each object to be processed Construct a set of search lines Each search line is referenced by a projection point of a contour. Let its coordinates be given, and extend it along its normal vector. Extend in both directions; on each search line, select a set of high-confidence candidate edge points. The reprojection error for each candidate edge point compared to the registered baseline pose correction between Jacobian matrix It can be represented as: ; Among them, the derivative term The chain rule can be expanded as follows: In the formula, and These are the three-dimensional coordinates of the contour points in the camera coordinate system and the world coordinate system, respectively. The Jacobian matrix can be represented as follows: ; In the formula, It is a three-dimensional coordinate. The three components represent the position of the point on the X, Y, and Z axes. It is a 3x3 rotation matrix for transforming from the camera coordinate system to the world coordinate system. It is based on vectors Constructed antisymmetric matrix.

[0077] To enhance robustness and accelerate convergence, the overall optimization employs an image pyramid method. This process performs a fixed number of iterations at coarser pyramid levels; Perform 8 iterations at the specified resolution. Four iterations are performed at each level; ultimately, only the original resolution is considered. Convergence detection is performed horizontally; during a maximum of 4 iterations, if the pose update magnitude... Less than the predefined residual value If the optimization stops, then the optimization will cease.

[0078] The embodiments disclosed herein were verified on the aforementioned physical experimental platform for rocket modules with an inner diameter of approximately 2.5 meters. To ensure the reliability of the evaluation results, the measurement method used (error calibration via a physical positioning ball) was first independently tested. The results showed that the average measurement error of the evaluation method itself was 1.65 mm, and the maximum error was 2.41 mm, which is much smaller than the order of magnitude of the registered error, thereby ensuring the accuracy of the subsequent verification data. Experimental data strongly demonstrates that the overall design of the method in this disclosure and the technical processing of each step directly and inevitably contribute to improving registration accuracy. On the experimental platform, when using only existing single-object registration, the average error is as high as 34.5 mm, which is far from meeting industrial requirements. Based on this, after introducing the pre-calibration processing of the rotation vector in this disclosure, the average error is significantly reduced to 16–20 mm, proving the effectiveness of this pre-calibration step in correcting large rotation errors. Furthermore, by using multi-object joint processing to correct the rotational posture, the average error is further reduced to approximately 10 mm. This series of data shows that the hierarchical framework of this disclosure, through the progressive steps, ensures the global consistency and high accuracy of the final registration.

[0079] To further verify the method of this disclosure embodiment, this disclosure embodiment was also compared with the high-performance multi-marker mutual calibration method in related technologies; the results show that the final average registration error of the method of this disclosure embodiment is 9.4mm, which is more than 40% higher than the accuracy of the marker method; this result shows that the embodiment of this disclosure uses a multi-object joint approach to correct the rotational attitude correction pose, and by constructing a unified energy function and utilizing the dense geometric constraints of multiple objects to be processed in the scene, it fundamentally solves the problem of rotational error amplification with distance during single-point reference registration, thereby improving the 3D registration quality.

[0080] Furthermore, this invention offers convenience in deployment and use. Since no physical markers are required, this invention reduces the difficulty and cost of deployment in complex industrial environments, making it more applicable. Simultaneously, experiments demonstrate that this method is insensitive to the number and spatial distribution of constrained objects, exhibiting good robustness and facilitating practical industrial deployment.

[0081] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term "computer storage medium" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A three-dimensional registration method, applied to a scene containing three or more objects to be processed, the objects to be processed including a registration target object and constraint objects, characterized in that, include: The rotational pose correction pose of the registered target object and the constraint object is projected onto the image to be processed, and multiple candidate edge points of the registered target object and the constraint object are obtained by contour search. The rotational pose correction pose is the pose after the rotational pose deviation is corrected. The image to be processed includes images containing one or more objects to be processed, which are collected from two or more views of augmented reality (AR). The residual function corresponding to each candidate edge point is determined according to the predetermined energy function, where the energy function is the sum of the weighted log-likelihood loss generated by all candidate edge points, and the residual function is the error term generated by the candidate edge points; Based on each determined residual function, the corresponding Jacobian matrix with respect to the pose correction is determined, wherein the Jacobian matrix contains the pre-determined relative pose information of the constrained object relative to the registered target object. Based on the residual function, Jacobian matrix and pre-determined weight of each residual corresponding to each candidate edge point, a normalization equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. Solve the constructed normalized equation to obtain the pose correction amount; If the pose correction amount of the registered target object is determined to be less than the preset iteration termination condition, the rotational attitude correction pose of the registered target object and the constraint object is updated according to the pose correction amount, and the pose correction amount is obtained again based on the updated rotational attitude correction pose; if the iteration termination condition is determined to be met, the pose of the registered target object is corrected according to the pose correction amount of the registered target object. Based on the registered target object whose pose correction has been completed, perform 3D registration of the scene.

2. The method according to claim 1, characterized in that, The expression for the energy function is: ； In the formula, The parameters represent the rotational pose to be corrected, where i represents all anchor points mi projected onto the image to be processed from the 3D model; j represents the candidate edge points found along the normal search direction of each anchor point mi. ;w Indicates candidate edge points The weights; It is a smooth step function used to smoothly transition the foreground and background probabilities inside and outside the projected contour; Indicates candidate edge points The symbolic distance to the projected contour; Represents candidate edge points calculated based on a pre-determined appearance model. The posterior probability of belonging to the foreground; Represents candidate edge points calculated based on the appearance model. The posterior probabilities belonging to the foreground and background.

3. The method according to claim 1, characterized in that, The residual function is the error term generated by the candidate edge points, and the residual is... The expression is: 。 4. The method according to claim 1, characterized in that, The Jacobian matrix is ​​the partial derivative of the residual function with respect to the pose correction.

5. The method according to claim 1, characterized in that, The weights are set based on the geometric and / or appearance attributes of the candidate edge points.

6. The method according to claim 5, characterized in that, When the weight is set based on the geometric and appearance attributes of the candidate edge points, its expression is: ； In the formula, and P( is a pre-defined hyperparameter used to balance the influence of geometric and appearance properties; |C) indicates the anchor point The probability corresponding to the actual edge point reflects the appearance confidence of the candidate edge point; Indicates candidate edge points Its anchor point The normalized distance between them reflects the geometric confidence of the candidate edge point.

7. The method according to any one of claims 1 to 6, characterized in that, The construction of the normalized equation based on the negative values ​​of the Hessian approximation matrix and gradient vector includes: The residual of each candidate edge point, the weight of each residual, and the Jacobian matrix are accumulated to the negative value of the pre-established Hessian approximation matrix and gradient vector. The normalization equation is constructed based on the accumulated Hessian approximation matrix and the negative value of the gradient vector; The expression for the normalization equation is as follows: In the formula, H represents the accumulated Hessian approximation matrix, and b represents the negative value of the accumulated gradient vector. represents the pose correction amount.

8. The method according to any one of claims 1 to 6, characterized in that, The image to be processed includes an image obtained by downsampling the original image containing the registered target object.

9. The method according to any one of claims 1 to 6, characterized in that, Before projecting the rotational pose correction poses of the registered target object and constraint object onto the image to be processed and obtaining multiple candidate edge points through contour search, the method further includes a pre-calibration process to obtain the rotational pose correction pose: Based on the pose correction result of each of the objects to be processed, the relative position of each constrained object with respect to the registered target object is obtained and denoted as the observation vector. The design true value of the corresponding object to be processed, which is stored in the pre-created database, is read and denoted as the reference vector. By rotating and transforming, the rotational attitude of the registered target object is iteratively corrected, so that the error angle between the observation vector and the reference vector gradually decreases until it is lower than a preset threshold or the maximum number of iterations is reached, and the rotational attitude correction pose is obtained.

10. The method according to claim 9, characterized in that, Before performing the pre-calibration process to obtain the rotational attitude correction pose, the method further includes the following process to correct the pose of each object to be processed: Contour projection is performed on the original image according to the initial pose of each of the objects to be processed; Based on a pre-determined segmentation mask, a contour search is performed on the contour projection using segmentation guidance. The initial pose of the object to be processed is corrected based on the results of the contour search.

11. A computer storage medium storing a computer program that, when executed by a processor, implements the method of three-dimensional registration as described in any one of claims 1 to 10.

12. A terminal, comprising: A memory and a processor, wherein the memory stores a computer program; wherein, The processor is configured to execute computer programs in memory; When the computer program is executed by the processor, it implements the method of three-dimensional registration as described in any one of claims 1 to 10.

13. A three-dimensional registration device, applied to a scene containing three or more objects to be processed, the objects to be processed including a registration target object and constraint objects, characterized in that, include: The system comprises an edge search unit, a residual determination unit, a Jacobian matrix determination unit, an equation construction unit, a solution unit, a decision and correction unit, and a registration unit; among which, The edge search unit is set to project the rotational attitude correction pose of the registered target object and the constraint object onto the image to be processed, and obtain multiple candidate edge points of the registered target object and the constraint object through contour search. The rotational attitude correction pose is the pose after the rotational attitude deviation is corrected. The image to be processed includes images containing one or more objects to be processed, which are collected from two or more views of augmented reality (AR). The residual unit is determined as follows: the residual function corresponding to each candidate edge point is determined according to the predetermined energy function, where the energy function is the sum of the weighted log-likelihood loss generated by all candidate edge points, and the residual function is the error term generated by the candidate edge points; The Jacobian matrix unit is set as follows: for each determined residual function, the corresponding Jacobian matrix with respect to the pose correction is determined respectively, wherein the Jacobian matrix contains the pre-determined relative pose information of the constrained object relative to the registered target object; The equation building unit is set as follows: based on the residual function, Jacobian matrix and pre-determined weight of each residual corresponding to each candidate edge point, a normalized equation based on the negative values ​​of the Hessian approximation matrix and gradient vector is constructed. The solution element is set to solve the constructed normalized equations to obtain the pose correction. The determination and correction unit is set as follows: when the pose correction amount of the registered target object is determined to be less than the preset iteration termination condition, the rotational attitude correction pose of the registered target object and the constraint object is updated according to the pose correction amount, and the pose correction amount is obtained again based on the updated rotational attitude correction pose; when the iteration termination condition is determined to be met, the pose of the registered target object is corrected according to the pose correction amount of the registered target object. The registration unit is set to perform 3D registration of the scene based on the target object whose pose correction has been completed.

Images