Virtual-real fusion method and device based on multi-scale registration

Abstract

This invention discloses a virtual-real fusion method and apparatus based on multi-scale registration, relating to the fields of computer vision and 3D object pose estimation. It enables precise registration of real-time endoscopic images with CBCT 3D models, providing anatomical visualization navigation for impacted wisdom tooth extraction and reducing the risk of nerve damage. This method requires no feature point detection and matching, has low texture requirements, and is suitable for scenarios where endoscopic images lack rich texture. Multi-scale optimization strategies and adaptive regularization mechanisms improve optimization efficiency and stability, while trajectory consistency detection and correction mechanisms enhance the system's robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and three-dimensional object pose estimation, in particular to a virtual-real fusion method and device based on multi-scale registration based on template rendering and contour optimization. BACKGROUND

[0002] In computer vision and robot vision applications, accurately estimating the 6D pose (3D rotation and 3D translation) of a three-dimensional object relative to a camera is the basis for many tasks, including augmented reality, robot grasping, medical image navigation, etc. Traditional pose estimation methods are mainly divided into feature point-based indirect methods and direct method-based methods.

[0003] Feature point-based methods (such as ORB-SLAM, SuperPoint-SuperGlue, etc.) require detecting and matching feature points, which have high requirements for image texture. In scenes with insufficient texture, uneven lighting, and repeated texture, they are prone to failure or mismatch, resulting in failed or inaccurate pose estimation.

[0004] Direct method-based methods (such as DSO, PWP3D, etc.) directly use image intensity or contour information for registration, which has lower requirements for texture. However, traditional direct methods have the following limitations: First, they lack robustness in the presence of lighting changes, occlusions, and rapid motion, easily falling into local optimal solutions. Second, they lack effective registration quality assessment mechanisms, making it difficult to determine the reliability of current pose estimation. Third, the optimization process lacks an adaptive mechanism, making it impossible to dynamically adjust the optimization strategy based on scene conditions. Finally, they lack trajectory consistency detection and correction mechanisms, making it difficult to effectively handle abnormal situations in pose estimation. SUMMARY

[0005] In view of the above problems, the present application provides a virtual-real fusion method and device based on multi-scale registration for overcoming the above problems or at least partially solving the above problems. It is used to realize real-time tracking of the 6D pose of a three-dimensional object in a monocular image sequence. This method realizes real-time pose estimation without feature point detection and matching by pre-rendering a large number of template views, constructing a foreground-background histogram model, and using a multi-scale iterative optimization strategy. It is particularly suitable for applications in scenes with insufficient texture, frequent lighting changes, and the need for high-precision pose tracking.

[0006] The present application provides the following solutions:

[0007] A virtual-real fusion method based on multi-scale registration, comprising: Preprocessing a monocular image sequence of a target three-dimensional object to generate a three-dimensional mesh model containing object surface information, uniformly sampling view angles from the three-dimensional model and rendering to generate a plurality of template views, each of which includes a set of contour points, a direction vector, and depth information; A feature extraction network is used to extract multi-scale feature maps from the input image; For each contour point, image pixels are sampled along its normal vector direction to construct a foreground histogram and a background histogram respectively. The probability separation degree between the foreground and background is calculated based on the Bayesian formula to evaluate the matching quality of the current pose. An optimization strategy based on a multi-layer image pyramid structure from coarse to fine is adopted, and pose iterative optimization is performed at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation and pose update. An adaptive regularization mechanism is adopted to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of contour detection, thereby controlling the optimization step size; Monitor the speed, gradient and confidence index of pose estimation, detect anomalies, and use historical poses for smooth fusion and Kalman filtering to smooth and correct abnormal pose estimations. The optimized 6D pose of the target 3D object is output for real-time fusion display.

[0008] Preferably, the template view is generated by uniformly sampling 2000 viewpoints using the Fibonacci spherical sampling method and then rendered using the OpenGL rendering engine.

[0009] Preferably, the contour point set is obtained by uniformly sampling 200 three-dimensional points along the object contour, and includes position coordinates, normal vectors, and foreground and background distance information.

[0010] Preferably, the encoder of the feature extraction network uses the MobileOne-S0 architecture, the decoder of the feature extraction network adopts the U-Net structure, and the output of the feature extraction network has three scale feature maps, corresponding to one-quarter, one-half, and one times the resolution of the original image, respectively.

[0011] Preferably, the feature extraction network is trained using a self-supervised learning method, and is trained by the contour matching error between template rendering and real images. The loss function is the negative log-likelihood of the foreground-background histogram separation.

[0012] Preferably, the foreground histogram and the background histogram adopt an RGB three-channel joint histogram. For each contour point, pixel values ​​are sampled in the image along its normal vector direction to construct the foreground histogram and the background histogram respectively, and updated in the sequence frames by exponential moving average.

[0013] Preferably, the multi-layer image pyramid structure includes four layers of image pyramids. In the coarse-scale optimization stage, the image pyramids converge quickly to the approximate pose. In the meso-scale optimization stage, the image pyramids are refined. In the fine-scale optimization stage, the image pyramids are precisely located. Multiple iterations of optimization are performed in the fine-scale stage. The pose update uses the Gauss-Newton method, which obtains the pose increment by solving a system of linear equations and updates the pose through exponential mapping.

[0014] Preferably, the adaptive regularization mechanism includes: For the confidence calculation of each contour point, the edge is searched within the sector region along the normal vector direction of each contour point, taking into account factors such as edge strength, continuity, and directional consistency; the adaptive regularization formula uses a nonlinear mapping.

[0015] Preferably, the velocity estimation uses an exponentially weighted moving average method; the anomaly detection rules include two cases: low confidence combined with high gradient and velocity anomaly combined with high gradient; the correction strategy adopts a smoothing fusion method. When an anomaly is detected, the weighted average of the most recent 3 historical poses is calculated, linearly interpolated with the current pose, and then smoothed by Kalman filtering.

[0016] A virtual-real fusion device based on multi-scale registration, used to execute the above-described virtual-real fusion method based on multi-scale registration, the device comprising: The data preprocessing unit is used to preprocess the monocular image sequence of the target 3D object, generate a 3D mesh model containing the object surface information, and generate multiple template views from the 3D model by uniformly sampling the viewpoint and rendering. Each template view includes a set of contour points, a direction vector and depth information. The feature extraction unit is used to extract multi-scale feature maps of the input image using a feature extraction network; The foreground and background histogram construction unit is used to sample image pixels along the normal vector direction for each contour point, construct the foreground and background histograms respectively, and calculate the foreground and background probability separation degree based on Bayes' theorem to evaluate the matching quality of the current pose. The multi-scale optimization unit is used to adopt an optimization strategy based on a multi-layer image pyramid structure from coarse to fine, and to perform pose iterative optimization at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation and pose update. The adaptive regularization unit is used to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of the contour detection using an adaptive regularization mechanism, thereby controlling the optimization step size. The trajectory consistency detection unit is used to monitor the speed, gradient and confidence index of pose estimation, detect abnormal situations, and use historical poses for smooth fusion and Kalman filtering to smooth and correct abnormal pose estimation. The 6D pose output unit is used to output the optimized 6D pose of the target 3D object for real-time fusion display.

[0017] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This application provides a virtual-real fusion method and apparatus based on multi-scale registration, which can achieve precise registration between real-time endoscopic images and CBCT 3D models, providing anatomical structure visualization navigation for impacted wisdom tooth extraction and reducing the risk of nerve damage. This method requires no feature point detection and matching, has low texture requirements, and is suitable for scenarios where endoscopic images have limited texture. Multi-scale optimization strategies and adaptive regularization mechanisms improve optimization efficiency and stability, while trajectory consistency detection and correction mechanisms enhance the system's robustness.

[0018] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a flowchart of the virtual-real fusion method based on multi-scale registration provided in an embodiment of the present invention; Figure 2 This is a system framework diagram provided in an embodiment of the present invention; Figure 3 This is a flowchart of the real-time registration message interaction provided in the embodiments of the present invention; Figure 4 This is a functional module structure diagram provided in an embodiment of the present invention; Figure 5 This is a flowchart of the multi-scale optimization processing module provided in an embodiment of the present invention; Figure 6 This is a flowchart of the adaptive regularization processing module provided in an embodiment of the present invention; Figure 7 This is a flowchart of the trajectory consistency detection and correction module provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the virtual-real fusion method device based on multi-scale registration provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the virtual-real fusion method device based on multi-scale registration provided in an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0022] See Figure 1 This invention provides a virtual-real fusion method based on multi-scale registration, such as... Figure 1 As shown, the method may include: S101: Preprocess the monocular image sequence of the target 3D object to generate a 3D mesh model containing the object's surface information. Generate multiple template views from the 3D model by uniformly sampling the viewpoints and rendering them. Each template view includes a set of contour points, a direction vector, and depth information. In a specific implementation, this application embodiment can provide that the template view is generated by uniformly sampling 2000 viewpoints using the Fibonacci spherical sampling method and then rendered using an OpenGL rendering engine.

[0023] The contour point set is obtained by uniformly sampling 200 three-dimensional points along the object contour, and includes position coordinates, normal vectors, and foreground and background distance information.

[0024] S102: A feature extraction network is used to extract multi-scale feature maps of the input image; in specific implementation, the encoder of the feature extraction network can use the MobileOne-S0 architecture, the decoder of the feature extraction network can use the U-Net structure, and the output of the feature extraction network can be a feature map of three scales, corresponding to one-quarter, one-half and one times the resolution of the original image, respectively.

[0025] The feature extraction network is trained using a self-supervised learning approach, employing the contour matching error between template rendering and real images. The loss function is the negative log-likelihood of the foreground-background histogram separation.

[0026] S103: For each contour point, sample image pixels along its normal vector direction to construct a foreground histogram and a background histogram respectively, and calculate the foreground-background probability separation degree based on Bayes' theorem to evaluate the matching quality of the current pose; In specific implementation, the embodiments of this application can provide that the foreground histogram and the background histogram adopt an RGB three-channel joint histogram. For each contour point, sample pixel values ​​in the image along its normal vector direction to construct a foreground histogram and a background histogram respectively, and update them in the sequence frames using an exponential moving average method.

[0027] S104: An optimization strategy based on a multi-layer image pyramid structure from coarse to fine is adopted. The pose is iteratively optimized at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation, and pose update. In specific implementation, the multi-layer image pyramid structure in this application embodiment may include a 4-layer image pyramid. The coarse-scale optimization stage quickly converges to the approximate pose, the meso-scale optimization stage refines the pose, and the fine-scale optimization stage performs precise localization. Multiple iterative optimizations are performed at the fine scale. The pose update uses the Gauss-Newton method, which obtains the pose increment by solving a system of linear equations and updates the pose through exponential mapping.

[0028] S105: An adaptive regularization mechanism is used to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of the contour detection, thereby controlling the optimization step size; in specific implementation, the adaptive regularization mechanism provided in this application embodiment may include: For the confidence calculation of each contour point, the edge is searched within the sector region along the normal vector direction of each contour point, taking into account factors such as edge strength, continuity, and directional consistency; the adaptive regularization formula uses a nonlinear mapping.

[0029] S106: Monitor the velocity, gradient, and confidence index of pose estimation, detect anomalies, and use historical poses for smooth fusion and Kalman filtering to correct abnormal pose estimations; in specific implementation, this application embodiment can provide velocity estimation using an exponentially weighted moving average method; the anomaly detection rules include two cases: low confidence combined with high gradient and velocity anomaly combined with high gradient; the correction strategy adopts a smooth fusion method, when an anomaly is detected, calculates the weighted average of the most recent 3 frames of historical poses, performs linear interpolation with the current pose, and then smooths it using Kalman filtering.

[0030] S107: Output the optimized 6D pose of the target 3D object for real-time fusion display.

[0031] The virtual-real fusion method based on multi-scale registration provided in this application avoids real-time rendering overhead by pre-rendering a large number of template views (2000 viewpoints) while ensuring the uniformity and integrity of viewpoint coverage. It constructs a foreground-background histogram model and judges the matching quality of the current pose by evaluating the probability separation degree of the foreground and background around the contour points. This is an efficient registration quality evaluation method that does not require feature points. A multi-scale iterative optimization strategy is adopted, gradually refining the pose from coarse to fine. The coarse scale quickly converges to the approximate pose, while the fine scale performs precise positioning. An adaptive regularization mechanism is introduced to dynamically adjust the optimization step size based on the confidence of contour detection, ensuring rapid optimization of high-confidence frames while improving the stability of low-confidence frames. A trajectory consistency detection and correction mechanism is designed to detect anomalies in pose estimation by monitoring indicators such as velocity, gradient, and confidence, and uses historical poses for smooth fusion and Kalman filtering smoothing to effectively correct abnormal pose estimations.

[0032] Experimental results show that the method achieves sub-millimeter and sub-degree accuracy in both translation and rotation errors, meeting real-time requirements.

[0033] The following is a detailed description of the virtual-real fusion method based on multi-scale registration provided in the embodiments of this application.

[0034] This application provides a virtual-real fusion method based on template rendering and contour optimization and multi-scale registration, such as... Figure 2 , Figure 3 , Figure 4 As shown, the system implementing this method mainly includes a data preprocessing module, a feature extraction module, a registration optimization module, an error correction module, and a real-time fusion display module. Each module works collaboratively through its sub-modules to achieve a complete registration process from data preprocessing to real-time fusion display. The data preprocessing module is responsible for endoscopic image acquisition, CBCT data preprocessing, and template view pre-rendering; the feature extraction module contains a feature extraction network; the registration optimization module includes multi-scale optimization; and the error correction module includes trajectory consistency detection. These modules work together to achieve accurate registration between real-time endoscopic images and the CBCT 3D model.

[0035] This method avoids real-time rendering overhead by pre-rendering a large number of template views, uses foreground-background histogram separation to evaluate registration quality, adopts a multi-scale iterative optimization strategy to gradually refine the pose from coarse to fine, and combines adaptive regularization mechanism and trajectory consistency detection to improve system robustness.

[0036] 1. Data preprocessing: The target 3D object is preprocessed to generate a 3D mesh model containing surface information. From the 3D mesh model, 2000 view directions are uniformly sampled using the Fibonacci spherical sampling method. A corresponding template view is generated using the OpenGL rendering engine. Each template view contains a set of contour points, orientation vectors, and depth information. The contour point set is obtained by uniformly sampling 200 3D points along the object's contour, containing position coordinates, normal vectors, and foreground and background distance information. A monocular image sequence is acquired, and image sharpness is optimized using a dehazing algorithm. The Region of Interest (ROI) containing the target object is extracted and normalized.

[0037] 2. Feature extraction network: A lightweight design is adopted to meet real-time requirements. The encoder uses the MobileOne-S0 architecture, and the decoder uses the U-Net structure, outputting feature maps at three scales, corresponding to one-quarter, one-half, and one times the resolution of the original image, respectively. The network is trained using a self-supervised learning approach, trained by the contour matching error between template rendering and real images, with the loss function being the negative log-likelihood of the foreground-background histogram separation.

[0038] 3. Foreground and background histogram model: This is used to evaluate the foreground-background probability separation around contour points, determining the matching quality of the current pose. The histogram model employs a joint RGB three-channel histogram. For each contour point, pixel values ​​are sampled along its normal vector direction in the image to construct foreground and background histograms respectively. Probability calculation uses Bayes' theorem, and the separation evaluation is obtained by subtracting the average foreground probability of background sampling points from the average foreground probability of foreground sampling points. The histogram model is initialized in the first frame and updated in subsequent frames using an exponential moving average.

[0039] 4. Multi-scale optimization strategy: A four-layer image pyramid structure is constructed, progressing from coarse to fine. The coarse-scale optimization stage rapidly converges to the approximate pose, the meso-scale optimization stage refines the pose, and the fine-scale optimization stage performs precise localization, with multiple iterative optimizations at the fine scale. Each iteration includes template view selection, contour projection, gradient calculation, Hessian calculation, and pose update. The pose update uses the Gauss-Newton method, obtaining the pose increment by solving a system of linear equations, and then updating the pose through an exponential mapping.

[0040] like Figure 5 As shown, Figure 5The flowchart for the multi-scale optimization processing module is as follows. The main processing steps of this module are: Step 1, template selection and contour projection; Step 2, gradient / Hessian calculation and adaptive regularization; Step 3, pose update. Then, it is determined whether convergence has occurred. If convergence has not occurred and the maximum number of iterations has not been reached, return to Step 1 to continue iteration; otherwise, proceed to the next scale optimization or terminate the optimization.

[0041] 5. Adaptive regularization mechanism: The regularization strength of the Hessian matrix is ​​dynamically adjusted based on the confidence level of the contour detection, thereby controlling the optimization step size. For each contour point, the confidence level is calculated by searching for edges within a sector along its normal vector direction, considering three factors: edge strength, continuity, and directional consistency. The adaptive regularization formula employs a nonlinear mapping. When the average confidence level is high, the regularization strength approaches its minimum, resulting in a large optimization step size and rapid convergence. Conversely, when the average confidence level is low, the regularization strength approaches its maximum, resulting in a small optimization step size and a conservative strategy to avoid divergence.

[0042] like Figure 6 As shown, Figure 6 The flowchart shows the adaptive regularization processing module. The module's processing flow mainly includes the following steps: Step 1, edge search and confidence calculation; Step 2, calculation of adaptive regularization strength; Step 3, construction of the regularization matrix and optimization of the pose.

[0043] 6. Trajectory Consistency Detection Module: Anomalies in pose estimation are detected by monitoring metrics such as velocity, gradient, and confidence. Velocity estimation uses an exponentially weighted moving average method. Anomaly detection rules include two cases: low confidence combined with high gradient and velocity anomalies combined with high gradient. The correction strategy employs a smoothing fusion method. When an anomaly is detected, the system calculates a weighted average of the last three historical poses, performs linear interpolation with the current pose, and then smooths the result using a Kalman filter.

[0044] like Figure 7 As shown, Figure 7 This is a flowchart of the trajectory consistency detection and correction module. The processing flow of this module mainly includes the following steps: Step 1, velocity estimation and smoothing. Step 2, anomaly detection. Then, it is determined whether an anomaly is detected. If no anomaly is detected, the current pose is maintained; if an anomaly is detected, proceed to Step 3 for pose correction and smoothing, and output the corrected pose.

[0045] The specific workflow is as follows: The entire real-time registration message interaction process is as follows: Figure 2As shown, upon system startup, data preprocessing and template view pre-rendering are performed. For each frame of endoscopic image, the system sequentially performs feature extraction, registration optimization, and error correction, with the final output used for real-time fusion display. Each module works collaboratively in the workflow sequence to achieve real-time registration.

[0046] Specifically, upon system startup, the 3D model and pre-rendered template view are loaded first. During the initialization phase, the initial pose is set, and a foreground / background histogram is constructed during the first frame processing. For subsequent frames, the system uses the final pose of the previous frame as the initial pose for the current frame, selects the closest template view based on the initial pose, and performs image preprocessing and feature extraction. Multi-scale optimization starts at a coarse scale, proceeds sequentially to medium and fine scales, and the fine scale undergoes multiple iterations to achieve precise localization. Each iteration includes template selection, contour projection, gradient calculation, Hessian calculation, adaptive regularization, and pose update. Trajectory consistency detection is performed after all scale optimizations are completed, detecting and correcting anomalies. Histogram updates use the pixel values ​​of the current frame, updated using an exponential moving average. Finally, the system uses the optimized pose for real-time fusion and display.

[0047] As can be seen, the template rendering and contour optimization method does not require feature point detection and matching, has low texture requirements, and is suitable for scenarios where endoscopic images have limited texture. The multi-scale optimization strategy gradually refines the pose from coarse to fine, improving optimization efficiency and accuracy. The adaptive regularization mechanism dynamically adjusts the optimization step size based on confidence, ensuring rapid optimization of high-confidence frames while improving the stability of low-confidence frames. The trajectory consistency detection and correction mechanism effectively corrects abnormal pose estimations, improving the system's robustness.

[0048] Experimental results show that the method achieves sub-millimeter and sub-degree accuracy in both translation and rotation errors, meeting real-time requirements.

[0049] In summary, the virtual-real fusion method based on multi-scale registration provided in this application can achieve accurate registration between real-time endoscopic images and CBCT 3D models, providing anatomical visualization navigation for impacted wisdom tooth extraction and reducing the risk of nerve injury. This method requires no feature point detection and matching, has low texture requirements, and is suitable for scenarios where endoscopic images lack rich texture. The multi-scale optimization strategy and adaptive regularization mechanism improve optimization efficiency and stability, while the trajectory consistency detection and correction mechanism enhances the system's robustness.

[0050] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. The technical solutions and processing methods of the present invention are equally applicable to other surgical navigation applications. Furthermore, the technical solutions and methods of the present invention are also applicable to various endoscopic surgical scenarios, including but not limited to laparoscopic surgery, arthroscopic surgery, and nasal endoscopy, and therefore these application scenarios also fall within the scope of the claims of this invention.

[0051] See Figure 8 This application embodiment can also provide a virtual-real fusion device based on multi-scale registration, such as... Figure 8 As shown, the apparatus for performing the above-described virtual-real fusion method based on multi-scale registration may include: The data preprocessing unit 801 is used to preprocess the monocular image sequence of the target three-dimensional object to generate a three-dimensional mesh model containing the surface information of the object, and to generate multiple template views from the three-dimensional model by uniformly sampling the viewpoint and rendering. Each template view includes a set of contour points, a direction vector and depth information. Feature extraction unit 802 is used to extract multi-scale feature maps of the input image using a feature extraction network; The foreground and background histogram construction unit 803 is used to sample image pixels along the normal vector direction for each contour point, construct foreground and background histograms respectively, and calculate the foreground and background probability separation degree based on Bayes' theorem to evaluate the matching quality of the current pose. The multi-scale optimization unit 804 is used to adopt an optimization strategy based on a multi-layer image pyramid structure from coarse to fine, and to perform pose iterative optimization at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation and pose update. The adaptive regularization unit 805 is used to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of the contour detection using an adaptive regularization mechanism, thereby controlling the optimization step size. The trajectory consistency detection unit 806 is used to monitor the speed, gradient and confidence index of pose estimation, detect abnormal situations, and use historical poses for smooth fusion and Kalman filtering to smooth and correct abnormal pose estimation. The 6D pose output unit 807 is used to output the optimized 6D pose of the target 3D object for real-time fusion display.

[0052] This application embodiment can also provide a virtual-real fusion device based on multi-scale registration, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the above-described virtual-real fusion method based on multi-scale registration according to the instructions in the program code.

[0053] like Figure 9 As shown in the figure, the virtual-real fusion device based on multi-scale registration provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0054] In this embodiment, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0055] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiment of the virtual-real fusion method based on multi-scale registration.

[0056] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions: A monocular image sequence of the target 3D object is preprocessed to generate a 3D mesh model containing the object's surface information. Multiple template views are generated from the 3D model by uniformly sampling the viewpoint and rendering. Each template view includes a set of contour points, a direction vector, and depth information. A feature extraction network is used to extract multi-scale feature maps from the input image; For each contour point, image pixels are sampled along its normal vector direction to construct a foreground histogram and a background histogram respectively. The probability separation degree between the foreground and background is calculated based on the Bayesian formula to evaluate the matching quality of the current pose. An optimization strategy based on a multi-layer image pyramid structure from coarse to fine is adopted, and pose iterative optimization is performed at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation and pose update. An adaptive regularization mechanism is adopted to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of contour detection, thereby controlling the optimization step size; Monitor the speed, gradient and confidence index of pose estimation, detect anomalies, and use historical poses for smooth fusion and Kalman filtering to smooth and correct abnormal pose estimations. The optimized 6D pose of the target 3D object is output for real-time fusion display.

[0057] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0058] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0059] Communication interface 12 can be an interface for a communication model, used to connect with other devices or systems.

[0060] Of course, it should be noted that, Figure 9 The structure shown does not constitute a limitation on the virtual-real fusion device based on multi-scale registration in the embodiments of this application. In practical applications, virtual-real fusion devices based on multi-scale registration can include more than Figure 9 More or fewer components as shown, or combinations of certain components.

[0061] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described virtual-real fusion method based on multi-scale registration.

[0062] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0063] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0064] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0065] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A virtual-real fusion method based on multi-scale registration, characterized in that, include: A monocular image sequence of the target 3D object is preprocessed to generate a 3D mesh model containing the object's surface information. Multiple template views are generated from the 3D model by uniformly sampling the viewpoint and rendering. Each template view includes a set of contour points, a direction vector, and depth information. A feature extraction network is used to extract multi-scale feature maps from the input image; For each contour point, image pixels are sampled along its normal vector direction to construct a foreground histogram and a background histogram respectively. The probability separation degree between the foreground and background is calculated based on the Bayesian formula to evaluate the matching quality of the current pose. An optimization strategy based on a multi-layer image pyramid structure from coarse to fine is adopted, and pose iterative optimization is performed at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation and pose update. An adaptive regularization mechanism is adopted to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of contour detection, thereby controlling the optimization step size; Monitor the speed, gradient and confidence index of pose estimation, detect anomalies, and use historical poses for smooth fusion and Kalman filtering to smooth and correct abnormal pose estimations. The optimized 6D pose of the target 3D object is output for real-time fusion display.

2. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, The template view is generated by uniformly sampling 2000 viewpoints using the Fibonacci spherical sampling method and then rendered using the OpenGL rendering engine.

3. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, The contour point set is obtained by uniformly sampling 200 three-dimensional points along the object contour, and includes position coordinates, normal vectors, and foreground and background distance information.

4. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, The encoder of the feature extraction network uses the MobileOne-S0 architecture, and the decoder of the feature extraction network adopts the U-Net structure. The output of the feature extraction network consists of feature maps at three scales, corresponding to one-quarter, one-half, and one times the resolution of the original image, respectively.

5. The virtual-real fusion method based on multi-scale registration according to claim 4, characterized in that, The feature extraction network is trained using a self-supervised learning approach, employing the contour matching error between template rendering and real images. The loss function is the negative log-likelihood of the foreground-background histogram separation.

6. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, The foreground histogram and the background histogram adopt a joint RGB three-channel histogram. For each contour point, pixel values ​​are sampled in the image along its normal vector direction to construct the foreground histogram and the background histogram respectively, and updated in the sequence frames by exponential moving average.

7. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, The multi-layer image pyramid structure includes four layers of image pyramids. In the coarse-scale optimization stage, the image pyramids converge quickly to the approximate pose. In the meso-scale optimization stage, the image pyramids are refined. In the fine-scale optimization stage, the image pyramids are precisely located. Multiple iterations of optimization are performed in the fine-scale stage. The pose update uses the Gauss-Newton method, which obtains the pose increment by solving a system of linear equations and updates the pose through exponential mapping.

8. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, The adaptive regularization mechanism includes: For the confidence calculation of each contour point, the edge is searched within the sector region along the normal vector direction of each contour point, taking into account factors such as edge strength, continuity, and directional consistency; the adaptive regularization formula uses a nonlinear mapping.

9. The virtual-real fusion method based on multi-scale registration according to claim 1, characterized in that, Velocity estimation uses an exponentially weighted moving average method; anomaly detection rules include two cases: low confidence combined with high gradient and velocity anomaly combined with high gradient; the correction strategy adopts a smoothing fusion method. When an anomaly is detected, the weighted average of the most recent 3 historical poses is calculated, linearly interpolated with the current pose, and then smoothed by Kalman filtering.

10. A virtual-real fusion device based on multi-scale registration, characterized in that, The apparatus for performing the virtual-real fusion method based on multi-scale registration as described in any one of claims 1-9, the apparatus comprising: The data preprocessing unit is used to preprocess the monocular image sequence of the target 3D object, generate a 3D mesh model containing the object surface information, and generate multiple template views from the 3D model by uniformly sampling the viewpoint and rendering. Each template view includes a set of contour points, a direction vector and depth information. The feature extraction unit is used to extract multi-scale feature maps of the input image using a feature extraction network; The foreground and background histogram construction unit is used to sample image pixels along the normal vector direction for each contour point, construct the foreground and background histograms respectively, and calculate the foreground and background probability separation degree based on Bayes' theorem to evaluate the matching quality of the current pose. The multi-scale optimization unit is used to adopt an optimization strategy based on a multi-layer image pyramid structure from coarse to fine, and to perform pose iterative optimization at multiple scales from coarse to fine. Each iteration includes template selection, contour projection, gradient and Hessian calculation and pose update. The adaptive regularization unit is used to dynamically adjust the regularization strength of the Hessian matrix based on the confidence level of the contour detection using an adaptive regularization mechanism, thereby controlling the optimization step size. The trajectory consistency detection unit is used to monitor the speed, gradient and confidence index of pose estimation, detect abnormal situations, and use historical poses for smooth fusion and Kalman filtering to smooth and correct abnormal pose estimation. The 6D pose output unit is used to output the optimized 6D pose of the target 3D object for real-time fusion display.

Images