Thermal imaging and three-dimensional model registration and temperature mapping method and electronic device

Abstract

The application discloses a thermal imaging and three-dimensional model registration and temperature mapping method and an electronic device, and relates to the technical field of industrial detection and computer vision. The thermal imaging and three-dimensional model registration and temperature mapping method first acquires a known three-dimensional model and single thermal imaging data of a target; then, a physically driven matching function is constructed, which integrates three constraints of geometric structure consistency, edge physical consistency and temperature distribution consistency; then, in the pose space of the three-dimensional model, a two-stage optimization strategy from coarse to fine is adopted to search for a pose that optimizes the matching function; finally, based on the optimal pose, the temperature information of the thermal imaging is mapped to the surface of the three-dimensional model. The application does not require additional geometric sensors or multiple images, effectively overcomes problems such as lack of thermal imaging texture and edge blur through a physically driven optimization framework, and realizes automatic registration and temperature mapping with high precision and high robustness.

Description

Technical Field

[0001] This application relates to the fields of industrial inspection and computer vision technology, and in particular to a thermal imaging and three-dimensional model registration and temperature mapping method and electronic device. Background Technology

[0002] In the field of industrial thermal analysis, achieving high-precision mapping between two-dimensional thermal imaging and three-dimensional digital models is a crucial prerequisite for spatial temperature field analysis, hotspot localization, and process optimization. Current mainstream technologies suffer from the following inherent drawbacks:

[0003] Multi-sensor fusion reconstruction methods rely on the rigid integration of infrared thermal imagers with additional geometric sensors (such as depth cameras and LiDAR). While they can obtain geometrically accurate temperature field models, the system is complex, hardware costs are high, and the calibration and synchronization processes between multiple sensors are cumbersome. Furthermore, their stability in vibrating and high-temperature industrial environments is questionable.

[0004] Reconstruction methods based on multi-view stereo vision attempt to recover 3D geometry using only multiple frames of thermal imaging sequences. However, thermal imaging generally suffers from a lack of texture features and blurred edges due to thermal diffusion, making feature matching difficult in traditional SfM / MVS algorithms. This results in high noise and poor integrity in the reconstructed model, making it difficult to guarantee mapping accuracy and robustness.

[0005] Emerging methods based on deep learning and neural rendering utilize neural networks (such as NeRF and 3DGS) to implicitly learn scenarios; however, they heavily rely on massive amounts of training data with precise pose labels, resulting in extremely high acquisition and annotation costs; the models have huge computational loads and weak generalization capabilities, and are currently still far from stable and efficient industrial-grade applications.

[0006] In other words, existing technologies have failed to strike a balance between the three core requirements of "single low-cost sensor", "no need for massive data" and "high precision and high robustness". Summary of the Invention

[0007] To address the aforementioned issues, this application provides a thermal imaging and 3D model registration and temperature mapping method and electronic device. Through physical-driven consistency evaluation, the complex cross-modal registration problem is transformed into an optimization search problem for the pose of a known 3D model, thereby achieving high-precision and robust temperature mapping that relies solely on a single thermal image and a known 3D model.

[0008] The first technical solution adopted in this application is: providing a method for thermal imaging registration with a three-dimensional model and temperature mapping, including:

[0009] Acquire a known three-dimensional model of the target under test and a single thermal imaging data collected for the target under test;

[0010] A physics-driven matching function is constructed to quantitatively evaluate the degree of matching between the projection information of the 3D model and the thermal imaging data under any assumed pose. The matching function is constructed collaboratively based on three types of physical constraints: geometric consistency, edge physical consistency, and temperature distribution consistency.

[0011] Based on the matching function, an optimization search is performed in the pose space of the three-dimensional model to obtain the optimal registration pose.

[0012] Based on the optimal registration pose, the temperature information in the thermal imaging data is mapped onto the surface of the three-dimensional model to generate a three-dimensional digital model with temperature field information.

[0013] The matching function is a comprehensive scoring function obtained by fusing geometric structure consistency scoring items, edge physical consistency scoring items, and temperature distribution consistency scoring items in a weighted manner; wherein, the weight coefficient of each scoring item can be dynamically adjusted according to at least one of the following factors: optimization stage, prior knowledge certainty, or data quality.

[0014] An adaptive weighting strategy is used to dynamically adjust the weight coefficients of each scoring item, including:

[0015] In the coarse positioning optimization stage, the geometric structure consistency scoring item is assigned the highest weight;

[0016] During the fine-tuning phase, the edge physical consistency scoring item is assigned the highest weight;

[0017] When the 3D model lacks predefined semantic labels, the weight of the temperature distribution consistency score item is increased.

[0018] In an optional embodiment, the optimization search employs a two-stage strategy from coarse to fine, including:

[0019] In the first stage, the pose search space is constrained based on physical priors, and a global optimization algorithm is used to perform a coarse search within the constrained space to obtain the coarse localization pose.

[0020] In the second stage, using the coarse localization pose as the initial value, a local optimization algorithm is used for fine iterative optimization until convergence, thereby obtaining the optimal registration pose.

[0021] In an optional embodiment, after obtaining the optimal registration pose and before performing the temperature information mapping, a result verification and rollback step is further included:

[0022] If the comprehensive matching score corresponding to the optimal registration pose is lower than a preset threshold, a rollback mechanism is triggered.

[0023] The backoff mechanism includes: generating multiple sets of dispersed initial poses in the pose space, performing coarse search in parallel, retaining multiple better poses to form a set, and performing fine optimization starting from the poses in the set, and selecting the pose with the best matching score from all results as the new optimal registration pose.

[0024] In an optional embodiment, the step of constructing a local thermal pattern template library is also included;

[0025] The local thermal pattern template library is used to provide expected thermal pattern feature data for predefined key local structures in the 3D model. The expected thermal pattern feature data is obtained based on historical calibration data and is used to assist in calculating at least one scoring item in the matching function.

[0026] In an optional embodiment, the geometric consistency score is comprehensively evaluated based on the overall contour matching degree between the projection of the 3D model and the thermal imaging data, as well as the thermal pattern matching degree of key local structures; wherein the thermal pattern matching degree of the key local structures is calculated based on the local thermal pattern template library.

[0027] In an alternative embodiment, the edge physical consistency score is calculated by measuring the spatial correlation between the projected geometric edges of the 3D model and significant temperature gradient feature points extracted from the thermal imaging data.

[0028] In an optional embodiment, the temperature distribution consistency score is calculated by comparing the similarity between a macroscopic thermophysical statistical feature vector extracted from the thermal imaging data and a simulation feature vector extracted from the model projected temperature map based on the current assumed pose.

[0029] The second technical solution adopted in this application is: providing an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the thermal imaging and three-dimensional model registration and temperature mapping method as described in any of the preceding claims.

[0030] Due to the adoption of the above technical solution, this application has at least one of the following beneficial effects compared with the prior art:

[0031] 1. By designing a physics-driven matching function, the traditional reliance on clear outlines and textures in thermal imaging is bypassed. By utilizing multi-physics constraints of geometry, edge, and temperature distribution, reliable registration can still be achieved in scenarios with low texture, blurred edges, local heating, or interference.

[0032] 2. By integrating local feature temperature constraints with global temperature distribution pattern consistency assessment, it can accurately handle typical industrial scenarios where the target part generates heat, effectively distinguish the target body from irrelevant heat sources, and ensure the physical rationality of the registration results.

[0033] 3. By adopting a two-stage optimization strategy from coarse to fine and combining scene geometric priors, it achieves precise alignment at the thermal imaging resolution level while ensuring global search capabilities and avoiding getting trapped in local optima. The entire process requires no manual intervention and is suitable for integration into automated production lines.

[0034] 4. The solution relies only on single-frame thermal imaging data and the existing 3D model of the target object, without the need to add expensive auxiliary sensing devices such as depth cameras and laser scanners, and also avoids the cumbersome on-site multi-sensor calibration process, which greatly reduces the system complexity and implementation cost. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0036] Figure 1 This is a flowchart illustrating a thermal imaging and 3D model registration and temperature mapping method provided in an embodiment of this application.

[0037] Figure 2 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0039] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0040] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0041] Existing thermal imaging and 3D model registration techniques typically rely on image feature matching or multi-sensor fusion. The former struggles to extract stable features when thermal imaging textures are scarce, edges are blurred, or the target is only locally heated, leading to registration failure. The latter requires additional hardware such as depth cameras or LiDAR, resulting in complex systems, high costs, and difficulties in on-site calibration. These drawbacks make existing methods unreliable in real-world industrial scenarios, with high deployment costs and low automation levels.

[0042] In view of this, the thermal imaging and 3D model registration and temperature mapping method of this application constructs a physically driven matching function based on geometric structure consistency, edge physical consistency, and temperature distribution consistency, and performs optimization search directly in the pose space, without relying on image feature extraction or multi-source sensor data. This effectively overcomes the aforementioned shortcomings and achieves highly robust, low-cost, and fully automatic accurate registration and temperature mapping under single-frame thermal imaging. Figure 1 As shown, Figure 1 A flowchart illustrating a thermal imaging and 3D model registration and temperature mapping method provided in an embodiment of this application includes the following steps:

[0043] The process involves acquiring a known 3D model of the target under test, along with a single thermal imaging image of the target. The 3D model is a precise computer-aided design (CAD) model of the target, typically stored in common 3D formats such as GLB, OBJ, and STL. The 3D model contains a complete geometric structure. In a preferred embodiment, the 3D model also contains semantic information: for example, cold zones are parts determined to maintain low temperatures based on physical structure (such as the mold base), and hot zones are parts expected to be heated during the process (such as the working surface of the mold cavity). Semantic tags serve as prior knowledge to guide optimization and verification.

[0044] Single thermal imaging data includes a single thermal imaging temperature matrix. And thermal imager internal parameters; the temperature matrix of a single thermal image is a two-dimensional data matrix obtained from the infrared thermal imager. Each element in the matrix This represents the temperature measurement value of the corresponding pixel on the image plane. Additionally, the intrinsic parameter matrix of the thermal imager must be provided. It is used to establish the projection relationship between the pixel coordinates of a two-dimensional image and the normalized coordinates in the three-dimensional camera coordinate system.

[0045] A physics-driven matching function is constructed to quantitatively evaluate the degree of matching between the projection information of the 3D model and the thermal imaging data under any assumed pose. The matching function is constructed collaboratively based on three types of physical constraints: geometric consistency, edge physical consistency, and temperature distribution consistency. By constructing a physics-driven matching function, the direct dependence on the appearance features of thermal imaging is bypassed, and instead, the robust physical properties contained therein and the known geometric information of the 3D model are utilized to construct a consistency evaluation function with multiple physical constraints.

[0046] Based on the matching function, an optimization search is performed in the pose space of the 3D model to obtain the optimal registration pose. The optimal registration pose is the pose that makes the matching function value optimal in the six-degree-of-freedom pose space of the model (three translations and three rotations). In this embodiment, the pose with the optimal matching function value is the pose with the largest matching function value. In other embodiments, the pose with the optimal matching function value can be selected from other options, and no limitation is made on this.

[0047] Based on the optimal registration pose, the temperature information from the thermal imaging data is mapped onto the surface of the 3D model, generating a 3D digital model with temperature field information. The detailed steps for generating the 3D digital model are described below:

[0048] For each vertex of the 3D model surface, its optimal pose and camera intrinsic parameters are used to project it onto the thermal imaging 2D plane to obtain sub-pixel coordinates. A bilinear interpolation algorithm is then used to extract these coordinates from the temperature matrix. A high-precision temperature value is obtained from the data and assigned to the vertex. After traversing all vertices, a three-dimensional digital model with continuous temperature field information is constructed.

[0049] In summary, the thermal imaging and 3D model registration and temperature mapping method of this embodiment first acquires a known 3D model of the target and a single thermal imaging image; then, it constructs a physics-driven matching function that integrates triple constraints of geometric consistency, edge physical consistency, and temperature distribution consistency; next, within the pose space of the 3D model, a two-stage optimization strategy from coarse to fine is used to search for the pose that optimizes the matching function; finally, based on this optimal pose, the temperature information of the thermal imaging is mapped onto the surface of the 3D model. This application does not require additional geometric sensors or multiple frames of images. Through a physics-driven optimization framework, it effectively overcomes problems such as lack of thermal imaging texture and blurred edges, achieving high-precision and highly robust automatic registration and temperature mapping.

[0050] The matching function is a comprehensive scoring function obtained by weightedly fusing geometric structure consistency scoring items, edge physical consistency scoring items, and temperature distribution consistency scoring items; wherein, the weight coefficient of each scoring item can be dynamically adjusted according to at least one of the following factors: optimization stage, prior knowledge certainty, or data quality. In this embodiment, the matching function is as follows:

[0051] .

[0052] in, For geometric consistency scoring items, For edge physical consistency scoring items, This is a scoring item for temperature distribution consistency. , and These are the corresponding adaptive weight coefficients; and they satisfy... .

[0053] The edge physical consistency scoring item uses the first-order physical quantity of the temperature field—the gradient—to define and match the boundary; the temperature distribution consistency scoring item uses the statistical physical properties of the temperature field (such as mean, variance, and spatial distribution) for macroscopic pattern matching; the local matching in the geometric structure consistency scoring item is based on the statistical thermal pattern feature vector (such as average temperature and gradient direction) exhibited by the local region in thermal imaging.

[0054] The adjustment of the weighting coefficients is based on the following principle:

[0055] Optimization phase: Different weight configuration strategies are used in the coarse search and fine optimization phases.

[0056] Prior knowledge certainty: Adjustments are made based on whether the 3D model contains reliable semantic labels such as cold and / or hot zones.

[0057] Data quality: Fine-tuned based on confidence indicators such as real-time calculated thermal imaging gradient sharpness and noise level.

[0058] An adaptive weighting strategy is used to dynamically adjust the weight coefficients of each scoring item, including:

[0059] In the coarse positioning optimization stage, the geometric structure consistency score item is given the highest weight;

[0060] During the fine-tuning phase, the edge physical consistency scoring item is given the highest weight;

[0061] When the 3D model lacks predefined semantic labels, increase the weight of the temperature distribution consistency score item.

[0062] When the 3D model contains complete semantic labels, the strategy-based scheduling based on the optimization phase is as follows:

[0063] The goal of the coarse localization stage is to quickly approximate a physically plausible solution space. At this stage, geometric consistency is given the highest weight (e.g., ...). =0.7, =0.1, =0.2); leveraging its hard constraint characteristics rooted in the deterministic model, it provides strong initial anchoring and directional guidance for the search. The temperature distribution consistency score serves as a soft constraint to provide auxiliary screening, while the weight of the edge physical consistency score is intentionally reduced because it is sensitive to initial pose errors, and emphasizing it too early can easily lead to instability.

[0064] The goal of the fine-tuning phase is to achieve pixel-level precise alignment within the neighborhood of the high-quality initial value. The weighting configuration undergoes a fundamental shift: the weight of the edge physical consistency scoring item is significantly increased (e.g., =0.2, =0.7, =0.1), making it the core engine driving pose fine-tuning and achieving sharp edge alignment; the weight of the geometric consistency scoring item is moderately reduced, but its veto power of hard constraints is retained; the weight of the temperature distribution consistency scoring item is further reduced, mainly playing the role of final physical rationality verification.

[0065] When the 3D model lacks semantic labels or process knowledge, the weight of the temperature distribution consistency score item is increased to make it the main guide for macroscopic thermal pattern matching; the weight of the geometric structure consistency score item is reduced accordingly, relying only on its most basic overall contour constraints.

[0066] Regardless of whether the 3D model contains complete semantic labels or lacks semantic labels, the weights are fine-tuned based on the confidence index calculated in real time. For example, if the current thermal imaging gradient is detected to be clear and the noise is low, the real-time weight of the edge physical consistency score item is slightly increased; if significant non-uniform noise is detected in the image, causing the statistical feature confidence of the temperature distribution consistency score item to decrease, its weight is automatically slightly decreased.

[0067] The specific values ​​of the weighting coefficients can be pre-calibrated through offline analysis of a representative sample set and can be fine-tuned online during actual deployment; no limitations are imposed on this.

[0068] The search optimization employs a two-stage strategy, from coarse to fine, including:

[0069] In the first stage, the pose search space is constrained based on physical priors, and a global optimization algorithm is used to perform a coarse search within this constrained space to obtain the coarse localization pose. The steps for obtaining the coarse localization pose are described in detail below:

[0070] The pitch angle of the 3D model relative to the camera is limited to a typical observation range (e.g., 30° to 60°), and the roll angle is constrained to a very small range close to zero (e.g., ±5°).

[0071] The translation search is mainly focused on the direction of the degree of freedom with the greatest possibility of change (e.g., the horizontal X-axis), while the translation amounts in the other two directions (Y, Z) are limited to a small physical range (e.g., ±10 mm) that characterizes mechanical installation tolerances or minor vibrations.

[0072] Subsequently, within the feasible solution space defined by the aforementioned prior constraints, multiple sets (e.g., 50-100 sets) of different initial pose hypotheses are randomly generated. Then, an optimization algorithm suitable for parallel processing and with strong global search capabilities (such as Particle Swarm Optimization (PSO) or Differential Evolution (DE)) is selected, using the matching function S(Pose) as the fitness function, to drive these pose hypotheses to approach higher-scoring regions in the solution space. After a predetermined number of iterations, the pose with the highest matching score is selected as the output of this stage, denoted as the coarse localization pose.

[0073] In the second stage, using the coarse positioning pose as the initial value, a local optimization algorithm is used for fine iterative optimization until convergence, obtaining the optimal registration pose and achieving final alignment at the thermal imaging resolution level.

[0074] In this stage, the complete matching function is calculated, and iterative fine-tuning is performed using an efficient local optimization algorithm based on gradient information (such as quasi-Newton methods (e.g., L-BFGS) or the Levenberg-Marquardt (LM) algorithm). This algorithm utilizes the local curvature information of the objective function S(Pose) to quickly and stably converge to the optimal solution within that local region. The optimization process continues until convergence is determined according to the standard criteria of the optimization algorithm (usually based on the improvement of the pose parameter increment or the objective function value being lower than a preset tolerance). At this point, the corresponding pose is output as the final optimal registration pose.

[0075] After obtaining the optimal registration pose and before performing temperature information mapping, the process also includes result verification and rollback steps:

[0076] If the overall matching score corresponding to the optimal registration pose is lower than the preset threshold, a backoff mechanism is triggered. It should be noted that this application does not limit the specific size of the preset threshold. The preset threshold is determined based on the statistical analysis of the scoring results of a large number of known correctly registered samples.

[0077] The backoff mechanism includes: generating multiple sets of dispersed initial poses in the pose space, performing coarse search in parallel, retaining multiple better poses to form a set, and performing fine optimization starting from the poses in this set, selecting the pose with the best matching score from all results as the new optimal registration pose; the backoff mechanism includes the following process:

[0078] Analyze the physical rationality of the convergence trajectory. Check whether the local structural thermal mode consistency score in the geometric consistency score item is abnormally low or whether the global thermal zone distribution in the temperature distribution consistency score item seriously violates the prior heat source direction. If the score is high but the key physical sub-items are abnormal, it is determined that it has fallen into a physically unreasonable local optimum.

[0079] Within a predefined pose prior constraint space, N sets (N≥5) of initial pose seeds that are far apart from each other (e.g., sampled using the maximum-minimum distance method) are generated simultaneously. Subsequently, a simplified first-stage (coarse localization) optimization is performed in parallel or serially to obtain N candidate poses.

[0080] The top M pose assumptions with the highest scores (e.g., M=3) are retained to form an elite set. Starting with each pose in this set, the complete second stage is performed to obtain M refined optimization results.

[0081] Select the pose with the highest comprehensive score from these M results as the final output of the new optimal registration pose for this backtracking, and then re-verify it.

[0082] If a pose that meets the conditions cannot be obtained after the preset maximum number of backtracking attempts (e.g., 3 times), the system determines that the registration has failed and reports a warning.

[0083] The thermal imaging and 3D model registration and temperature mapping method also includes the step of constructing a local thermal pattern template library;

[0084] The local thermal pattern template library is used to provide expected thermal pattern feature data for predefined key local structures in a 3D model. The expected thermal pattern feature data is obtained statistically based on historical calibration data and is used to assist in calculating at least one scoring item in the matching function.

[0085] The local thermal pattern template library is a pre-built knowledge base using historical calibration data, designed to describe the typical behavior of specific key local structures on a known 3D model in thermal imaging. These local structures include, but are not limited to, process-sensitive features such as gates, venting channels, and thin-walled regions. The template library establishes one or more thermal pattern templates for each predefined key local structure to adapt to changes in different viewpoints or process conditions. Each template entry contains the following information:

[0086] (1) Structural identifier: A unique identifier corresponding to a predefined local structure in a 3D model.

[0087] (2) Expected thermal feature vector: This vector is obtained by analyzing historical thermal imaging data from multiple frames at known correct registration poses. It is used to quantitatively describe the typical thermal performance of the local structure under standard process conditions. The expected thermal feature vector may include features such as the average temperature, temperature standard deviation, maximum temperature, principal direction of the temperature gradient, local contrast, thermal centroid coordinates, and low-frequency energy distribution of the Fourier spectrum of the temperature field (characterizing the smoothness of the overall thermal distribution). The principal direction of the gradient should correspond to the heat dissipation path or material interface direction of the local structure.

[0088] (3) Feature statistics: The template may contain the statistical properties of the expected hot feature vector (such as mean, covariance matrix) to be used to calculate robustness measures such as Mahalanobis distance during matching.

[0089] (4) Process importance weight: The weight coefficient is pre-set according to the process criticality of the local structure and is used for weighted fusion in the comprehensive score.

[0090] The geometric consistency score is comprehensively evaluated based on the overall contour matching degree between the projection of the 3D model and the thermal imaging data, as well as the thermal pattern matching degree of key local structures; among which, the thermal pattern matching degree of key local structures is calculated based on a local thermal pattern template library.

[0091] The geometric consistency score is used to evaluate the degree of matching between the 3D model projection and the thermal imaging data under the current assumed pose, based on geometric constraints and thermophysical priors.

[0092] This scoring criterion is implemented through multi-level structure matching, including global contour matching and local feature matching. In global contour matching, the visible region mask of the 3D model in the current pose is calculated. Contour mask of thermal image foreground region extracted from thermal imaging The geometric similarity between Sim( , ); wherein, the geometric similarity measure includes at least one of the intersection-union ratio (IoU), the Dice coefficient, or the overlap area ratio.

[0093] In local feature matching, at least one key local region visible from the current viewpoint is determined based on the predefined key local structures and their semantic information in the 3D model. For each visible key local region, the corresponding expected thermal pattern features are obtained from the local thermal pattern template library based on its structural identifier, and the observed thermal pattern features are extracted in the corresponding projection area of ​​the thermal image. The matching degree score of the local region is obtained by calculating the similarity between the expected thermal pattern features and the observed thermal pattern features. The similarity calculation method includes at least one of cosine similarity or Mahalanobis distance.

[0094] The geometric consistency score is calculated using the following formula:

[0095] .

[0096] Where Sim(·) represents the global contour similarity measure (such as IoU) between the model projection mask and the thermal image foreground region mask. This represents the matching score of the k-th key local region. The process importance weights corresponding to the k-th local region are derived from the template library. It is a dynamic or static weighting coefficient between 0 and 1, used to balance the contribution ratio of overall contour matching and local feature matching to the total score. A larger value can be set during the initial registration stage or when the global contour is clear. This value enhances overall guidance; it can be reduced during the later stages of registration or when fine alignment is required. The value serves to highlight local features.

[0097] By providing initial spatial constraints through overall contour similarity and combining thermal pattern matching of key local regions for detail calibration, this scoring item can achieve stable evaluation and optimization guidance of the pose relationship between the 3D model and the thermal image even when the global contour of the thermal image is blurred or the local features are significant.

[0098] The edge physical consistency score is calculated by measuring the spatial correlation between the projected geometric edges of the 3D model and the significant temperature gradient feature points extracted from the thermal imaging data.

[0099] The edge physical consistency score is used to assess the spatial correlation between the projected edges of the 3D model and the physical evidence from thermal imaging, based on the temperature gradient field.

[0100] To address the edge blurring issue caused by thermal diffusion in thermal imaging, this scoring item is implemented as follows: First, the set of projected geometric edge points of the 3D model is obtained based on the current assumed pose. Secondly, a set of significant thermal gradient feature points is extracted from the thermal imaging temperature matrix. Each feature point Includes its position coordinates and corresponding gradient magnitude .

[0101] For edge point set Every point in Find its spatial nearest neighbor in the gradient feature point set Q. And calculate the correlation contribution value of the point pair according to the following formula:

[0102] .

[0103] in, This is the preset distance tolerance parameter.

[0104] This calculation method uses an exponential term. A soft distance constraint is implemented, allowing model edges to be associated with gradient features within a finite neighborhood to accommodate geometric offsets caused by thermal diffusion; simultaneously, through the gradient magnitude term... By implementing evidence strength weighting, associations are more likely to occur in regions with significant temperature gradients.

[0105] To eliminate background interference, the evaluation scope was limited to the common observation region R. The final edge physical consistency score was obtained by summing the correlation contribution values ​​of all model edge points within this region and then normalizing them.

[0106] .

[0107] in, This represents the global maximum value in the thermal imaging gradient magnitude map.

[0108] By shifting the matching criteria from relying on morphologically variable isothermal contours to relying on temperature gradient fields that directly reflect differences in heat flow, this scoring term can more robustly establish spatial correspondence between the geometric edges of the 3D model and the physical evidence from thermal imaging, thereby improving the accuracy and stability of registration.

[0109] The temperature distribution consistency score is calculated by comparing the similarity between the macroscopic thermophysical statistical feature vector extracted from thermal imaging data and the simulation feature vector extracted from the model projected temperature map based on the current assumed pose.

[0110] The temperature distribution consistency score is used to evaluate the consistency between the expected thermophysical state determined by the 3D model and the macroscopic temperature distribution pattern actually observed in thermal imaging under the current assumed pose.

[0111] To address the complexity of the temperature field in thermal imaging caused by local hot spots, interference from irrelevant heat sources, and thermal diffusion effects, this scoring item is implemented through the following steps: First, based on the thermal imaging temperature matrix, a set of normalized macroscopic thermophysical statistical features are extracted within the effective observation area to form an observation feature vector. Simultaneously, the surface of the 3D model is projected onto the thermal imaging plane based on the current pose, and the projected temperature map of the model is obtained through sampling. Simulation feature vectors were extracted in this region using the same method. .

[0112] To quantify global pattern consistency, the temperature distribution consistency score is calculated using the following formula. :

[0113] .

[0114] in, Representing the observed eigenvectors and simulated eigenvectors The cosine similarity between them has a range of . This metric focuses on the directional consistency of feature vectors in multidimensional space, is insensitive to changes in absolute numerical scale, and can capture the relative pattern similarity of temperature distribution; it maps them to the intuitive [0, 1] interval through a linear transformation (1 / 2)(· + 1). Representing the observed eigenvectors and simulated eigenvectors Mahalanobis distance between them This is the preset attenuation coefficient.

[0115] This scoring formula, by integrating cosine similarity and Mahalanobis distance, takes into account both the shape similarity of temperature distribution patterns and the statistical closeness of eigenvalues. This allows for a robust assessment of the consistency of the overall thermophysical pattern and provides robustness against local outliers or overall scaling.

[0116] By introducing this macroscopic model consistency assessment, pose assumptions that significantly contradict the observation data in terms of overall thermophysical characteristics can be effectively eliminated during the optimization process, and the pose can be guided to converge in a direction that conforms to thermodynamic laws and process priors, thereby ensuring the physical rationality and reliability of the registration results.

[0117] Regarding the above embodiments, this application provides an electronic device; please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The computer device includes a memory and a processor, wherein the memory and the processor are coupled to each other, the memory stores program data, and the processor is used to execute the program data to implement the steps of any embodiment of the above-described thermal imaging and three-dimensional model registration and temperature mapping method.

[0118] In this embodiment, the processor may also be referred to as a CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for thermal imaging registration with a 3D model and temperature mapping, characterized in that, include: Acquire a known three-dimensional model of the target under test and a single thermal imaging data collected for the target under test; A physics-driven matching function is constructed to quantitatively evaluate the degree of matching between the projection information of the 3D model and the thermal imaging data under any assumed pose. The matching function is constructed collaboratively based on three types of physical constraints: geometric consistency, edge physical consistency, and temperature distribution consistency. Based on the matching function, an optimization search is performed in the pose space of the three-dimensional model to obtain the optimal registration pose. Based on the optimal registration pose, the temperature information in the thermal imaging data is mapped onto the surface of the three-dimensional model to generate a three-dimensional digital model with temperature field information. The matching function is a comprehensive scoring function obtained by fusing geometric structure consistency scoring items, edge physical consistency scoring items, and temperature distribution consistency scoring items in a weighted manner; wherein, the weight coefficient of each scoring item can be dynamically adjusted according to at least one of the following factors: optimization stage, prior knowledge certainty, or data quality. An adaptive weighting strategy is used to dynamically adjust the weight coefficients of each scoring item, including: In the coarse positioning optimization stage, the geometric structure consistency scoring item is assigned the highest weight; During the fine-tuning phase, the edge physical consistency scoring item is assigned the highest weight; When the 3D model lacks predefined semantic labels, the weight of the temperature distribution consistency score item is increased.

2. The method according to claim 1, characterized in that, The optimization search employs a two-stage strategy, from coarse to fine, including: In the first stage, the pose search space is constrained based on physical priors, and a global optimization algorithm is used to perform a coarse search within the constrained space to obtain the coarse localization pose. In the second stage, using the coarse localization pose as the initial value, a local optimization algorithm is used for fine iterative optimization until convergence, thereby obtaining the optimal registration pose.

3. The method according to claim 1, characterized in that, After obtaining the optimal registration pose and before performing the temperature information mapping, the process also includes result verification and rollback steps: If the comprehensive matching score corresponding to the optimal registration pose is lower than a preset threshold, a rollback mechanism is triggered. The backoff mechanism includes: generating multiple sets of dispersed initial poses in the pose space, performing coarse search in parallel, retaining multiple better poses to form a set, and performing fine optimization starting from the poses in the set, and selecting the pose with the best matching score from all results as the new optimal registration pose.

4. The method according to claim 1, characterized in that, It also includes the step of building a local thermal pattern template library; The local thermal pattern template library is used to provide expected thermal pattern feature data for predefined key local structures in the 3D model. The expected thermal pattern feature data is obtained based on historical calibration data and is used to assist in calculating at least one scoring item in the matching function.

5. The method according to claim 4, characterized in that, The geometric consistency score is comprehensively evaluated based on the overall contour matching degree between the projection of the 3D model and the thermal imaging data, as well as the thermal pattern matching degree of key local structures; wherein, the thermal pattern matching degree of key local structures is calculated based on the local thermal pattern template library.

6. The method according to claim 1, characterized in that, The edge physical consistency score is calculated by measuring the spatial correlation between the projected geometric edges of the 3D model and significant temperature gradient feature points extracted from the thermal imaging data.

7. The method according to claim 1, characterized in that, The temperature distribution consistency score is calculated by comparing the similarity between the macroscopic thermophysical statistical feature vector extracted from the thermal imaging data and the simulation feature vector extracted from the model projected temperature map based on the current assumed pose.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

Images