A camera pose optimization method and system for intelligent space scenarios
By constructing a 3D Gaussian model and iteratively adjusting the pose parameters, the accuracy and stability issues of camera pose optimization in industrial scenarios with approximate structures were solved, improving the consistency between virtual images and actual photographs and enhancing the reliability of spatial intelligent tasks.
Patent Information
- Application Number
- CN202611381377.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2046-09-08
AI Technical Summary
In industrial scenarios with approximate structures, existing camera pose optimization methods lack accuracy and stability, leading to deviations in the correspondence between the actual captured images and the 3D scene. This affects the alignment of virtual and real images and the reliability of spatial intelligent tasks.
By constructing a 3D Gaussian model, rendering virtual images using initial pose parameters, and extracting anchor point sets in overlapping areas, the pose parameters are iteratively adjusted to optimize camera pose by combining the spatial layout relationship of the workshop and the gradient direction of the image. By using color and marker contour differences to identify areas of abrupt changes in appearance, candidate pose parameters are corrected to ensure the consistency between the image and the 3D scene.
It improves the accuracy and stability of camera pose optimization, reduces the impact of local appearance changes on the alignment of virtual and real images, improves the correspondence between virtual images and actual photos, and enhances the reliability of tasks in spatial intelligent scenarios.
Smart Images

Figure CN122906227B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer vision technology, and in particular relates to a camera pose optimization method and system for spatial intelligent scenarios. Background Technology
[0002] Camera pose optimization is a fundamental method for achieving accurate correspondence between actual images and 3D scenes, virtual scenes, or spatial coordinate data in spatial intelligence scenarios. It typically estimates the camera's translation and rotation parameters through methods such as image feature matching, structural edge matching, reference object recognition, or error iterative optimization. This method can be applied to scenarios such as digital twin workshops, industrial visual inspection, equipment positioning, path planning, and spatial interaction, providing fundamental data support for subsequent spatial intelligence tasks.
[0003] Existing camera pose optimization methods typically utilize image features, semantic map information, depth information, or motion constraint information from keyframes to establish a correspondence between the image and the 3D scene, and iteratively adjust the camera pose based on the matching error or loss value. These methods can improve the alignment accuracy between the captured image and the 3D scene when the scene structure has significant discriminative power and the effective observation range is relatively complete.
[0004] However, in large stamping workshops, assembly workshops, and other industrial settings, the similarity in workshop layout and equipment structure can lead to similar image observation results at different spatial locations. When there are deviations in the initial camera pose, incomplete fields of view, or insufficient effective difference information, the correspondence between the actual captured image and the 3D scene is prone to deviation, thus affecting the alignment of the virtual and real images and the reliability of subsequent spatial intelligent tasks. Therefore, existing technologies suffer from insufficient accuracy and stability in camera pose optimization under industrial scenarios with similar structures. Summary of the Invention
[0005] The purpose of this application is to provide a camera pose optimization method, system, electronic device and storage medium for spatial intelligent scenarios, so as to solve the problem of insufficient accuracy and stability of camera pose optimization in industrial scenarios with similar structures in the prior art.
[0006] To address the aforementioned technical problems, in a first aspect, this application provides a camera pose optimization method for spatial intelligence scenarios, comprising: Multiple on-site photos from different angles in the workshop and actual photos taken by the camera to be calibrated were acquired. A three-dimensional Gaussian model including three-dimensional Gaussian spatial primitives with spatial coordinates and texture gradient directions was constructed. The three-dimensional Gaussian model was used for three-dimensional Gaussian sputtering rendering. The virtual image is rendered using an initial pose parameter that includes 3D translation and 3D rotation. When the overlapping area of the virtual image and the actual photo is greater than the area of the actual photo by a preset overlap area ratio threshold, the first anchor point set and the second anchor point set corresponding to each other are extracted from the overlapping area and the pose parameter pair is solved based on the projection relationship between the two. The pose parameter pair includes two candidate pose parameters, each composed of a three-dimensional translation and a three-dimensional rotation. Based on the mirrored equipment areas on both sides of the workshop, the symmetrical reference plane and asymmetrical area of the workshop are determined. The three-dimensional Gaussian space primitives in the asymmetrical area are mapped to the two-dimensional coordinate space of the actual photo using two candidate pose parameters, and a directional distribution map recording the projection position and projection direction is constructed. The candidate pose parameters corresponding to the direction distribution map with the smallest overall deviation from the gradient direction of the image region corresponding to the actual photo are determined as the target pose parameters. Based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, the target pose parameters are iteratively adjusted until the convergence condition is met, and the optimized pose parameters are obtained.
[0007] In one feasible implementation, before mapping the three-dimensional Gaussian space primitives in the asymmetric region to the two-dimensional coordinate space of the actual photograph using two candidate pose parameters, the method further includes: Based on the differences in color distribution and marker outlines between the actual photograph and the virtual image within the overlapping area, areas of abrupt appearance change are identified. Based on the directional deviation between the image gradient direction of the actual photo within the appearance mutation region and the texture gradient direction of the three-dimensional Gaussian space primitives projected within the appearance mutation region, the two candidate pose parameters in the pose parameter pair are corrected respectively to obtain the corrected pose parameter pair.
[0008] In one feasible implementation, regions of abrupt appearance change are determined based on differences in color distribution and marker outlines between the actual photograph and the virtual image within the overlapping region, including: Divide the overlapping region into multiple comparison blocks; For each comparison block, the pixel percentage of the actual photo and the virtual image in each preset color interval within the comparison block is calculated, and the absolute values of the difference between the pixel percentage of the actual photo and the virtual image in the same preset color interval are summed to obtain the color histogram difference. Extract the labeled edges of the actual photo and the virtual image within the comparison block; calculate the number of non-overlapping edge pixels between the labeled edges of the actual photo and the labeled edges of the virtual image; The ratio of the number of non-overlapping edge pixels to the sum of the number of labeled edge pixels in the actual photo and the number of labeled edge pixels in the virtual image is defined as the label contour difference. If the color histogram difference is greater than the preset color difference threshold, and the identifier outline difference is greater than the preset identifier difference threshold, then the comparison block is identified as an appearance change region.
[0009] In one feasible implementation, based on the directional deviation between the image gradient direction of the actual photograph within the abrupt change region and the texture gradient direction of the 3D Gaussian space primitives projected within the abrupt change region, the two candidate pose parameters in the pose parameter pair are corrected respectively, resulting in the corrected pose parameter pair, including: The pose correction weights corresponding to the color distribution differences in the appearance mutation area and the pose correction weights corresponding to the differences in the marker contour are both set to preset weights, which are less than the pose correction weights of the color features and marker contour features in the non-appearance mutation area. For each candidate pose parameter, the three-dimensional Gaussian space primitives are mapped to the two-dimensional coordinate space of the actual image according to the candidate pose parameter; Select a three-dimensional Gaussian space primitive whose projection position is located within the appearance abrupt change region, and calculate the directional deviation between the texture gradient direction of the selected three-dimensional Gaussian space primitive and the image gradient direction at the projection position. Preset translation changes and preset rotation changes are applied to the three-dimensional translation of the candidate pose parameters, respectively, to obtain the adjusted candidate pose parameters. The adjusted candidate pose parameters whose directional deviation is less than that of the original candidate pose parameters are determined as the corrected candidate pose parameters, forming a corrected pose parameter pair.
[0010] In one feasible implementation, extracting corresponding first and second anchor point sets from the overlapping region and solving for pose parameter pairs based on their projection relationship includes: In the overlapping area of the actual photo, the corner points of the device outline are extracted by corner point detection, and the intersection points of the device structure are extracted by detecting the intersection of the edge lines of the device structure. The corner points of the device outline and the intersection points of the device structure are combined to form the first anchor point set. In the overlapping area of the virtual image, extract the virtual device contour corner points corresponding to the device contour corner points in the first anchor point set, and the virtual device structure intersection points corresponding to the device structure intersection points in the first anchor point set, and form the virtual device contour corner points and virtual device structure intersection points into a second anchor point set. Determine the spatial coordinates of the three-dimensional Gaussian space primitive corresponding to each virtual anchor point in the second anchor point set and use them as the three-dimensional anchor point coordinates; The pixel coordinates of the first anchor point corresponding to each 3D anchor point coordinate are used as the 2D anchor point coordinates. Based on the correspondence between the 3D anchor point coordinates and the projected coordinates of the 2D anchor point coordinates, the pose parameter pairs are solved.
[0011] In one feasible implementation, determining the symmetrical reference plane and asymmetrical region of the workshop based on the mirrored equipment arrangement areas on both sides of the workshop includes: The symmetry reference plane of the workshop is determined by the mirror symmetry plane between the equipment areas arranged in mirror on both sides of the workshop. Using the workshop symmetry reference plane as a reference, a local region is determined in the three-dimensional Gaussian model, and a mirror region is obtained by mirroring the local region with respect to the workshop symmetry reference plane. The number of 3D Gaussian space primitives in the local region and the number of 3D Gaussian space primitives in the mirror region are counted separately to obtain the number of local primitives and the number of mirror primitives; the absolute value of the difference between the number of local primitives and the number of mirror primitives is determined as the difference in the number of primitives. Calculate the angle between the texture gradient direction of the 3D Gaussian space primitive in the local region and the 3D Gaussian space primitive in the mirror region whose spatial coordinates are symmetrical about the workshop symmetry reference plane, and average all the angles to obtain the average angle of the texture gradient direction. If the difference in the number of primitives is greater than a preset number threshold, and the average included angle is greater than a preset angle threshold, then the local area is determined as an asymmetric region.
[0012] In one feasible implementation, three-dimensional Gaussian spatial primitives within the asymmetric region are mapped to the two-dimensional coordinate space of the actual photograph using two candidate pose parameters, respectively, to construct a direction distribution map recording the projection position and projection direction, including: The two-dimensional coordinate space of the actual photo is divided into multiple grid units according to the preset grid scale; For each candidate pose parameter, the direction of the texture gradient of the mapped 3D Gaussian space primitive in the 2D coordinate space is taken as the projection direction, and the projection direction is filled into the mesh cell where the projection position is located. The grid direction value of the grid cell is obtained by averaging the unit direction vectors of each projection direction within the same grid cell. Based on the projection position of each 3D Gaussian space primitive and the mesh orientation value of its corresponding mesh element, an orientation distribution map corresponding to the candidate pose parameters is constructed.
[0013] In one feasible implementation, the candidate pose parameters corresponding to the direction distribution map that minimizes the overall deviation from the gradient direction of the region corresponding to the actual photograph are determined as the target pose parameters, including: For each candidate pose parameter, determine the image gradient direction of the same grid cell in the actual image; Calculate the angle difference between the grid orientation value of each grid cell in the orientation distribution map corresponding to the candidate pose parameters and the image gradient direction of the same grid cell in the actual photo; The average of the angle differences corresponding to each candidate pose parameter is calculated to obtain the overall deviation of each candidate pose parameter. The candidate pose parameter with the smallest overall deviation is determined as the target pose parameter.
[0014] In one feasible implementation, the convergence condition is that the absolute value of the difference between two adjacent reprojection residuals is less than a preset convergence threshold. Based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, the target pose parameters are iteratively adjusted until the convergence condition is met, resulting in optimized pose parameters, including: Calculate the pixel coordinate distance between each anchor point in the first anchor point set and the matching anchor point in the updated second anchor point set, and determine the reprojection residual based on the average value of all pixel coordinate distances; A positive and negative preset translational perturbation is applied to the three-dimensional translation in the target pose parameters, and a positive and negative preset rotational perturbation is applied to the three-dimensional rotation in the target pose parameters to obtain multiple perturbation pose parameters. The perturbation pose parameter whose reprojection residual after rendering a virtual image using a 3D Gaussian model driven by the perturbation pose parameter is less than the reprojection residual corresponding to the target pose parameter is selected as the new target pose parameter. The virtual image is re-rendered and updated based on the new target pose parameters until the convergence condition is met.
[0015] Secondly, this application provides a camera pose optimization system for spatial intelligent scenarios, including: The acquisition module is used to acquire multiple on-site photos from different angles in the workshop and actual photos taken by the camera to be calibrated, and to construct a three-dimensional Gaussian model including three-dimensional Gaussian spatial primitives with spatial coordinates and texture gradient directions. The three-dimensional Gaussian model is used for three-dimensional Gaussian sputtering rendering. The rendering module is used to drive the rendering of virtual images of a 3D Gaussian model with initial pose parameters including 3D translation and 3D rotation. The solution module is used to extract the corresponding first anchor point set and second anchor point set from the overlapping area when the area ratio of the overlapping area of the virtual image and the actual photo is greater than the preset overlap area ratio threshold, and solve the pose parameter pair based on the projection relationship between the two. The pose parameter pair includes two candidate pose parameters, each composed of a three-dimensional translation and a three-dimensional rotation. The module is used to determine the symmetrical reference plane and asymmetrical region of the workshop based on the mirrored equipment arrangement areas on both sides of the workshop. It uses two candidate pose parameters to map the three-dimensional Gaussian space primitives in the asymmetrical region to the two-dimensional coordinate space of the actual photo, and constructs a direction distribution map that records the projection position and projection direction. The determination module is used to determine the candidate pose parameters corresponding to the direction distribution map that has the smallest overall deviation from the gradient direction of the image region corresponding to the actual photo as the target pose parameters. The determination module is also used to iteratively adjust the target pose parameters until the convergence condition is met based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, so as to obtain the optimized pose parameters.
[0016] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute computer programs to implement the steps of the camera pose optimization method for spatial intelligent scenarios as described in the first aspect above.
[0017] The camera pose optimization method for spatial intelligent scenes provided in this application constructs a 3D scene model that reflects the spatial structure and texture features of a workshop, and generates a virtual image corresponding to the actual photograph based on the initial pose. This allows for obtaining image correspondence information for pose solving even with effective overlapping areas, thereby forming candidate pose results. Addressing the situation where similar structures in industrial workshops may result in similar image observations at different locations, the method utilizes the spatial layout relationship of the workshop to determine scene regions with distinguishing functions. By comparing the projection direction information of this region under different candidate poses with the image direction information in the actual photograph, a pose result with a higher degree of correspondence to the actual shooting area can be determined from the candidate pose results. Iterative adjustments to the determined pose result based on the reprojection residuals between image anchor points gradually bring the correspondence between the virtual image and the actual photograph closer to consistency. Therefore, this application can improve the accuracy and stability of camera pose optimization in industrial workshops with approximate structures for spatial intelligent scenes, providing a foundation for accurate correspondence between actual images, virtual scenes, and spatial coordinate data.
[0018] Furthermore, by comparing the differences in color distribution and marker contours between actual photographs and virtual images within overlapping areas, this application can identify local areas in the workshop that are inconsistent with the existing 3D scene representation, thereby reflecting the impact of equipment status, on-site markers, or local environmental changes on image correspondence. Based on this, according to the directional deviation between the image gradient direction of the actual photograph and the texture gradient direction after the 3D scene primitives are projected into the local area, the candidate pose parameters are corrected to make the candidate pose more consistent with the local structural representation in the actual photograph. The corrected candidate pose provides a more reliable initial basis for subsequent pose discrimination and iterative optimization, reducing the impact of local appearance changes on the alignment effect between virtual and real images. Therefore, this application can improve the accuracy and stability of camera pose optimization when there are local appearance changes in the workshop scene under spatial intelligent scenarios, and improve the reliability of virtual-real spatial alignment. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A flowchart illustrating a camera pose optimization method for spatial intelligent scenarios provided in this application embodiment; Figure 2 A schematic diagram of a directional distribution map provided in an embodiment of this application; Figure 3 A flowchart illustrating an iterative method for determining optimized pose parameters provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a camera pose optimization system for intelligent spatial scenarios provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation
[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0022] To address the problems of existing technologies, embodiments of this application provide a camera pose optimization method, apparatus, device, computer storage medium, and computer program product for spatial intelligent scenarios. The camera pose optimization method for spatial intelligent scenarios provided in this application embodiment will be described first below.
[0023] Figure 1 This illustration shows a flowchart of a camera pose optimization method for spatial intelligent scenarios according to an embodiment of this application. Figure 1 As shown, the method includes steps S110 to S160.
[0024] S110. Acquire multiple on-site photos from different angles in the workshop and actual photos taken by the camera to be calibrated. Construct a three-dimensional Gaussian model including three-dimensional Gaussian spatial primitives with spatial coordinates and texture gradient directions. The three-dimensional Gaussian model is used for three-dimensional Gaussian sputtering rendering.
[0025] Spatial intelligent scene refers to an application environment that utilizes images, 3D scenes, and spatial coordinate data to achieve workshop environment perception, virtual-real alignment, and inspection or positioning tasks. Multiple on-site photos from different angles in the workshop are two-dimensional color images collected from multiple observation positions within the workshop, directed towards equipment, tracks, columns, and workstations. On-site photos from any adjacent observation positions have a common visible area that can be used for multi-view reconstruction.
[0026] The camera to be calibrated is an imaging device whose extrinsic parameters are to be determined and whose intrinsic parameters are known or can be obtained through calibration. The actual photograph is a two-dimensional color image acquired by the camera to be calibrated at the moment of positioning. The intrinsic parameter calibration and distortion correction of the camera to be calibrated are completed before pose optimization. The shooting pose corresponding to the field photograph, the spatial coordinates of the three-dimensional Gaussian space primitives, and the pose parameters of the camera to be calibrated are all represented using the same workshop world coordinate system. The workshop world coordinate system is established through fixed and known positioning points within the workshop, and the transformation relationship between the field photograph reconstruction coordinate system and the workshop world coordinate system is determined based on the positioning points.
[0027] A 3D Gaussian model is an explicit 3D representation of a workshop, composed of multiple 3D Gaussian spatial primitives. 3D Gaussian Splatting (3DGS) rendering refers to the process of projecting visible 3D Gaussian spatial primitives onto a 2D imaging plane according to a given pose, and then synthesizing pixel colors based on depth order and opacity. A 3D Gaussian spatial primitive is an anisotropic 3D Gaussian unit used to explicitly represent the local appearance and spatial distribution of a workshop. It can include spatial coordinates, covariance parameters, opacity, color parameters, and texture gradient direction. The spatial coordinates are the 3D position of the primitive's center in the workshop's world coordinate system, and the texture gradient direction is the unit direction vector of the corresponding local texture within the local tangent plane.
[0028] In one specific embodiment, a set of on-site photographs collected at multiple observation locations along the workshop aisle is formed, the first of which is a photographic set. Zhang's on-site photos are recorded as follows ,in These are the sequence numbers of the photos taken at the scene. The total number of photos taken at the scene is [number missing]. This ensures that each area to be modeled is covered by multiple on-site photos that share a common visible area.
[0029] First, examine the photos from each scene. Camera intrinsic parameter calibration, distortion correction, exposure normalization, feature matching, and geometric consistency screening were performed. Various scene photographs were obtained through multi-view triangulation and beam adjustment. Corresponding shooting position The sparse point cloud is initialized with multiple three-dimensional Gaussian spatial primitives. ,in The primitive number is used for each three-dimensional Gaussian space primitive. The local tangent plane is determined based on the neighboring point cloud, and the field photograph containing the projection position of the primitive is used. Extract the local image gradient, map the image gradient direction to a 3D direction vector in the camera coordinate system parallel to the imaging plane, and then determine the shooting pose. Transform to the workshop world coordinate system and project onto the local tangent plane. Then, perform a weighted average and normalization on multiple effective observation directions according to the image gradient magnitude to obtain the texture gradient direction. .
[0030] Next, a collection of photos from the scene. The data is used as supervisory data for 3D Gaussian sputtering rendering, and the spatial coordinates are updated based on the color difference between the rendered image and the corresponding on-site photograph. The covariance parameter, opacity, and color parameter are updated, and the texture gradient direction is recalculated after the primitive parameters are updated. A three-dimensional Gaussian model is formed; the camera to be calibrated takes actual photos in the workshop, and its intrinsic parameters are obtained through pre-calibration. The shooting poses and spatial coordinates of the three-dimensional Gaussian spatial primitives corresponding to the on-site photos are unified to the workshop world coordinate system.
[0031] Each round of primitive parameter updates uses the color difference between the rendered image and the corresponding on-site photograph as the optimization criterion. The reserved verification photograph in the on-site photograph is not included in the current round of parameter updates; it is only used to compare the rendering color difference before and after the parameter update. Primitive parameter updates stop when the color difference of the verification photograph does not decrease in three consecutive rounds of parameter updates. After each update, the texture gradient direction is recalculated according to the updated spatial coordinates and local tangent plane.
[0032] S120. Drive the three-dimensional Gaussian model to render virtual images using initial pose parameters including three-dimensional translation and three-dimensional rotation. Initial pose parameters refer to the initial six-degree-of-freedom spatial transformation parameters of the camera to be calibrated relative to the workshop world coordinate system, which may include initial three-dimensional translation. and initial three-dimensional rotation Initial 3D translation This is the translation offset during the transformation from workshop world coordinates to the coordinates of the camera to be calibrated. It has three components, and its length unit is consistent with the spatial coordinate unit of the three-dimensional Gaussian space primitive. Initial three-dimensional rotation amount. This is a rotation vector, whose direction represents the rotation axis direction, and whose magnitude represents the rotation angle, with the angle unit being radians; the initial three-dimensional rotation amount Transform into the initial rotation matrix using the Rodrigues transform. Initial rotation matrix Used to transform coordinates in the workshop world coordinate system to the coordinate system of the camera to be calibrated.
[0033] Virtual images refer to images based on the intrinsic parameter matrix of the camera to be calibrated. The initial pose parameters are used to render a 2D color image from a 3D Gaussian model using 3D Gaussian sputtering. The pixel size, principal point position, and pixel coordinate orientation of the virtual image are consistent with those of the actual photograph. The initial pose parameters can be the approximate spatial position and orientation of the camera to be calibrated, so that the virtual image and the actual photograph form an overlapping area.
[0034] In one specific embodiment, the pre-calibrated intrinsic parameter matrix is obtained. In addition to the actual photographs taken by the camera to be calibrated, based on the installation location record of the camera to be calibrated, preliminary positioning results, or an approximate observation position determined manually, the initial three-dimensional translation amount when transforming the workshop world coordinates to the coordinates of the camera to be calibrated is obtained. Based on the approximate orientation of the optical axis of the camera to be calibrated and the vertical direction of the image, the initial three-dimensional rotation is obtained. And transform to obtain the initial rotation matrix. .
[0035] For each three-dimensional Gaussian space primitive spatial coordinates according to Transform to the coordinate system of the camera to be calibrated, where Three-dimensional Gaussian space primitive Camera coordinates; using intrinsic parameter matrix Camera coordinates Perform perspective projection to obtain the two-dimensional projection position. and projection depth Three-dimensional Gaussian space primitives with a projection depth of no more than zero or whose two-dimensional projection positions are outside the imaging range are removed. The remaining three-dimensional Gaussian space primitives are sorted from near to far according to their projection depth, and pixel-by-pixel color synthesis is performed based on the projection covariance parameter, opacity, and color parameters to obtain a virtual image. At the same time, the index of the three-dimensional Gaussian space primitive and the two-dimensional projection position of each pixel are recorded. For example, when the camera to be calibrated is installed on one side of the workshop aisle and faces the equipment area A, the approximate translation amount corresponding to the installation position and the initial rotation amount corresponding to the optical axis orientation can be used as the initial pose parameters to render a virtual image containing the equipment area A and adjacent columns.
[0036] S130. When the area of the overlapping region between the virtual image and the actual photo is greater than the area of the actual photo by a preset overlap area ratio threshold, extract the corresponding first anchor point set and second anchor point set from the overlapping region and solve the pose parameter pair based on their projection relationship. The pose parameter pair includes two candidate pose parameters, each composed of a three-dimensional translation and a three-dimensional rotation.
[0037] The overlapping region refers to the effective pixel area corresponding to the edge of the same device, column, or structure in the virtual image and the actual photograph. The overlap area ratio is the proportion of pixels in the overlapping region to the total number of pixels in the actual photograph. The preset overlap area ratio threshold can be determined based on calibration images at different installation locations to be the minimum area ratio that can stably detect a preset number of non-collinear device / structure anchor points. The pose parameter pair includes the first candidate 3D translation amount. First candidate 3D rotation amount Second candidate three-dimensional translation Second candidate three-dimensional rotation amount .
[0038] In one feasible implementation, step S130, which involves extracting corresponding first and second anchor point sets from the overlapping region and solving for pose parameter pairs based on their projection relationship, includes: In the overlapping area of the actual photo, the corner points of the device outline are extracted by corner point detection, and the intersection points of the device structure are extracted by detecting the intersection of the edge lines of the device structure. The corner points of the device outline and the intersection points of the device structure are combined to form the first anchor point set. The corner point of the device outline refers to the pixel position where two or more edge directions intersect or make a significant turn in the outer outline of the device. It can be the corner of the outer frame, the turning point of the column edge, or the turning point of the panel edge.
[0039] The intersection of equipment structures refers to the location where the edges of different equipment structures meet in an actual photograph; it can be the intersection of the edge line of a column and the edge line of a beam. The first anchor point set is a two-dimensional feature point set composed of equipment outline corner points and equipment structure intersections. The pixel coordinates of the first anchor point are defined as two-dimensional anchor point coordinates. subscript This is the anchor point number.
[0040] In practice, the actual photos within the overlapping area are converted to grayscale and noise is suppressed. The minimum eigenvalue corner detection method is used to obtain the device contour corners; edge detection methods are used to obtain structural edges, and line detection methods are used to obtain edge segments. Edge segments with a direction difference less than a preset merging threshold and an endpoint spacing less than a preset connection threshold are merged. The intersection positions of edge segments in different directions are calculated to obtain the device structure intersection points. Feature points with local contrast below a preset contrast threshold, located at image boundaries, or with a distance from adjacent anchor points less than a preset spacing threshold are removed. The pixel coordinates, point type, and local texture description information of the retained feature points are written into the first anchor point set.
[0041] In the overlapping area of the virtual image, extract the virtual device contour corner points corresponding to the device contour corner points in the first anchor point set, and the virtual device structure intersection points corresponding to the device structure intersection points in the first anchor point set, and form the virtual device contour corner points and virtual device structure intersection points into a second anchor point set. Virtual device outline corner points refer to the corner points of the same entity structure in the virtual image that correspond to the virtual device outline corner points. Virtual device structure intersection points refer to the intersection points of the same entity structures in the virtual image that correspond to the virtual device structure intersection points. The second anchor point set is a two-dimensional feature point set composed of virtual device outline corner points and virtual device structure intersection points. The pixel coordinates of the second anchor point are used as virtual anchor point coordinates. Two-dimensional anchor point coordinates With virtual anchor point coordinates This indicates the two-dimensional observation position of the same physical structure in both actual photographs and virtual images.
[0042] In practice, the same corner detection and edge intersection detection methods as those used in actual photographs are employed to extract the virtual device contour corners and virtual device structure intersections from the overlapping areas of the virtual images. Based on point type consistency, local texture description similarity, and pixel distances corresponding to the initial pose parameters, the coordinates of the two-dimensional anchor points are determined. With virtual anchor point coordinates Perform bidirectional matching. Retain anchor point pairs whose forward and reverse matching results are consistent, whose description similarity is greater than a preset description threshold, and whose pixel distance is less than a preset distance threshold; then remove anchor point pairs that do not satisfy the local geometric relationship through a random sampling consistency method to form a second anchor point set.
[0043] Determine the spatial coordinates of the three-dimensional Gaussian space primitive corresponding to each virtual anchor point in the second anchor point set and use them as the three-dimensional anchor point coordinates; 3D anchor point coordinates This refers to the coordinates of the virtual anchor point. The corresponding spatial coordinates of the center of the 3D Gaussian space primitive in the workshop world coordinate system. The major contributing 3D Gaussian space primitive refers to the 3D Gaussian space primitive with the largest cumulative contribution weight to the pixel color within the neighborhood of the virtual anchor point. The cumulative contribution weight is the weighted result of the opacity contribution and color contribution of the 3D Gaussian space primitive at each neighboring pixel.
[0044] In practice, the main contributing 3D Gaussian space primitive index and 2D projection position of each pixel in the virtual image are read, and the virtual anchor point coordinates are used. The neighborhood pixels are determined around the center, and the contribution weights of each 3D Gaussian space primitive within the neighborhood pixels are accumulated. The index of the 3D Gaussian space primitive with the largest accumulated contribution weight is determined as the main contribution index. The main contribution index Corresponding spatial coordinates Determined as three-dimensional anchor point coordinates When the difference between the largest cumulative contribution weight and the second largest cumulative contribution weight is less than the preset weight difference threshold, the corresponding virtual anchor point is removed to avoid spatial coordinate uncertainty caused by occlusion of boundaries or mixed regions of multiple primitives.
[0045] The pixel coordinates of the first anchor point corresponding to each 3D anchor point coordinate are used as the 2D anchor point coordinates. Based on the correspondence between the 3D anchor point coordinates and the projected coordinates of the 2D anchor point coordinates, the pose parameter pairs are solved.
[0046] The projection coordinate correspondence refers to the geometric relationship between the 3D anchor point coordinates, after being transformed to the coordinate system of the camera to be calibrated and subjected to perspective projection through candidate pose parameters, and the corresponding 2D anchor point coordinates. For the first or second candidate pose parameter, the candidate rotation matrix is obtained by transforming the candidate 3D rotation amount, and combined with the candidate 3D translation amount and intrinsic parameter matrix, the 2D predicted coordinates corresponding to the 3D anchor point coordinates are calculated.
[0047] No. The average reprojection residual of each candidate pose parameter is denoted as . , can be represented as: Among them, the number of effective anchor points The number of corresponding 2D and 3D anchor point coordinates used in pose determination, and the 2D predicted coordinates. 3D anchor point coordinates According to the The pixel coordinates obtained by projecting the candidate pose parameters, and the candidate number. for or Mean reprojection residual This represents the average pixel coordinate distance for all valid anchor point pairs.
[0048] In practice, the smallest set of non-collinear 3D anchor point coordinates is selected from the effective anchor point pairs. A perspective three-point pose calculation method is used to generate multiple pose hypotheses that satisfy the orthographic projection depth condition. For each pose hypothesis, the average reprojection residual and the number of inliers are calculated using all effective anchor point pairs. Inliers are anchor point pairs whose pixel coordinate distance is less than a preset pixel error threshold. The pose hypotheses are sorted according to the number of inliers (from most to least) and the average reprojection residual (from smallest to largest). Two pose hypotheses with a high number of inliers and a small average reprojection residual are selected to obtain the first candidate 3D translation. First candidate 3D rotation amount Second candidate three-dimensional translation Second candidate three-dimensional rotation amount For example, when there are pairs of structures with similar shapes in device region A, different minimum anchor point groups can form two pose assumptions that both satisfy pixel error constraints, and the two pose assumptions together constitute a pose parameter pair.
[0049] In one feasible implementation, before mapping the three-dimensional Gaussian space primitives in the asymmetric region to the two-dimensional coordinate space of the actual photograph using two candidate pose parameters in step S140, the method further includes: S131. Based on the differences in color distribution and marker outline between the actual photograph and the virtual image within the overlapping area, determine the areas of abrupt changes in appearance.
[0050] S132. Based on the directional deviation between the image gradient direction of the actual photo within the appearance mutation region and the texture gradient direction of the three-dimensional Gaussian space primitives whose projection positions are located within the appearance mutation region, correct the two candidate pose parameters in the pose parameter pair respectively to obtain the corrected pose parameter pair.
[0051] Abrupt appearance changes refer to image regions within an overlapping area where the actual photograph and the virtual image exhibit significant differences in local color distribution and marker outlines. These differences can be caused by device surface updates, temporary occlusions, marker replacements, or changes in local lighting. Color distribution differences represent the color composition differences between two images within the same comparison block, while marker outline differences represent the degree of non-overlapping of marker edges in the two images. Pose correction weights refer to the degree of influence of image features when adjusting candidate pose parameters; the pose correction weights for abrupt appearance changes are less than those for non-abrupt appearance changes.
[0052] In one feasible implementation, step S131 determines the appearance abrupt change region based on the color distribution difference and marker outline difference between the actual photograph and the virtual image within the overlapping region, including: Divide the overlapping region into multiple comparison blocks; A comparison block refers to a local image unit obtained by dividing the overlapping area according to a preset grid scale. Each comparison block has the same pixel range in both the actual photograph and the virtual image. The preset grid scale can be determined based on the typical size of device logos, control panel graphics, or safety color bars in the image, enabling a single comparison block to cover the entire logo or local outline.
[0053] In practice, a two-dimensional grid is created based on the pixel size of the actual photograph. Grid cells within the overlapping area that are entirely within the effective pixel range are identified as comparison blocks. For grid cells located at the boundary of the overlapping area, the effective pixel percentage is calculated, and grid cells with an effective pixel percentage greater than a preset coverage threshold are retained as comparison blocks. The same block number is assigned to the same comparison block in both the actual photograph and the virtual image, enabling color features and marker edges to be compared in the same spatial location.
[0054] For each comparison block, the pixel percentage of the actual photo and the virtual image in each preset color interval within the comparison block is calculated, and the absolute values of the difference between the pixel percentage of the actual photo and the virtual image in the same preset color interval are summed to obtain the color histogram difference. A preset color range refers to the range of color values divided within a unified color space, which can be constructed based on preset segments of the luminance and chrominance components. The color histogram differences of the comparison blocks are denoted as... , representing the sum of the absolute values of the differences in pixel proportions between the actual photograph and the virtual image within each preset color range, can be expressed as: Among them, the comparison block number Indicates the current local image unit, color range index. This indicates the current preset color range and the total number of color ranges. This indicates the total number of preset color ranges, representing the actual percentage of pixels in the photo. Indicates the actual photo is in the [number]th position. Within the comparison block, it falls into the first... The percentage of pixels in each color range, the percentage of pixels in the virtual image. This indicates the percentage of corresponding pixels in a virtual image.
[0055] In practice, the actual photo and the virtual image are converted to the same color space, and the number of pixels in each comparison block is counted according to the preset color range to obtain the pixel ratio of the actual photo. and virtual image pixel ratio For each preset color range, the absolute difference between the proportions of two pixels is calculated and summed to obtain the color histogram difference. The preset color range can be determined based on the color distribution of the workshop calibration image, so that the equipment coating color, warning color, label background color and background color fall into different color ranges.
[0056] Extract the labeled edges of the actual photo and the virtual image within the comparison block; calculate the number of non-overlapping edge pixels between the labeled edges of the actual photo and the labeled edges of the virtual image; Label edges refer to the local edges formed between equipment nameplates, text graphics, warning signs, safety color bars, or operation markings and the surrounding background, excluding structural edges formed solely by the equipment's geometric contours. Non-coincident edge pixels refer to label edge pixels in the actual photograph and label edge pixels in the virtual image that do not form a matching edge pixel at the same coordinates in the vicinity.
[0057] In practice, local color contrast and closed region features are used to locate candidate regions for identification from the comparison blocks. Edge detection is then performed on these candidate regions to obtain the actual photo identification edges and the virtual image identification edges. Using a preset pixel distance as the edge matching tolerance, the nearest edge pixel in the virtual image is searched for each edge pixel in the actual photo identification edge; edge pixels with a nearest edge pixel distance greater than the preset pixel distance are identified as non-overlapping edge pixels in the actual photo. Similarly, for each edge pixel in the virtual image identification edge, the nearest edge pixel in the actual photo is searched; edge pixels with a nearest edge pixel distance greater than the preset pixel distance are identified as non-overlapping edge pixels in the virtual image. For continuous identifications such as safety color bars, adjacent edge pixels can be connected into edge segments first, and then the endpoint positions and orientations of the edge segments can be verified for consistency.
[0058] The ratio of the number of non-overlapping edge pixels to the sum of the number of labeled edge pixels in the actual photo and the number of labeled edge pixels in the virtual image is defined as the label contour difference. Difference in the identifier outline of the k-th comparison block This is the ratio of the sum of non-overlapping edge pixels in the actual photo and the virtual image to the sum of the number of labeled edge pixels in the actual photo and the virtual image. When no labeled edges are detected in either the actual photo or the virtual image within the comparison block, the outline difference is labeled. If only one image detects a marker edge, all marker edge pixels in that image are identified as non-overlapping edge pixels. Marker contour difference. The larger the value, the more obvious the difference between the actual photograph and the virtual image in terms of shape, location, or content.
[0059] In practice, the number of pixels representing the actual photo's edge, the number of pixels representing the virtual image's edge, and the number of pixels representing non-overlapping edges are counted separately. The number of pixels representing non-overlapping edges is then divided by the sum of the number of pixels representing the two edge markers to obtain the difference in the marker outline. When no marker edges are detected in either the actual photograph or the virtual image within the comparison block, the outline difference will be marked. The value is set to zero; when only one image detects a marker edge, all detected marker edge pixels are determined as non-overlapping edge pixels.
[0060] If the color histogram difference is greater than the preset color difference threshold, and the identifier outline difference is greater than the preset identifier difference threshold, then the comparison block is identified as an appearance change region.
[0061] The preset color difference threshold is determined based on the difference in color histograms between calibrated images without appearance changes. The preset marker difference threshold is determined based on the difference in marker outlines between calibrated images without appearance changes. Regions of abrupt appearance changes consist of comparison blocks where both the color histogram difference and the marker outline difference are greater than the preset marker difference threshold.
[0062] In practice, actual photographs and virtual images of equipment with stable appearances within the workshop are selected as calibration image pairs. The color histogram differences and marker outline differences for each comparison block are calculated. Based on the color histogram differences and marker outline differences for each comparison block in the stable calibration image pair, the mean and standard deviation of the corresponding differences are calculated. The preset color difference threshold is the mean of the color histogram differences plus twice the standard deviation, and the preset marker difference threshold is the mean of the marker outline differences plus twice the standard deviation. For each comparison block, the color histogram differences are... Differences in logo outline The comparison is compared with the corresponding threshold. If both conditions are met, the comparison block is identified as the appearance mutation region. Adjacent appearance mutation regions are connected to obtain the complete appearance mutation region.
[0063] In one feasible implementation, step S132 corrects the two candidate pose parameters in the pose parameter pair based on the directional deviation between the image gradient direction of the actual photograph within the appearance change region and the texture gradient direction of the three-dimensional Gaussian space primitives projected within the appearance change region, to obtain the corrected pose parameter pair, including: The pose correction weights corresponding to the color distribution differences in the appearance mutation area and the pose correction weights corresponding to the differences in the marker contour are both set to preset weights, which are less than the pose correction weights of the color features and marker contour features in the non-appearance mutation area. The pose correction weight corresponding to color distribution differences refers to the degree of influence of color features when adjusting candidate pose parameters, while the pose correction weight corresponding to marker contour differences refers to the degree of influence of marker contour features when adjusting candidate pose parameters. The comprehensive pose correction weights of the three-dimensional Gaussian spatial primitives are denoted as follows: It can be determined based on the comparison block where the three-dimensional Gaussian space primitive projection position is located.
[0064] In specific implementation, the same first preset weight is set for the weight corresponding to the color distribution difference and the weight corresponding to the mark outline difference in the appearance mutation area, and the same second preset weight is set for the color feature weight and the mark outline feature weight in the non-appearance mutation area, with the first preset weight being less than the second preset weight. Comprehensive pose correction weights for three-dimensional Gaussian space primitives The weights can be obtained by multiplying the color feature weights and the marker contour feature weights corresponding to the three-dimensional Gaussian space primitives, thus reducing the impact of color and marker changes in the appearance abrupt change region on pose adjustment. In this embodiment, the color feature weights and marker contour feature weights in the appearance abrupt change region can both be set to 0.25, while the color feature weights and marker contour feature weights in the non-appearance abrupt change region can both be set to 1.
[0065] For each candidate pose parameter, the three-dimensional Gaussian space primitives are mapped to the two-dimensional coordinate space of the actual image according to the candidate pose parameter; The projected position corresponding to the candidate pose parameter refers to the two-dimensional pixel position obtained after transforming the spatial coordinates of the 3D Gaussian space primitives to the coordinate system of the camera to be calibrated using candidate 3D translation and candidate 3D rotation, and then performing perspective projection using the intrinsic parameters of the camera to be calibrated. The first candidate pose parameter is determined by the first candidate 3D translation. and the first candidate 3D rotation amount The second candidate pose parameters are composed of the second candidate three-dimensional translation. Second candidate three-dimensional rotation amount constitute.
[0066] In specific implementation, the first candidate three-dimensional rotation amount will be... Second candidate three-dimensional rotation amount The coordinates of each 3D Gaussian space primitive are converted into the corresponding rotation matrix, and the spatial coordinates of each primitive are sequentially mapped to the coordinate system of the camera to be calibrated and the 2D coordinate space of the actual photograph. 3D Gaussian space primitives whose projection depth does not meet the orthographic projection conditions or whose projection position is outside the imaging range of the actual photograph are discarded, while 3D Gaussian space primitives whose projection position is located in the region of abrupt change in appearance are retained.
[0067] Select a three-dimensional Gaussian space primitive whose projection position is located within the appearance abrupt change region, and calculate the directional deviation between the texture gradient direction of the selected three-dimensional Gaussian space primitive and the image gradient direction at the projection position. The projection direction refers to the two-dimensional direction obtained by transforming the texture gradient direction of a 3D Gaussian space primitive through candidate pose parameters and projecting it onto the two-dimensional coordinate space of the actual photograph. The image gradient direction refers to the two-dimensional direction calculated in the actual photograph at the projection position of the 3D Gaussian space primitive. For the j-th 3D Gaussian space primitive, under the h-th candidate pose parameters, the center of the primitive and its adjacent spatial points offset along its texture gradient direction are projected onto the actual photograph, and the two-dimensional projection direction is determined based on the line connecting the two projection positions. The acute angle between the two-dimensional projection direction and the two-dimensional image gradient direction at the corresponding position in the actual photograph is determined as the direction deviation. The candidate number h is either a or b.
[0068] In practice, for each retained 3D Gaussian space primitive, the center of the 3D Gaussian space primitive and its neighboring spatial points offset along the texture gradient direction are projected. The direction of the line connecting the two 2D projection positions is determined as the projection direction. The image gradient direction at the corresponding projection position in the actual image is calculated using the local gradient operator. After normalizing the projection direction and the image gradient direction, the angle between them is calculated to obtain the direction deviation. When the angle between the projection direction and the image gradient direction exceeds a right angle, the angle is converted into a supplementary angle to eliminate the directional difference caused by the opposite directions of the same edge.
[0069] Preset translation changes and preset rotation changes are applied to the three-dimensional translation of the candidate pose parameters, respectively, to obtain the adjusted candidate pose parameters. The preset translational change refers to the positive or negative change applied to a single component of the candidate 3D translation, and the preset rotational change refers to the positive or negative change applied to a single component of the candidate 3D rotation. The adjusted candidate pose parameters are the pose parameters obtained by applying a preset change to a 3D translation component or a 3D rotation component based on the current candidate pose parameters.
[0070] In practice, a preset translational variation is determined based on the spatial resolution of the 3D Gaussian model, and a preset rotational variation is determined based on the field of view and image resolution of the camera to be calibrated. For the first and second candidate pose parameters, positive and negative preset translational variations are applied to the three translational components, and positive and negative preset rotational variations are applied to the three rotational components, resulting in multiple adjusted candidate pose parameters. For each adjusted candidate pose parameter, the 3D Gaussian spatial primitive mapping and orientation deviation calculation are re-executed.
[0071] The initial preset translation change is twice the median distance between the centers of adjacent 3D Gaussian spatial primitives in the 3D Gaussian model. The initial preset rotation change is a rotation that moves the projected position near the image center by two pixels. If no perturbation pose parameter with a smaller residual exists, both the preset translation and preset rotation changes are adjusted to half of their current values.
[0072] The adjusted candidate pose parameters whose directional deviation is less than that of the original candidate pose parameters are determined as the corrected candidate pose parameters, forming a corrected pose parameter pair.
[0073] Weighted direction deviation of the h-th candidate pose parameter The directional deviation of all preserved three-dimensional Gaussian space primitives whose projection positions are located within the region of abrupt change in appearance. The evaluation value is obtained by weighted averaging. (Integrated pose correction weights) The color feature weights and marker contour feature weights of the comparison block where the three-dimensional Gaussian space primitive projection position is located are used to determine the pose parameters. The candidate pose parameters and their adjusted candidate pose parameters are all calculated using the same set of retained primitives, weight determination method, and orientation deviation calculation method to obtain the corresponding weighted orientation deviation. .
[0074] In practice, the weighted directional deviations of the first candidate pose parameters, the second candidate pose parameters, and each adjusted candidate pose parameter are calculated separately. For the first candidate pose parameter, from the adjusted candidate pose parameters whose weighted direction deviation is less than the original first candidate pose parameter's weighted direction deviation, the adjusted candidate pose parameter with the smallest weighted direction deviation is selected as the corrected first candidate pose parameter; the corrected second candidate pose parameter is obtained in the same way. If no adjusted candidate pose parameter with a smaller weighted direction deviation exists for any candidate pose parameter, then that candidate pose parameter is retained, and the two corrected results together constitute the corrected pose parameter pair.
[0075] S140. Based on the mirrored equipment areas on both sides of the workshop, determine the symmetrical reference plane and asymmetrical area of the workshop. Using two candidate pose parameters, map the three-dimensional Gaussian space primitives in the asymmetrical area to the two-dimensional coordinate space of the actual photo to construct a direction distribution map that records the projection position and projection direction.
[0076] The workshop symmetry reference plane refers to the mirror-symmetric plane between the mirrored equipment areas on both sides of the workshop, used to establish the spatial mirror relationship between the two equipment areas. An asymmetric region refers to a local area in the 3D Gaussian model that differs significantly from its mirrored counterpart in both the number of 3D Gaussian spatial primitives and the direction of the texture gradient. An orientation distribution map is a data map that records the projected positions, projection directions, mesh orientation values, and mesh confidence scores of 3D Gaussian spatial primitives according to the mesh cells in the 2D coordinate space of the actual photograph.
[0077] like Figure 2 As shown, the orientation distribution map represents the two-dimensional coordinate space of the actual photograph using a regular grid, with each grid cell corresponding to a local pixel region in the actual photograph. The black dots in the diagram represent grid center reference points, used to indicate the center position of the grid cell; short line segments passing through the grid center reference points represent grid orientation values, and the extension orientation of these short line segments indicates the dominant texture direction within the grid cell, without distinguishing between opposite directions. The black dots in the diagram do not represent the projection positions of the 3D Gaussian space primitives; the actual projection positions of the 3D Gaussian space primitives are recorded as associated data in the corresponding grid cells.
[0078] In one feasible implementation, step S140, determining the symmetrical reference plane and asymmetrical region of the workshop based on the mirrored equipment arrangement areas on both sides of the workshop, includes: The symmetry reference plane of the workshop is determined by the mirror symmetry plane between the equipment areas arranged in mirror on both sides of the workshop. A mirrored symmetry plane refers to a plane in which an equipment area on one side of a workshop can be mirrored and corresponds to an equipment area on the other side in terms of spatial location and structural layout. The equipment center point refers to the spatial center location of the equipment body, column, track, or workstation structure. A mirrored equipment pair refers to two equipment areas located on opposite sides of a workshop that meet preset matching conditions in terms of equipment type, size, and installation height.
[0079] The workshop equipment layout diagram or equipment installation measurement record records the spatial extent of the equipment body, columns, and tracks in the workshop world coordinate system. Three-dimensional Gaussian spatial primitives whose spatial coordinates lie within the corresponding spatial extent are assigned to the corresponding candidate regions, and the connected region with the most primitives in the candidate region is retained as the high-density three-dimensional Gaussian spatial primitive region corresponding to the equipment body, columns, or tracks.
[0080] In practice, high-density 3D Gaussian spatial primitive regions corresponding to the main body of the equipment, columns, and tracks are extracted from the 3D Gaussian model, and the spatial center position and principal direction of each region are calculated. Based on the spatial center position, principal direction, and region size, mirror pairs of equipment are matched on both sides. The midpoint between the two equipment center points in each mirror pair is calculated, and the direction connecting the two equipment center points is determined as the candidate direction of the plane normal. A random sampling consensus method is used to screen for consistent midpoints and candidate plane normal directions, and then least-squares plane fitting is used to obtain the workshop symmetry reference plane.
[0081] Using the workshop symmetry reference plane as a reference, a local region is determined in the three-dimensional Gaussian model, and a mirror region is obtained by mirroring the local region with respect to the workshop symmetry reference plane. A local region refers to a finite spatial extent located on one side of the workshop symmetry reference plane in a 3D Gaussian model. A mirror region refers to the corresponding spatial extent obtained on the other side of the workshop symmetry reference plane after mirroring the spatial position within the local region around the workshop symmetry reference plane. The unit normal vector of the workshop symmetry reference plane is denoted as... The reference point on the symmetry reference plane of the workshop is denoted as . No. Spatial coordinates of a three-dimensional Gaussian space primitive The corresponding mirror coordinates are , can be represented as: , Among them, the planar distance value Representing spatial coordinates Signed distance relative to the workshop symmetry reference plane, mirror coordinates Representing spatial coordinates The spatial location obtained through mirror mapping.
[0082] In practice, the 3D Gaussian model is divided into multiple spatial grids according to a preset spatial scale, and each spatial grid and its neighboring spatial grids are combined into a local region. For each 3D Gaussian spatial primitive within a local region, the planar distance value is used to... and mirror coordinates Perform mirror mapping based on mirror coordinates Identify the mirror region. Remove local regions in the mirror region that exceed the effective modeling range of the 3D Gaussian model, retaining only region pairs where both the local region and the mirror region contain effective 3D Gaussian spatial primitives.
[0083] The number of 3D Gaussian space primitives in the local region and the number of 3D Gaussian space primitives in the mirror region are counted separately to obtain the number of local primitives and the number of mirror primitives; the absolute value of the difference between the number of local primitives and the number of mirror primitives is determined as the difference in the number of primitives. The number of local primitives refers to the number of 3D Gaussian space primitives within a local region, while the number of mirror primitives refers to the number of 3D Gaussian space primitives within a mirror region. The difference in the number of primitives is denoted as... , is the number of local primitives With the number of mirror primitives The absolute value of the difference can be expressed as: Among them, the number of local primitives The number of 3D Gaussian spatial primitives whose spatial coordinates lie within a local region, and the number of mirror primitives. The number of three-dimensional Gaussian spatial primitives whose spatial coordinates lie within the mirror region.
[0084] In practice, the number of 3D Gaussian space primitives within the local region is counted, and the number of 3D Gaussian space primitives within the mirror region is counted. The difference in the number of primitives is then used as the basis for calculation. This characterizes the difference in spatial coverage on both sides. The preset quantity threshold can be determined based on the distribution of the difference in the number of primitives in the known mirrored equipment areas within the workshop, ensuring that small fluctuations in the sampling density on the equipment surface are not identified as asymmetric structural differences.
[0085] Calculate the angle between the texture gradient direction of the 3D Gaussian space primitive in the local region and the 3D Gaussian space primitive in the mirror region whose spatial coordinates are symmetrical about the workshop symmetry reference plane, and average all the angles to obtain the average angle of the texture gradient direction. The mirrored texture gradient direction refers to the direction obtained by mirroring the texture gradient direction of the three-dimensional Gaussian space primitives within a local region according to a mirror-image reference plane. The mirror texture gradient direction of a three-dimensional Gaussian spatial primitive is denoted as . , can be represented as: , Among them, the directional projection value Indicates the direction of texture gradient Unit normal vector of the symmetrical reference plane in the workshop The projection result on the mirror texture gradient direction Indicates the direction of texture gradient The orientation after mirror transformation. The average angle of the texture gradient direction is denoted as... , can be represented as: Among them, the number of valid images The number of 3D Gaussian spatial primitive pairs and their orientation angles are required to complete spatial mirror matching. For mirror texture gradient direction The acute angle between the mirrored region and the corresponding texture gradient direction.
[0086] In practice, the spatial coordinates of each 3D Gaussian spatial primitive within a local area are mirrored to a mirror region. Within the mirror region, a 3D Gaussian spatial primitive with the smallest distance to its mirror coordinates and a distance less than a preset matching distance is searched, forming a valid mirror correspondence. For each valid mirror correspondence, the mirror texture gradient direction is used as the basis for further analysis. The angle between the calculated direction and the corresponding texture gradient direction within the mirror region Then, the angles in all directions The average angle of the texture gradient direction is obtained by averaging. .
[0087] If the difference in the number of primitives is greater than a preset number threshold, and the average included angle is greater than a preset angle threshold, then the local area is determined as an asymmetric region.
[0088] The preset angle threshold is determined based on the average angle of the texture gradient direction in the known mirrored equipment area within the workshop. Asymmetric regions are the differences in the number of primitives. The number exceeds a preset threshold, and the average angle of the texture gradient direction is greater than the threshold value. Local areas exceeding a preset angle threshold. When both conditions are met, the local area shows significant differences from the mirrored area in both spatial coverage and texture detail.
[0089] In practice, the difference in the number of primitives in each local region will be calculated separately. The average angle of the texture gradient direction is compared with a preset threshold. The results are compared with a preset angle threshold. Only when both comparisons meet the conditions is the corresponding local region identified as an asymmetric region. Adjacent local regions that meet the conditions are spatially connected to obtain a complete asymmetric region; isolated regions with an area smaller than the preset minimum region range are discarded as noise regions.
[0090] In one feasible implementation, step S140 maps the three-dimensional Gaussian space primitives in the asymmetric region to the two-dimensional coordinate space of the actual photograph using two candidate pose parameters, constructing a direction distribution map that records the projection position and projection direction, including: The two-dimensional coordinate space of the actual photo is divided into multiple grid units according to the preset grid scale; A grid cell refers to a regular image region obtained by dividing the two-dimensional coordinate space of an actual photograph according to a preset grid scale. The preset grid scale can be determined based on the resolution of the actual photograph and the projection range of asymmetrical regions, so that each grid cell can accommodate local texture direction information, while avoiding a single grid cell covering multiple unrelated device structures.
[0091] In practice, a two-dimensional regular grid is established based on the pixel size of the actual photograph, and a unique grid position is assigned to each grid cell. The preset grid scale can be determined by calibrating the image, ensuring high consistency in the projection orientation of the three-dimensional Gaussian spatial primitives within the same grid cell. Grid cells located at the boundary of the actual photograph and whose effective pixel percentage is less than a preset coverage threshold are not included in the orientation distribution map construction.
[0092] For each candidate pose parameter, the direction of the texture gradient of the mapped 3D Gaussian space primitive in the 2D coordinate space is taken as the projection direction, and the projection direction is filled into the mesh cell where the projection position is located. Projection position refers to the two-dimensional pixel position obtained by mapping a 3D Gaussian space primitive within an asymmetric region to the actual image according to the candidate pose parameters. Projection direction refers to the two-dimensional direction obtained after transforming the texture gradient direction of the 3D Gaussian space primitive to the coordinate system of the camera to be calibrated and mapping it to the two-dimensional coordinate space of the actual image. The first candidate pose parameter is derived from the first candidate 3D translation amount. and the first candidate 3D rotation amount The second candidate pose parameters are composed of the second candidate three-dimensional translation. Second candidate three-dimensional rotation amount constitute.
[0093] In practice, the spatial coordinates of the 3D Gaussian spatial primitives within the asymmetric region are mapped to the 2D coordinate space of the actual photograph using the first and second candidate pose parameters, respectively. 3D Gaussian spatial primitives whose projection depth does not meet the orthographic projection conditions or whose projection positions exceed the imaging range of the actual photograph are discarded. For each retained 3D Gaussian spatial primitive, the center of the 3D Gaussian spatial primitive and its neighboring spatial points offset along the texture gradient direction are projected, and the direction of the line connecting the two 2D projection positions is determined as the projection direction. The corresponding mesh element is determined based on the projection position, and the projection position and projection direction are written into the corresponding mesh element.
[0094] The grid direction value of the grid cell is obtained by averaging the unit direction vectors of each projection direction within the same grid cell. The mesh orientation value is the average of the unit orientation vectors of all projected directions within the same mesh cell, used to represent the dominant texture orientation within that mesh cell. The grid direction value of each grid cell is denoted as , can be represented as: Among them, the grid cell number This indicates the number of projection directions for the current mesh cell. Indicates the first The number of projection directions written within a grid cell, per unit projection direction Indicates the first A projection direction that has been normalized and aligned.
[0095] In practice, the first valid projection direction within each grid cell is selected as the reference direction. Projection directions whose dot product with the reference direction is less than zero are reversed to ensure consistency between the positive and negative directions of the same edge. The processed projection directions are then converted into unit projection directions. Based on grid direction values The average result is obtained by calculation.
[0096] like Figure 2 As shown, mesh cells with valid mesh orientation values are represented by short line segments passing through the mesh center reference point. Mesh cells without a written projection orientation are marked as blank meshes, and the mesh center reference point and mesh orientation value are not drawn in the figure. When the magnitude of the mesh orientation value is less than a preset consistency threshold, the corresponding mesh cell is marked as a low-confidence mesh, represented by a cross line in the figure; the original projection position and projection orientation record of the 3D Gaussian space primitive can be retained in the low-confidence mesh, but the mesh orientation value is not used for candidate pose parameter evaluation.
[0097] Based on the projection position of each 3D Gaussian space primitive and the mesh orientation value of its corresponding mesh element, an orientation distribution map corresponding to the candidate pose parameters is constructed.
[0098] The orientation distribution map includes grid position, grid orientation value, grid confidence, and the projection position of the 3D Gaussian space primitives written into the corresponding grid cell. Different candidate pose parameters correspond to different orientation distribution maps. The two orientation distribution maps use the same grid scale and the same 2D coordinate range of the actual photograph, which can reflect the differences in texture orientation distribution in asymmetric regions under candidate pose parameters on a grid-by-grid basis. Figure 2 A simplified representation of the orientation distribution map corresponding to one of the candidate pose parameters is shown. The figure only shows the mesh division, mesh center reference point, mesh orientation value, low confidence mesh and blank mesh, without plotting the three-dimensional Gaussian space primitive projection position recorded in each mesh cell.
[0099] In practice, for the first candidate pose parameter, the grid position, grid orientation value, grid confidence level, and 3D Gaussian space primitive projection position of each grid cell are written into the first orientation distribution map; for the second candidate pose parameter, they are written into the second orientation distribution map in the same way. Blank grids and low-confidence grids are retained as invalid markers and do not participate in the orientation consistency evaluation between the orientation distribution map and the gradient direction of the actual photographic image, thus obtaining the orientation distribution maps corresponding to the two candidate pose parameters respectively.
[0100] S150. The candidate pose parameters corresponding to the direction distribution map with the smallest overall deviation from the gradient direction of the image in the region corresponding to the actual photo are determined as the target pose parameters. The image gradient direction corresponding to the actual photograph refers to the dominant image gradient direction within a local region of the actual photograph that shares the same two-dimensional coordinate range as the effective grid cells in the orientation distribution map. The angle difference refers to the acute angle between the grid orientation value corresponding to the candidate pose parameter and the image gradient direction of the corresponding grid cell in the actual photograph. The overall deviation refers to the average of the angle differences for the same candidate pose parameter across all effective grid cells.
[0101] In one feasible implementation, step S150 determines the candidate pose parameters corresponding to the direction distribution map with the smallest overall deviation from the gradient direction of the image region corresponding to the actual photograph as the target pose parameters, including: For each candidate pose parameter, determine the image gradient direction of the same grid cell in the actual image; The actual grid image gradient direction refers to the dominant direction of the gradient direction of each effective pixel in a real photograph within a grid cell. The actual grid image gradient direction of each grid cell is denoted as . Grid cell number The grid cell number is consistent with that in the orientation distribution map. Valid image gradient pixels refer to pixels whose image gradient magnitude is greater than a preset gradient threshold, used to exclude flat background areas and weak texture areas.
[0102] In practice, the actual photograph is converted to grayscale and noise is suppressed. Gradient operators are used to calculate the horizontal and vertical gradients of each pixel, obtaining the effective image gradient pixels and their gradient directions. For each grid cell, the image gradient direction opposite to the main direction within the grid is reversed. Then, all effective image gradient directions are normalized and averaged to obtain the actual grid image gradient direction. Grid cells with fewer than a preset threshold number of valid image gradient pixels are marked as invalid grid cells.
[0103] Calculate the angle difference between the grid orientation value of each grid cell in the orientation distribution map corresponding to the candidate pose parameters and the image gradient direction of the same grid cell in the actual photo; Angle difference It refers to the first The candidate pose parameter corresponding to the first Each grid direction value Gradient direction of the actual grid image The acute angle between them, where the candidate number for or Grid direction value The gradient direction of the actual mesh image is derived from the orientation distribution map corresponding to the candidate pose parameters. The same grid cells from the actual photograph.
[0104] In practice, common grid cells that are not marked as empty or low-confidence grid cells in the orientation distribution map and are not marked as invalid grid cells in the actual photograph are selected. For each common grid cell, the grid orientation value is... Gradient direction of the actual grid image Normalization is performed to calculate the angle between two directions; when the angle exceeds a right angle, it is converted to a supplementary angle to obtain the angle difference. This process eliminates the influence of opposite directions on the angle difference at the same structural edge.
[0105] The average of the angle differences corresponding to each candidate pose parameter is calculated to obtain the overall deviation of each candidate pose parameter; the candidate pose parameter with the smallest overall deviation is determined as the target pose parameter.
[0106] No. The total deviation of the candidate pose parameters is denoted as , is the The average of the angle differences of each candidate pose parameter across all common mesh elements can be expressed as: Among them, the number of common grid cells For the first The number of grid cells with valid orientations in the orientation distribution map corresponding to each candidate pose parameter and the actual photograph, and the angle difference. For the first Angle difference in a common grid cell, total deviation Used to evaluate the degree of consistency between candidate pose parameters and the local structural orientation of the actual photograph.
[0107] In practice, the overall deviation corresponding to the first candidate pose parameter is calculated separately. Total deviation corresponding to the second candidate pose parameters Compare the overall deviation. Deviation from the total The candidate pose parameter corresponding to the smaller overall deviation is determined as the target pose parameter. If the difference between the two overall deviations is less than a preset discrimination threshold, the number of common mesh cells corresponding to the two candidate pose parameters is compared, and the candidate pose parameter with more common mesh cells is determined as the target pose parameter.
[0108] S160. Based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, iteratively adjust the target pose parameters until the convergence condition is met, and obtain the optimized pose parameters.
[0109] Updating the virtual image refers to re-driving the rendering of the 3D Gaussian model using the current target pose parameters. Updating the second anchor point set refers to extracting the virtual anchor point set from the updated virtual image and establishing a matching relationship with the first anchor point set. The reprojection residual is the average distance between the pixel coordinates in the first anchor point set and the corresponding virtual anchor point pixel coordinates in the updated second anchor point set. The convergence condition is that the absolute value of the difference between two adjacent reprojection residuals is less than a preset convergence threshold. Optimized pose parameters are the target pose parameters that satisfy the convergence condition.
[0110] In one feasible implementation, the convergence condition is that the absolute value of the difference between two adjacent reprojection residuals is less than a preset convergence threshold. In step S160, based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, the target pose parameters are iteratively adjusted until the convergence condition is met, thus obtaining the optimized pose parameters, such as... Figure 3 As shown, the process of determining the optimal pose parameters through iteration includes steps S161 to S168.
[0111] S161. Re-render and update the virtual image based on the current target pose parameters.
[0112] S162, Extract and update the second anchor set and match it with the first anchor set.
[0113] S163. Calculate the pixel coordinate distance between matching anchor points and determine the reprojection residual.
[0114] Calculate the pixel coordinate distance between each anchor point in the first anchor point set and the matching anchor point in the updated second anchor point set, and determine the reprojection residual based on the average value of all pixel coordinate distances; like Figure 3 As shown in the step of calculating the reprojection residual, for the updated virtual image obtained by re-rendering the current target pose parameters, the updated second anchor point set is extracted, and the pixel coordinate distance between each matching anchor point between the updated second anchor point set and the first anchor point set is calculated. The reprojection residual of the next iteration is denoted as... , is the average pixel coordinate distance between the coordinates of each two-dimensional anchor point in the first anchor point set and the coordinates of the matching virtual anchor points in the updated second anchor point set, which can be expressed as: Among them, the iteration number This indicates the number of pose adjustments and the number of valid anchor pairs. The number of anchor point pairs participating in the reprojection residual calculation, and the coordinates of the two-dimensional anchor points. For the first anchor point set Update the virtual anchor point coordinates. To update the second anchor point set and the coordinates of the two-dimensional anchor points Matched virtual anchor point coordinates. Reprojection residuals. As Figure 3 A unified evaluation data for screening perturbation pose parameters and judging convergence conditions.
[0115] In practice, the virtual image is rendered and updated using the current target pose parameters. The same corner detection and structural intersection detection methods as the second anchor point set are used to extract and update the second anchor point set. A matching relationship is established between the first anchor point set and the updated second anchor point set based on anchor point type, local texture description information, and pixel position. Anchor point pairs that do not meet the requirements of bidirectional matching and local geometric consistency are removed. For each retained anchor point pair, the pixel coordinate distance between the two-dimensional anchor point coordinates and the updated virtual anchor point coordinates is calculated, and the distance is then calculated based on the reprojection residual. The calculation method yields the current reprojection residual.
[0116] S164. Apply positive and negative perturbations to the translation and rotation to obtain the perturbation pose parameters.
[0117] S165. Render and calculate the corresponding reprojection residuals using each perturbation pose parameter.
[0118] A positive and negative preset translational perturbation is applied to the three-dimensional translation in the target pose parameters, and a positive and negative preset rotational perturbation is applied to the three-dimensional rotation in the target pose parameters to obtain multiple perturbation pose parameters. The preset translational perturbation refers to the positive and negative changes applied to the three translational components of the current target pose parameters. The preset rotational perturbation refers to the positive and negative changes applied to the three rotational components of the current target pose parameters. The perturbed pose parameters refer to the candidate pose parameters obtained by changing only one translational component or one rotational component based on the current target pose parameters.
[0119] In practice, a preset translational perturbation is determined based on the spatial resolution of the 3D Gaussian model, and a preset rotational perturbation is determined based on the field of view and image resolution of the camera to be calibrated. Positive and negative preset translational perturbations are applied to the three translational components of the current target pose parameters, and positive and negative preset rotational perturbations are applied to the three rotational components, resulting in multiple perturbed pose parameters. Each perturbed pose parameter is used to drive the 3D Gaussian model to render the corresponding virtual image and extract the corresponding updated second anchor point set.
[0120] S166. Select the perturbation pose parameter with smaller residual as the new target pose parameter.
[0121] The perturbation pose parameter whose reprojection residual after rendering a virtual image using a 3D Gaussian model driven by the perturbation pose parameter is less than the reprojection residual corresponding to the target pose parameter is selected as the new target pose parameter. The reprojection residual corresponding to the current target pose parameters refers to the reprojection residual obtained after rendering a 3D Gaussian model driven by the current target pose parameters. The reprojection residual corresponding to the perturbed pose parameters refers to the reprojection residual obtained after rendering a 3D Gaussian model driven by the perturbed pose parameters. The new target pose parameters refer to the perturbed pose parameters whose reprojection residual is smaller than the reprojection residual corresponding to the current target pose parameters.
[0122] In practice, the reprojection residual corresponding to each perturbation pose parameter is calculated separately, and the reprojection residual is set to be smaller than the current reprojection residual. The perturbation pose parameters are determined as valid perturbation pose parameters. Among all valid perturbation pose parameters, the perturbation pose parameter with the smallest reprojection residual is selected as the new target pose parameter; when no valid perturbation pose parameter exists, the current target pose parameter remains unchanged, and the preset translation perturbation and preset rotation perturbation are reduced according to a preset attenuation ratio. The virtual image is re-rendered and updated using the new target pose parameters, and the reprojection residual is recalculated.
[0123] When the reprojection residuals of multiple valid perturbation pose parameters are the same, the perturbation pose parameter with more valid matching anchor points is selected; when the number of valid matching anchor points is also the same, the perturbation pose parameter with smaller change relative to the current target pose parameter is selected. Perturbation pose parameters with fewer than six valid matching anchor points are not included in the selection of new target pose parameters.
[0124] S167. Determine whether the absolute value of the difference between two adjacent reprojection residuals is less than the preset convergence threshold.
[0125] S168. When the absolute value of the difference between two adjacent reprojection residuals is less than the preset convergence threshold, output the optimized pose parameters.
[0126] The virtual image is re-rendered and updated based on the new target pose parameters until the convergence condition is met.
[0127] The preset convergence threshold is denoted as This is the residual change threshold determined based on the anchor point pixel positioning accuracy. For example... Figure 3 As shown in the convergence judgment step, the current reprojection residual is compared with the previous reprojection residual. The iteration process is considered converged when the absolute value of the difference between two adjacent reprojection residuals meets the following condition: Among them, the current reprojection residual For the first The reprojection residual obtained in the next iteration is the reprojection residual obtained in the previous iteration. For the first The reprojection residual obtained in the next iteration is used to set a convergence threshold. Used to determine whether the changes in the reprojection residuals between two adjacent iterations tend to stabilize.
[0128] In practice, each time new target pose parameters are obtained and the current reprojection residual is calculated... Then, calculate the current reprojection residual. Residual from the previous reprojection The absolute value of the difference. For example... Figure 3 As shown, when the absolute value of the difference between two adjacent reprojection residuals is less than a preset convergence threshold, the current reprojection residual is not greater than twice the anchor point pixel positioning error, and the number of effectively matched anchor points is not less than six, the optimized pose parameters are output. If the above conditions are not met simultaneously and the number of iterations has not reached fifty, the process returns to the step of re-rendering and updating the virtual image based on the current target pose parameters, and continues to execute the steps of updating the second anchor point set extraction, reprojection residual calculation, perturbation pose parameter generation, and target pose parameter update. If the above conditions are still not met after fifty iterations, the pose parameter with the smallest reprojection residual is recorded as the pose parameter to be reviewed.
[0129] This application generates virtual images corresponding to actual photographs using a 3D Gaussian model. It constructs a correspondence between 3D and 2D anchor points using equipment contour corners and structural intersections to obtain candidate pose parameters. Addressing pose ambiguity caused by mirrored equipment arrangement on both sides of the workshop, it constructs a direction distribution map based on the texture gradient direction of asymmetrical regions and filters candidate pose parameters, thereby improving pose discrimination capabilities in industrial scenes with approximate structures. Simultaneously, by reducing the correction weight in areas of abrupt appearance changes, it weakens interference caused by equipment label updates, local occlusion, and lighting variations. Furthermore, it iteratively adjusts the target pose parameters based on the reprojection residual, thereby improving the alignment accuracy between the actual photograph and the 3D scene and the stability of camera pose optimization.
[0130] During verification, actual photographs taken by the camera to be calibrated, with fixed installation measurement records, were selected as verification photographs. For each verification photograph, the translational and rotational errors between the optimized pose parameters and the reference pose in the installation measurement record were recorded. The average reprojection residual, the number of effective matching anchor points, and the number of iterations required to reach convergence were also recorded. For actual photographs with updated equipment identification, temporary occlusion, or changes in local lighting, the pose optimization results with and without reducing the weights of appearance change regions were recorded, respectively. Each verification record was associated with the corresponding photograph number, equipment region number, appearance change type, camera intrinsic parameter version, and reference pose source.
[0131] Figure 4 This is a schematic diagram illustrating a specific implementation of a camera pose optimization system for spatial intelligent scenarios provided in this application. (Refer to...) Figure 4The system may include: The acquisition module 410 is used to acquire multiple on-site photos from different angles in the workshop and actual photos taken by the camera to be calibrated, and to construct a three-dimensional Gaussian model including three-dimensional Gaussian spatial primitives with spatial coordinates and texture gradient directions. The three-dimensional Gaussian model is used for three-dimensional Gaussian sputtering rendering. Rendering module 420 is used to drive a 3D Gaussian model to render a virtual image with initial pose parameters including 3D translation and 3D rotation. The solving module 430 is used to extract the corresponding first anchor point set and second anchor point set from the overlapping area when the area ratio of the overlapping area of the virtual image and the actual photo is greater than the preset overlap area ratio threshold, and solve the pose parameter pair based on the projection relationship between the two. The pose parameter pair includes two candidate pose parameters, each composed of a three-dimensional translation amount and a three-dimensional rotation amount. Module 440 is used to determine the symmetrical reference plane and asymmetrical region of the workshop based on the mirrored equipment arrangement areas on both sides of the workshop. It uses two candidate pose parameters to map the three-dimensional Gaussian space primitives in the asymmetrical region to the two-dimensional coordinate space of the actual photo, and constructs a direction distribution map that records the projection position and projection direction. The determination module 450 is used to determine the candidate pose parameters corresponding to the direction distribution map with the smallest overall deviation from the gradient direction of the image in the region corresponding to the actual photo as the target pose parameters. The determination module 450 is further used to iteratively adjust the target pose parameters until the convergence condition is met based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, so as to obtain the optimized pose parameters.
[0132] The camera pose optimization system for spatial intelligent scenarios in this application is used to implement the aforementioned camera pose optimization method for spatial intelligent scenarios. Therefore, the specific implementation of the camera pose optimization system for spatial intelligent scenarios can be found in the embodiment section of the camera pose optimization method for spatial intelligent scenarios above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.
[0133] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application is shown.
[0134] The electronic device may include a processor 510 and a memory 520 storing computer program instructions.
[0135] Specifically, the processor 510 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0136] The memory 520 may include a large-capacity memory for data or instructions.
[0137] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices.
[0138] The processor 510 reads and executes computer program instructions stored in the memory 520 to implement any of the camera pose optimization methods for spatial intelligent scenarios in the above embodiments.
[0139] In one example, the electronic device may also include a communication interface 530 and a bus 540. Wherein, such as Figure 5 As shown, the processor 510, memory 520, and communication interface 530 are connected through bus 540 and complete communication with each other.
[0140] Communication interface 530 is primarily used to enable communication between modules, devices, units, and / or equipment in the embodiments of this application. Bus 540 includes hardware, software, or both, coupling components of an electronic device together. Where appropriate, bus 540 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application contemplates any suitable bus or interconnection.
[0141] The electronic device can execute the camera pose optimization method for spatial intelligent scenes in the embodiments of this application, thereby realizing the camera pose optimization method for spatial intelligent scenes described in conjunction with the accompanying drawings.
[0142] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0143] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0144] The foregoing has provided a detailed description of a camera pose optimization method, system, and electronic device for spatial intelligent scenarios. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.
Claims
1. A method for camera pose optimization for spatially intelligent scene, characterized in that, include: Multiple on-site photos from different angles in the workshop and actual photos taken by the camera to be calibrated are acquired, and a three-dimensional Gaussian model including three-dimensional Gaussian spatial primitives with spatial coordinates and texture gradient directions is constructed. The three-dimensional Gaussian model is used for three-dimensional Gaussian sputtering rendering. The virtual image is rendered using an initial pose parameter that includes 3D translation and 3D rotation. When the overlapping area of the virtual image and the actual photo is greater than the area of the actual photo by a preset overlap area ratio threshold, the first anchor point set and the second anchor point set corresponding to each other are extracted from the overlapping area and the pose parameter pair is solved based on the projection relationship between the two. The pose parameter pair includes two candidate pose parameters, each composed of a three-dimensional translation and a three-dimensional rotation. Based on the mirrored equipment areas on both sides of the workshop, the symmetrical reference plane and asymmetrical area of the workshop are determined. The three-dimensional Gaussian space primitives in the asymmetrical area are mapped to the two-dimensional coordinate space of the actual photo using two candidate pose parameters, and a directional distribution map recording the projection position and projection direction is constructed. The candidate pose parameters corresponding to the direction distribution map with the smallest overall deviation from the gradient direction of the image region corresponding to the actual photo are determined as the target pose parameters. Based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, the target pose parameters are iteratively adjusted until the convergence condition is met, thereby obtaining the optimized pose parameters.
2. The method according to claim 1, characterized in that, Before mapping the three-dimensional Gaussian space primitives in the asymmetric region to the two-dimensional coordinate space of the actual photograph using two candidate pose parameters, the method further includes: Based on the differences in color distribution and marker outlines between the actual photograph and the virtual image within the overlapping area, areas of abrupt appearance changes are identified. Based on the directional deviation between the image gradient direction of the actual photograph within the appearance mutation region and the texture gradient direction of the three-dimensional Gaussian space primitives whose projection positions are located within the appearance mutation region, the two candidate pose parameters in the pose parameter pair are corrected respectively to obtain the corrected pose parameter pair.
3. The method according to claim 2, characterized in that, The step of determining the appearance abrupt change region based on the color distribution difference and marker outline difference between the actual photo and the virtual image within the overlapping region includes: The overlapping region is divided into multiple comparison blocks; For each comparison block, the pixel percentage of the actual photo and the virtual image in each preset color interval within the comparison block is calculated, and the absolute values of the differences between the pixel percentages of the actual photo and the virtual image in the same preset color interval are summed to obtain the color histogram difference. Extract the marker edges of the actual photo and the virtual image within the comparison block; calculate the number of non-overlapping edge pixels between the marker edges of the actual photo and the marker edges of the virtual image; The ratio of the number of non-overlapping edge pixels to the sum of the number of labeled edge pixels in the actual photo and the number of labeled edge pixels in the virtual image is determined as the label contour difference; If the color histogram difference is greater than a preset color difference threshold, and the identifier outline difference is greater than a preset identifier difference threshold, then the comparison block is determined as the appearance abrupt change region.
4. The method according to claim 2, characterized in that, Based on the directional deviation between the image gradient direction of the actual photograph within the appearance mutation region and the texture gradient direction of the three-dimensional Gaussian space primitives projected within the appearance mutation region, the two candidate pose parameters in the pose parameter pair are corrected respectively to obtain the corrected pose parameter pair, including: The pose correction weights corresponding to the color distribution differences in the appearance mutation area and the pose correction weights corresponding to the differences in the marker contour are both set to preset weights, which are less than the pose correction weights of the color features and marker contour features in the non-appearance mutation area. For each candidate pose parameter, the three-dimensional Gaussian space primitives are mapped to the two-dimensional coordinate space of the actual photograph according to the candidate pose parameters; Select a three-dimensional Gaussian space primitive whose projection position is located within the appearance abrupt change region, and calculate the directional deviation between the texture gradient direction of the selected three-dimensional Gaussian space primitive and the image gradient direction at the projection position; Preset translation changes and preset rotation changes are applied to the three-dimensional translation of the candidate pose parameters, respectively, to obtain the adjusted candidate pose parameters. The adjusted candidate pose parameters whose directional deviation is less than the directional deviation of the candidate pose parameters before adjustment are determined as the corrected candidate pose parameters, and thus form the corrected pose parameter pair.
5. The method according to claim 1, characterized in that, The step of extracting corresponding first and second anchor point sets from the overlapping region and solving for pose parameter pairs based on their projection relationship includes: In the overlapping area of the actual photos, the device outline corner points are extracted by corner point detection, and the device structure intersection points are extracted by detecting the intersection of the device structure edge lines. The device outline corner points and the device structure intersection points are combined to form the first anchor point set. In the overlapping area of the virtual image, extract the virtual device outline corner points corresponding to the device outline corner points in the first anchor point set, and extract the virtual device structure intersection points corresponding to the device structure intersection points in the first anchor point set, and combine the virtual device outline corner points and the virtual device structure intersection points to form the second anchor point set; Determine the spatial coordinates of the three-dimensional Gaussian space primitive corresponding to each virtual anchor point in the second anchor point set and use them as the three-dimensional anchor point coordinates; The pixel coordinates of the first anchor point corresponding to each of the three-dimensional anchor point coordinates are used as the two-dimensional anchor point coordinates. The pose parameter pairs are solved according to the correspondence between the three-dimensional anchor point coordinates and the projected coordinates of the two-dimensional anchor point coordinates.
6. The method according to claim 1, characterized in that, The process of determining the symmetrical reference plane and asymmetrical area of the workshop based on the mirrored equipment areas on both sides of the workshop includes: The workshop symmetry reference plane is determined by the mirror symmetry plane between the equipment areas arranged in mirror on both sides of the workshop. Using the workshop symmetry reference plane as a reference, a local region is determined in the three-dimensional Gaussian model, and a mirror region is obtained by mirror mapping the local region with respect to the workshop symmetry reference plane; The number of three-dimensional Gaussian space primitives in the local region and the number of three-dimensional Gaussian space primitives in the mirror region are counted respectively to obtain the number of local primitives and the number of mirror primitives; the absolute value of the difference between the number of local primitives and the number of mirror primitives is determined as the primitive number difference. Calculate the angle between the three-dimensional Gaussian space primitive in the local region and the texture gradient direction of the three-dimensional Gaussian space primitive in the mirror region whose spatial coordinates are symmetrical about the workshop symmetric reference plane, and average all the angles to obtain the average angle of the texture gradient direction; If the difference in the number of primitives is greater than a preset number threshold, and the average included angle is greater than a preset angle threshold, then the local region is determined as the asymmetric region.
7. The method according to claim 1, characterized in that, The process of mapping the three-dimensional Gaussian space primitives in the asymmetric region to the two-dimensional coordinate space of the actual photograph using two candidate pose parameters, and constructing an orientation distribution map that records the projection position and projection direction, includes: The two-dimensional coordinate space of the actual photograph is divided into multiple grid units according to a preset grid scale; For each candidate pose parameter, the direction of the texture gradient of the mapped 3D Gaussian space primitive in the 2D coordinate space is taken as the projection direction, and the projection direction is filled into the mesh cell where the projection position is located. The grid direction value of the grid cell is obtained by averaging the unit direction vectors of each projection direction within the same grid cell. Based on the projection position of each 3D Gaussian space primitive and the mesh orientation value of its corresponding mesh cell, an orientation distribution map corresponding to the candidate pose parameters is constructed.
8. The method according to claim 7, characterized in that, The step of determining the candidate pose parameters corresponding to the direction distribution map with the smallest overall deviation from the gradient direction of the image region corresponding to the actual photograph as the target pose parameters includes: For each candidate pose parameter, determine the image gradient direction of the same grid cell in the actual photograph; Calculate the angle difference between the grid orientation value of each grid cell in the orientation distribution map corresponding to the candidate pose parameters and the image gradient direction of the same grid cell in the actual photo; The average of the angle differences corresponding to each candidate pose parameter is calculated to obtain the overall deviation of each candidate pose parameter. The candidate pose parameter with the smallest overall deviation is determined as the target pose parameter.
9. The method according to claim 1, characterized in that, The convergence condition is that the absolute value of the difference between the residuals of two adjacent reprojections is less than a preset convergence threshold. The step of iteratively adjusting the target pose parameters until the convergence condition is met, based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, to obtain optimized pose parameters, includes: Calculate the pixel coordinate distance between each anchor point in the first anchor point set and the matching anchor point in the updated second anchor point set, and determine the reprojection residual based on the average value of all pixel coordinate distances; A positive preset translational perturbation and a negative preset translational perturbation are applied to the three-dimensional translation in the target pose parameters, and a positive preset rotational perturbation and a negative preset rotational perturbation are applied to the three-dimensional rotation in the target pose parameters to obtain multiple perturbation pose parameters. The perturbation pose parameter whose reprojection residual after rendering a virtual image using the perturbation pose parameter is less than the reprojection residual corresponding to the target pose parameter is selected as the new target pose parameter. The updated virtual image is re-rendered based on the new target pose parameters until the convergence condition is met.
10. A camera pose optimization system for intelligent spatial scenarios, characterized in that, include: The acquisition module is used to acquire multiple on-site photos from different angles in the workshop and actual photos taken by the camera to be calibrated, and to construct a three-dimensional Gaussian model including three-dimensional Gaussian spatial primitives with spatial coordinates and texture gradient directions. The three-dimensional Gaussian model is used for three-dimensional Gaussian sputtering rendering. The rendering module is used to drive the rendering of virtual images of a 3D Gaussian model with initial pose parameters including 3D translation and 3D rotation. The solution module is used to extract a first anchor point set and a second anchor point set corresponding to each other from the overlapping area when the proportion of the overlapping area of the virtual image and the actual photo to the area of the actual photo is greater than a preset overlap area proportion threshold, and solve the pose parameter pair based on the projection relationship between the two. The pose parameter pair includes two candidate pose parameters, each composed of a three-dimensional translation and a three-dimensional rotation. The module is used to determine the symmetrical reference plane and asymmetrical region of the workshop based on the mirrored equipment arrangement areas on both sides of the workshop. It uses two candidate pose parameters to map the three-dimensional Gaussian space primitives in the asymmetrical region to the two-dimensional coordinate space of the actual photo, and constructs a direction distribution map that records the projection position and projection direction. The determination module is used to determine the candidate pose parameters corresponding to the direction distribution map that has the smallest overall deviation from the gradient direction of the image region corresponding to the actual photo as the target pose parameters. The determining module is further configured to iteratively adjust the target pose parameters until the convergence condition is met based on the reprojection residual between the first anchor point set and the updated second anchor point set extracted from the updated virtual image obtained by re-rendering based on the target pose parameters, thereby obtaining optimized pose parameters.
Citation Information
Patent Citations
Camera pose solving method, system and terminal based on 3DGS
CN119048603A
Three-dimensional scene reconstruction method and device based on three-dimensional skewed Gaussian sputtering
CN121564223A
