Camera positioning method and system under semantic constraint
By combining multi-view image data and semantically constrained camera localization models, the problem of localization drift caused by single sensor data is solved, achieving high-precision camera localization and semantic reconstruction in complex scenes, and improving VR/AR interaction and robot autonomous navigation capabilities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-07
AI Technical Summary
Existing camera localization methods rely on single sensor data or simple fusion, leading to localization drift and a lack of semantic information constraints. This makes it difficult to maintain robustness in scenarios with missing textures or dynamic object occlusion, affecting VR/AR interactive experiences and robot autonomous decision-making.
By collecting multi-view image data, constructing a block image sample set and labeling objects, using a semantically constrained camera localization model to perform geometric and object semantic reconstruction, and combining texture, geometric and sequence feature fusion, a more reliable initial camera pose is output.
It enables more accurate camera positioning in complex scenes, enhances virtual-real interaction and robot navigation capabilities in VR/AR, and provides high-level semantic information to ensure accurate overlay of virtual objects with real space and execution of complex tasks.
Smart Images
Figure CN121810779A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a camera localization method and system under semantic constraints. Background Technology
[0002] In fields such as virtual reality and augmented reality, camera positioning accuracy directly affects the realism and immersion of scene interaction. Current technologies often rely on single-sensor data or simple fusion of multi-source data. Due to significant differences in sampling frequencies and error models among different sensors, positioning drift is easily caused. Furthermore, traditional positioning methods only solve pose based on geometric features (such as feature point matching and photometric consistency), failing to utilize scene semantic information (such as spatial rules like "the table is above the ground" or "doors and windows are embedded in the wall"). In scenes with missing textures or dynamic object occlusion, geometric features are prone to failure, significantly reducing positioning robustness.
[0003] Furthermore, existing scene reconstructions mostly generate pure geometric maps (such as point clouds and meshes), which lack semantic attributes such as object categories and spatial relationships. This makes it impossible to provide high-level constraints for camera pose optimization, making it difficult to achieve collaborative optimization of "localization-mapping" and hindering the improvement of VR / AR interactive experience and robot autonomous decision-making capabilities.
[0004] Therefore, there is an urgent need for a camera localization method that incorporates semantic constraints. Summary of the Invention
[0005] In view of this, the present invention proposes a camera localization method and system under semantic constraints, which can achieve data security isolation and privacy protection, and ensure the security and compliant use of data throughout the entire process.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A camera localization method under semantic constraints includes: Collect image data from multiple perspectives, input the image data into a pre-constructed camera localization model, and output the initial camera pose of the image data; Based on the initial camera pose, perform geometric reconstruction and object semantic reconstruction operations on the pre-constructed semantic map.
[0007] Based on the above technical solution, the present invention can be further improved as follows: Optionally, before inputting the image data into the pre-built camera localization model, the following steps are included: Collect historical image data from multiple perspectives, and divide the historical image data into blocks to obtain multiple block images; Based on the segmented images, a segmented image sample set is constructed.
[0008] Optionally, before the step of constructing a segmented image sample set based on the segmented images, the following steps are included: Objects are marked on the segmented image; Calculate the camera pose for each image block; Dense point cloud computing is performed on the segmented images to obtain the occlusion relationship between different detected objects, and segmented images with a matching degree of less than a preset threshold with the global model features are removed.
[0009] Optionally, after the step of constructing a segmented image sample set based on the segmented images, the following steps are included: Extract image feature points from the segmented image, and transform the image feature points to the world coordinate system based on the camera pose; The corresponding points of the image feature points are calculated using motion reconstruction structures; Calculate the feature distance and coordinate distance between the image feature points and their corresponding points, and calculate the average feature point distance and average coordinate distance for each sample; Set a double threshold condition, and select the image blocks that simultaneously meet the double threshold condition to form the final dataset; The camera localization model is trained based on the final dataset.
[0010] Optionally, the camera localization model under semantic constraints includes an input layer, a texture feature extraction layer, a geometric feature extraction layer, a sequence feature extraction layer, a multimodal feature fusion layer, a coordinate decoder, and an output layer; The image data is received through the input layer; Texture features are extracted through the texture feature extraction layer; Geometric features are extracted through the geometric feature extraction layer; Sequence features are extracted through the sequence feature extraction layer; The texture features, geometric features, and sequence features are fused through a multimodal feature fusion layer to obtain fused features; The fused features are input into the coordinate decoder to obtain the relative pose residual between the current frame and the previous frame. Based on the relative pose residual and the known camera pose of the previous frame, the initial camera pose of the current frame is calculated. The initial camera pose is output through the output layer.
[0011] Optionally, the geometric reconstruction operation of the pre-constructed semantic map based on the initial camera pose includes: Geometric reconstruction of the pre-constructed semantic map is performed based on the neural radiation field and the initial camera pose.
[0012] Optionally, the step of performing object semantic reconstruction on the pre-constructed semantic map based on the initial camera pose includes: Based on the initial camera pose, the 2D bounding box of the object detected in the multi-view is transformed to the world coordinate system; Each object in the world coordinate system is reprojected onto the image plane, and the reprojected 2D bounding box is compared with the 2D bounding box generated by instance segmentation to calculate the error between the camera and the object. By merging the 2D bounding boxes of the same object from multiple views, a 3D bounding box of the object is obtained. Object semantic reconstruction is performed on a pre-built semantic map based on the object's 3D bounding box and the error between the camera and the object.
[0013] A camera localization system under semantic constraints, comprising: Multi-view image acquisition module, used to acquire image data from multiple perspectives; A semantically constrained camera localization module is used to input the image data into a pre-built camera localization model to output the initial camera pose of the image data. The semantic map reconstruction module is used to perform geometric reconstruction and object semantic reconstruction operations on the pre-constructed semantic map based on the initial camera pose.
[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method described herein.
[0015] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method when executed by a processor.
[0016] The present invention has the following advantages: The camera localization method under semantic constraints in this invention integrates complementary information from different perspectives by inputting multi-view image data into a pre-constructed camera localization model, reducing the limitations of single-view observation and outputting a more reliable initial camera pose. Based on the initial camera pose, geometric reconstruction (constructing the three-dimensional spatial structure of the scene) and object semantic reconstruction (annotating object categories, associated spatial attributes, etc. in the pre-constructed semantic map) are performed simultaneously, so that the semantic map has both "precise geometric coordinates" and "understandable semantic information". For example, in VR / AR scenarios, it ensures the accurate superposition of virtual objects and real space, and enables devices to recognize semantic regions such as "table" and "wall", achieving a more natural virtual-real interaction. In robot navigation, complex tasks such as obstacle avoidance and target area recognition can be achieved based on the semantic map. Attached Figure Description
[0017] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the camera localization method under semantic constraints in an embodiment of the present invention. Figure 2 This is a schematic diagram of the main components of the camera positioning system under semantic constraints in an embodiment of the present invention; Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0019] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0020] It should be noted that, where there is no conflict, the embodiments and features of the present invention can be combined with each other. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0021] Figure 1 This is a flowchart illustrating the camera localization method under semantic constraints in an embodiment of the present invention, as shown below. Figure 1 As shown, the camera localization method under semantic constraints provided in this embodiment of the invention includes the following steps S101 to S102.
[0022] S101 acquires multi-view image data, inputs the image data into a pre-built camera positioning model, and outputs the initial camera pose of the image data.
[0023] Before the step of inputting the image data into the pre-built camera localization model, the following steps are included: Collect historical image data from multiple perspectives, and divide the historical image data into blocks to obtain multiple block images; Based on the segmented images, a segmented image sample set is constructed.
[0024] Before the step of constructing a segmented image sample set based on the segmented images, the following steps are included: Objects are marked on the segmented image; Calculate the camera pose of each image block and use the Structure From Motion (SFM) method to estimate the camera pose of the image blocks. The segmented images are processed using dense point cloud computing through Multi-View Stereo (MVS) and surface reconstruction methods to obtain the occlusion relationships between different detected objects. In order to detect outliers, the original SFM pipeline can remove samples that do not match the global model features well enough and remove segmented images whose matching degree with the global model features is lower than a preset threshold.
[0025] After the step of constructing a segmented image sample set based on the segmented images, the following steps are included: Extract image feature points from the segmented image, and transform the image feature points to the world coordinate system based on the camera pose; The corresponding points of the image feature points are calculated using motion reconstruction structures; Calculate the feature distance and coordinate distance between the image feature points and their corresponding points, and calculate the average feature point distance and average coordinate distance for each sample; Set a double threshold condition, and select the image blocks that simultaneously meet the double threshold condition to form the final dataset; The camera localization model is trained based on the final dataset.
[0026] S102 performs geometric reconstruction and object semantic reconstruction operations on the pre-constructed semantic map based on the initial camera pose.
[0027] The semantically constrained camera localization model includes an input layer, a texture feature extraction layer, a geometric feature extraction layer, a sequence feature extraction layer, a multimodal feature fusion layer, a coordinate decoder, and an output layer. The image data is received through the input layer; Texture features are extracted through the texture feature extraction layer; Geometric features are extracted through the geometric feature extraction layer; Sequence features are extracted through the sequence feature extraction layer; The texture features, geometric features, and sequence features are fused through a multimodal feature fusion layer to obtain fused features; The fused features are input into the coordinate decoder to obtain the relative pose residual between the current frame and the previous frame. Based on the relative pose residual and the known camera pose of the previous frame, the initial camera pose of the current frame is calculated. Assumption , , , This indicates the rotation and translation (from the camera to the world) between the previous and current frames. This represents the XYZ coordinates (initial camera pose) of the current frame within the camera system, obtained by uniformly sampling the depth image. The definition is as follows: ; Here, E represents the identity matrix.
[0028] The initial camera pose is output through the output layer.
[0029] Geometric reconstruction of the pre-constructed semantic map is performed based on the neural radiation field and the initial camera pose.
[0030] Based on the initial camera pose, the 2D bounding box of the object detected in the multi-view is transformed to the world coordinate system; Each object in the world coordinate system is reprojected onto the image plane, and the reprojected 2D bounding box is compared with the 2D bounding box generated by instance segmentation to calculate the error between the camera and the object. If the camera pose, feature points, and object are denoted as C*, P*, and O* respectively, then the semantic Bundle Adjustment can be represented as the following nonlinear optimization problem.
[0031] ; in, , and These represent the i-th camera pose, the j-th feature point, and the k-th object, respectively.
[0032] E ( Pi , Ci ) 、E ( Pi , Oi )and E ( Oi , CiThe errors between feature points and camera pose, feature points and object pose, and camera pose and object pose, respectively, are used to reproject each object in the world coordinate system onto the image plane. The reprojected 2D bounding box is compared with the 2D bounding box generated by instance segmentation to calculate the error between the camera and the object, as shown in the following formula.
[0033] ; ; ; in, To detect error, This represents the transformation matrix from the world coordinate system to the camera coordinate system. Let be the 3D representation of the i-th object under certain condition c, where c may be related conditions such as scene and camera view. For pose error, Let be the pose transformation matrix, and π represent the transformation matrix from the camera coordinate system to the pixel coordinate system. Oi This represents the bounding box for instance segmentation detection. O 2 d This represents the bounding box generated by two-dimensional projection.
[0034] By merging the 2D bounding boxes of the same object from multiple views, a 3D bounding box of the object is obtained. Object semantic reconstruction is performed on a pre-built semantic map based on the object's 3D bounding box and the error between the camera and the object.
[0035] Figure 2 This is a schematic diagram of the main components of the camera positioning system under semantic constraints in an embodiment of the present invention. Figure 2 As shown, the semantically constrained camera localization system 1 provided in this embodiment of the invention includes a multi-view image acquisition module 10, a semantically constrained camera localization module 20, and a semantic map reconstruction module 30.
[0036] Multi-view image acquisition module 10 is used to acquire image data from multiple perspectives; The semantically constrained camera localization module 20 is used to input the image data into a pre-constructed camera localization model to output the initial camera pose of the image data; The semantic map reconstruction module 30 is used to perform geometric reconstruction and object semantic reconstruction operations on the pre-constructed semantic map based on the initial camera pose.
[0037] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3As shown, the electronic device 40 includes: a processor 401, a memory 402, and a bus 403; The processor 401 and the memory 402 communicate with each other via the bus 403. The processor 401 is used to call program instructions in the memory 402 to execute the methods provided in the above-described method embodiments, and to execute the methods provided in the embodiments of the present invention.
[0038] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions, which cause a computer to execute the method provided in this embodiment of the invention.
[0039] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.
[0040] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A camera localization method under semantic constraints, characterized in that, include: Collect image data from multiple perspectives, input the image data into a pre-constructed camera localization model, and output the initial camera pose of the image data; Based on the initial camera pose, perform geometric reconstruction and object semantic reconstruction operations on the pre-constructed semantic map.
2. The camera localization method under semantic constraints according to claim 1, characterized in that, Before the step of inputting the image data into the pre-built camera localization model, the following steps are included: Collect historical image data from multiple perspectives, and divide the historical image data into blocks to obtain multiple block images; Based on the segmented images, a segmented image sample set is constructed.
3. The camera localization method under semantic constraints according to claim 2, characterized in that, Before the step of constructing a segmented image sample set based on the segmented images, the following steps are included: Objects are marked on the segmented image; Calculate the camera pose for each image block; Dense point cloud computing is performed on the segmented images to obtain the occlusion relationship between different detected objects, and segmented images with a matching degree of less than a preset threshold with the global model features are removed.
4. The camera localization method under semantic constraints according to claim 3, characterized in that, After the step of constructing a segmented image sample set based on the segmented images, the following steps are included: Extract image feature points from the segmented image, and transform the image feature points to the world coordinate system based on the camera pose; The corresponding points of the image feature points are calculated using motion reconstruction structures; Calculate the feature distance and coordinate distance between the image feature points and their corresponding points, and calculate the average feature point distance and average coordinate distance for each sample; Set a double threshold condition, and select the image blocks that simultaneously meet the double threshold condition to form the final dataset; The camera localization model is trained based on the final dataset.
5. The camera localization method under semantic constraints according to claim 1, characterized in that, The semantically constrained camera localization model includes an input layer, a texture feature extraction layer, a geometric feature extraction layer, a sequence feature extraction layer, a multimodal feature fusion layer, a coordinate decoder, and an output layer. The image data is received through the input layer; Texture features are extracted through the texture feature extraction layer; Geometric features are extracted through the geometric feature extraction layer; Sequence features are extracted through the sequence feature extraction layer; The texture features, geometric features, and sequence features are fused through a multimodal feature fusion layer to obtain fused features; The fused features are input into the coordinate decoder to obtain the relative pose residual between the current frame and the previous frame. Based on the relative pose residual and the known camera pose of the previous frame, the initial camera pose of the current frame is calculated. The initial camera pose is output through the output layer.
6. The camera localization method under semantic constraints according to claim 1, characterized in that, The geometric reconstruction operation of the pre-constructed semantic map based on the initial camera pose includes: Geometric reconstruction of the pre-constructed semantic map is performed based on the neural radiation field and the initial camera pose.
7. The camera localization method under semantic constraints according to claim 1, characterized in that, The object semantic reconstruction operation based on the pre-constructed semantic map using the initial camera pose includes: Based on the initial camera pose, the 2D bounding box of the object detected in the multi-view is transformed to the world coordinate system; Each object in the world coordinate system is reprojected onto the image plane, and the reprojected 2D bounding box is compared with the 2D bounding box generated by instance segmentation to calculate the error between the camera and the object. By merging the 2D bounding boxes of the same object from multiple views, a 3D bounding box of the object is obtained. Object semantic reconstruction is performed on a pre-built semantic map based on the object's 3D bounding box and the error between the camera and the object.
8. A camera localization system under semantic constraints, characterized in that, include: Multi-view image acquisition module, used to acquire image data from multiple perspectives; A semantically constrained camera localization module is used to input the image data into a pre-built camera localization model to output the initial camera pose of the image data. The semantic map reconstruction module is used to perform geometric reconstruction and object semantic reconstruction operations on the pre-constructed semantic map based on the initial camera pose.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.