Positioning mapping method and system in bank dynamic illumination environment and storage medium

Abstract

The invention discloses a positioning mapping method and system in a bank dynamic illumination environment and a storage medium, and relates to the technical field of robot mapping, and the method comprises the steps: obtaining an RGB image, a depth image and inertial measurement data in a bank environment; performing illumination equalization processing on the RGB image, and separating an illumination component and a reflection component through a multi-scale algorithm to obtain a stable reflection image; performing defect repair on the depth image, extracting feature points based on the reflection image and the repaired depth image, obtaining three-dimensional coordinates of the feature points, and performing pose estimation in combination with inertial measurement data; loopback detection is carried out through geometric matching in combination with auxiliary constraint, and effective loopback is screened; the pose estimation result and the effective loop are fused to execute pose map optimization, and a corrected pose is obtained; and constructing a three-dimensional environment map based on the corrected pose and the repaired depth image. The method adapts to the special environment of a bank, effectively improves the positioning robustness and the mapping precision, and can provide reliable support for subsequent robot inspection, space planning and other applications.

Description

Technical Field

[0001] This invention relates to the field of robot mapping technology, and in particular to a positioning and mapping method, system, and storage medium for dynamic lighting environments in banks. Background Technology

[0002] With the expansion of the application of intelligent technology in the financial field, the demand for autonomous positioning and 3D mapping technology in bank lobbies is becoming increasingly urgent. This technology can provide basic support for subsequent applications such as robot inspection and spatial planning.

[0003] Existing RGB-DSLAM (Simultaneous Localization and Mapping) algorithms already possess certain localization and mapping capabilities in static, well-lit general indoor environments. However, bank lobbies, as typical commercial indoor settings, generally exhibit environmental characteristics where dynamic lighting and highly reflective surfaces coexist. On the one hand, large floor-to-ceiling windows, high-brightness spotlights, and other facilities cause frequent changes in ambient light intensity, resulting in local overexposure, underexposure, or abrupt shadow changes, affecting the feature stability of RGB images. On the other hand, highly reflective materials such as glass counters and polished marble floors can cause deviations in the depth signals received by RGB-D cameras, leading to holes or noise in the depth images.

[0004] The aforementioned environmental factors directly cause a decline in the quality of image data acquired by the existing RGB-DSLAM algorithm, which in turn leads to problems such as distortion or lack of detail in the 3D map, making it difficult to meet the high-precision positioning and mapping requirements of the bank lobby scene. Summary of the Invention

[0005] To address the technical problem of decreased accuracy of constructed 3D maps due to dynamic lighting environments in banks in existing technologies, this invention provides a positioning and mapping method, system, and storage medium for banks under dynamic lighting environments.

[0006] The technical solution adopted in this invention is:

[0007] The first aspect of this application provides a method for positioning and mapping under dynamic lighting conditions in banks, including the following steps:

[0008] Step 1: Acquire image data and inertial measurement data in the banking environment, where the image data includes RGB images and depth images;

[0009] Step 2: Perform illumination equalization processing on the RGB image, and separate the illumination component and reflection component of the image through a multi-scale algorithm to obtain the reflection image after illumination equalization;

[0010] Step 3: Perform defect repair processing on the depth image to obtain the repaired depth image; extract feature points based on the reflection image and the repaired depth image, obtain the three-dimensional coordinates of the feature points through the repaired depth image, and perform pose estimation in combination with inertial measurement data to obtain the pose estimation result;

[0011] Step 4: Perform loop closure detection using geometric matching combined with auxiliary constraints to filter out valid loops;

[0012] Step 5: Combine the pose estimation results with the effective loop closures to perform pose graph optimization and obtain the corrected pose;

[0013] Step 6: Based on the corrected pose and repaired depth image, construct a 3D environment map.

[0014] Preferably, the defect repair processing performed on the depth image to obtain the repaired depth image includes the following:

[0015] Semantic segmentation is performed on the RGB image to obtain the first semantic segmentation result, and high reflectivity regions are marked based on the first semantic segmentation result;

[0016] Obtain the depth value of this region in the current depth image. If the depth value is zero or exceeds the effective measurement range of the camera, the region is determined to be an invalid region of the current depth image.

[0017] Before and after acquiring the current depth image, multiple frames of depth images of the banking environment are continuously acquired. The depth information corresponding to the areas in the multiple frames of depth images of the banking environment whose depth values ​​are neither zero nor exceed the effective measurement range of the camera is selected to form multiple frames of historical effective depth information.

[0018] The effective depth information from multiple historical frames is projected onto the coordinate system of the current depth image, and the invalid areas of the current depth image are filled using a weighted average method to obtain the repaired depth image.

[0019] Preferably, the step of extracting feature points based on the reflection image and the restored depth image, obtaining the three-dimensional coordinates of the feature points through the restored depth image, and performing pose estimation in conjunction with inertial measurement data, yields the following pose estimation results:

[0020] Feature extraction algorithms are used to extract feature points from the reflection image. Three-dimensional coordinates are assigned to the feature points using the repaired depth image. Semantic trust weights are assigned to the feature points based on the semantic segmentation results. The semantic trust weights are used as weighting factors for the reprojection error. Weighted nonlinear optimization is performed by combining inertial measurement data and three-dimensional coordinates to obtain the camera relative pose as the pose estimation result.

[0021] Preferably, loop closure detection is performed by combining geometric matching with auxiliary constraints, and the valid loop closures obtained include the following:

[0022] The current frame corresponding to the current image data is determined, and the current frame is matched with the historical frame set to obtain multiple potential loop closure candidate frames and the geometric matching score corresponding to each potential loop closure candidate frame; the historical frame set consists of historical frames corresponding to multiple frames of historical image data.

[0023] Semantic segmentation is performed on the RGB image of the current frame to obtain the second semantic segmentation result of the current frame; semantic segmentation is performed on the RGB images of each potential loop closure candidate frame to obtain the third semantic segmentation result of each potential loop closure candidate frame.

[0024] The semantic tag matching degree of each potential loop closure candidate frame is calculated based on the second semantic segmentation result and the third semantic segmentation result.

[0025] The geometric matching score and the semantic label matching degree are weighted and fused according to a preset balanced weight to obtain the loop closure score corresponding to each potential loop closure candidate frame.

[0026] Each loop closure score is compared with a preset threshold. If the loop closure score exceeds the preset threshold, the corresponding potential loop closure candidate frame is determined to be a valid loop closure.

[0027] Preferably, the step of combining the pose estimation results with the effective loop closure to perform pose graph optimization, resulting in the corrected pose, includes the following:

[0028] By combining the odometry constraints corresponding to the pose estimation results, the loop closure constraints corresponding to the effective loop closure, and the IMU constraints corresponding to the inertial measurement data, a total error function is constructed.

[0029] By minimizing the total error function, a globally consistent camera pose sequence is obtained, and this camera pose sequence is used as the corrected pose.

[0030] Preferably, the 3D environment map constructed based on the corrected pose and repaired depth image includes the following:

[0031] Using the corrected pose as a spatial reference and combining it with the repaired depth image, a preset 3D mapping algorithm is used for fusion processing to construct a 3D environment map.

[0032] Preferably, the assignment of semantic trust weights to feature points based on semantic segmentation results includes the following:

[0033] The semantic region type of each feature point is determined based on the semantic segmentation results; among which, the semantic region types include high reflectivity region, weak texture region, semantically significant object region, and strong texture region;

[0034] If a feature point is located in a high-reflectivity area or a weak-texture area, a semantic trust weight within a first preset weight range is assigned to that feature point.

[0035] If a feature point is located in a semantically significant object region or a strong texture region, a semantic trust weight within a second preset weight range is assigned to the feature point; the values ​​of the first preset weight range are all less than the values ​​of the second preset weight range.

[0036] Preferably, the effective depth information from multiple historical frames is projected onto the coordinate system of the current depth image, and an invalid region of the current depth image is filled using a weighted average method. Specifically, this includes the following:

[0037] Acquire the pose data of the current frame and the pose data of each historical frame, and calculate the pose similarity between each historical frame and the current frame; the pose similarity is obtained by weighting the rotation angle deviation and translation distance deviation between the current frame and the historical frames.

[0038] If the pose similarity between a historical frame and the current frame is higher than a preset similarity threshold, then the historical frame is determined to be a frame with the same viewpoint, and its corresponding historical effective depth information is assigned the first fusion weight.

[0039] If the pose similarity between a historical frame and the current frame is lower than a preset similarity threshold, the historical frame is determined to be an out-of-view frame, and its corresponding historical effective depth information is assigned a second fusion weight; the first fusion weight is greater than the second fusion weight.

[0040] Based on the first and second fusion weights allocated above, a weighted average calculation is performed on the effective depth information of multiple historical frames to complete the filling of the current invalid region.

[0041] The second aspect of this application provides a positioning and mapping system for a bank under dynamic lighting conditions, applied to the aforementioned positioning and mapping method for a bank under dynamic lighting conditions, comprising:

[0042] A multi-source data acquisition module is used to acquire image data and inertial measurement data in the banking environment, wherein the image data includes RGB images and depth images;

[0043] An adaptive illumination preprocessing module is used to perform illumination equalization processing on an RGB image. It separates the illumination component and reflection component of the image through a multi-scale algorithm to obtain an illumination equalized reflection image.

[0044] The semantic depth pose module is used to perform defect repair processing on the depth image to obtain a repaired depth image; extract feature points based on the reflection image and the repaired depth image, obtain the three-dimensional coordinates of the feature points through the repaired depth image, and perform pose estimation in combination with inertial measurement data to obtain the pose estimation result.

[0045] A constraint-weighted loop closure detection module is used to perform loop closure detection by combining geometric matching with auxiliary constraints, and to filter out valid loops.

[0046] The pose graph optimization module is used to combine the pose estimation results with the effective loop closures to perform pose graph optimization and obtain the corrected pose.

[0047] A map building module is used to construct a three-dimensional environment map based on the corrected pose and the repaired depth image.

[0048] A third aspect of this application provides a readable storage medium having computer instructions stored thereon, characterized in that, when executed by a processor, the instructions implement the positioning and mapping method for a bank under dynamic lighting conditions as described above.

[0049] The beneficial effects of the present invention are at least one of the following: the illumination equalization processing performed on RGB images can effectively separate the illumination component and the reflection component of the image, and improve the characteristic stability of RGB images under dynamic lighting conditions.

[0050] Defect repair processing of depth images can reduce the problem of missing or abnormal depth data caused by highly reflective materials, improve the integrity of depth images, and help improve the accuracy of the three-dimensional coordinates of feature points.

[0051] Combining RGB images, depth images, and inertial measurement data for pose estimation, along with loop closure detection and pose graph optimization, helps improve the consistency and reliability of autonomous localization and alleviates localization deviations caused by poor image data quality.

[0052] A 3D environment map can be constructed based on the corrected pose and repaired depth image, which can improve the integrity and accuracy of the 3D map and adapt to the application requirements of positioning and mapping in the bank lobby scene. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention;

[0054] Figure 2 This is a system structure block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0055] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] Example 1

[0057] Existing RGB-SLAM algorithms are not adapted to the special environment of dynamic lighting and highly reflective surfaces in bank lobbies. This results in poor stability of RGB image features and invalid holes in depth images, leading to insufficient autonomous localization accuracy and distorted or missing details in the 3D map, making it difficult to meet the high-precision localization and mapping requirements of bank scenarios. To address these issues, this embodiment provides a localization and mapping method for dynamic lighting environments in banks, such as... Figure 1 As shown, it includes the following steps:

[0058] Step 1: Acquire image data and inertial measurement data in the banking environment, where the image data includes RGB images and depth images.

[0059] It should be noted that RGB images are two-dimensional images containing scene color information captured by an RGB-D camera. Each pixel's color characteristics are described by three channels: red (R), green (G), and blue (B). Depth images are two-dimensional images captured by an RGB-D camera containing distance information from each point in the scene to the camera's optical center. The grayscale value or numerical value of each pixel directly corresponds to the actual depth of that point. Inertial measurement data are physical quantities reflecting the device's motion state, including acceleration and angular velocity data, collected by an IMU, used to help improve the stability of pose estimation.

[0060] For example, an RGB-D camera and an IMU are fixedly mounted on a mobile device, such as an autonomous mobile robot, which moves at a constant speed within a bank lobby along a preset path. The RGB-D camera synchronously acquires RGB images and depth images at a preset acquisition frequency, while the IMU acquires inertial measurement data. During the acquisition process, the device's built-in storage module stores all raw data in real time and uses timestamps to achieve precise synchronization of the three types of data, ensuring that multi-source data from the same time point can be matched during subsequent processing.

[0061] Step 2: Perform illumination equalization processing on the RGB image. Separate the illumination component and reflection component of the image through a multi-scale algorithm to obtain the reflection image after illumination equalization.

[0062] It should be noted that the illumination equalization processing is an enhancement process performed on RGB images to address the problem of local overexposure and underexposure caused by dynamic lighting in the bank lobby. The purpose is to improve the overall brightness consistency and detail clarity of the image.

[0063] The illumination component is the brightness component in an RGB image determined by ambient lighting conditions, reflecting the light intensity distribution of the scene and easily affected by dynamic lighting changes. The reflection component is the brightness component in an RGB image determined by the inherent properties of objects in the scene, reflecting the color and texture characteristics of the objects themselves. It is stable and is the core basis for subsequent feature extraction.

[0064] In this embodiment, the multi-scale algorithm is the multi-scale Retinex algorithm (MSR) based on Retinex theory. It estimates the illumination component through Gaussian filter functions of multiple different scales, thereby achieving the separation of the illumination component and the reflection component.

[0065] In the specific implementation process, the RGB image acquired in step 1 undergoes grayscale preprocessing, converting the three-channel color image into a single-channel grayscale image to simplify subsequent computational complexity. Then, the multi-scale Retinex algorithm is used to perform illumination equalization processing, assuming the grayscale RGB image is... , its satisfaction ,in For pixel coordinates, For the irradiation component, This is the reflection component.

[0066] Using multiple Gaussian filtering functions of different scales Estimated irradiation component The Gaussian filter function satisfies ,in Let be the standard deviation at the m-th scale. Final reflection image. The calculation method is as follows:

[0067]

[0068] in The scale value ranges from 3 to 5. For the first The weights of each scale, and , Represents the relationship between the Gaussian filter function and the image. The convolution operation is then performed. Finally, the calculated reflection image is subjected to grayscale stretching to map the pixel values ​​to the standard range of 0-255, resulting in a light-equalized reflection image.

[0069] Step 3: Perform defect repair processing on the depth image to obtain the repaired depth image; extract feature points based on the reflection image and the repaired depth image, obtain the three-dimensional coordinates of the feature points through the repaired depth image, and perform pose estimation in combination with inertial measurement data to obtain the pose estimation result.

[0070] Considering that the highly reflective materials in bank lobbies can cause RGB-D cameras to receive incorrect infrared reflection signals, resulting in large areas of holes or noise in the depth image, i.e., invalid regions, these invalid regions affect the accuracy of the 3D coordinates of feature points, thereby reducing pose estimation accuracy. In one possible implementation, the defect repair processing performed on the depth image yields a repaired depth image including the following:

[0071] Semantic segmentation is performed on the RGB image to obtain the first semantic segmentation result, and high reflectivity regions are marked based on the first semantic segmentation result.

[0072] It should be noted that the first semantic segmentation result is obtained by performing semantic segmentation on the RGB image acquired in step 1. It is used to accurately label regions of different semantic types in the image, especially high-reflectivity regions. High-reflectivity regions are areas in the bank lobby made of highly reflective materials, including glass counters, glass doors, polished marble floors, metal signs, etc. These areas are prone to causing invalid data in the depth image.

[0073] For example, a lightweight deep learning model, such as MobileNetV2-DeeplabV3+, is used to perform semantic segmentation on the RGB image acquired in step 1, and outputs the first semantic segmentation result. In this result, each pixel corresponds to a semantic label, and the high reflectivity areas composed of glass countertops, glass doors, polished marble floors, metal signs, etc. are accurately marked based on the semantic labels.

[0074] Obtain the depth value of this region in the current depth image. If the depth value is zero or exceeds the effective measurement range of the camera, the region is determined to be an invalid region of the current depth image.

[0075] It should be noted that the depth data of invalid regions cannot reflect the actual distance information in the scene.

[0076] For example, the current depth image acquired in step 1 is read, and the depth values ​​of all pixels in the high reflectivity area are extracted. If the depth value of a pixel is zero or exceeds the effective measurement range of the RGB-D camera, the area where the pixel is located is determined to be an invalid area of ​​the current depth image.

[0077] Before and after acquiring the current depth image, multiple frames of depth images of the banking environment are continuously acquired. The depth information corresponding to the areas in the multiple frames of depth images of the banking environment whose depth values ​​are neither zero nor exceed the effective measurement range of the camera is selected to form multiple frames of historical effective depth information.

[0078] It should be noted that historical effective depth information refers to the depth information of the regions whose depth values ​​are not zero and do not exceed the effective measurement range of the camera, which are among the multiple depth images continuously acquired before and after the current depth image.

[0079] For example, M frames of historical depth images continuously acquired before and after the current depth image are obtained, where M ranges from 5 to 10. The validity of each frame of historical depth image is judged, and the depth information corresponding to the area with a depth value that is not zero and does not exceed the effective measurement range of the camera is selected to form multiple frames of valid historical depth information.

[0080] The effective depth information from multiple historical frames is projected onto the coordinate system of the current depth image, and the invalid areas of the current depth image are filled using a weighted average method to obtain the repaired depth image.

[0081] In the specific implementation process, a coordinate transformation matrix is ​​used to project the historical effective depth information of each frame onto the coordinate system of the current depth image to ensure precise spatial alignment between the historical depth data and the current image. A weighted average method is then used to fuse the projected historical effective depth information from multiple frames, calculating the repaired depth value for each pixel within the current invalid region, as shown in the following formula:

[0082]

[0083] in, The current frame number. For the first Frame history images in The effective depth value of a pixel. For the first A validity mask for frames, where a valid depth value corresponds to 1 and an invalid value corresponds to 0. The number of valid historical frames participating in the fusion is used to obtain the complete restored depth image.

[0084] Considering that the degree of reflection interference in highly reflective areas varies under different shooting angles, the closer the viewing angles of historical frames and the current frame are, the higher the matching degree of their depth data with the current scene, and the higher the restoration accuracy. The above implementation method does not consider viewing angle differences and uses uniform weighted fusion, which may lead to additional errors introduced by data from different viewing angles. To solve this problem, in another possible implementation method, valid depth information from multiple historical frames is projected onto the coordinate system of the current depth image, and a weighted average method is used to fill in the invalid areas of the current depth image. Specifically, this includes the following:

[0085] The pose data of the current frame and the pose data of each historical frame are obtained, and the pose similarity between each historical frame and the current frame is calculated. The pose similarity is obtained by weighting the rotation angle deviation and translation distance deviation between the current frame and the historical frames.

[0086] It should be noted that pose similarity is a parameter used to quantify the difference in shooting perspective between the current frame and historical frames. It is calculated by weighting the rotation angle deviation and translation distance deviation, and the value ranges from 0 to 1. The closer the value is to 1, the closer the viewing perspective is.

[0087] In the specific implementation process, the pose data of the current frame and the pose data of each historical frame are obtained from the front-end odometry, and the rotation angle deviation between each historical frame and the current frame is calculated. Translational distance deviation pose similarity The calculation method is as follows:

[0088]

[0089] in and These are the weighting coefficients, and , This is the preset maximum translation distance threshold.

[0090] If the pose similarity between a historical frame and the current frame is higher than a preset similarity threshold, then the historical frame is determined to be a frame with the same viewpoint, and its corresponding historical effective depth information is assigned the first fusion weight.

[0091] For example, a preset similarity threshold is set. If the pose similarity of a certain historical frame If it is a frame from the same viewpoint, then the first fusion weight is assigned to its corresponding historical effective depth information. .

[0092] If the pose similarity between a historical frame and the current frame is lower than a preset similarity threshold, the historical frame is determined to be an out-of-view frame, and its corresponding historical effective depth information is assigned a second fusion weight; the first fusion weight is greater than the second fusion weight.

[0093] For example, if the pose similarity of a certain historical frame If it is a frame from a different viewpoint, then the second fusion weight is assigned to its corresponding historical effective depth information. .

[0094] Based on the first and second fusion weights allocated above, a weighted average calculation is performed on the effective depth information of multiple historical frames to complete the filling of the current invalid region.

[0095] For example, the historical valid depth information of each historical frame is projected onto the coordinate system of the current depth image through coordinate transformation, and then a weighted average method is used for fusion calculation to obtain the depth value of the invalid region in the repaired depth image. The calculation method is as follows:

[0096]

[0097] in For the first The fusion weight of historical frames, and the same view frame. frame capture from different perspectives , For the first Frame history frames The depth value of a pixel. For the first Frame history frame validity mask, when When the effective depth value is, Otherwise, it is 0. The frame number of the current frame.

[0098] In one possible implementation, the step of extracting feature points based on the reflection image and the restored depth image, obtaining the three-dimensional coordinates of the feature points through the restored depth image, and performing pose estimation in conjunction with inertial measurement data, yields a pose estimation result including the following:

[0099] Feature extraction algorithms are used to extract feature points from the reflection image. Three-dimensional coordinates are assigned to the feature points using the repaired depth image. Semantic trust weights are assigned to the feature points based on the semantic segmentation results. The semantic trust weights are used as weighting factors for the reprojection error. Weighted nonlinear optimization is performed by combining inertial measurement data and three-dimensional coordinates to obtain the camera relative pose as the pose estimation result.

[0100] It should be noted that feature extraction algorithms are used to extract stable and identifiable feature points from reflective images. In this embodiment, the ORB algorithm or SIFT algorithm is used. Feature points are clusters of pixels in a reflective image that have obvious texture or contour features. They can reflect the local features of an object and are the core basis for pose estimation.

[0101] 3D coordinates are the spatial coordinates of a feature point in the camera coordinate system, calculated from the pixel coordinates of the feature point and the corresponding depth value in the restored depth image. Semantic trust weights are weights assigned based on the semantic region type in which the feature point is located, used to distinguish the reliability of the feature point and affecting the optimization result of pose estimation.

[0102] Considering the weak texture areas in the bank lobby, such as sparse feature points on solid-color walls and low reliability of feature points in high-reflectivity areas, directly using all feature points for pose estimation can easily lead to increased cumulative errors and drift in the positioning trajectory. In one possible implementation, assigning semantic trust weights to feature points based on semantic segmentation results includes the following:

[0103] The semantic region type of each feature point is determined based on the semantic segmentation results; among them, the semantic region types include high reflectivity region, weak texture region, semantically significant object region, and strong texture region.

[0104] If a feature point is located in a high-reflectivity region or a weak-texture region, a semantic trust weight within a first preset weight range is assigned to that feature point.

[0105] If a feature point is located in a semantically significant object region or a strong texture region, a semantic trust weight within a second preset weight range is assigned to the feature point; the values ​​of the first preset weight range are all less than the values ​​of the second preset weight range.

[0106] It should be noted that the first preset weight range refers to the semantic trust weight range assigned to feature points in high-reflectivity or weak-texture regions; the values ​​are relatively small, reflecting lower reliability of these feature points. The second preset weight range refers to the semantic trust weight range assigned to feature points in semantically significant object regions or strong-texture regions; the values ​​are relatively large, reflecting higher reliability of these feature points.

[0107] In the specific implementation process, the ORB algorithm is used to extract feature points from the reflection image obtained in step 2. The maximum number of feature points is set, such as 2000, and the feature point threshold is set, such as 0.01. A set of feature points with stability and recognizability is extracted. Each feature point contains pixel coordinates and feature descriptors.

[0108] For each feature point in the feature point set, the corresponding depth value is read from the obtained restored depth image based on its pixel coordinates. Combined with the intrinsic parameters of the RGB-D camera, such as focal length and principal point coordinates, the 3D coordinates of the feature point in the camera coordinate system are calculated using the camera imaging model. The calculation formula is as follows:

[0109]

[0110] in These are the pixel coordinates of the feature point. Principal point coordinates , Focal length This refers to the depth value in the repaired depth image.

[0111] Based on the first semantic segmentation result obtained in the above implementation, the semantic region type of each feature point is determined. The semantic region types include high reflectivity region, weak texture region, semantically significant object region, and strong texture region. The weak texture region is determined by calculating the gray-level gradient variance of the 3×3 pixel region around the feature point. If the gray-level gradient variance is lower than a preset threshold (e.g., 50), it is a weak texture region. The semantically significant object region includes regions with obvious semantic features such as ATM machines, information screens, signs, and fixed furniture. The strong texture region is a region with rich texture and obvious gray-level gradient changes, other than the above three types of regions.

[0112] If the feature point is located in a high-reflectivity area or a weak-texture area, a semantic trust weight within the first preset weight range is assigned; if the feature point is located in a semantically significant object area or a strong-texture area, a semantic trust weight within the second preset weight range is assigned.

[0113] Combining the inertial measurement data acquired in step 1, such as acceleration and angular velocity, a visual-inertial fusion pose estimation model is constructed. The weighted projection error is used as the cost function, and the expression for the cost function is:

[0114]

[0115] in For the current frame Compared to the previous frame The relative pose of the camera, For the feature point set, For the first Semantic trust weights for each feature point For the first The three-dimensional coordinates of each feature point This is a camera projection function that projects 3D coordinates into pixel coordinates. For the first Each feature point in the current frame pixel coordinates, Let be the covariance matrix of the projection error. The cost function is minimized using the Gauss-Newton method or the Levenberg-Marquardt method to obtain the camera's relative pose. This is then fused with IMU pre-integration constraints to output the pose estimation result.

[0116] Step 4: Perform loop closure detection by combining geometric matching with auxiliary constraints to filter out valid loops.

[0117] It should be noted that geometric matching is a matching method based on the bag-of-words (BoW) model, which filters potential loop closure candidate frames by comparing the similarity of feature descriptors between the current frame and historical frames.

[0118] Effective loopback refers to the loop formed by the frame collected when the device actually returns to the historical position and the current frame, which can provide reliable constraints for backend pose graph optimization.

[0119] In one possible implementation, loop closure detection is performed using geometric matching combined with auxiliary constraints, and valid loop closures are selected, including the following:

[0120] The current frame corresponding to the current image data is determined, and the current frame is matched with the historical frame set to obtain multiple potential loop closure candidate frames and the geometric matching score corresponding to each potential loop closure candidate frame; the historical frame set consists of historical frames corresponding to multiple historical image data frames.

[0121] Semantic segmentation is performed on the RGB image of the current frame to obtain the second semantic segmentation result of the current frame; semantic segmentation is performed on the RGB images of each potential loop closure candidate frame to obtain the third semantic segmentation result of each potential loop closure candidate frame.

[0122] The semantic label matching degree of each potential loop closure candidate frame is calculated based on the second semantic segmentation result and the third semantic segmentation result.

[0123] The geometric matching score and the semantic label matching score are weighted and fused according to a preset balanced weight to obtain the loop closure score corresponding to each potential loop closure candidate frame.

[0124] Each loop closure score is compared with a preset threshold. If the loop closure score exceeds the preset threshold, the corresponding potential loop closure candidate frame is determined to be a valid loop closure.

[0125] It should be noted that the current frame refers to the frame corresponding to the image data acquired and processed at the current moment. The historical frame set refers to the collection of multiple frames of image data acquired and processed before the current frame. Potential loop closure candidate frames are historical frames selected through geometric matching that have a high similarity to the feature descriptor of the current frame; these may be frames acquired when the device returns to a historical position.

[0126] Geometric matching score: A parameter used to quantify the degree of geometric feature matching between the current frame and potential loop closure candidate frames; a higher score indicates a higher geometric feature similarity. Second semantic segmentation result: The result obtained by performing semantic segmentation on the RGB image of the current frame, used to calculate the semantic label information of the current frame. Third semantic segmentation result: The result obtained by performing semantic segmentation on the RGB images of each potential loop closure candidate frame, used to calculate the semantic label information of the potential loop closure candidate frames. Semantic label matching degree: A parameter used to quantify the consistency of semantic labels between the current frame and potential loop closure candidate frames, reflecting the degree of overlap and relative positional consistency of semantic regions in the two frames. Preset balance weight: A parameter used to balance the influence of geometric matching score and semantic label matching degree in loop closure scoring. Loop closure score: The final score obtained by weighted fusion of geometric matching score and semantic label matching degree, used to determine whether a potential loop closure candidate frame is a valid loop closure. Preset threshold: A critical value used to determine a valid loop closure; a loop closure score exceeding this threshold is considered a valid loop closure.

[0127] In the specific implementation process, geometric matching and potential loop closure candidate frame screening are performed as follows: A bag-of-words (BoW) model is constructed, and the feature descriptors of all images in the historical frame set are clustered to generate a bag-of-words dictionary. The feature descriptors of the reflection image of the current frame obtained in step 2 are extracted and matched with the bag-of-words dictionary to calculate the geometric matching score between the current frame and each historical frame in the historical frame set. t represents the current frame number, and k represents the historical frame number. A higher geometric matching score indicates a higher similarity in geometric features between the two frames. A geometric matching score threshold is set, and historical frames with geometric matching scores higher than this threshold are selected as potential loop closure candidate frames.

[0128] Using the same lightweight deep learning model as described above, semantic segmentation is performed on the RGB image of the current frame and the RGB images of each potential loop closure candidate frame to obtain a second semantic segmentation result and a third semantic segmentation result. Semantic label information of each frame is extracted from the two types of semantic segmentation results, including the category, location and range of the semantic region.

[0129] For each potential loopback candidate frame, the Jaccard similarity coefficient is used to calculate its semantic tag matching degree with the current frame. The calculation method is as follows:

[0130]

[0131] in This refers to the set of semantic regions in the second semantic segmentation result of the current frame. This refers to the set of semantic regions in the third semantic segmentation result of potential loop closure candidate frames. The intersection of two sets, that is, regions with the same semantic category and overlapping areas. It is the union of the two sets, that is, the total range of all semantic regions. The semantic tag matching degree ranges from 0 to 1, and the closer the value is to 1, the higher the consistency of the semantic tags between the two frames.

[0132] Set preset balance weights The geometric matching score and semantic tag matching score are weighted and fused to obtain the loop closure score. The calculation formula is:

[0133]

[0134] Set preset threshold The loop closure score of each potential loop closure candidate frame is compared with a preset threshold. If the loop closure score exceeds the preset threshold, the corresponding potential loop closure candidate frame is determined to be a valid loop closure.

[0135] Step 5: Combine the pose estimation results with the effective loop closures to perform pose graph optimization and obtain the corrected pose.

[0136] In one possible implementation, the step of combining the pose estimation results with effective loop closure to perform pose graph optimization to obtain the corrected pose includes the following:

[0137] A total error function is constructed by combining the odometry constraints corresponding to the pose estimation results, the loop closure constraints corresponding to the effective loop closure, and the IMU constraints corresponding to the inertial measurement data. By minimizing the total error function, a globally consistent camera pose sequence is obtained, and this camera pose sequence is used as the corrected pose.

[0138] It should be noted that odometry constraints refer to the constraints formed by the pose estimation results obtained in step 3, reflecting the pose relationship between adjacent frames. Loop closure constraints are formed by the valid loop closures obtained in step 4, reflecting the pose relationship between the current frame and historical valid loop closure frames. IMU constraints are obtained by pre-integrating the inertial measurement data acquired in step 1, reflecting the inertial characteristics of the device's motion. The total error function is a function constructed by combining the odometry constraint error, loop closure constraint error, and IMU constraint error; the pose sequence corresponding to its minimum value is a globally consistent pose sequence. The corrected pose is a globally consistent camera pose sequence obtained by minimizing the total error function, eliminating accumulated errors and achieving higher accuracy.

[0139] The pose estimation results obtained in step 3 contain accumulated errors. Over time, these errors can cause the pose sequence to deviate from the actual scene, affecting the accuracy of map construction. Effective loop closure can provide global constraints. By combining odometry constraints and IMU constraints to perform pose graph optimization, accumulated errors can be eliminated, resulting in a globally consistent corrected pose, providing a reliable pose benchmark for accurate mapping.

[0140] In the specific implementation process, the pose estimation results obtained in step 3 are converted into odometry constraints between adjacent frames. Each odometer constraint includes rotational and translational constraints; the valid laps obtained in step 4 are converted into lapsing constraints. Each loop closure constraint includes rotation and translation constraints between the current frame and historical valid loop closure frames; the inertial measurement data acquired in step 1 is pre-integrated to obtain the IMU constraints. This reflects the motion constraints of the device between adjacent timestamps.

[0141] Set the weight coefficients for each constraint, such as the weight of the odometer constraint. , loop constraint weight IMU constraint weights Construct the total error function The calculation formula is:

[0142]

[0143] in For camera pose sequence, The optimized camera pose sequence, This indicates summing over all constraints of the same kind.

[0144] A graph optimization method, such as the g2o framework, is used to solve for the total error function. The camera pose is treated as a node in the graph, and the constraints are treated as edges. The pose parameters of each node are adjusted iteratively to minimize the total error function. During the iteration process, the pose sequence is continuously updated until the total error converges to a preset range. The resulting pose sequence is the globally consistent corrected pose.

[0145] Step 6: Based on the corrected pose and repaired depth image, construct a 3D environment map.

[0146] In one possible implementation, the construction of a three-dimensional environment map based on the corrected pose and the repaired depth image includes the following: using the corrected pose as a spatial reference, combining it with the repaired depth image, and performing fusion processing using a preset three-dimensional mapping algorithm to construct a three-dimensional environment map.

[0147] It should be noted that the corrected pose refers to the globally consistent camera pose sequence obtained in step 5, providing a precise spatial reference for 3D mapping. The repaired depth image refers to the complete and accurate depth image obtained, providing reliable distance data for 3D mapping. The preset 3D mapping algorithm is used to fuse the corrected pose and the repaired depth image to construct a 3D environment map. In this embodiment, the truncated symbolic distance function (TSDF) or the semantic octree (OctoMap) algorithm is used. The 3D environment map refers to a 3D model that can reflect the actual spatial structure and object distribution of the bank lobby scene, containing distance and semantic information of the scene.

[0148] In the specific implementation process, the corrected pose obtained in step 5 is matched with the repaired depth image according to the timestamp to ensure that each frame of the repaired depth image corresponds to unique pose data. A preset 3D mapping algorithm, such as the semantic octree algorithm, is used to construct the 3D environment map. Using the corrected pose as the spatial reference, the depth information in each frame of the repaired depth image is converted into 3D point cloud data in the camera coordinate system. Coordinate transformation is then used to unify the 3D point cloud data of all frames into the global coordinate system. For each voxel in the global coordinate system, its distance information and semantic label are calculated by fusing multiple frames of 3D point cloud data. The semantic label is obtained from the first semantic segmentation result and stored in an octree structure. A truncated symbolic distance function is used to smooth the distance information of the voxels, eliminating noise interference and improving the smoothness and accuracy of the map.

[0149] The merged 3D environment map undergoes redundancy removal, eliminating duplicate voxel data and invalid data; map edges are smoothed and optimized to improve the map's visual appeal and usability; finally, a complete and accurate 3D environment map is output.

[0150] Example 2

[0151] This embodiment provides a positioning and mapping system for a bank under dynamic lighting conditions, applied to the aforementioned positioning and mapping method for a bank under dynamic lighting conditions, such as... Figure 2 As shown, it includes:

[0152] A multi-source data acquisition module is used to acquire image data and inertial measurement data in the banking environment, wherein the image data includes RGB images and depth images.

[0153] An adaptive illumination preprocessing module is used to perform illumination equalization processing on RGB images. It separates the illumination component and reflection component of the image through a multi-scale algorithm to obtain an illumination equalized reflection image.

[0154] The semantic depth pose module is used to perform defect repair processing on the depth image to obtain a repaired depth image; extract feature points based on the reflection image and the repaired depth image, obtain the three-dimensional coordinates of the feature points through the repaired depth image, and perform pose estimation in combination with inertial measurement data to obtain the pose estimation result.

[0155] A constraint-weighted loop closure detection module is used to perform loop closure detection by combining geometric matching with auxiliary constraints, and to filter out valid loops.

[0156] The pose graph optimization module is used to combine the pose estimation results with the effective loop closures to perform pose graph optimization and obtain the corrected pose.

[0157] A map building module is used to construct a three-dimensional environment map based on the corrected pose and the repaired depth image.

[0158] Example 3

[0159] This embodiment provides a readable storage medium storing computer instructions, characterized in that, when executed by a processor, the instructions implement the positioning and mapping method under dynamic lighting conditions in a bank as described above.

[0160] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for positioning and mapping under dynamic lighting conditions in banks, characterized in that, Includes the following steps: Step 1: Acquire image data and inertial measurement data in the banking environment, where the image data includes RGB images and depth images; Step 2: Perform illumination equalization processing on the RGB image, and separate the illumination component and reflection component of the image through a multi-scale algorithm to obtain the reflection image after illumination equalization; Step 3: Perform defect repair processing on the depth image to obtain the repaired depth image; extract feature points based on the reflection image and the repaired depth image, obtain the three-dimensional coordinates of the feature points through the repaired depth image, and perform pose estimation in combination with inertial measurement data to obtain the pose estimation result; Step 4: Perform loop closure detection using geometric matching combined with auxiliary constraints to filter out valid loops; Step 5: Combine the pose estimation results with the effective loop closures to perform pose graph optimization and obtain the corrected pose; Step 6: Based on the corrected pose and repaired depth image, construct a 3D environment map.

2. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 1, characterized in that: The defect repair processing performed on the depth image yields a repaired depth image including the following: Semantic segmentation is performed on the RGB image to obtain the first semantic segmentation result, and high reflectivity regions are marked based on the first semantic segmentation result; Obtain the depth value of this region in the current depth image. If the depth value is zero or exceeds the effective measurement range of the camera, the region is determined to be an invalid region of the current depth image. Before and after acquiring the current depth image, multiple frames of depth images of the banking environment are continuously acquired. The depth information corresponding to the areas in the multiple frames of depth images of the banking environment whose depth values ​​are neither zero nor exceed the effective measurement range of the camera is selected to form multiple frames of historical effective depth information. The effective depth information from multiple historical frames is projected onto the coordinate system of the current depth image, and the invalid areas of the current depth image are filled using a weighted average method to obtain the repaired depth image.

3. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 2, characterized in that, The effective depth information from multiple historical frames is projected onto the coordinate system of the current depth image, and an invalid region in the current depth image is filled using a weighted average method. This includes the following: Acquire the pose data of the current frame and the pose data of each historical frame, and calculate the pose similarity between each historical frame and the current frame; the pose similarity is obtained by weighting the rotation angle deviation and translation distance deviation between the current frame and the historical frames. If the pose similarity between a historical frame and the current frame is higher than a preset similarity threshold, then the historical frame is determined to be a frame with the same viewpoint, and its corresponding historical effective depth information is assigned the first fusion weight. If the pose similarity between a historical frame and the current frame is lower than a preset similarity threshold, the historical frame is determined to be an out-of-view frame, and its corresponding historical effective depth information is assigned a second fusion weight; the first fusion weight is greater than the second fusion weight. Based on the first and second fusion weights allocated above, a weighted average calculation is performed on the effective depth information of multiple historical frames to complete the filling of the current invalid region.

4. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 1, characterized in that, The process involves extracting feature points based on the reflection image and the restored depth image, obtaining the three-dimensional coordinates of the feature points from the restored depth image, and performing pose estimation using inertial measurement data. The resulting pose estimation results include the following: Feature extraction algorithms are used to extract feature points from the reflection image. Three-dimensional coordinates are assigned to the feature points using the repaired depth image. Semantic trust weights are assigned to the feature points based on the semantic segmentation results. The semantic trust weights are used as weighting factors for the reprojection error. Weighted nonlinear optimization is performed by combining inertial measurement data and three-dimensional coordinates to obtain the camera relative pose as the pose estimation result.

5. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 4, characterized in that, The assignment of semantic trust weights to feature points based on semantic segmentation results includes the following: The semantic region type of each feature point is determined based on the semantic segmentation results; among which, the semantic region types include high reflectivity region, weak texture region, semantically significant object region, and strong texture region; If a feature point is located in a high-reflectivity area or a weak-texture area, a semantic trust weight within a first preset weight range is assigned to that feature point. If a feature point is located in a semantically significant object region or a strong texture region, a semantic trust weight within a second preset weight range is assigned to the feature point; the values ​​of the first preset weight range are all less than the values ​​of the second preset weight range.

6. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 1, characterized in that, By performing loop closure detection using geometric matching combined with auxiliary constraints, valid loop closures are selected, including the following: Determine the current frame corresponding to the current image data, match the current frame with the historical frame set to obtain multiple potential loop closure candidate frames and the geometric matching score corresponding to each potential loop closure candidate frame; The historical frame set consists of historical frames corresponding to multiple frames of historical image data; Perform semantic segmentation on the RGB image of the current frame to obtain the second semantic segmentation result of the current frame; Semantic segmentation is performed on the RGB images of each potential loop closure candidate frame to obtain the third semantic segmentation result of each potential loop closure candidate frame; The semantic tag matching degree of each potential loop closure candidate frame is calculated based on the second semantic segmentation result and the third semantic segmentation result. The geometric matching score and the semantic label matching degree are weighted and fused according to a preset balanced weight to obtain the loop closure score corresponding to each potential loop closure candidate frame. Each loop closure score is compared with a preset threshold. If the loop closure score exceeds the preset threshold, the corresponding potential loop closure candidate frame is determined to be a valid loop closure.

7. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 1, characterized in that, The pose estimation results are combined with the effective loop closure to perform pose graph optimization, resulting in the corrected pose, which includes the following: By combining the odometry constraints corresponding to the pose estimation results, the loop closure constraints corresponding to the effective loop closure, and the IMU constraints corresponding to the inertial measurement data, a total error function is constructed. By minimizing the total error function, a globally consistent camera pose sequence is obtained, and this camera pose sequence is used as the corrected pose.

8. The method for positioning and mapping under dynamic lighting conditions in banks according to claim 1, characterized in that, Based on the corrected pose and repaired depth images, a 3D environment map is constructed, including the following: Using the corrected pose as a spatial reference and combining it with the repaired depth image, a preset 3D mapping algorithm is used for fusion processing to construct a 3D environment map.

9. A positioning and mapping system for banks under dynamic lighting conditions, characterized in that, The positioning and mapping method applied to the dynamic lighting environment of a bank as described in any one of claims 1-8 includes: A multi-source data acquisition module is used to acquire image data and inertial measurement data in the banking environment, wherein the image data includes RGB images and depth images; An adaptive illumination preprocessing module is used to perform illumination equalization processing on an RGB image. It separates the illumination component and reflection component of the image through a multi-scale algorithm to obtain an illumination equalized reflection image. The semantic depth pose module is used to perform defect repair processing on the depth image to obtain a repaired depth image; extract feature points based on the reflection image and the repaired depth image, obtain the three-dimensional coordinates of the feature points through the repaired depth image, and perform pose estimation in combination with inertial measurement data to obtain the pose estimation result. A constraint-weighted loop closure detection module is used to perform loop closure detection by combining geometric matching with auxiliary constraints, and to filter out valid loops. The pose graph optimization module is used to combine the pose estimation results with the effective loop closures to perform pose graph optimization and obtain the corrected pose. A map building module is used to construct a three-dimensional environment map based on the corrected pose and the repaired depth image.

10. A readable storage medium having computer instructions stored thereon, characterized in that, When executed by the processor, this instruction implements the location mapping method for a bank under dynamic lighting conditions as described in any one of claims 1-8.

Images