Reflection column scene abnormal depth detection method based on multi-modal fusion

Abstract

The invention discloses a reflective column scene abnormal depth detection method based on multi-modal fusion, and belongs to the technical field of abnormal depth detection. The method comprises the following steps: firstly, aligning RGB and IR images with different sizes, extracting a ground area in the RGB image through a segmentation network, and mapping the ground area into a depth map to accurately identify the ground; for the problem that the ground near the reflective column is difficult to distinguish, more reliable ground identification is realized by introducing a segmentation network. Meanwhile, for other abnormal noise points in the depth map, a method for screening by using the standard deviation relative difference of the two-frame depth difference and the single-frame standard deviation is provided, so that the robustness of random noise is enhanced. According to the method, the defects of an existing multi-mode method in a high-reflection scene are overcome, and a new solution thought is provided for quality improvement of the ToF depth data in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of abnormal depth detection, and particularly relates to a light reflection column scene abnormal depth detection method based on multi-modal fusion. BACKGROUND

[0002] In the navigation and environment perception of mobile robots (such as automatic guided vehicles, AGVs), fusing vision and depth information is a key means to achieve high-precision autonomous positioning, mapping and path planning. In current industrial and logistics scenarios, AGVs are often equipped with RGB-D cameras based on active optics (structured light, indirect or direct ToF, etc.) to obtain real-time dense three-dimensional information, assisting in completing tasks such as obstacle avoidance, target recognition and semantic understanding.

[0003] In actual application environments, light reflection columns are often designed as navigation reference markers due to their distinctive geometric structure and visual features, which are used to assist AGVs in achieving autonomous positioning. Light reflection columns are usually wrapped with high-reflectivity materials, which can provide good visibility in complex lighting environments. However, the high reflectivity and non-Lambertian characteristics of the surface of the light reflection column can cause significant degradation of the depth estimation of the RGB-D camera. Since the RGB-D camera estimates depth by actively emitting infrared light or structured light and receiving reflected signals, the saturation highlights of light, the pupil deflection caused by specular reflection, and the multi-path interference (MPI) of near-far paths can all destroy the physical assumptions of depth solving, further inducing artifacts such as depth holes, outliers, false depth stripes, and local point cloud drift or breakage. These artifacts not only affect local geometric reconstruction and surface normal estimation, but also amplify trajectory deviations and reduce safety margins in map construction, obstacle detection and path planning.

[0004] This abnormality is manifested as follows: in the image, the light reflection area forms an excessively saturated highlight patch, resulting in outliers or false depth data around the light reflection column, as shown in Figs. 15(a)-15(d), where it can be clearly observed that there are unrealistic depths around the light reflection column. On the point cloud level, such abnormal data will manifest as abnormal point clouds, structure damage or spatial drift, severely affecting the robustness and accuracy of downstream tasks such as map construction, obstacle detection and navigation path planning.

[0005] Therefore, how to accurately detect abnormal depth in light reflection column scenarios, while retaining the light reflection column as a navigation target, and suppressing the abnormal depth information and point cloud interference it brings, has become an important research problem for improving the reliability of AGV navigation. SUMMARY

[0006] The technical problems to be solved by the present application are: how to realize accurate detection of abnormal depth in a light column scene, and further, under the premise of retaining the light column as a navigation target, suppress the abnormal depth information and point cloud interference brought by the light column, and provide a light column scene abnormal depth detection method based on multi-modal fusion.

[0007] The present application solves the above technical problems by the following technical solutions, and comprises the following steps:

[0008] S1: multi-modal image input

[0009] Input the collected RGB image, IR image and original depth map;

[0010] S2: image registration

[0011] Establish a geometric mapping relationship from the IR image pixel coordinates to the RGB image pixel coordinates, realize pixel-level registration of the IR image and the RGB image based on the geometric mapping relationship, and obtain the registered RGB image;

[0012] S3: potential abnormal point screening

[0013] In the original depth map, the effective depth region after removing invalid values is sorted in ascending order, and the pixels of the set percentage are retained as potential abnormal points;

[0014] S4: light column detection

[0015] The Grounded-SAM segmentation network is used to segment the column in the registered RGB image, to obtain a plurality of candidate column regions, then the IR radiation intensity analysis and pixel ratio judgment are used to determine the light column candidate region, and then the brightness threshold judgment is used to obtain the light column candidate column, and then the light column region is obtained, the boundary box of the light column region is expanded to the surrounding, to obtain a detection buffer zone, and the potential abnormal points appearing in the detection buffer zone are determined as abnormal depth regions;

[0016] S5: complete depth map

[0017] The depth value of the pixel in the abnormal depth region in step S4 is directly set as the maximum value, and for the pixel with a depth value of 0, a neighborhood window of a set size is extracted with the pixel as the center, if there is at least one effective pixel with a depth value greater than 0 in the neighborhood window, the depth value of the effective pixel is completed as the average value of all effective depth values in the neighborhood, and then the completed depth map is obtained;

[0018] S6: abnormal point judgment

[0019] With the completed depth map in step S5, the standard deviation of a single frame depth map and the standard deviation of the difference between two frames of depth maps are calculated, and a standard deviation relative difference threshold is judged. Pixels greater than the threshold are considered abnormal points, and pixels less than the threshold are considered real points, and the coordinates are stored respectively;

[0020] S7: Selection of coordinates of ground points

[0021] The Grounded-SAM segmentation network is used to segment the ground area in the registered RGB image, and the ground area is mapped to the depth map. The abnormal depth caused by protrusions is screened out, the pixels belonging to the ground are selected, and the coordinates corresponding to the pixels are read;

[0022] S8: Result visualization

[0023] The results of steps S6 and S7 are visualized and output, and the abnormal depth detection in the reflective column scene is completed.

[0024] Further, in the step S4, the specific processing process is as follows:

[0025] S41: Using a semantic segmentation network based on deep learning to perform instance segmentation on objects in the RGB image, and segmenting multiple candidate column regions;

[0026] S42: Based on the registration mapping relationship between the RGB image and the IR image established in step S2, the candidate column region is mapped to the corresponding position in the IR image;

[0027] S43: Perform pixel-level statistical analysis on each candidate column region in the IR image, calculate the proportion of pixels with infrared intensity exceeding a set value, and select the candidate box with a proportion exceeding a threshold as a reflective column candidate region, and map the reflective column candidate region to the original depth map;

[0028] S44: In the reflective column candidate region, by comparing the sum of IR values of each column of pixels in the region with the global average column brightness, when the brightness of any column exceeds a set threshold, the column is determined as a reflective column candidate column, and the longest continuous segment in the reflective column candidate column is finally selected as the reflective column region;

[0029] S45: Expand the boundary box of the reflective column region to the surrounding area to obtain a detection buffer zone, and determine the potential abnormal points in the buffer zone as abnormal depth regions.

[0030] Further, in the step S5, the entire completion process is based on the original depth map, and the interpolated values are not affected by the completed values; the set maximum value is a fixed numerical value.

[0031] Further, in the step S6, the position information of each pixel in the completed depth map is (x, y, z, r, c), wherein x, y and z are the spatial position of the pixel in the three-dimensional coordinate system of the camera; z is the depth value d of the pixel; r and c are the row coordinate and column coordinate of the pixel in the depth image matrix.

[0032] Further, in the step S6, the specific processing process is as follows:

[0033] S61: Calculate the standard deviation of single-frame depth value and the weight

[0034] For the position of the potential abnormal point, the standard deviation is directly assigned as 1 and the weight is directly assigned as 1; for the pixel with a depth value less than or equal to 1, it is considered to be located in the undetected area or the position of the light column body, and the standard deviation is directly assigned as 0; for the pixel with a depth value greater than 1, a fixed-size neighborhood window is extracted around the pixel, all valid pixels with a depth value greater than 0 in the neighborhood window are extracted, and the standard deviation is calculated , that is, the single-frame standard deviation is obtained; at the same time, a neighborhood validity weight is calculated for each pixel position, which is defined as the fourth power of the ratio of the number of pixels with valid depth values in the neighborhood of the pixel to the total number of pixels in the neighborhood, that is:

[0035] ;

[0036] wherein is the number of pixels with valid depth values, is the total number of pixels in the neighborhood;

[0037] S62: Calculate the depth difference between two frames and the weight

[0038] When calculating the depth difference , only the pixels with valid depth values at the corresponding positions in the two completed depth maps are processed, and the depth difference is defined as the absolute value of the difference between the two; if any pixel has no valid depth value in any one of the two maps, the depth difference is set to a maximum value to represent an invalid or unavailable difference value, and a neighborhood validity weight is calculated for the pixel position, and the calculation method is the same as that of the neighborhood validity weight in step S61;

[0039] S63: Calculate the standard deviation of the depth difference between two frames

[0040] For the pixels effective for depth difference, a fixed size neighborhood window is extracted centered on the pixel, all effective pixels with depth value greater than 0 are screened in the neighborhood window, and the standard deviation is calculated based on the depth difference of the effective pixels, that is:

[0041] ;

[0042] Wherein, i is each pixel in the neighborhood, and N is the total number of pixels in the neighborhood;

[0043] S64: Calculate the relative difference of standard deviation

[0044] When the standard deviation And the standard deviation Both are not equal to 0, the calculation is as follows:

[0045]

[0046]

[0047] After calculating the relative difference of standard deviation , a threshold is set, pixels greater than the threshold are considered as abnormal points, and pixels less than the threshold are considered as real points, and the coordinates are stored respectively.

[0048] Compared with the prior art, the present application has the following advantages: the light column scene abnormal depth detection method based on multi-modal fusion firstly aligns the RGB and IR images of different sizes, extracts the ground area in the RGB image through the segmentation network, and maps it to the depth map to accurately identify the ground; for the problem that the ground near the light column is difficult to distinguish, a more reliable ground identification is realized by introducing the Grounded-SAM segmentation network. At the same time, for other abnormal noise points in the depth map, a method of using the standard deviation of the standard deviation of two frames of depth difference and single frame standard deviation for screening is proposed to enhance the robustness to random noise. This method not only makes up for the shortcomings of existing multi-modal methods in high light reflection scenes, but also provides a new solution for improving the quality of ToF depth data in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is a flowchart of the light column scene abnormal depth detection method based on multi-modal fusion in the embodiment of the present application;

[0050] Figure 2 is an overlap chart of the RGB image and the IR image when not registered in the embodiment of the present application;

[0051] Figure 3 is a schematic diagram of the image registration process in the embodiment of the present application;

[0052] Figure 4is an overlapping image of the registered RGB image and IR image in embodiments of the present application;

[0053] Figure 5 is a depth distribution result image of the depth map in embodiments of the present application;

[0054] Figure 6 is an effect image of screening the abnormal area of pseudo-depth of the depth map in embodiments of the present application;

[0055] Figure 7 is a schematic diagram of the selection process of the light column by the segmentation network in embodiments of the present application;

[0056] Figure 8 is a schematic diagram of the selection process of the pseudo-depth abnormal pixels in the light column buffer area in embodiments of the present application;

[0057] Figure 9 is a partial depth completion image example in embodiments of the present application;

[0058] Figure 10 is a schematic diagram of the abnormal depth screening result near and far from the light column in embodiments of the present application;

[0059] Fig. 11(a) is a scene depth map in embodiments of the present application;

[0060] Fig. 11(b) is a scene RGB image in embodiments of the present application;

[0061] Figure 12 is a selection result image of the ground pixels in embodiments of the present application;

[0062] Figure 13 is a screening effect image (green part) of the ground pixels in embodiments of the present application;

[0063] Fig. 14(a) is a screening effect image of the ground pixels in a gymnasium in embodiments of the present application;

[0064] Fig. 14(b) is a screening effect image of the ground pixels in a runway in embodiments of the present application;

[0065] Fig. 14(c) is a screening effect image of the ground pixels in a dance room in embodiments of the present application;

[0066] Fig. 15(a) is a depth map example in the prior art;

[0067] Fig. 15(b) is a point cloud image example in the prior art;

[0068] Fig. 15(c) is an IR image example in the prior art;

[0069] Fig. 15(d) is an RGB image example in the prior art. DETAILED DESCRIPTION

[0070] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0071] like Figure 1 As shown, this embodiment provides a technical solution: a method for detecting abnormal depth in reflective pillar scenes based on multimodal fusion, including the following main contents:

[0072] In the process of removing abnormal depths, considering the significant differences in depth data across different regions, this invention does not employ a uniform global filtering strategy. Instead, it divides the overall depth into three sub-parts and processes them separately based on their respective characteristics. Specifically, regions near reflective pillars often produce localized false depths or holes due to direct specular reflection, requiring targeted outlier suppression. Regions far from reflective pillars are easily affected by indirect specular reflection, manifesting as outliers or drifting point clouds, requiring removal through distributional filtering. Ground transition areas connected to reflective pillars, due to reflection interference, often make it difficult to effectively distinguish the ground from adjacent low-lying objects; therefore, they are separated and subjected to a dedicated filtering strategy. This step-by-step filtering method based on region division allows for more targeted processing of depth artifacts from different sources, improving point cloud quality while preserving effective information to the greatest extent possible.

[0073] Firstly, regarding the area near the reflective pillars, based on the measured and analyzed depth map results, this region, as the primary specular reflective surface, is directly affected by multipath interference and signal saturation, often inducing anomalous depth values ​​in its vicinity. These anomalous values ​​spatially overlap with or are adjacent to actual objects, but their depth is much greater than the actual distance to the object, exhibiting a significant deviation from the camera's "pseudo-long-distance" representation. This phenomenon mainly stems from the interference or multipath reflection of the emitted light path of the active optical depth sensor when encountering highly reflective surfaces, causing the received signal to be incorrectly interpreted as originating from a distant location, thus generating anomalous long-distance depth points. Further analysis revealed that these anomalous depths are typically concentrated in the region with the highest depth values ​​in the entire image, and their numerical characteristics significantly differ from the true depth distribution in the scene. Therefore, by sorting the depth map or setting a high-level threshold, the maximum value region at the top layer of the depth distribution can be directly extracted and removed, effectively filtering out anomalous depth points caused by reflective pillars.

[0074] Secondly, areas far from the reflective pillars are affected by indirect specular reflection (stray light) from them, leading to a decrease in signal-to-noise ratio and random fluctuations and outliers in depth measurements. Considering the randomness and instability of depth value fluctuations caused by reflective interference, a time-series inter-frame stability index is further introduced for discrimination. Specifically, the standard deviation of the depth value of the target region in the current frame and the standard deviation of the depth difference between corresponding regions in adjacent frames are calculated, and the relative difference of their standard deviations is constructed as an anomaly index. Experiments show that anomalous depth points fluctuate significantly over time, and their inter-frame difference standard deviation is significantly higher than the intra-frame fluctuation level. Therefore, the larger the relative difference of this standard deviation, the higher the probability that the region is an anomaly caused by reflective interference.

[0075] Finally, specifically regarding ground depth, due to the strong reflective properties of the reflective surface, TOF cameras are prone to multipath reflections during measurement, causing the depth values ​​around the reflective pillars to be misjudged as greater than the true values. When converting this depth data into a point cloud, points with larger depth values ​​have larger Y values, appearing as if the point is elevated in the 3D point cloud. For example, some point clouds that should belong to the ground may be misidentified as ground protrusions or structures, and this error is particularly noticeable in ground areas near the reflective pillars. Therefore, a combination of segmentation networks and depth maps is used to filter ground pixels.

[0076] By using the above step-by-step filtering method, it is basically possible to filter out abnormal depth areas caused by reflective pillars, as well as the areas where real objects and the ground are located.

[0077] The specific processing steps of the method of the present invention are as follows:

[0078] 1. Image registration

[0079] In multimodal image processing scenarios, due to the inherent characteristics of camera hardware and differences in imaging principles, the acquired RGB images and IR (infrared) images often exhibit size inconsistencies, as shown below. Figure 2 As shown, this size difference severely affects the accuracy of subsequent image-based operations, such as feature point selection, target localization, and multimodal data fusion. To ensure accurate selection of corresponding positions and precise spatial correlation between the two modal image data during subsequent processing, perspective transformation is needed to geometrically correct the RGB image. This eliminates the spatial offset caused by differences in imaging size and viewing angle, achieving high-precision alignment of the RGB and IR images within the same geometric reference frame. This minimizes spatial misalignment caused by differences in image size and viewing angle, laying a solid foundation for subsequent multimodal image analysis and processing.

[0080] To achieve strict registration between the two images, a geometric mapping relationship needs to be established from the pixel coordinates of the infrared image to the pixel coordinates of the RGB image. This mapping depends on the intrinsic and extrinsic parameters of the two cameras, as well as the depth information provided by the infrared images. The specific process is as follows: Figure 3 As shown.

[0081] To align the IR image with the RGB image at the pixel level, a perspective mapping between pixels and camera coordinates is first established using a pinhole camera model. The imaging process of a single camera can be described by the following perspective projection model:

[0082] ;

[0083] in, Let be the pixel coordinates on the image plane, s be the scale factor, and in perspective projection... , Let K be the 3D coordinates of the point in the camera coordinate system, and K be the camera intrinsic parameter matrix.

[0084] ;

[0085] in, , The focal lengths are the horizontal and vertical focal lengths. The coordinates of the main point.

[0086] Infrared cameras provide depth information for each pixel. That is, the distance of the spatial point corresponding to the pixel in the infrared camera coordinate system. Let the infrared pixel coordinates be... Based on the pinhole model, it can be back-projected onto the infrared camera coordinate system:

[0087] ;

[0088] ;

[0089] ;

[0090] in, , The focal length of the infrared camera in the x and y directions; , These are the principal point coordinates of the infrared camera, i.e., the offset of the image center in the pixel coordinate system; , , The coordinates of the point in space are the three-dimensional coordinates of the infrared camera in the coordinate system.

[0091] This allows us to obtain the three-dimensional point of the pixel in the infrared camera coordinate system:

[0092] .

[0093] Then, using the extrinsic parameters (rotation matrix R and translation vector T) obtained from the dual-target calibration, the points in the infrared camera coordinate system are transformed to the RGB camera coordinate system:

[0094] ;

[0095] in, , , The coordinates of the point in space are the three-dimensional coordinates in the RGB camera coordinate system.

[0096] In obtaining Then, through the intrinsic parameters of the RGB camera Projecting the 3D points back onto the RGB image plane yields the corresponding pixel coordinates:

[0097] ;

[0098] ;

[0099] in, , These are the pixel coordinates in the RGB image; , The focal length of the RGB camera in the x and y directions; , These are the coordinates of the principal point of the RGB camera.

[0100] At the implementation level, the depth validity of each infrared pixel should first be determined (e.g., ), and check after projection And whether the projected coordinates fall within the pixel range of the RGB image (boundary check). During sampling, the color values ​​can be read from the RGB image using nearest neighbor or bilinear interpolation and assigned back to the output image based on the IR pixel coordinates, thus obtaining the color alignment result in the IR coordinate system.

[0101] Registration quality is quantitatively evaluated using metrics such as reprojection error (MRE), inlier ratio, and mutual information (MI). Under normal circumstances, for cross-modal images, an inlier ratio ≥ 0.6 indicates good matching quality, 0.4–0.6 is acceptable, and below 0.4 indicates poor matching. An MRE ≤ 2 is generally considered to indicate high geometric alignment accuracy, with 2–4 being acceptable. The MI value varies with modal differences; its absolute value is small, but there should be a significant improvement after registration compared to unregistered images. Experimental results show that the registration results in this embodiment have an inlier ratio between 0.42 and 0.63, an MRE between 0.66 and 2.3 pixels, and an MI between 0.22 and 0.35. Therefore, it can be considered that the image registration method used in this embodiment, while ensuring modal consistency, possesses good geometric accuracy and stability, meeting the accuracy requirements of practical engineering applications such as multimodal image fusion and target detection.

[0102] Minor spatial deviations can still be observed in the registration results, primarily due to the combined effects of the imaging model and the actual environment. First, ToF cameras inevitably exhibit radial and tangential distortion. Even with distortion parameters introduced for correction during calibration, limitations in calibration accuracy, temperature variations, and optical component stability may leave slight distortion errors, thus violating the ideal pinhole imaging assumption and causing a shift between the theoretical projection position calculated based on intrinsic and extrinsic parameters and the actual imaging coordinates. Second, highly reflective cylindrical surfaces in the scene can induce strong specular reflections, leading to ranging saturation and multipath propagation in the infrared depth sensor, resulting in instability or anomalies in local depth values. Since the registration process heavily relies on depth information for 3D backprojection and projection mapping, these local depth errors will directly propagate to the final reprojection result, manifesting as subtle misalignments or deformations in local areas. Therefore, even with high overall registration accuracy, non-negligible sub-pixel-level deviations may still occur in areas with highly reflective materials.

[0103] 2. Screening of potential outliers

[0104] Visual analysis of numerous depth maps reveals that, near highly reflective structures (such as reflective pillars), some pixels exhibit significantly higher depth values ​​than the actual depth in the scene, displaying a clear "pseudo-distant" characteristic. These anomalous depth values ​​are typically caused by infrared interference, sensor noise, or multiple surface reflections, severely impacting the spatial consistency of the depth map and the accuracy of subsequent processing. To eliminate this unreliable data, this invention employs a depth value screening strategy based on statistical distribution: valid depth regions (excluding invalid values, such as depth values ​​of 0 or 1) are sorted in ascending order, retaining the last 8% of pixels as potential pseudo-depth anomaly regions. This percentage is determined based on the statistical characteristics of depth value distribution: in numerous depth maps, anomalous responses typically manifest as pixels at the tail end of the distribution, with a stable proportion of approximately 5–10%. By comparing the removal effects of different thresholds, this invention selects 8% as a compromise, covering the vast majority of pseudo-distant points while avoiding excessive damage to genuine valid pixels. Figure 5 As shown.

[0105] Experiments show that selecting only the last 8% of depth values ​​can effectively identify and eliminate most abnormal depth responses without requiring external prior information, thereby improving the overall quality and robustness of the depth map. The results are shown in the image below. Figure 6 As shown.

[0106] 3. Selection of reflectors

[0107] Analysis of the experimental results from step 2 shows that the pseudo-depth anomaly region is mainly distributed around the reflective pillars. Therefore, accurately identifying the spatial location of the reflective pillars is a prerequisite for effective anomaly screening. This invention uses a Grounded-SAM-based instance segmentation network to segment objects in the registered RGB image, obtaining the bounding box coordinates of various objects. Although current mainstream segmentation networks still have certain limitations in fine-grained category recognition and cannot directly identify the specific category of "reflective pillar," they can stably and with high confidence segment its superior semantic category "pillar."

[0108] Based on the established RGB-IR image registration mapping relationship, the "pillar" regions detected in the RGB image can be accurately mapped to their corresponding positions in the IR image. Building upon this, and considering the radiometric response characteristics of the IR image, this invention proposes using infrared radiation intensity as the criterion for identifying reflective pillars. Specifically, pixel-level statistical analysis is performed on each candidate "pillar region," calculating the percentage of pixels with infrared intensity values ​​exceeding 1600, and selecting candidate boxes with a percentage exceeding a threshold as reflective pillar candidate regions. Since the depth map and IR map have the same spatial resolution and pixel correspondence, the identified reflective pillar candidate regions can be further mapped to the original depth map. Within the reflective pillar candidate region, the sum of the IR values ​​of each column of pixels within that region is compared with the global average column brightness. When the brightness of a certain column exceeds a predetermined threshold (0.5 times the average brightness), that column is identified as an accurate reflective pillar candidate column. Finally, the longest continuous segment in the reflective pillar candidate column is selected as the precise location of the reflective pillar. The process of selecting reflective pillars is as follows: Figure 7 As shown.

[0109] Based on this, the bounding box of the reflector is expanded outwards to construct a suitable detection buffer (such as...). Figure 8 (in the yellow box), and identify potential anomalies in the buffer that match the description in step 2 as anomaly depth regions (in red) to achieve high-confidence deep anomaly removal within the target region.

[0110] This step, through multimodal data fusion and radiation characteristic analysis, enabled the precise location of the reflective pillars, providing reliable spatial constraints for subsequent anomaly point screening.

[0111] 4. Complete the depth map

[0112] First, for the potential abnormal points in the abnormal depth area identified in step 3, the depth value of these coordinate points is directly set to 10000mm.

[0113] For a pixel with a depth value of 0, extract a 3×3 neighborhood window centered on it. If there is at least one valid pixel with a depth value greater than 0 within this neighborhood, pad the pixel's depth value to the average of all valid depth values ​​(i.e., greater than 0) within the neighborhood. Partial depth padding is as follows: Figure 9 As shown. The entire completion process is based on the original depth map, ensuring that the interpolation is not affected by the already completed values.

[0114] 5. Single-frame depth value

[0115] Read the position information (x, y, z, r, c) of each pixel in the completed depth map (excluding the background, the reflector itself and the abnormal points, whose depth values ​​are defined as 1).

[0116] x, y, z represent the spatial position of the pixel in the camera's three-dimensional coordinate system;

[0117] z is the depth value of that pixel. ;

[0118] r and c represent the row and column coordinates of the pixel in the depth image matrix.

[0119] 6. Calculate the standard deviation of the depth values ​​in a single frame. and weight

[0120] For outliers (the top 8% of pixels in the depth value filtering step), a standard deviation of 1 and a weight of 1 are directly assigned (this is done to prevent outliers from being ignored when their standard deviation is 0 when distinguishing coordinate point attributes later by the relative difference in standard deviation). Pixels with a depth value less than or equal to 1 are considered to be located in undetected areas or at the location of the reflective pillar. Since the depth values ​​after completion in these areas are not representative, they are not included in the standard deviation calculation and are directly assigned a standard deviation of 0. For pixels with a depth value greater than 1, a fixed-size neighborhood window (e.g., 9×9) is extracted centered on that point. All valid pixels with a depth value greater than 0 are extracted from this neighborhood, and their standard deviation is calculated as a measure of the depth fluctuation of that pixel.

[0121] Simultaneously, a neighborhood validity weight is calculated for each pixel location. This weight is defined as the fourth power of the ratio of the number of pixels in the surrounding neighborhood (e.g., a 9×9 region) that have valid depth values ​​in both images to the total number of pixels in the neighborhood (typically 81), i.e.:

[0122]

[0123] in, The number of pixels representing the effective depth value. This represents the total number of pixels in the neighborhood.

[0124] 7. Calculate the depth difference between two frames. and weight

[0125] The pixel-by-pixel difference and neighborhood weights are jointly calculated for two completed depth maps.

[0126] In calculating depth difference Only pixels with valid depth values ​​(greater than 0 and not equal to 10000) at corresponding positions in both depth maps are processed, and their depth difference is defined as the absolute value of the difference between the two. If a pixel has no valid depth value in either map, its depth difference is set to a preset maximum value (such as 10000) to represent an invalid or unusable difference. Simultaneously, the weight of the valid value at that position within its neighborhood is calculated (using the same method as the weight calculation for single-frame depth).

[0127] 8. Calculate the standard deviation of the depth difference between two frames.

[0128] This process aims to evaluate the standard deviation and neighborhood weights of an image based on the difference information between two depth maps.

[0129] If the depth difference at a certain pixel location is a maximum value (set to 10000, as explained in step 4), it means that at least one pixel at that location has an invalid value (depth of 0) in the depth maps of the two frames, and there is no basis for difference analysis. Therefore, the standard deviation of this pixel location is directly set to the maximum value (10000), and its neighborhood weight is set to 1 to ensure numerical stability and comparability in subsequent processing. For pixels with valid depth differences (i.e., not 10000), a fixed-size neighborhood (e.g., 9×9) is extracted centered on that pixel. All valid pixels with depth values ​​greater than 0 are filtered within this neighborhood, and the standard deviation is calculated based on the depth differences of these pixels, i.e.:

[0130]

[0131] Where i represents each pixel in the neighborhood, and N represents the total number of pixels in the neighborhood;

[0132] Divide the standard deviation by This ensures reasonable scaling normalization when performing relative standard deviation analysis with other types of standard deviations. Weight calculation is the same as above.

[0133] 9. Calculate the relative difference in standard deviation.

[0134] Only when the standard deviation and standard deviation Calculations are performed when all values ​​are not equal to 0:

[0135]

[0136]

[0137] relative differences in standard deviation After calculation, a threshold is set. Pixels larger than the threshold are considered outliers (red), and pixels smaller than the threshold are considered real pixels (blue). Their coordinates are stored. The filtering results are as follows: Figure 10As shown.

[0138] 10. Selection of ground point coordinates

[0139] The foregoing analysis reveals that, due to the reflective properties, the depth information of the part where the column contacts the ground is difficult to estimate accurately. This error is particularly noticeable in the ground area near the reflective column. As shown in Figures 11(a) and 11(b), the area within the red boxes in both figures clearly shows a flat ground beneath the reflective column in the RGB image, but in the depth image, the point cloud in this area is more likely to be interpreted as an object on the ground. (This phenomenon occurs both indoors and outdoors, but is more pronounced indoors; therefore, an indoor scene is chosen as an example here.)

[0140] To address this issue, this invention introduces a ground detection method based on a Grounded-SAM segmentation network to accurately extract ground regions from RGB images, thereby compensating for the lack of depth information. The segmentation network maps ground regions to a depth map, effectively and accurately identifying depth pixels belonging to the ground. Figure 12 As shown.

[0141] This method avoids misclassifying ground points as foreground objects due to abnormally high ground depth caused by highly reflective materials (such as reflective pillars). In other words, this strategy can distinguish between ground height deviations caused by surface reflections and targets that actually exist on the ground, thereby improving the accuracy of ground detection and object recognition. Figure 13 As shown.

[0142] 11. Filtering for abnormal depths

[0143] Based on the aforementioned depth value distribution filtering strategy, experimental results show that this method can accurately extract the true depth of cardboard boxes or flower pots in the images used in this embodiment, and effectively suppress false depth caused by strong reflections from reflective pillars. Further verification experiments on multiple scenes and combinations of objects, such as stadiums, running tracks, and dance studios, demonstrate that this method has good generalization and filtering effects, as shown in Figures 14(a)-14(c).

[0144] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multi-modal fusion-based light column scene abnormal depth detection method, characterized in that, The method comprises the following steps: S1: multi-modal image input Input the collected RGB image, IR image and original depth map; S2: image registration Establish a geometric mapping relationship from the IR image pixel coordinates to the RGB image pixel coordinates, realize pixel-level registration of the IR image and the RGB image based on the geometric mapping relationship, and obtain a registered RGB image; S3: potential abnormal point screening In the original depth map, the valid depth region after removing invalid values is sorted in ascending order, and a set percentage of pixels are retained as potential abnormal points; S4: light column detection The Grounded-SAM segmentation network is used to segment the column in the registered RGB image to obtain a plurality of candidate column regions, then the IR radiation intensity analysis and pixel ratio judgment are used to determine the light column candidate region, and then the light column candidate column is obtained through the brightness threshold judgment, and then the light column region is obtained, and the light column region boundary box is expanded to the surrounding to obtain a detection buffer zone, and the potential abnormal points appearing in the detection buffer zone are determined as abnormal depth regions; S5: complete the depth map The depth value of the pixel in the abnormal depth region in step S4 is directly set as the maximum value, and for the pixel with a depth value of 0, a neighborhood window of a set size is extracted centered on the pixel, if there is at least one valid pixel with a depth value greater than 0 in the neighborhood window, the depth value of the valid pixel is completed to the average value of all valid depth values in the neighborhood, and then the completed depth map is obtained; S6: abnormal point judgment Using the completed depth map in step S5, the standard deviation of a single frame depth map and the standard deviation of the difference between two frames of depth maps are calculated, and the standard deviation relative difference threshold is judged, the pixels greater than the threshold are regarded as abnormal points, and the pixels less than the threshold are regarded as real points, and the coordinates are stored respectively; S7: coordinate selection of ground points The Grounded-SAM segmentation network is used to segment the ground region in the registered RGB image, the ground region is mapped to the depth map, the abnormal depth caused by the protrusion is screened out, the pixels belonging to the ground are selected, and the coordinates corresponding to the pixels are read; S8: result visualization The results in steps S6 and S7 are visualized and output, and the abnormal depth detection in the light column scene is completed.

2. The method according to claim 1, wherein, In the step S4, the specific processing process is as follows: S41: using a semantic segmentation network based on deep learning to perform instance segmentation on the objects in the RGB image, and segmenting a plurality of candidate column regions; S42: based on the registration mapping relationship between the RGB image and the IR image established in step S2, mapping the candidate column regions to the corresponding positions in the IR image; S43: performing pixel-level statistical analysis on each candidate column region in the IR image, calculating the pixel ratio of infrared intensity exceeding a set value, and selecting the candidate box with a ratio exceeding a threshold as a light column candidate region, and mapping the light column candidate region to the original depth map; S44: In the light column candidate region, by comparing the sum of IR value of each column pixel in the region with the global average column brightness, when the brightness of any column exceeds the set threshold, the column is determined as a light column candidate column, and finally the longest continuous segment in the light column candidate column is selected as the light column region; S45: The light column region boundary box is expanded to the surrounding to obtain a detection buffer zone, and potential abnormal points appearing in the buffer zone are determined as abnormal depth regions.

3. The method according to claim 1, wherein, In the step S5, the entire completion process is based on the original depth map, and is not affected by the completed values during interpolation; the set maximum value is a fixed number.

4. The method according to claim 1, wherein, In the step S6, the position information of each pixel in the completed depth map is (x, y, z, r, c), wherein x, y and z are the spatial positions of the pixel in the three-dimensional coordinate system of the camera; z is the depth value d of the pixel; r and c are the row coordinate and column coordinate of the pixel in the depth image matrix.

5. The method according to claim 4, wherein, In the step S6, the specific processing process is as follows: S61: Calculate standard deviation of single frame depth value and weights For the position of potential outliers, directly assign the standard deviation as 1 and the weight as 1; for the pixels with depth value less than or equal to 1, consider that they are located in the undetected area or the position of the light column body, directly assign the standard deviation as 0; for the pixels with depth value greater than 1, extract a fixed size neighborhood window centered on the pixel, extract all valid pixels with depth value greater than 0 in the neighborhood window, and calculate the standard deviation , that is, obtain the single-frame standard deviation; at the same time, calculate a neighborhood validity weight for each pixel position , which is defined as the fourth power of the ratio of the number of pixels in the neighborhood around the pixel that have valid depth values in both the two completed depth maps to the total number of pixels in the neighborhood, that is: ； wherein, the number of pixels for effective depth value, is the total number of pixels in the neighborhood; S62: Calculate the depth difference between two frames and weights In calculating the depth difference , only the pixels with valid depth values in both corresponding positions of the two completed depth maps are processed, and the depth difference is defined as the absolute value of the difference between the two; if any pixel has no valid depth value in any one of the two maps, the depth difference is set to a maximum value to represent an invalid or unavailable difference, and a neighborhood validity weight for the pixel position is calculated in the same way as the calculation of the neighborhood validity weight in step S61. S63: Calculate standard deviation of the difference of two frames depth For the depth difference effective pixel, a neighborhood window of a fixed size is extracted with the pixel as the center, all effective pixels with a depth value greater than 0 are screened in the neighborhood window, and the standard deviation is calculated based on the depth difference of the effective pixels, that is: ； Wherein i is each pixel in the neighborhood, and N is the total number of pixels in the neighborhood; S64: Calculate the standard deviation relative difference when the standard deviation and the standard deviation are not equal to 0: ； ； Relative difference of standard deviation After the calculation, a threshold is set, and pixels greater than the threshold are considered abnormal points, and pixels less than the threshold are considered real points, and the coordinates are stored respectively.

Images