A self-detection method and device for shelf settlement

Abstract

This application relates to the field of shelving inspection technology, and more particularly to a self-detection method and device for shelving settlement. The proposed scheme involves acquiring a top-view image using an image sensor positioned directly above the top of the shelving unit. The image is divided into multiple grid cells, and a rectangular geometric shape is located using Hough transform. Multiple test areas are then selected based on edge features. For each test area, homography registration and orthorectification are performed based on a reference image. Subpixel-level dense registration is used to obtain the displacement field, and robust estimation yields the geometric disparity component field corresponding to the imaging parameters. Under the assumption of small deformation, the disparity is inverted into pixel normal distance, and local distance measurements are obtained by weighted aggregation of local areas. By iteratively merging adjacent grids, the boundary of the settlement area is determined, and the final settlement detection result is output. This application can achieve millimeter-level settlement quantization in a single frame image, avoiding hardware modifications, and is suitable for automated detection and long-term monitoring of shelving units of various specifications.

Description

Technical Field

[0001] This application relates to the field of shelving inspection technology, and in particular to a self-detection method and device for shelving settlement. Background Technology

[0002] In existing technologies, various solutions have been proposed for detecting rack settlement. For example, using laser ranging radar, while providing high ranging accuracy under ideal conditions, often results in fluctuating or false points in practical applications, making it difficult to form a stable settlement assessment. Adding reference marks to rack components can achieve clear positioning results in the short term, but it is prone to failure in long-term operation, has high maintenance costs, and is not suitable for large-scale deployment. Using multi-view cameras or point cloud scanning can acquire three-dimensional information of the rack surface to some extent, but in complex warehousing scenarios, the point cloud distribution is often incomplete, leading to inaccurate representation of local settlement. Some studies have attempted to determine settlement through global image comparison, but this method typically only obtains overall distance changes and struggles to reveal subtle local deformations, easily resulting in insufficient detection accuracy.

[0003] To address the above issues, this application presents a self-detection method and device for shelf settlement. Summary of the Invention

[0004] The technical problem this application aims to solve is to address the shortcomings of existing technologies by providing a self-detection method and apparatus for shelf settlement. This method involves acquiring a top-view image using an image sensor positioned directly above the shelf top, dividing the image into multiple grid cells, and using Hough transform to locate the rectangular geometric shape. Multiple test areas are then selected based on edge features. For each test area, homography registration and orthorectification are performed based on a reference image. Subpixel-level dense registration is used to obtain the displacement field, and robust estimation yields the geometric disparity component field corresponding to the imaging parameters. Under the assumption of small deformation, the disparity is inverted into pixel normal distance, and local areas are weighted and aggregated to obtain local distance measurement results. By iteratively merging adjacent grids, the boundaries of the settlement area are determined, and the final settlement detection result is output. This application can achieve millimeter-level settlement quantization in a single frame image, avoiding hardware modifications, and is suitable for automated detection and long-term monitoring of multi-specification shelves.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A self-detection method for shelf settlement is applied to a detection device, the detection device including an image sensor installed directly above the shelf, the field of view of the image sensor in a single shot being the same size as the top of the shelf, the method comprising:

[0007] Acquire an image of the top of the shelf captured by the image sensor, wherein the image of the top of the shelf includes multiple grid cells;

[0008] The rectangular geometry of the top image of the shelf is located by Hough transform, and multiple test areas are selected from the multiple grid cells;

[0009] Perform iterative operations to measure the local distance of the currently selected test area. If the local distance measurement result is less than the preset distance threshold, merge the adjacent grid cells of the currently selected test area into the currently selected test area and replace the currently selected test area, until the local distance measurement result is greater than or equal to the preset distance threshold, and output the currently selected test area.

[0010] The results of the settlement detection are summarized from all currently selected areas to be tested.

[0011] The process involves locating the rectangular geometry of the top image of the shelf using Hough transform, and then selecting multiple test areas from the multiple grid cells, including:

[0012] The image of the top of the shelf is preprocessed by converting it into a binary edge map using an edge detection operator. The preprocessing includes edge enhancement and noise reduction.

[0013] The rectangular geometry of the top of the shelf is obtained by analyzing the binary edge map using Hough transform.

[0014] Based on the rectangular geometric shape, the binary edge image is subjected to perspective correction so that the aspect ratio of the binary edge image is consistent with the standard size of the shelf, thus obtaining a standard binary edge image;

[0015] Based on the standard binary edge map corresponding to the grid cells of the shelf top image, calculate the Euclidean distance and local gradient direction change rate of the edge pixels of each grid cell to the preset reference boundary, and generate candidate scores for each grid cell.

[0016] Based on the candidate scores, multiple regions to be tested are selected.

[0017] Perspective correction is performed on the binary edge map based on the rectangular geometry, including:

[0018] Based on the rectangular geometric shape, determine the positions of the four corner points of the shelf in the preset standard geometric model, and calculate the perspective matrix based on the positions;

[0019] The binary edge map is subjected to perspective correction based on the perspective matrix.

[0020] The step of performing local distance measurement on the currently selected area to be measured includes:

[0021] A reference image is acquired, and based on the rectangular geometric shape, the image block corresponding to the currently selected test area is homography registered with the corresponding image block in the reference image, so that the image block corresponding to the currently selected test area is orthophoto-aligned with the corresponding image block in the reference image in a preset ideal plane.

[0022] Subpixel-level dense registration is performed within the currently selected test area after orthophoto alignment to obtain the displacement field corresponding to the currently selected test area.

[0023] A robust estimation of the displacement field is performed to obtain the geometric disparity component field corresponding to the image sensor, wherein the robust estimation is based on the brightness consistency, gradient direction consistency and local structural continuity of the currently selected test area to correct the displacement field;

[0024] A deformation assumption is made on the geometric disparity component field, and the normal distance of each pixel in the currently selected test area is solved. The normal distances are then weighted and summed to obtain the local distance measurement result. The deformation assumption treats the geometric disparity of the geometric disparity component field as a perturbation relative to the ideal plane, and a mapping model based on geometric disparity and normal distance is constructed based on the perturbation.

[0025] The subpixel-level dense registration within the currently selected test area after orthophoto alignment includes:

[0026] The currently selected test region is preprocessed by normalization, and the gradient magnitude map and gradient direction map are extracted.

[0027] Based on the normalized preprocessing results, the currently selected test area is divided into multiple scale layers by a preset multi-scale image pyramid, and the corresponding initial displacement field of each scale layer is calculated by a block matching algorithm.

[0028] For each scale layer, the brightness consistency residual and gradient direction consistency residual are used as the joint objective function. The gradient magnitude map and gradient direction map are sampled by sub-pixel interpolation. The initial displacement field is iteratively updated according to the sampling results until the preset convergence condition is reached, so as to obtain the dense displacement field corresponding to each scale layer.

[0029] The dense displacement field is updated and passed from bottom to top layer by layer, and the displacement field corresponding to the currently selected test area is output.

[0030] Robust estimation of the displacement field yields the geometric disparity component field corresponding to the image sensor, including:

[0031] A consistency check is performed on the pixel displacement vectors in the displacement field, and pixel displacement vectors that do not meet the consistency requirement are removed.

[0032] For the remaining pixel displacement vectors, a weighted average is performed based on the brightness consistency residual, gradient direction consistency residual, and neighborhood structure continuity to obtain the corrected pixel displacement result.

[0033] The corrected pixel displacement result is normalized by combining it with the imaging parameters of the image sensor to obtain the geometric disparity component field corresponding to the imaging geometry of the image sensor, wherein the imaging parameters include focal length, pixel size and optical axis position.

[0034] Assuming deformation in the geometric disparity component field, the normal distance of each pixel in the currently selected region to be measured is calculated, including:

[0035] Using the ideal plane as a zero-position reference, a one-to-one mapping relationship between geometric parallax and normal distance is established, wherein the mapping relationship is constructed based on the imaging parameters of the image sensor;

[0036] For each pixel in the geometric disparity component field, a local solution window centered on the pixel is constructed. The geometric disparity component in the local solution window is expressed as a linear first-order approximate response of the corresponding normal distance in the local solution window, thus obtaining a least squares solution model with normal distance as the unknown and geometric disparity as the observed quantity.

[0037] The observation weights are determined based on the brightness consistency residual, gradient direction consistency residual, and local texture energy. The mapping relationship is used as the direction weight to perform a weighted solution on the least squares solution model to obtain the estimated normal distance of the local solution window.

[0038] By summing up the estimated normal distances of all pixels, the normal distance of each pixel in the currently selected test area is obtained.

[0039] The step of merging adjacent grid cells of the currently selected test area into the currently selected test area includes:

[0040] Obtain the normal distance distribution map of the currently selected test area, and calculate the gradient field of the normal distance distribution map;

[0041] Covariance analysis is performed on the vector set of the gradient field to determine the unit vector along the direction of maximum variance as the principal direction;

[0042] At the outer boundary of the currently selected test area, a candidate set of adjacent mesh cells is determined, and for each candidate mesh cell, the displacement vector of the geometric center of the candidate mesh cell relative to the centroid of the currently selected test area is calculated.

[0043] Calculate the angle between the displacement vector and the main direction; when the angle is less than a preset direction consistency threshold, determine the corresponding mesh element as having the same direction;

[0044] Mesh cells that are determined to be in the same direction will be merged into the currently selected test area as adjacent mesh cells.

[0045] A self-detection device for shelf settlement, the device comprising:

[0046] Shelf body;

[0047] An image sensor is installed directly above the shelf body to capture images of the top of the shelf, and the field of view of a single image capture covers the entire area of ​​the top of the shelf;

[0048] The processor is electrically connected to the image sensor.

[0049] The processor includes:

[0050] The image processing module is used to acquire the image of the top of the shelf captured by the image sensor, preprocess the image of the top of the shelf, divide the image of the top of the shelf into multiple grid cells, locate the rectangular geometric shape of the image of the top of the shelf based on Hough transform, and select multiple areas to be tested from the multiple grid cells.

[0051] The local measurement module is used to perform homography registration and perspective correction between the reference image and the image of the top of the shelf, perform subpixel-level dense registration on the area to be measured to obtain a displacement field, and perform robust estimation on the displacement field to obtain a geometric disparity component field corresponding to the imaging geometry of the image sensor; based on the deformation assumption, the geometric disparity component field is mapped to the normal distance of each pixel, and the normal distance is weighted and aggregated to output the local distance measurement result of the area to be measured;

[0052] The iterative merging module is used to compare the local distance measurement results with a preset distance threshold during the iteration process. When the local distance measurement results are less than the preset distance threshold, the adjacent grid cells of the area to be measured are merged into the area to be measured according to the criteria of direction consistency and distance continuity, and the merged area replaces the area to be measured, until the local distance measurement results are greater than or equal to the preset distance threshold; and all the final areas to be measured are summarized to output the settlement detection results of the shelf.

[0053] Compared with the prior art, the beneficial effects of this application are:

[0054] This application acquires a top-view image directly above the shelf in a single capture, combines Hough transform and mesh generation to filter the area to be measured, and then constructs a geometrical disparity component field corresponding to the camera's imaging geometry through benchmark registration, sub-pixel-level dense registration, and robust estimation. Under the assumption of small deformation, this is inverted into pixel-level normal distance, thus achieving millimeter-level quantization of local settlement. This application requires no additional sensors or structural modifications, relying solely on images for detection, avoiding the shortcomings of existing technologies such as unstable detection results, incomplete point clouds, and the inability of overall information to reflect local deformation. By iteratively merging adjacent meshes, settlement boundaries can be adaptively delineated, avoiding missed detections and reducing the probability of over-detection. The reliability and applicability of the detection results are improved, meeting the practical needs of automated inspection and long-term trend monitoring. Attached Figure Description

[0055] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 An exemplary application scenario diagram provided for an embodiment of this application;

[0057] Figure 2 A schematic diagram of a processor module provided in an embodiment of this application;

[0058] Figure 3 A flowchart illustrating a self-detection method for shelf settlement provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the principle of the area to be tested provided in an embodiment of this application;

[0060] Figure 5 This application provides an example of an iterative diagram of the region to be tested. Figure 1 ;

[0061] Figure 6 This application provides an example of an iterative diagram of the region to be tested. Figure 2 . Detailed Implementation

[0062] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0063] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0064] In typical high-bay warehousing and automated stacking operations, the geometric stability of the racking system is not static. Long-term loads, slight ground deformation, loose component connections, and temperature and humidity cycles can all accumulate millimeter-level settlement or beam deflection in localized areas. These small but critical deviations are often amplified first at the interface between the forks and the pallet-beam, manifesting as jamming during loading and unloading, abnormal deceleration of the stacker crane, or even protective shutdown. Field engineers' intuitive experience often relies on temporary string lines, manual levels, or visual inspection of edges. However, when the racking is located at a high level, with narrow aisles, or in unmanned operations, these methods may pose operational risks or be insufficient in terms of time cost and reproducibility to meet the needs of continuous equipment operation.

[0065] Several alternative solutions have been tried in the existing technology:

[0066] For example, measurements can be taken using laser ranging radar. Laser ranging radar relies on high-frequency pulse signals and time-of-flight calculations, and under ideal conditions, it can provide millimeter-level distance measurement accuracy. Therefore, theoretically, it is suitable for monitoring distance changes between the top of the shelf and a reference surface. However, in warehousing scenarios, this method faces several limitations. First, the surfaces of shelf beams and uprights are mostly galvanized or spray-coated metal, possessing high reflectivity. Laser echo signals are prone to strong saturation, scattering, and multipath interference, leading to jitter or false points in the ranging results. Second, warehouse aisles are narrow, and shelf heights are high. Laser radar is typically deployed in fixed locations, collecting data through point or line scanning. It is difficult to cover the entire top of the storage location in a single inspection, often requiring multiple scans or mechanical scanning compensation, resulting in insufficient efficiency and real-time performance.

[0067] More importantly, the installation and calibration of lidar require additional hardware and fixed benchmarks. Deploying and modifying lidar in existing operational warehousing systems is costly, and ensuring consistency during long-term maintenance is difficult. This means that while lidar can provide high-precision distance detection under experimental conditions, its applicability and stability are limited in real-world scenarios, making it difficult to serve as a universal solution.

[0068] For example, attaching reference marks to components seems like a way to quickly pinpoint the location of the marks through visual recognition or optical scanning, thereby indirectly estimating the deformation of the shelving structure. However, in practical applications, this approach also has significant limitations: First, reference marks are easily affected by dust, oil, or light reflection in the storage environment, causing recognition accuracy to decrease over time and requiring periodic cleaning or replacement. Second, frequent warehouse operations mean that mechanical collisions or airflow disturbances during the movement of goods can cause the marks to fall off or shift, rendering subsequent measurement data meaningless once the mark position becomes unreliable. Third, as the number of shelving layers and aisles increases, the number of marks that need to be placed grows exponentially, leading to a sharp increase in maintenance costs and manpower. Furthermore, relying on reference marks is essentially an additional method; its detection capability is highly dependent on the presence and integrity of the marks themselves, failing to meet the requirements of low maintenance and long-term operation in long-term unmanned warehousing scenarios.

[0069] This application thus returns to a simpler, more engineering-oriented observation:

[0070] Shelves are typically cuboids, and their top surface geometry can be initially considered as a regular rectangle. If a static image is captured from directly above with a field of view equal to the top surface of the shelf during the period when the storage location is empty, the complex spatial problem can be transformed into a more controllable visual measurement problem of the geometric deviation of the rectangular plane on the image plane.

[0071] Based on this premise, the method proposed in this application does not rely on real-time 3D reconstruction or multi-source sensing, but instead uses a single top-view image as the detection carrier:

[0072] First, Hough transform is used to lock the rectangular boundary of the shelf top surface in the image, and perspective correction is performed to bring it back to the orthographic domain consistent with the shelf size. Then, a meshing approach is used to filter out locations that may be subject to local deformation from the whole, avoiding wasted computation in indiscriminate areas. Subsequently, the settlement-free reference map retained during the installation / commissioning phase is used. The residual displacements remaining after the single-reading registration of the corresponding mesh blocks represent the non-planar parallax outside the planar interpretation. After robust estimation to remove illumination artifacts, these residuals, combined with the imaging geometry of the image sensor and the known camera-shelf installation distance, can be inverted into the normal distance of the mesh center relative to the ideal plane under the assumption of small deformation.

[0073] It should be noted that, in order to align with the experience that on-site problems are usually localized and spread along the direction of the components, the embodiment further extracts the main direction on the normal distance distribution, allowing the area merging to proceed along the settlement direction most likely to extend, which not only suppresses false expansion but also quickly delineates the sections that truly need maintenance.

[0074] The application scenarios of this approach are not limited to a specific shelf type, but rather focus on the common constraints of operation and imaging:

[0075] High-level narrow aisles, top-mounted cameras, single-frame top view when the storage area is empty, potentially high reflectivity of component surfaces, and discourage additional labeling or installation of expensive sensors on-site. This embodiment of the application can be applied to shuttle racking, stacker crane aisle racking, cold chain galvanized beam racking, or long-span shelves in medium-sized warehouses, provided the above observation conditions are met.

[0076] It is important to emphasize that this application uses a single-frame image as the core input, and the edge computing end can complete matching, residual decomposition, and distance inversion. The output is a compact settlement heat zone and quantization results, rather than a large volume of video or point cloud that requires high-bandwidth backhaul. Therefore, whether it is running in parallel with the stacker crane controller or performing batch inspections during / at the end of a shift, it can be stably deployed with existing network and computing resources.

[0077] refer to Figure 1 , Figure 1 This is an exemplary application scenario diagram provided for an embodiment of this application.

[0078] Figure 1 The application scenario is shown, which includes the shelf body, image sensor and processor, where the dashed line represents the field of view of the image sensor.

[0079] It is understandable that making the field of view of an image sensor the same size as the top of the shelf in a single shot is not limited to a single method.

[0080] In one optional implementation, during the installation or commissioning phase, an imaging ratio is established based on the nominal top surface dimensions of the shelf and the camera imaging parameters. The installation height and pitch are selected to ensure that the four edges of the shelf top surface fall precisely within the effective boundaries of the pixel array in a single exposure. When using a zoom lens, a field of view equal to that of the shelf can be obtained by zooming to the target imaging ratio and locking the focal length. To eliminate installation tolerances and minor on-site deviations, the processor can use the rectangular boundaries detected by the Hough transform during power-on self-calibration to slightly compensate for the camera's pitch / roll angles. A standard imaging window corresponding one-to-one with the shelf top surface is defined on the image plane. Through electronic cropping and perspective correction, the effective field of view is precisely limited to the top area of ​​the shelf. For scenarios with multiple shelf specifications, a set of proportional parameters and homography mappings can be pre-stored for different specifications. The processor automatically calls the corresponding parameters based on the location identifier, ensuring that the final effective field of view remains consistent with the corresponding shelf top.

[0081] The aforementioned methods can be used individually or in combination. Their common goal is to achieve an effective field of view of the same size as the top of the shelf by using optical matching as the primary method and software cropping and correction as a secondary method in a single shooting.

[0082] It is understood that the specific application scenario of this application can be understood as a self-detection device for shelf settlement provided in the embodiments of this application, the device comprising:

[0083] Shelf body;

[0084] An image sensor is installed directly above the shelf body to capture images of the top of the shelf. The field of view of the image sensor in a single shot covers the entire area of ​​the top of the shelf.

[0085] The processor is electrically connected to the image sensor.

[0086] refer to Figure 2 , Figure 2 A schematic diagram of the processor module provided in this application embodiment.

[0087] The processor includes:

[0088] The image processing module is used to acquire the image of the top of the shelf captured by the image sensor, preprocess the image of the top of the shelf, divide the image of the top of the shelf into multiple grid cells, locate the rectangular geometric shape of the image of the top of the shelf based on Hough transform, and select multiple areas to be tested from the multiple grid cells.

[0089] The local measurement module is used to perform homography registration and perspective correction between the reference image and the image of the top of the shelf, perform subpixel-level dense registration on the area to be measured to obtain a displacement field, and perform robust estimation on the displacement field to obtain a geometric disparity component field corresponding to the imaging geometry of the image sensor; based on the deformation assumption, the geometric disparity component field is mapped to the normal distance of each pixel, and the normal distance is weighted and aggregated to output the local distance measurement result of the area to be measured;

[0090] The iterative merging module is used to compare the local distance measurement results with a preset distance threshold during the iteration process. When the local distance measurement results are less than the preset distance threshold, the adjacent grid cells of the area to be measured are merged into the area to be measured according to the criteria of direction consistency and distance continuity, and the merged area replaces the area to be measured, until the local distance measurement results are greater than or equal to the preset distance threshold; and all the final areas to be measured are summarized to output the settlement detection results of the shelf.

[0091] Next, with reference to the accompanying drawings, the self-detection method for shelf settlement provided in the embodiments of this application will be further elaborated. Figure 3 The method shown is applied to a detection device, which includes an image sensor mounted directly above a shelf. The image sensor's field of view in a single shot is the same size as the top of the shelf. The method includes:

[0092] S1: Acquire an image of the top of the shelf captured by the image sensor, wherein the image of the top of the shelf includes multiple grid cells;

[0093] In this embodiment, the top image of the shelf is captured in a single shot by a camera installed directly above the shelf. The captured image range is corrected for installation height and lens parameters to precisely cover the top surface of the shelf, ensuring that the image completely includes the boundary of the shelf.

[0094] To facilitate subsequent localized detection, the image of the top of the shelf was divided into multiple regular grid cells, with each grid cell serving as a potential detection area.

[0095] Understandably, transforming a large area of ​​the shelf surface into several fine-grained computational objects can significantly improve the positioning accuracy of local settlement in actual testing, while reducing interference from invalid areas.

[0096] Those skilled in the art will understand that the grid division method can be flexibly adjusted according to the actual size of the shelf, and is not limited to a fixed number or proportion of grids.

[0097] S2: Locate the rectangular geometry of the top image of the shelf using Hough transform, and filter out multiple test areas from the multiple grid cells;

[0098] In this embodiment, the Hough transform is used to extract straight line boundaries in the image, thereby locating the rectangular geometry of the shelf top surface. After completing the geometric localization, candidate scores are calculated by combining the edge pixels and gradient direction features of the grid cells, and multiple test areas with high anomaly probabilities are selected from all grid cells. This avoids redundant point-by-point calculations on the entire image, instead focusing on areas more likely to experience settlement based on geometric constraints, thereby improving detection efficiency and overall reliability.

[0099] S3: Perform iterative operation to measure the local distance of the currently selected area to be measured;

[0100] In this embodiment, the local distance measurement process first uses a settlement-free reference map acquired during the installation and commissioning phase. Homography registration is used to align the reference map with the current image on an ideal plane. Subsequently, sub-pixel-level dense registration is performed on the local area to obtain the residual displacement field, and a robust estimation method is used to eliminate the effects of illumination variations and reflection interference. Combining imaging parameters such as the image sensor's focal length, pixel size, and principal point position, the displacement field is inverted into geometric parallax components corresponding to the line-of-sight direction, and then mapped to the normal distance under the assumption of small-amplitude deformation.

[0101] S4: If the local distance measurement result is greater than the preset distance threshold, merge the adjacent grid cells of the currently selected test area into the currently selected test area and replace the currently selected test area.

[0102] In this embodiment, when the local distance exceeds a preset threshold, it indicates that the current area has shown a subsidence trend. The processor further determines the directional consistency and distance continuity of its adjacent grid cells, merging these adjacent cells only when the conditions are met. In this way, the detection range can be expanded along the main direction of subsidence, ensuring coverage of the subsidence-affected area while avoiding the erroneous merging of irrelevant grids, thus maintaining the targeting and robustness of the detection.

[0103] S5: Output the currently selected area to be measured until the local distance measurement result is less than or equal to the preset distance threshold;

[0104] In this embodiment, when the local distance of the merged area reaches or exceeds a preset threshold, it indicates that the settlement amplitude of that area has entered a recognizable stable state, the iterative merging process terminates, and the area result at this point is taken as the final output. This effectively prevents the detection area from expanding indefinitely, while ensuring that the settlement detection results have stable boundaries and quantifiable values, which is beneficial for subsequent maintenance and data archiving.

[0105] S6: Summarize all currently selected test areas and output the settlement detection results;

[0106] In this embodiment, all the areas to be measured determined through the iterative process are uniformly summarized to form a settlement distribution map and quantitative results of the top of the shelf.

[0107] Settlement detection results can be used to generate local heat maps to help on-site personnel quickly locate risk points, or they can be output to the upper management system in the form of structured data for long-term trend monitoring and maintenance decisions.

[0108] Those skilled in the art will understand that the data type of the output can be an image, a table, or a list of values, as long as it can at least reflect the settlement location and degree of settlement. This application does not impose any further limitations.

[0109] Before delving into the specific technical details of the steps, the embodiments of this application need to be emphasized again.

[0110] Those skilled in the art will understand that the existing imaging-range measurement mechanism is based on a camera imaging model. The three-dimensional points of the captured object are projected onto the image plane, and their pixel coordinates and the scale of the real space are constrained by the focal length, principal point position, pixel size, and imaging pose.

[0111] Conventional image ranging often establishes a global scale relationship based on the entire image and estimates the average physical distance from the pixel to the camera. However, in scenarios with local subsidence, the statistical value of the global distance cannot reveal the local deformation distribution, and may even mask key deviations due to the mutual cancellation between different regions.

[0112] Based on this, this embodiment refines the ranging object to the grid level. By combining the registration parallax remaining after global geometric alignment with the camera imaging parameters under the assumption of linear small perturbation, the normal distance is inverted, thereby directly obtaining the local geometric deviation of each grid. This preserves the geometric consistency of the entire image while placing the measurement on a local scale that can indicate the structural health.

[0113] Given the local normal distance, region growing and result determination are not based on pixel intensity or edge quantity, but rather on whether the geometric distance deviates from the plane as the core metric.

[0114] Specifically, the initial test area originates from the intersection of geometric constraints and candidate scores. If the local distance is significantly larger, it reflects the actual sinking relative to the camera at that location. Subsequently, the area is expanded only along adjacent grids that are continuous in distance and consistent in direction. The normal distance of the newly incorporated area is checked for consistency with the boundary prediction value. As the expansion range increases, the local deformation is diluted by the surrounding stable area, and the statistics of the local distance tend to be planar until it is reduced to near the distance threshold.

[0115] It is easy to understand that the logic behind this application's use of this as the termination point for iteration is as follows:

[0116] Settlement is usually localized, rather than a uniform downward movement across the entire surface. Once a local anomaly is completely enveloped and borders a normal area, the local measurement will naturally return to the normal fluctuation range allowed by the equipment. Further expansion is neither necessary nor will it introduce uncertainty.

[0117] It is important to emphasize that, unlike the direct threshold determination of overall distance, the iterative strategy of this application emphasizes the transition from anomaly to normal. When the local distance measurement result is less than or equal to a preset threshold as the termination condition for iteration, it does not mean that settlement is not detected, but rather that within the existing merging range, the geometric deviation has returned to the allowable zone, indicating that the spatial boundary of the abnormal area has been sufficiently delineated. If it continues to expand, the boundary will indiscriminately intrude into the healthy area, distorting the result boundary. Therefore, the termination essentially provides the smallest possible sufficient envelope, reducing both over-detection and under-detection.

[0118] Those skilled in the art will understand that the distance threshold can be set to a millimeter range based on site accuracy requirements and historical fluctuation zones. The methods for collecting and statistically analyzing operational data can be selected according to the actual deployment, as long as they meet the minimum requirements for state discrimination. This application does not impose any further limitations.

[0119] Consider two special scenarios on-site. First, if the local distance of the initial test area is less than (or equal to) the distance threshold, it indicates that the area is well consistent with the nominal plane, and can be immediately skipped and moved to the next candidate area to avoid unnecessary area growth. Second, if the local distance is consistently greater than the distance threshold during the merging process and does not increase with expansion, it indicates that the deviation is not a local phenomenon and is very likely due to overall settlement or large-scale deflection. In this case, the entire surface verification process can be triggered.

[0120] The system outputs low-order surface fitting results of the entire normal distance distribution, such as the overall downward displacement and principal deflection direction, and replaces local alarms with global alarms, making it easier for maintenance personnel to determine whether basic or large components need to be inspected. By distinguishing the iterative performance of local anomaly convergence and global anomaly non-convergence, structural changes at different scales can be taken into account in the same process, reducing misjudgments and omissions.

[0121] Next, the technical content of the method of the embodiments of this application regarding the Hough transform will be further elaborated.

[0122] It is understandable that, although the field of view of the image sensor in a single shot is limited to the same size as the top of the shelf, slight tilting, rotation or positional errors are inevitable during the camera installation process. At the same time, factors such as optical distortion, ambient lighting conditions, and background interference during the shooting process may also cause the image boundary to be misaligned with the actual boundary of the shelf. The actual image acquired will not naturally and strictly coincide with the top of the shelf.

[0123] Based on this, this embodiment uses Hough transform to extract and locate the straight line features in the image. Since the top of the shelf is initially a regular rectangle, its boundary in the image will inevitably appear as an approximate set of straight lines. The Hough transform searches for a family of straight lines in the parameter space using an accumulator, and can robustly extract the four boundary lines of the shelf even in the presence of noise and local occlusion, thereby recovering the rectangular geometry. In this way, the spatial positions of the four corner points and boundaries of the shelf can be accurately obtained. Combined with perspective correction and comparison with the standard model, this ensures that subsequent mesh cell division and local distance inversion calculations are all based on the correct geometric framework.

[0124] In one example, the process of locating the rectangular geometry of the shelf top image using Hough transform, and filtering out multiple test areas from the plurality of grid cells, includes:

[0125] S2.1: Preprocess the image of the top of the shelf by converting the preprocessed image of the top of the shelf into a binary edge map using an edge detection operator, wherein the preprocessing includes edge enhancement and noise reduction.

[0126] Specifically, to ensure the stability of subsequent line detection and geometric fitting, it is necessary to first stabilize the radiometric and structural information of the image. The original image may contain interference such as uneven exposure, specular highlights on metal surfaces, minor dust scratches, and weak texture variations. These factors can lead to discontinuous or over-enhanced edge responses, thus affecting the reliable extraction of line parameters. The preprocessing stage, without changing the physical geometric information, unifies the grayscale dynamic range, suppresses high-frequency noise, and highlights the transition band of the macroscopic structure, so that the edge operator maintains a high response to the real geometric boundary while maintaining a low response to highlights and texture noise.

[0127] It is understood that the preprocessing and edge detection operators in this application can be implemented using conventional techniques, and will not be elaborated upon here.

[0128] S2.2: The binary edge map is analyzed by Hough transform to obtain the rectangular geometry of the top of the shelf;

[0129] Specifically, the ideal shape of the shelf top surface can be viewed on the image plane as a set of pairwise approximately parallel and mutually approximately orthogonal boundary lines. Due to slight deviations in camera mounting posture, local occlusion, and false edges caused by reflection, single threshold or extreme value detection is insufficient to stably locate the four boundaries. The voting mechanism of Hough space can accommodate a certain proportion of missing and noise, and reflects the global consistency of structural lines through parameter aggregation, thus recovering the main geometric structure even when incomplete edges and background interference coexist.

[0130] In this embodiment, probabilistic Hough line detection is performed on the binary edge map, limiting the lower limit of line length and the interval sampling step size to control the false alarm rate. The obtained line candidates are clustered according to direction, divided into two groups of nearly orthogonal direction clusters, and collinear merging and endpoint extension are performed within each direction cluster to form a candidate set of two outer boundaries and a candidate set of two inner boundaries. Then, based on the edge point support and the circumscribed relationship with the image boundary, four lines enclosing the rectangle are selected from each set, and the four corner points are obtained by calculating the pairwise intersection points. To avoid false selection caused by long shadows or residual storage space, a geometric consistency check is introduced: the allowable range of the deviation of the corner interior angle from the right angle, the allowable range of the ratio of opposite side lengths and parallelism, and the allowable range of the deviation of the aspect ratio from the nominal size ratio. When there are multiple candidates that meet the conditions, the one with the highest combined score of accumulator score, edge coverage, and geometric consistency is selected as the rectangular output.

[0131] S2.3: Perform perspective correction on the binary edge image according to the rectangular geometric shape so that the aspect ratio of the binary edge image is consistent with the standard size of the shelf, thereby obtaining a standard binary edge image;

[0132] Specifically, when there are tolerances in the installation height and orientation of the top-view camera, a regular rectangle will appear as a general quadrilateral on the image plane. If the side length or internal position is measured directly in this coordinate system, it will introduce proportional distortion caused by projection, affecting subsequent statistics based on distance and orientation. Perspective correction establishes an orthographic domain on the image plane that is consistent with the nominal top surface of the shelf, making the boundaries two sets of parallel straight lines with the aspect ratio consistent with the nominal size, thus providing a unified scale and orientation reference for global and local measurements.

[0133] In one example, perspective correction is performed on the binary edge map based on the rectangular geometry, including:

[0134] Based on the rectangular geometric shape, determine the positions of the four corner points of the shelf in the preset standard geometric model, and calculate the perspective matrix based on the positions;

[0135] The binary edge map is subjected to perspective correction based on the perspective matrix.

[0136] S2.4: Based on the grid cells of the standard binary edge map corresponding to the top image of the shelf, calculate the Euclidean distance and local gradient direction change rate of the edge pixels of each grid cell to the preset reference boundary, and generate a candidate score for each grid cell.

[0137] Specifically, the ideal shelf top surface within the orthophoto domain should exhibit a region with clear boundaries, sparse internal texture, and stable directionality. Once local settlement or bending occurs, new light-dark transitions and micro-contours will appear within, causing changes in the spatial distribution of edge pixels from the reference boundary and the concentration of gradient directions. Using grid cells as the unit of measurement, this statistical difference caused by geometric perturbation can be transformed into a comparable scoring index, thereby identifying priority areas more likely to deform without relying on manual labeling.

[0138] In this embodiment, a distance field to the reference boundaries is first constructed on a standard binary edge map, and the minimum Euclidean distance to the four reference boundaries is given for each edge pixel. The distribution characteristics of this distance are statistically analyzed within the grid cell, including the mean, quantiles, and spatial dispersion, and combined with the edge pixel density to give a quantitative characterization of the abnormal expansion of the edge inward. Subsequently, the gradient direction histogram of the grid cell is extracted in the corresponding gray-scale domain or gradient domain, and the concentration and multimodality of the direction distribution are calculated. The lower the direction concentration and the more abrupt changes in local directions, the more likely it is to correspond to surface microstructures or shadow edges caused by micro-deformation. To reduce the error introduced by texture noise, the gradient direction can be constrained to regional consistency, and only directions with intensity exceeding the threshold and consistent with the neighborhood are counted. The aforementioned distance statistics and direction statistics are combined into candidate scores according to weights, and the weights can be automatically adjusted according to the shelf specifications, lighting conditions, and historical stability. For grids near the mid-span or column connection, position weights can be added to reflect the prior structural risk distribution.

[0139] S2.5: Based on the candidate scores, filter multiple regions to be tested;

[0140] Understandably, the candidate score provides a quantified probability that each grid cell is a suspected deformation region. The candidate score is compared with a preset scoring threshold. If the score is greater than the threshold, the corresponding grid cell is marked as a region to be tested.

[0141] refer to Figure 4 , Figure 4 This is a schematic diagram of the principle of the area to be tested provided in an embodiment of this application.

[0142] Figure 4 The image shows six marked test areas, including test area one, test area two, test area three, test area four, test area five and test area six. These areas are divided and located in a grid pattern in the top image of the shelf for subsequent local distance measurement and settlement judgment.

[0143] Understandably, in engineering practice, rack settlement typically occurs first at locations where the structure is weaker in terms of stress and stability, such as the mid-span of beams, the base of uprights, or near connection points. The mid-span area is prone to deflection due to bending deformation, while upright connection points may experience localized settlement due to uneven ground or loose connections. Based on this engineering experience, when selecting areas to be tested, priority should be given to the mid-span of the structure and the locations of upright supports to more quickly detect early signs of settlement.

[0144] It is easy to understand that, Figure 4This is merely an illustrative example; the selection of specific test areas needs to be automated through an image processing workflow. For instance, as mentioned above, the Hough transform is used to obtain the geometric boundaries of the rectangle, and then gridding, candidate score calculation, and connected component analysis are used to filter out several areas suspected of having settlement. In different scenarios, the number, location, and shape of the test areas can be dynamically adjusted according to the shelf size, imaging angle, and scoring threshold.

[0145] Those skilled in the art will understand that the area to be tested is not limited to Figure 4 The six shown can also be any number of distribution areas, as long as they can meet the needs of subsequent local distance measurement and settlement trend analysis. This application does not impose any further restrictions.

[0146] Next, the technical content of the method of the embodiments of this application regarding local distance measurement will be further elaborated.

[0147] In one example, the local distance measurement of the currently selected area to be measured includes:

[0148] S3.1: Obtain a reference image, and in conjunction with the rectangular geometric shape, perform homography registration between the image block corresponding to the currently selected test area and the corresponding image block in the reference image, so that the image block corresponding to the currently selected test area is orthophoto-aligned with the corresponding image block in the reference image in a preset ideal plane;

[0149] Specifically, a single frame image simultaneously contains differences in camera pose and the actual deformation of the measured surface, and pixel differences alone cannot distinguish the origins of these two factors. Aligning the currently selected area to be measured with a reference image in a non-settling state under ideal plane constraints can absorb the geometric differences caused by shooting pose, lens projection, and perspective into the same homography map, so that subsequent residual offsets mainly come from real undulations outside the plane.

[0150] In this embodiment, the reference image is acquired during the installation or commissioning phase and bound to the location identifier. After the current image is transformed by Hough to obtain the four corner points, a one-to-one mapping is established between the corner points and the corresponding vertices of the reference image. Robust matching and RANSAC are used to obtain the homography matrix. To weaken the interference of illumination changes on matching, radiometric normalization and guided filtering are performed first, and then scale-invariant features or densely correlated blocks in the neighborhood of the corner points are extracted. Preferably, high-confidence features drive homography estimation, while low-confidence features are only used for verification. The calculated homography map is used to reproject the current image block into the orthographic domain, outputting a registration result with the same resolution and physical scale as the reference image. Subpixel-level interpolation and boundary-fidelity resampling are used to ensure that the edge positions are not smoothed.

[0151] S3.2: Perform subpixel-level dense registration within the currently selected test area after orthophoto alignment to obtain the displacement field corresponding to the currently selected test area;

[0152] Specifically, local settlement and bending correspond to subtle geometric perturbations. Relying solely on sparse features or integer pixel-level alignment can easily lead to the loss of millimeter-level deformation cues. Dense, subpixel resolution registration can provide a continuous displacement description at each pixel, enabling subsequent statistical analysis and inversion to achieve sufficient signal-to-noise ratio and spatial resolution.

[0153] Understandingly, a displacement field can be interpreted as a two-dimensional vector field describing the pixel-by-pixel correspondence between a reference image and the current image. Each pixel location corresponds to a displacement vector, which represents the spatial projection difference between that location in the reference state and the current state. In other words, the displacement field not only reflects the translation of local areas on the imaging plane but also includes subtle offset information caused by real geometric undulations, perspective perturbations, and local reflections. When there is settlement or bending at the top of the shelf, these deformations accumulate as non-zero residual displacements in local areas, causing the distribution of the displacement field to exhibit regular clustering or diffusion.

[0154] In one example, the subpixel-level dense registration within the currently selected test region after orthophoto alignment includes:

[0155] S3.2.1: Perform normalization preprocessing on the currently selected test region, and extract the gradient magnitude map and gradient direction map;

[0156] Specifically, performing pixel-level matching directly can easily lead to mismatches in areas with weak textures and high reflectivity. The purpose of normalization preprocessing is to stabilize the grayscale-structure relationship, so that brightness variations do not participate in subsequent matching as much as possible; gradient magnitude and direction can better describe the geometric boundary morphology, and as structural features for registration, they can resist illumination fluctuations to a certain extent.

[0157] It is understandable that the specific methods for extracting gradient magnitude maps and gradient direction maps can be achieved through conventional techniques, and this application will not elaborate on them here.

[0158] S3.2.2: Based on the normalized preprocessing results, the currently selected test area is divided into multiple scale layers by a preset multi-scale image pyramid, and the corresponding initial displacement field of each scale layer is calculated by a block matching algorithm.

[0159] Specifically, the actual image plane displacement corresponding to settlement is often small, but local textures, noise, and reflections can cause non-convex residuals, which can easily get trapped in local minima when optimizing directly at the original resolution. The multi-scale pyramid is gradually refined from top to bottom, which can expand the convergence domain and provide stable initial values ​​for dense fields; the block-level initial displacement can quickly provide a reliable seed in areas where the texture is discernible, and then propagate to the entire domain through interpolation, reducing the number of iterations for subsequent fine registration.

[0160] In this embodiment, the pyramid scale factor is preferably set to 0.5, and the number of layers is automatically determined based on the requirement that the minimum side size is not less than 32 pixels. Anti-aliasing filtering is performed on each layer before downsampling. Block matching uses normalized cross-correlation or phase correlation to search for relevant peaks within the candidate window. The block size is adaptively selected based on the local texture energy, with smaller blocks used in areas of strong texture to improve resolution and larger blocks used in areas of weak texture to improve robustness. The search radius decreases layer by layer, with a radius of 4-6 pixels for coarse layers and 1-2 pixels for fine layers. The peak value, the ratio of major to minor peaks, and the peak width of each block are output as a quality score. The displacement of high-quality blocks is interpolated globally using thin-plate splines or bicubic splines to obtain an initial displacement field that ranges from sparse to dense.

[0161] S3.2.3: For each scale layer, the brightness consistency residual and the gradient direction consistency residual are used as the joint objective function. The gradient magnitude map and gradient direction map are sampled by sub-pixel interpolation. The initial displacement field is iteratively updated according to the sampling results until the preset convergence condition is reached, so as to obtain the dense displacement field corresponding to each scale layer.

[0162] Specifically, the assumption of single brightness consistency is prone to failure under illumination changes and reflection interference. Introducing gradient direction consistency can enhance the constraint on structural boundaries. At the same time, sub-pixel interpolation can provide continuous sampling during the update process, enabling a smooth transition of displacement increments. The joint objective is solved iteratively at each layer, and the displacement field gradually approximates the true correspondence.

[0163] In this embodiment, the displacement after upsampling from the previous layer is used as the initial value of the current layer, and a deformation-sampling-update loop is performed on the current layer:

[0164] First, the region to be measured is registered to the reference domain according to the current displacement field. Then, the brightness and gradient direction are read using bicubic interpolation on the registered coordinates, and the brightness difference and direction difference are calculated. Pixel weights are then formed by combining structural weights and initial quality scores. The update step employs an iterative reweighting strategy, using Gauss-Newton or equivalent numerical methods to obtain the displacement increment. The displacement increment is limited by a line search to avoid excessive step size oscillations, until the maximum increment is less than a set threshold or the iteration upper limit is reached. To maintain the spatial rationality of the field, an edge-preserving smoothing regularization is introduced.

[0165] Reduce the smoothing weights at boundaries with high gradient magnitudes and increase them in uniform regions to prevent cross-boundary leakage.

[0166] S3.2.4: Update and pass the dense displacement field layer by layer from bottom to top, and output the displacement field corresponding to the currently selected test area;

[0167] Specifically, the layer-by-layer transmission of the pyramid needs to maintain both the details of the displacement field and global consistency. Coarse layers provide large-scale trends, while fine layers supplement high-frequency details. If upsampling is performed directly without correction, the errors in the coarse layers can easily be amplified to high resolution. Layer refinement and quality control work together to ensure that the final displacement is both continuous and reliable.

[0168] In this embodiment, the lower layer displacement is upsampled to the current layer scale using bicubic interpolation as the initial value for iterative optimization in S3.2.3. After optimization is completed at the finest layer, the consistency of the bidirectional registration results is checked, with values ​​within the threshold being acceptable and values ​​outside the threshold being invalid. Edge-guided interpolation is used to repair invalid or void regions, with the reference gradient direction during interpolation causing streamlines to propagate along the structural boundary to avoid cross-boundary smoothing. Subsequently, a small-scale joint bilateral smoothing is performed on the entire field to suppress isolated noise points while preserving deformation edges. Physically reasonable limits are set for the displacement amplitude (calculated based on the installation spacing and imaging scale), and values ​​exceeding the limits are backed up according to the nearest feasible range to prevent extreme values ​​from contaminating the aggregate statistics.

[0169] S3.3: Robustly estimate the displacement field to obtain the geometric disparity component field corresponding to the image sensor, wherein the robust estimation corrects the displacement field based on the brightness consistency, gradient direction consistency and local structural continuity of the currently selected test area;

[0170] Specifically, the displacement field contains both non-geometric drift components caused by illumination variations, surface reflection, and high light saturation, and geometric components related to the actual height variations. If these components are used directly for depth inversion without differentiation, errors can be easily amplified. Normalizing and decomposing the displacement components under imaging geometry, preserving the geometric parallax signal coupled with the line-of-sight direction, and suppressing non-geometric factors are key steps in reliably converting two-dimensional displacement into three-dimensional distance.

[0171] In one example, robust estimation of the displacement field yields the geometric disparity component field corresponding to the image sensor, including:

[0172] S3.3.1: Perform a consistency check on the pixel displacement vectors in the displacement field and remove pixel displacement vectors that do not meet the consistency requirements;

[0173] S3.3.2: For the remaining pixel displacement vectors, a weighted average is performed based on the brightness consistency residual, gradient direction consistency residual, and neighborhood structure continuity to obtain the corrected pixel displacement result;

[0174] S3.3.3: The corrected pixel displacement result is normalized by combining it with the imaging parameters of the image sensor to obtain the geometric disparity component field corresponding to the imaging geometry of the image sensor, wherein the imaging parameters include focal length, pixel size and optical axis position.

[0175] S3.4: Apply a deformation assumption to the geometric disparity component field, solve for the normal distance of each pixel in the currently selected region to be measured, and sum the weighted average of the normal distances to obtain the local distance measurement result. The deformation assumption uses the geometric disparity of the geometric disparity component field as a perturbation relative to the ideal plane, and constructs a mapping model based on geometric disparity and normal distance based on the perturbation.

[0176] Specifically, the ideal top surface within the orthophoto domain is considered as a null reference, and geometric parallax can be understood as a small deviation relative to this reference plane. By approximating this deviation with a small perturbation under the given imaging geometry of the camera-plane nominal mounting spacing, focal length, and principal viewing direction, a parallax-normal distance mapping independent of external markers can be established.

[0177] In one example, a deformation assumption is made on the geometric disparity component field, and the normal distance of each pixel in the currently selected region to be measured is calculated, including:

[0178] S3.4.1: Using the ideal plane as a zero-position reference, establish a one-to-one mapping relationship between geometric parallax and normal distance, wherein the mapping relationship is constructed based on the imaging parameters of the image sensor;

[0179] Specifically, if two-dimensional displacement is not correlated with imaging geometry, it is impossible to distinguish between in-plane projected slip and actual vertical undulations. Taking the orthophoto plane corresponding to the top surface of the shelf in a non-settlement state as the zero-position reference, the residual geometric parallax can be understood as a slight deviation relative to this reference plane. This deviation has a definite relationship with the angle of the imaging principal ray, focal length, pixel size, and nominal installation distance between the camera and the reference plane. Therefore, it is necessary to establish a pixel-level parallax-normal distance correspondence with imaging parameters as constraints.

[0180] In this embodiment, during the acquisition phase, the camera's intrinsic parameters are calibrated and the focal length, principal point position, and pixel physical size are recorded. During the assembly phase, the nominal installation distance between the camera and the top surface of the shelf, as well as the main viewing direction, are measured. After the image enters the orthophoto domain, the sensitivity coefficient for each pixel is calculated based on the angle between the vector direction from the pixel to the principal point and the main viewing direction, and a projection coefficient lookup table consistent with the parallax direction is formed. The geometric parallax obtained after the displacement field is normalized and decomposed by S3.3 is converted pixel by pixel into the normal distance response at the same physical scale according to the lookup table, so that the parallax of the same frame at different image positions is comparable.

[0181] S3.4.2: For each pixel in the geometric disparity component field, a local solution window centered on the pixel is constructed. The geometric disparity component in the local solution window is expressed as a linear first-order approximate response of the corresponding normal distance in the local solution window, thus obtaining a least squares solution model with normal distance as the unknown and geometric disparity as the observed quantity.

[0182] Specifically, shelf settlement exhibits a gradually varying spatial characteristic, and the parallax information of a single pixel is easily affected by noise and reflection disturbances, making it unstable. By treating the normal distance as a continuous function within a small window, geometric parallax can be seen as the linear response of this function after being mapped by the imaging geometric sensitivity. By using redundant observations in the neighborhood to constrain the unknowns of the central pixel, the noise resistance can be significantly improved and the ill-posedness caused by sparse texture can be reduced.

[0183] In this embodiment, a square window of adaptive size is constructed with each pixel as the center. The side length of the window is adaptively adjusted according to the structural weight and texture energy. A smaller size is used when there is sufficient texture, and a larger size is used when there is insufficient texture. Valid observation points are selected within the window, and the geometric parallax is projected onto the normal direction according to the sensitivity coefficient of S3.4.1 to obtain a set of linear responses to the normal distance of the center pixel. To avoid model mismatch caused by crossing the real deformation boundary, the window size is automatically reduced in the edge neighborhood with high gradient magnitude, and the participation weight of observation points that cross the boundary is reduced.

[0184] S3.4.3: Determine the observation weights based on the brightness consistency residual, gradient direction consistency residual, and local texture energy, and use the mapping relationship as the direction weights to perform a weighted solution on the least squares solution model to obtain the estimated normal distance of the local solution window;

[0185] Specifically, the reliability of different observation points is not the same. Affected by changes in illumination, reflection saturation, and poor texture, the parallax error of some points is larger. Direct equal-weight solution will cause abnormal observations to dominate the results. Therefore, it is necessary to construct weights that reflect the quality of observations and match the effective components of imaging geometry in the direction so that the solution is focused on observations with high information content and direction matching.

[0186] In this embodiment, the observation weights consist of three parts: observations with small brightness consistency residuals have higher weights, observations with gradient directions consistent with the baseline have higher weights, and observations with high local texture energy have higher weights. Simultaneously, directional weights are introduced to give higher weights to disparity components consistent with the normal sensitivity direction, while suppressing components related to in-plane slip. After normalization, the weights are used to solve the weighted least squares model, calculating the normal distance to the center pixel and outputting an uncertainty estimate based on the residual variance. The solution process employs an iterative reweighting strategy, suppressing outlier observations by successively updating the weights, thus ensuring the results are stable against anomalous lighting and local reflections.

[0187] S3.4.4: Summarize the estimated normal distance values ​​of all pixels to obtain the normal distance of each pixel in the currently selected test area;

[0188] Specifically, pixel-by-pixel local solutions yield discrete solution sets with uncertainty. Without spatial consistency integration, minor noise or holes may appear at the boundaries. Confidence-guided spatial aggregation of the solution sets can maintain the clarity of the true boundaries and obtain a smooth and reliable normal distance distribution in uniform regions, thus providing stable input for regional aggregation and subsequent merging strategies.

[0189] In this embodiment, a pixel-level confidence map is first generated based on the uncertainty, and low-confidence pixels are marked as needing repair. An edge-guided interpolation method is used for continuous low-confidence regions to propagate the solution value along the gradient direction to avoid crossing the true boundary. Subsequently, a small-scale anisotropic smoothing is performed on the entire region, with the smoothing principal axis perpendicular to the local gradient to protect the edge position. To eliminate isolated outliers, median filtering is superimposed to limit the range to one or two pixel neighborhoods. Finally, a continuous, well-defined normal distance distribution map with confidence information is formed.

[0190] In one example, merging adjacent grid cells of the currently selected test area into the currently selected test area includes:

[0191] Obtain the normal distance distribution map of the currently selected test area, and calculate the gradient field of the normal distance distribution map;

[0192] Covariance analysis is performed on the vector set of the gradient field to determine the unit vector along the direction of maximum variance as the principal direction;

[0193] At the outer boundary of the currently selected test area, a candidate set of adjacent mesh cells is determined, and for each candidate mesh cell, the displacement vector of the geometric center of the candidate mesh cell relative to the centroid of the currently selected test area is calculated.

[0194] Calculate the angle between the displacement vector and the main direction; when the angle is less than a preset direction consistency threshold, determine the corresponding mesh element as having the same direction;

[0195] Mesh cells that are determined to be in the same direction will be merged into the currently selected test area as adjacent mesh cells.

[0196] refer to Figure 5 , Figure 5 This application provides an example of an iterative diagram of the region to be tested. Figure 1 .

[0197] like Figure 5As shown, during the iteration process, the local distance measurement results of test regions three and four are greater than the preset distance threshold. Therefore, test regions three and four are deleted, and the iteration operation of the next test region begins.

[0198] It is understandable that the final termination condition of the iterative process of this application can be understood as the number of regions to be tested being zero.

[0199] refer to Figure 6 , Figure 6 This application provides an example of an iterative diagram of the region to be tested. Figure 2 .

[0200] Figure 6 The diagram shows the expansion and merging of the test region six during the iteration process. Light gray represents the currently selected test region, dark gray represents adjacent grid cells that are merged during the iteration process, and white represents other grids that are not involved in the merging.

[0201] It is understandable that in local settlement detection, the measurement value of a single grid may exhibit abnormal deviations due to local noise, light reflection, or insufficient texture. If the measurement is terminated after only one measurement, it can easily lead to insufficient detection area or incomplete boundaries. Therefore, this application adopts an iterative approach, gradually introducing adjacent grids that are consistent with the current area to be measured in terms of geometric direction and measurement trend, so that the final merged area more closely matches the actual settlement distribution. The core principle behind this is that settlement deformation is often not a sudden occurrence at a single point, but rather gradually expands along a certain direction. The iterative merging process can accurately reproduce this evolutionary pattern.

[0202] In this embodiment, the iterative logic is based on a local distance threshold: when the measurement result of the currently selected area is less than a preset threshold, it indicates that it is still within the subsidence influence range. At this time, the algorithm will search for adjacent grids around it and filter them according to directional consistency and distance continuity. Grids that meet the conditions are incorporated into the current area, such as... Figure 6 As shown above, as the merged area expands, the settlement area gradually intersects with the normal area, and the overall statistical local distance result will gradually recover. When it recovers to a value not less than a preset threshold, the iteration terminates, and the area at this point is output as the settlement boundary. If the initial grid itself is already greater than or equal to the threshold, it is skipped directly, and the next area to be measured is entered; if the threshold cannot be reached during the iteration process, it means that the settlement has expanded globally, and an overall alarm needs to be output.

[0203] It is easy to understand that, Figure 6For illustrative purposes only, in actual iteration, the merging direction, number of grids, and number of iterations are all determined by the measured normal distance distribution and the characteristic consistency of adjacent units. Those skilled in the art will understand that under different shelf structures or lighting conditions, the iteration strategy parameters, such as threshold range, directional consistency judgment criteria, and continuity indicators, can be flexibly adjusted, as long as the final merged area can stably cover the actual settlement range and avoid erroneous expansion. This application does not impose further limitations.

[0204] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for self-detecting a shelf sagging, applied to a detection device, characterized in that, The detection device comprises an image sensor installed above the shelf, a single shooting field of view of the image sensor is the same size as the top of the shelf, and the method comprises: obtaining a shelf top image shot by the image sensor, wherein the shelf top image comprises a plurality of grid cells; locating a rectangular geometric shape of the shelf top image by Hough transform, and screening a plurality of test regions from the plurality of grid cells; performing an iteration operation, performing local distance measurement on a currently selected test region, if the local distance measurement result is greater than a preset distance threshold, merging adjacent grid cells of the currently selected test region into the currently selected test region and replacing the currently selected test region, until the local distance measurement result is less than or equal to the preset distance threshold, and outputting the currently selected test region; collecting all currently selected test regions and outputting a sedimentation detection result; the local distance measurement on the currently selected test region comprises: obtaining a reference image, combining the rectangular geometric shape, performing homographic registration on an image block corresponding to the currently selected test region and a corresponding image block in the reference image, so that the image block corresponding to the currently selected test region is orthorectified and aligned with the corresponding image block in the reference image in a preset ideal plane; performing sub-pixel level dense registration in the orthorectified currently selected test region to obtain a displacement field corresponding to the currently selected test region; performing robust estimation on the displacement field to obtain a geometric parallax component field corresponding to the image sensor, wherein the robust estimation corrects the displacement field based on brightness consistency, gradient direction consistency and local structure continuity of the currently selected test region; performing deformation assumption on the geometric parallax component field to solve normal distance of each pixel in the currently selected test region, performing weighted average summation on the normal distance to obtain a local distance measurement result, wherein the deformation assumption takes the geometric parallax of the geometric parallax component field as a disturbance relative to the ideal plane, and constructs a mapping model based on geometric parallax and normal distance according to the disturbance.

2. The method of self-detecting a shelf sag according to claim 1, wherein, the locating of the rectangular geometric shape of the shelf top image by Hough transform and the screening of a plurality of test regions from the plurality of grid cells comprise: preprocessing the shelf top image, converting the preprocessed shelf top image into a binary edge image through an edge detection operator, wherein the preprocessing comprises edge enhancement and denoising processing; analyzing the binary edge image by Hough transform to obtain a rectangular geometric shape of the shelf top; performing perspective correction on the binary edge image according to the rectangular geometric shape, so that the aspect ratio of the binary edge image is consistent with the standard size of the shelf, to obtain a standard binary edge image; calculating the Euclidean distance of the edge pixels of each grid cell to a preset reference boundary and the local gradient direction change rate according to the grid cells corresponding to the shelf top image of the standard binary edge image, to generate a candidate score of each grid cell; screening a plurality of test regions according to the candidate score.

3. The method of self-detecting a shelf sag according to claim 2, wherein, The perspective correction of the binary edge map according to the rectangular geometric shape comprises: According to the rectangular geometric shape, the positions of four corner points of the shelf in a preset standard geometric model are determined, and a perspective matrix is calculated according to the positions; The binary edge map is corrected according to the perspective matrix.

4. The method of self-detecting a shelf sag according to claim 1, wherein, The sub-pixel level dense registration in the current selected to-be-measured region after orthographic alignment comprises: The current selected to-be-measured region is normalized and preprocessed, and a gradient amplitude map and a gradient direction map are extracted; Based on the normalized preprocessing result, the current selected to-be-measured region is divided into multiple scale layers through a preset multi-scale image pyramid, wherein an initial displacement field of each scale layer is calculated through a block matching algorithm; For each scale layer, the brightness consistency residual and the gradient direction consistency residual are taken as a joint objective function, the gradient amplitude map and the gradient direction map are sampled through sub-pixel interpolation, and the initial displacement field is iteratively updated according to the sampling result until a preset convergence condition is reached, so as to obtain a dense displacement field corresponding to each scale layer; The dense displacement field is updated and transmitted layer by layer from the bottom to the top, and the displacement field corresponding to the current selected to-be-measured region is output.

5. The method of self-detecting a shelf sag according to claim 1, wherein, The robust estimation of the displacement field is performed to obtain a geometric parallax component field corresponding to the image sensor, comprising: The pixel displacement vectors in the displacement field are subjected to consistency inspection, and the pixel displacement vectors that do not satisfy the consistency are removed; The remaining pixel displacement vectors are subjected to weighted average based on the brightness consistency residual, the gradient direction consistency residual and the neighborhood structure continuity, so as to obtain corrected pixel displacement results; The corrected pixel displacement results are combined with imaging parameters of the image sensor to be normalized, so as to obtain a geometric parallax component field corresponding to the imaging geometry of the image sensor, wherein the imaging parameters comprise focal length, pixel size and optical axis position.

6. The method of self-detecting a shelf sag of claim 1, wherein, The geometric parallax component field is subjected to deformation assumption, and the normal distance of each pixel in the current selected to-be-measured region is solved, comprising: The ideal plane is taken as a zero-bit reference, and a one-to-one mapping relationship between geometric parallax and normal distance is established, wherein the mapping relationship is constructed based on the imaging parameters of the image sensor; For each pixel in the geometric parallax component field, a local solving window centered on the pixel is constructed, the geometric parallax components in the local solving window are expressed as a linear first-order approximation response of the corresponding normal distance in the local solving window, so as to obtain a least squares solving model taking the normal distance as an unknown quantity and taking the geometric parallax as an observation; The observation weight is determined according to the brightness consistency residual, the gradient direction consistency residual and the local texture energy, and the mapping relationship is taken as a direction weight, so as to perform weighted solving on the least squares solving model, and a normal distance estimation value of the local solving window is solved. The normal distance estimation values of all pixels are summarized, so as to obtain the normal distance of each pixel in the current selected to-be-measured region.

7. The method of self-detecting a shelf sag according to claim 1, wherein, The adjacent grid cells of the current selected to-be-measured region are merged into the current selected to-be-measured region, comprising: A normal distance distribution map of the current selected to-be-measured region is obtained, and a gradient field of the normal distance distribution map is calculated. Performing covariance analysis on the vector set of the gradient field to determine a unit vector along a direction of maximum variance as a principal direction; Determining a candidate set of adjacent grid cells at an outer boundary of the currently selected to-be-measured region, and calculating a displacement vector of a geometric center of each candidate grid cell relative to a centroid of the currently selected to-be-measured region; Calculating an included angle between the displacement vector and the principal direction; when the included angle is less than a preset direction consistency threshold, determining the corresponding grid cell as direction-consistent; Merging the grid cells determined as direction-consistent as adjacent grid cells to the currently selected to-be-measured region.

8. A self-detecting device for shelf settlement, for implementing a self- detecting method for shelf settlement according to any one of claims 1 to 7, characterized in that, The device comprises: a shelf body; an image sensor mounted directly above the shelf body for collecting images of the top of the shelf, a single shooting field of view range of the images covering all areas of the top of the shelf; a processor electrically connected to the image sensor.

9. The self-checking device for detecting the settlement of a shelf according to claim 8, wherein The processor comprises: an image processing module for acquiring the images of the top of the shelf taken by the image sensor, pre-processing the images of the top of the shelf, dividing the images of the top of the shelf into a plurality of grid cells, and locating rectangular geometric shapes of the images of the top of the shelf based on Hough transformation, and screening a plurality of to-be-measured regions from the plurality of grid cells; a local measurement module for performing homographic registration and perspective correction between a reference image and the images of the top of the shelf, performing sub-pixel level dense registration on the to-be-measured regions, obtaining a displacement field, and performing robust estimation on the displacement field to obtain a geometric parallax component field corresponding to the imaging geometry of the image sensor; mapping the geometric parallax component field to normal distance of each pixel based on deformation assumption, and performing weighted aggregation on the normal distance to output local distance measurement results of the to-be-measured regions; an iterative merging module for comparing the local distance measurement results with a preset distance threshold in an iterative process, when the local distance measurement results are less than the preset distance threshold, merging adjacent grid cells of the to-be-measured regions to the to-be-measured regions according to direction consistency and distance continuity criteria, and replacing the to-be-measured regions with the merged regions until the local distance measurement results are greater than or equal to the preset distance threshold; and aggregating all final to-be-measured regions to output a settlement detection result of the shelf.

Images