Automatic identification and deviation correction method for material pallets of asrs system

Abstract

The application provides a material tray automatic identification and deviation correction method for an ASRS system, which comprises the following steps: triggering a multi-modal sensor to collect a tray gray scale image, a three-dimensional point cloud, a label material code and an ultrasonic wave front distance value to form a multi-modal data cache after a stacker is in place; extracting a tray four-corner point sub-pixel coordinate from the multi-modal data cache through equalization and edge detection, and obtaining a normal vector and a distance through point cloud filtering and plane fitting; obtaining a fusion confidence through weighted fusion of an RFID confidence, a visual matching and a geometric verification score, and outputting a tray pose parameter set; solving a six-degree-of-freedom pose based on EPnP and LM optimization, comparing the six-degree-of-freedom pose with a standard picking pose to obtain a deviation and a deflection angle, selecting a deviation correction strategy according to a comprehensive deviation amplitude, and performing differentiated compensation on different deviation levels. The application improves identification reliability through multi-modal weighted fusion, balances between response time and success rate through deviation grading self-adaptive deviation correction and iterative closed-loop verification.

Description

Technical Field

[0001] This application relates to the fields of automated warehouse control and machine vision technology, specifically to a method and apparatus for automatic identification and correction of material pallets in an ASRS system. Background Technology

[0002] In an Automated Storage and Retrieval System (ASRS), a stacker crane must precisely align its forks with the material pallets within the storage location to perform the retrieval operation. The actual position of the pallet within the storage location is affected by a combination of factors, including placement deviations from the previous retrieval operation, cumulative positioning errors after multiple stacker crane runs, minor deformations of the rack structure, and equipment vibrations. These factors result in translational and rotational deviations from the system's preset standard retrieval position. If these deviations are not detected and compensated for before the forks are used to perform the retrieval operation in the preset position, the forks will scrape, jam, or even collide with the pallet's positioning slots, causing material damage and equipment downtime.

[0003] Existing ASRS systems primarily rely on barcode scanning or single RFID reading for pallet identification. When pallet tags are soiled, obstructed, or in densely stacked situations, the success rate of single-modal identification significantly decreases. The reading range of RFID tags is reduced due to diffraction effects at the edges of metal pallets, and the decoding failure rate increases after barcodes are scratched. Repeated readings and manual intervention due to identification failures prolong the single retrieval cycle, reducing the effective utilization rate of the stacker crane.

[0004] Existing pallet positioning methods mostly employ fixed photoelectric sensor arrays or limit switch combinations, which can only detect the presence or coarse boundaries of the pallet, failing to acquire its precise six-DOF spatial pose. The few solutions using industrial cameras are mostly monocular two-dimensional detection, only able to acquire the pallet's translation and rotation within the image plane, unable to measure the pallet's depth offset and spatial tilt angle. Regarding forklift correction, existing methods mostly employ open-loop preset compensation or simple threshold-triggered secondary positioning strategies, failing to select differentiated motion control parameters and compensation strategies based on the continuous magnitude of the deviation. In scenarios with small deviations, motion redundancy takes too long, while in scenarios with large deviations, the single-wheel compensation accuracy is insufficient.

[0005] A more prominent problem is that in existing technologies, the three stages of tray recognition, pose detection, and correction control operate independently. The recognition results do not effectively guide feature selection and initial value setting in pose calculation, and the pose calculation results do not directly drive the differentiated selection of correction strategies and the adaptive configuration of motion parameters, resulting in a long cycle from sensor triggering to correction completion. Summary of the Invention

[0006] To address the issues of insufficient reliability in pallet recognition, low accuracy in pose detection, and lack of adaptability in correction strategies in existing technologies, this application provides a method and apparatus for automatic pallet recognition and correction in ASRS systems. By employing multimodal fusion recognition, precise six-degree-of-freedom pose estimation, and adaptive correction control based on deviation levels, the method achieves a synergistic improvement in pallet recognition reliability, positioning accuracy, and correction success rate.

[0007] To solve at least one of the above problems, this application provides the following technical solution: Firstly, this application provides a method for automatic identification and correction of material pallets in an ASRS system, comprising: triggering a multimodal sensor to collect a grayscale image, a 3D point cloud, a label material code, and an ultrasonic leading edge distance value of the pallet after the stacker crane is in position; aligning the collected data by timestamp and registering channel marks to form a multimodal data buffer and outputting it; reading the grayscale image from the multimodal data buffer, extracting the sub-pixel coordinates of the four corner points of the pallet through adaptive equalization and Gaussian filtering, and then extracting the sub-pixel coordinates of the four corner points of the pallet through Canny edge detection and Hough transform; reading the 3D point cloud from the multimodal data buffer, obtaining the normal vector and distance by fitting the pallet plane through RANSAC after outlier filtering and voxel downsampling; and then combining the RFID confidence level, ... The visual matching and geometric verification scores are weighted and summed according to preset weights to obtain the fusion confidence score, and the pallet pose parameter set is output when it is not lower than the preset threshold condition. The pallet pose parameter set is read, and the six-degree-of-freedom pose is solved based on EPnP and LM optimization. The six-degree-of-freedom pose is compared with the standard picking pose to obtain the lateral offset, fork extension direction offset and deflection angle. According to the comprehensive offset magnitude of the three deviation components, the correction strategy is selected and single compensation is performed for micro deviations, coarse and fine two-stage compensation is performed for medium deviations, and iterative compensation with weight measurement is performed for large deviations. After correction, a second acquisition is triggered to calculate the residual deviation. The correction is considered successful when each residual deviation does not exceed the convergence threshold. If it exceeds the threshold, iterative correction is performed and the number of iterations does not exceed the preset upper limit.

[0008] Furthermore, it also includes: an industrial camera acquiring a grayscale image of the tray after a preset exposure time following a valid trigger signal, and writing it into an image cache after low-pass filtering and noise reduction; a structured light depth sensor projecting an coded structured light pattern in the same time slot, and an infrared receiver acquiring the reflected pattern and converting it into a three-dimensional point cloud through triangulation; an RFID reader polling and reading the material code extracted from the tray tag according to a preset transmission power; an ultrasonic sensor array sequentially transmitting pulses according to a preset time sequence and acquiring the echo flight time of each channel, converting it into a distance value; aligning all sensor acquisition data to a unified timestamp based on the system bus global clock; detecting the completion and verification flags of each channel's data and registering the results as the channel acquisition flag; triggering re-acquisition when any channel is marked as failed, and marking it as unavailable when continuously failing; and encapsulating all channel data into the multimodal data cache and writing it into shared memory.

[0009] Furthermore, it also includes: reading the grayscale image from the image portion of the multimodal data cache, performing adaptive histogram equalization to compensate for uneven illumination, dividing the image into blocks according to a preset number of sub-blocks and limiting the contrast amplification factor to not exceed a preset upper limit, smoothing the image with a Gaussian kernel after equalization, and then extracting the edge gradient map through double-threshold Canny edge detection; accumulating line segments on the edge gradient map through polar coordinate Hough transform and extracting the longitudinal and transverse main edges of the tray by angle clustering, determining the initial pixel positions of the four corner points by the intersection points of the two pairs of main edges, and iteratively solving the sub-pixel coordinates of the four corner points of the tray within their neighborhood window according to the grayscale gradient weighted method.

[0010] Furthermore, it also includes: reading the three-dimensional point cloud from the point cloud portion of the multimodal data cache, traversing each point to calculate its average distance to K points in its neighborhood, and constructing a statistical outlier criterion based on the global mean and standard deviation to filter out outliers; performing downsampling on the filtered point cloud according to a preset voxel side length grid; inputting the downsampled point cloud into RANSAC plane fitting, randomly selecting three non-collinear points to form a candidate plane and statistically analyzing interior points according to the distance threshold condition, taking the plane corresponding to the largest set of interior points as the pallet bearing surface and solving its unit normal vector and distance; and multiplying the RFID confidence score by a first weighting coefficient, adding the visual matching score by a second weighting coefficient, adding the geometric verification score by a third weighting coefficient to obtain the fusion confidence score.

[0011] Furthermore, it also includes: using the sub-pixel coordinates of the four corner points of the tray and the plane normal vector as EPnP inputs, constructing a linear equation to solve for the control point coordinates through the correspondence of the four control points in the camera coordinate system and the world coordinate system, restoring the initial pose value of the tray corner points in the camera coordinate system, and then inputting it into LM optimization and iteratively refining the six-degree-of-freedom pose based on the cost function of reprojection error and plane constraint; comparing the refined six-degree-of-freedom pose with the standard picking pose, calculating the lateral offset as the difference between the current X coordinate and the standard X coordinate, the fork extension direction offset as the difference between the current Z coordinate and the standard Z coordinate, and the deflection angle as the rotation angle of the tray normal vector around the Y axis, to obtain the three deviation components for selection of the correction strategy.

[0012] Furthermore, it also includes: weighting and summing the absolute values ​​of the three deviation components according to a preset normalization coefficient to obtain the comprehensive offset amplitude; when the comprehensive offset amplitude is within a preset first threshold, a micro-deviation single-time compensation strategy is selected to directly superimpose each deviation component onto the servo position command of each axis and perform a single-time compensation motion at normal speed; when the comprehensive offset amplitude is between the first threshold and the second threshold, a medium-deviation coarse and fine two-stage compensation strategy is selected, with coarse adjustment at a high rate to compensate until the deviation is within the micro-deviation range, and fine adjustment at a low rate to compensate until convergence; when it exceeds the second threshold, a large-deviation iterative compensation strategy is selected and pose retesting is triggered after each round of compensation.

[0013] Furthermore, it also includes: after the correction compensation command is executed, a second multimodal acquisition with the same configuration as the initial acquisition is triggered. The acquisition range is reduced to a smaller area centered on the corrected coordinates to reduce processing latency. The grayscale image and 3D point cloud acquired by the second multimodal acquisition are processed in the same way as the initial acquisition to extract the corrected corner coordinates and planar parameters. The corrected six-degree-of-freedom pose is solved based on the corrected corner coordinates and planar parameters and compared with the standard picking pose to obtain the residual deviation. When the absolute value of each residual deviation does not exceed the preset convergence threshold, the correction is determined to be successful and a completion signal is output. If the number of iterations does not reach the preset upper limit, the number of iterations is increased and a new round of correction is returned.

[0014] Secondly, this application provides an automatic pallet identification and correction device for an ASRS system, comprising: a multimodal sensing trigger module, used to trigger a multimodal sensor to collect a pallet grayscale image, a 3D point cloud, an RFID tag material code, and an ultrasonic leading edge distance value after the stacker crane is in position; aligning the collected data by timestamp and registering channel acquisition marks to form a multimodal data buffer and outputting it; a multimodal fusion identification module, used to read the grayscale image from the multimodal data buffer, extract the sub-pixel coordinates of the four corner points of the pallet through equalization and edge detection, obtain the plane normal vector and distance from the point cloud through filtering and plane fitting, and obtain the fusion confidence score by weighting and summing the RFID confidence score, visual matching score, and geometric verification score according to preset weights and outputting the pallet pose parameter set; and a pose solving and deviation calculation module, used to read the pallet pose parameter set based on EPn. The P and LM optimization solves the six-DOF pose and compares it with the standard picking pose to obtain the lateral offset, fork extension direction offset, and deflection angle; the correction strategy adaptation and execution module is used to select the correction strategy based on the comprehensive offset magnitude of the three deviation components, perform single compensation for micro deviations, perform coarse and fine two-stage compensation for medium deviations, and perform iterative compensation with weight measurement for large deviations, map each deviation component to each axis servo correction command and execute the correction action through the motion control system; the secondary verification and iterative convergence module is used to trigger secondary acquisition to calculate the residual deviation after correction. If the residual deviation does not exceed the convergence threshold, the correction is considered successful. If it exceeds the threshold, iterative correction is performed and the number of iterations does not exceed the preset upper limit; the anomaly handling and alarm module is used to trigger reacquisition or manual intervention when the fusion confidence is lower than the preset threshold. If the correction iteration reaches the preset upper limit and still does not converge, an anomaly alarm is triggered.

[0015] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method for automatic identification and correction of material pallets for an ASRS system.

[0016] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the described method for automatic identification and correction of material pallets for an ASRS system.

[0017] As can be seen from the above technical solution, this application provides a method and apparatus for automatic identification and correction of material pallets in ASRS systems. It constructs a complete multimodal data cache by synchronously triggering multimodal sensor hardware and aligning with unified time base data. It improves the identification success rate of pallets under label damage and occlusion conditions by weighted fusion of RFID, vision and geometric three-modal confidence. It improves the pose detection accuracy by cross-validating EPnP and LM optimized pose estimation algorithms with joint constraints of two-dimensional corner points and three-dimensional planes. It achieves a balance between correction response time and success rate by using a deviation-level adaptive correction strategy and pose retest closed-loop iteration. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0019] Figure 1 This is a flowchart illustrating the automatic pallet identification and correction method for an ASRS system according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the process of multimodal sensor synchronous triggering and data alignment in an embodiment of this application; Figure 3 This is a schematic diagram of the image preprocessing and point cloud processing process in the embodiments of this application; Figure 4 This is a schematic diagram illustrating the multimodal fusion recognition decision and confidence level determination in an embodiment of this application; Figure 5 This is a schematic diagram of the six-degree-of-freedom pose calculation and visual-point cloud cross-validation process in the embodiments of this application; Figure 6 This is a schematic diagram of the deviation classification adaptive correction strategy and motion parameter configuration in the embodiments of this application; Figure 7 This is a flowchart illustrating the correction verification and iterative closed loop in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the automatic pallet identification and correction device for the ASRS system in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0021] Considering that existing ASRS systems suffer from insufficient reliability of single-modal recognition, low accuracy of pose detection, and lack of adaptability in correction strategies in the three stages of pallet recognition, pose detection, and correction control, this application provides a method and apparatus for automatic pallet recognition and correction in ASRS systems. Through a closed-loop mechanism of multimodal fusion recognition, accurate six-degree-of-freedom pose estimation, and adaptive correction based on deviation classification, the method and apparatus achieve a synergistic improvement in recognition reliability, positioning accuracy, and correction success rate.

[0022] To integrate multimodal fusion recognition, deviation classification correction, and iterative closed-loop verification into a complete pallet recognition and correction closed loop, this application provides an embodiment of an automatic pallet recognition and correction method for ASRS systems. See [link to embodiment]. Figure 1 The method for automatic pallet identification and correction in the ASRS system specifically includes the following: Step S110: After the stacker crane is in position, the multimodal sensor is triggered to collect the pallet grayscale image, 3D point cloud, label material code and ultrasonic leading edge distance value. The collected data is aligned by timestamp and the channel mark is registered to form a multimodal data buffer and output.

[0023] In this embodiment, the target cargo location signal is read from the stacker crane motion control system as the starting trigger for the multimodal acquisition process. After receiving the storage / retrieval task instruction from the warehouse management system, the stacker crane moves along the aisle to the encoder position corresponding to the target cargo location. The arrival criterion is that the difference between the encoder feedback position and the target position is less than a preset positioning window.

[0024] The arrival signal is converted into a hardware trigger pulse by the programmable logic controller and distributed in parallel through shielded twisted-pair cables of equal length to the external trigger ports of the industrial camera, structured light depth sensor, RFID reader and ultrasonic sensor array. The reception time difference of each port is controlled within the microsecond level.

[0025] The industrial camera initiates the exposure acquisition sequence upon detecting the rising edge of the trigger pulse. The camera is configured with a global shutter CMOS sensor with a resolution of 2048 x 1536 pixels, and the lens focal length is adapted to the working distance from the stacker crane's loading platform to the front surface of the pallet. The exposure time is adaptively adjusted based on the average grayscale value of the previous frame, using a proportional-integral controller, with the target grayscale value being the 8-bit grayscale median.

[0026] Upon receiving a trigger pulse, the structured light depth sensor initiates an coded structured light projection and acquisition sequence. The projector projects coded structured light patterns with different spatial frequencies onto the tray surface in a preset order, while the infrared receiver simultaneously acquires the reflected patterns modulated by the tray surface height. Using phase unfolding and triangulation principles, the phase shift of the reflected patterns is mapped to the depth value of each pixel, forming a dense depth map which is then converted into a 3D point cloud format.

[0027] The RFID reader operates in the UHF band and transmits inventory commands via a circularly polarized antenna in a polling manner. Upon receiving a trigger pulse, the reader initiates a full-antenna inventory check, recording the material code and received signal strength value of each responding tag. The ultrasonic sensor array consists of multiple transceiver ultrasonic probes evenly arranged along the front edge of the forklift platform.

[0028] After the trigger pulse arrives, each ultrasonic probe emits pulses sequentially according to a preset timing sequence and collects the echo flight time of each channel, converting it into the distance value from the probe to the front edge surface of the tray. After completing their respective acquisitions, the four sensor subsystems write the raw data into their corresponding data buffers, with each data channel carrying a timestamp based on the sensor's local time base.

[0029] After all sensors have completed data acquisition, the data from all channels are aligned to a unified timestamp using the system bus global clock as a reference. Subsequently, a data integrity check is performed on each channel, verifying the data acquisition completion flag and the verification pass flag, and the check results are recorded as acquisition markers for each channel.

[0030] When any channel is marked as failed, a single re-acquisition of that channel is triggered. If the re-acquisition also fails, the channel is marked as unavailable, and the corresponding bit is set in the unavailable channel mask. All data after alignment and verification are encapsulated into a multimodal data cache using the material code as the index key and written to shared memory for subsequent steps to read.

[0031] Step S120: Read the grayscale image from the multimodal data cache, extract the sub-pixel coordinates of the four corner points of the tray through adaptive equalization and Gaussian filtering, and extract the sub-pixel coordinates of the four corner points of the tray through Canny edge detection and Hough transform. Read the three-dimensional point cloud from the multimodal data cache, obtain the normal vector and distance by fitting the tray plane through RANSAC after outlier filtering and voxel downsampling. Calculate the fusion confidence score by weighting the RFID confidence score, visual matching and geometric verification scores according to preset weights, and output the tray pose parameter set when it is not lower than the preset threshold condition.

[0032] In this embodiment, grayscale image frames and 3D point cloud frames are read from the multimodal data cache output in step S110, serving as the initial inputs for the image processing pipeline and the point cloud processing pipeline. The two pipelines run in parallel on different processor cores of the edge computing node to shorten the total time spent in the preprocessing stage.

[0033] The image processing pipeline first divides the grayscale image frame into several sub-blocks. The size of each sub-block is determined based on the image resolution and the typical imaging size of the tray in the image. Each sub-block slides with a preset step size to ensure sufficient coverage. A grayscale histogram is independently calculated for each sub-block, and a cumulative distribution function is generated as the grayscale mapping function. The contrast amplification factor is limited to a preset upper limit to suppress local noise.

[0034] The equalized image is smoothed using a Gaussian filter to remove high-frequency pixel noise. The combination of the filter kernel size and standard deviation parameter strikes a balance between noise suppression and edge gradient preservation. The filtered image is then input into the Canny edge detection operator, which uses a dual threshold criterion of low and high thresholds. Pixels with gradient magnitudes below the low threshold are suppressed, while pixels above the high threshold are identified as strong edges and preserved. Pixels in the middle range are identified as weak edges based on connectivity.

[0035] The Hough transform represents lines in polar coordinate parameter space, with angle search covering 0 to 180 degrees and distance search covering 0 to the length of the image diagonal. Line segment parameters corresponding to the highest peak values ​​of the accumulator are identified in the accumulation matrix. The detected line segments are clustered into two approximately orthogonal direction clusters based on their direction angles. The two line segments with the highest accumulator values ​​in each cluster are selected as a pair of parallel edges of the tray in that direction.

[0036] After writing the four main edge lines into a general equation, we find their intersections pairwise. We then remove the intersections located outside the image boundary, and the remaining four valid intersections are the initial pixel coordinates of the four corner points of the tray. Using each corner point as the center, we iteratively solve for the sub-pixel precision position within a preset neighborhood window using a gray-level gradient weighted method. The iteration terminates when the displacement is less than a preset threshold for two consecutive iterations.

[0037] The 3D point cloud processing pipeline executes in parallel with the image pipeline. After reading point cloud frames from the multimodal data cache, all points are traversed, and the average distance of each point to its K nearest neighbors is calculated. The average distances of all K nearest neighbors form a sample set with an approximate Gaussian distribution. After calculating the global mean and standard deviation, points that exceed the mean plus a preset standard deviation are marked as statistical outliers and filtered out.

[0038] After outlier removal, the point cloud is downsampled using a predefined voxel size, retaining the point closest to the centroid within each cell. The downsampled point cloud is then used for RANSAC plane fitting. In each iteration, three non-collinear points are randomly selected to calculate the candidate plane equation. Points whose distance to the candidate plane is less than a distance threshold are identified as inliers. After iteration, the plane corresponding to the largest set of inliers is taken as the pallet bearing surface, and its unit normal vector and directed distance are calculated.

[0039] The tray plane normal vector and planar distance, along with the corner coordinates output by the image pipeline and the RFID reading results, are combined and entered into the multimodal fusion recognition stage. The fusion confidence score is calculated by multiplying the RFID confidence score by a first weighting coefficient, adding the visual matching score by a second weighting coefficient, and adding the geometric verification score by a third weighting coefficient. If the fusion confidence score is not lower than a preset threshold, the recognition is considered successful, and the corner coordinates, normal vector, and planar distance are packaged into a tray pose parameter set and output.

[0040] Step S130: Read the pallet pose parameter set, optimize and solve the six-DOF pose based on EPnP and LM, compare the six-DOF pose with the standard picking pose to obtain the lateral offset, fork extension direction offset and deflection angle, select the correction strategy according to the comprehensive offset magnitude of the three deviation components, perform single compensation for micro deviations, perform coarse and fine two-stage compensation for medium deviations, and perform iterative compensation with weight measurement for large deviations. After correction, trigger secondary acquisition to calculate residual deviation. If each residual deviation does not exceed the convergence threshold, the correction is considered successful. If it exceeds the threshold, iterative correction is performed and the number of iterations does not exceed the preset upper limit.

[0041] In this embodiment, the sub-pixel corner coordinates, pallet plane normal vector, and planar distance are read from the pallet pose parameter set output in step S120 and used as the starting input for pose calculation and deviation calculation. The coordinates of the corner points of the pallet 3D model in the pallet coordinate system are used as known 3D reference points, forming 4 pairs of 3D-2D correspondences with the 4 corresponding sub-pixel corner points, thus forming a perspective n-point problem.

[0042] Pose calculation employs the EPnP algorithm, representing four 3D reference points as a weighted combination of four virtual control points. A system of linear equations about the control point coordinates is constructed using the projection equations between the 3D reference points and the 2D projection points. Singular value decomposition is then used to obtain the initial coordinates of the control points in the camera coordinate system through least-squares solutions.

[0043] The initial values ​​of EPnP are then substituted into the LM nonlinear optimization framework. The optimization objective function is constructed using the cost function of reprojection error and planar constraints. The cost is continuously reduced through iteration until convergence, and the refined six-degree-of-freedom pose is output, containing three translational components and three rotational components.

[0044] The refined six-DOF pose is compared component-by-component with the standard picking pose pre-stored in the system configuration database. The standard picking pose is indexed by the location coordinates and is entered during system deployment through a teaching calibration process. Deviation calculation extracts three deviation components directly related to the fork access action: lateral offset, fork extension direction offset, and deflection angle.

[0045] The selection of the correction strategy is based on the combined offset magnitude of the three deviation components. The combined offset magnitude is the weighted sum of the absolute values ​​of the three deviation components using normalization coefficients. If it does not exceed a preset first threshold, it is classified as a minor deviation, and a single fast compensation is performed: the three deviation components are directly converted into servo position correction values ​​for each axis and written to the target position register of the servo driver in one go.

[0046] When the overall offset exceeds the first threshold but does not exceed the second threshold, it is classified as a medium deviation, and a two-stage compensation process of coarse adjustment and fine adjustment is performed. In the coarse adjustment stage, the deviation component is multiplied by a preset coarse adjustment ratio coefficient as the target value, and the target value is quickly approximated at a relatively high speed. After the coarse adjustment is in place, an intermediate position verification is triggered, the actual position of the encoder is read back, and the coarse adjustment residual deviation is calculated. In the fine adjustment stage, the residual deviation is used as the target value, and the final compensation is completed at a low speed and with high precision.

[0047] When the overall offset exceeds the second threshold, it is classified as a large deviation, and an iterative compensation strategy with remeasurement is executed. After each round of compensation motion is completed, a second multimodal acquisition with the same configuration as the initial acquisition is automatically triggered. The acquisition area is shrunk to the corrected coordinates to reduce processing latency. The same processing flow is performed on the second acquisition data to extract the corrected corner coordinates and planar parameters.

[0048] Read the corner coordinates and planar parameters after correction, and solve the six-DOF pose after correction using the same EPNP and LM optimization process. Compare the pose with the standard pose to calculate the residual deviation components. Compare the absolute value of each residual deviation component with the convergence threshold condition. If all residual deviations meet the convergence threshold condition, the correction is considered successful, a completion confirmation signal is output, and the fork access operation execution permission is unlocked.

[0049] When any residual deviation component exceeds the convergence threshold, the iterative logic is initiated. The cumulative number of correction iterations for the current task is checked; if it is less than a preset upper limit, the iteration count is increased, and the current residual deviation is used as the input for a new round of correction, returning to the correction strategy adaptation stage. If convergence is not achieved even after reaching the preset upper limit, correction is deemed a failure, triggering an anomaly alarm pushed to the operation interface and maintenance terminal. The initial deviation vector, the compensation parameter sequence for each round, and the residual deviation are written into the correction failure record. The storage location is marked as having a mechanical anomaly pending inspection, and subsequent storage and retrieval tasks are suspended.

[0050] As can be seen from the above description, the material pallet automatic identification and correction method for ASRS system provided in this application constructs a complete multimodal data cache by synchronously triggering multimodal sensor hardware and aligning with unified time base data. It improves the identification success rate of pallets under label damage and occlusion conditions by weighted fusion of RFID, vision and geometric three-modal confidence. It improves the pose detection accuracy by cross-validating EPnP and LM optimized pose estimation algorithms with joint constraints of two-dimensional corner points and three-dimensional planes. It achieves a balance between correction response time and success rate by deviation hierarchical adaptive correction strategy and pose retest closed-loop iteration.

[0051] In one embodiment of the automatic pallet identification and correction method for an ASRS system in this application, see [link to relevant documentation]. Figure 2 It can also specifically include the following: Step S210: After the trigger signal is valid, the industrial camera acquires the grayscale image of the tray according to the preset exposure time, and writes it into the image buffer after low-pass filtering and noise reduction. The structured light depth sensor projects the coded structured light pattern in the same time slot, and the infrared receiver collects the reflection pattern and converts it into a three-dimensional point cloud through triangulation. The RFID reader polls and reads the material code from the tray tag according to the preset transmission power. The ultrasonic sensor array transmits pulses in sequence according to the preset time sequence and collects the echo flight time of each channel and converts it into a distance value.

[0052] In this embodiment, the hardware trigger pulses output from the stacker crane's programmable logic controller arrive at the external trigger ports of each sensor, serving as the start of the independent acquisition process for each channel. The trigger pulses are connected to the trigger inputs of the four sensor subsystems via shielded twisted-pair cables of equal length, ensuring the synchronization of triggering across channels.

[0053] The industrial camera initiates the exposure and readout sequence upon detecting a trigger pulse. Adaptive adjustment of the exposure time is achieved through a closed-loop proportional-integral controller. The control cycle is synchronized with the camera's frame rate, and the controller output is limited within a preset range to prevent overexposure or underexposure that could degrade image quality.

[0054] After exposure, the industrial camera reads the grayscale pixel matrix from the CMOS sensor. After removing readout noise through an on-chip low-pass filter, the matrix is ​​written to the image buffer in its original pixel format. The low-pass filter is used to suppress pixel-level random noise generated under dark current conditions, and the filter strength can be configured to achieve a balance between noise suppression and image detail preservation.

[0055] Upon receiving a trigger pulse, the structured light depth sensor initiates an coded structured light projection and acquisition sequence. The projector sequentially projects coded structured light patterns with different spatial frequencies onto the tray surface in a preset order, while the infrared receiver synchronously acquires the reflected patterns during the projection of each pattern.

[0056] The sensor's built-in processor performs a phase unwrapping algorithm on multiple deformable fringe images, establishing a depth map for each pixel based on the unfolded continuous phase distribution and the triangular geometry between the projector and receiver. The depth mapping result is output as a dense depth map in millimeters, which is then converted into a 3D point cloud frame through camera intrinsic backprojection and written to the point cloud buffer.

[0057] The RFID reader initiates a full-antenna inventory upon receiving a trigger pulse at a preset transmission power. For each responding tag, the reader records its material code and signal strength value. For repeated readings of the same tag across multiple antenna channels, the record corresponding to the highest signal strength is retained, while other duplicate records are discarded. The data is then sorted and written to the RFID data buffer.

[0058] The ultrasonic sensor array consists of multiple transceiver probes, which are evenly arranged at preset intervals on the front crossbeam of the forklift platform. Upon the arrival of the trigger pulse, the control circuit sequentially selects the probe channels according to a predetermined routing table. After each channel transmits a pulse train, it switches to receiving mode and uses a threshold detection method to detect the arrival time of the first wave in the echo signal, calculating the distance from that probe to the front edge of the pallet. After multiple rounds of data acquisition, the median distance is taken as the final output and written to the ultrasonic data buffer.

[0059] Step S220: Align all sensor data to a unified timestamp based on the system bus global clock, detect the completion flag and verification flag of each channel data and register the results as the channel acquisition flag. If any channel is marked as failed, trigger re-acquisition and mark it as unavailable if it continues to fail. Encapsulate all channel data into the multimodal data cache and write it to shared memory.

[0060] In this embodiment, the raw acquisition data and their respective acquisition timestamps of each channel are read from the four independent buffers written in step S210, and the data alignment module performs data synchronization in the time dimension. The global clock reference is provided by a precision time protocol synchronization module installed in the control cabinet.

[0061] Each sensor node completes time base calibration through this synchronization module during system power-on initialization, ensuring clock synchronization accuracy between nodes is controlled within microseconds. When writing acquired data to the buffer, each sensor simultaneously writes a timestamp of the acquisition time to its local time base; the timestamp format supports millisecond-level resolution.

[0062] The alignment strategy uses the midpoint of the industrial camera exposure as the unified reference time for this acquisition frame. The start and end times of the camera exposure are recorded by the camera firmware in the metadata of the acquisition frame, and the midpoint of the exposure is taken as the arithmetic mean of the two. The time difference between the acquisition time of other sensor data and the reference time is handled by time slot alignment.

[0063] Time slot alignment uses a preset slot width as the unit, assigning the acquisition time of each channel's data to the corresponding time slot, and grouping data within the same time slot into the same data packet. After the time slots are determined, a monotonically increasing sequence number is assigned to the data packet. If the same sensor has multiple acquisition results within the same time slot, the result with the timestamp closest to the center of that time slot is selected.

[0064] Data completeness and validity checks are performed independently for each sensor channel. The completeness check verifies whether the sensor acquisition completion flag is true; this flag is set by the sensor firmware after the acquired data has been completely written to the cache. Validity checks vary depending on the sensor type and include image resolution consistency checks, depth map effective pixel ratio checks, and ultrasonic distance value range checks.

[0065] When any channel is marked as failed, the data alignment module is responsible for initiating the hardware reacquisition process for that channel. The programmable logic controller (PLC) re-outputs an independent trigger pulse for the failed channel, driving the corresponding sensor to perform a single reacquisition under the same acquisition parameters. Upon successful reacquisition, the acquisition status flag is updated and replaces the original data.

[0066] If re-acquisition still fails, the channel is marked as unavailable, and the corresponding bit is set in the unavailable channel mask. After all channels have been acquired and verified, the grayscale image frame data segment, 3D point cloud frame data segment, RFID tag read record list data segment, and ultrasonic distance value array data segment are assembled into a multimodal data cache structure using the RFID read master material code as the index key.

[0067] The assembled data cache is written to the shared memory area of ​​the edge computing node. After the write operation is complete, a semaphore mechanism is used to notify the image processing pipeline and the point cloud processing pipeline to start reading. The unavailable channel mask is synchronously passed to the downstream module for subsequent fusion and recognition stages to adjust the confidence weights of each modality.

[0068] In one embodiment of the automatic pallet identification and correction method for an ASRS system in this application, see [link to relevant documentation]. Figure 3 It can also specifically include the following: Step S310: Read the grayscale image from the image portion of the multimodal data cache, perform adaptive histogram equalization to compensate for uneven illumination, divide the image into blocks according to a preset number of sub-blocks and limit the contrast amplification factor to not exceed a preset upper limit, and after equalization, smooth the image with a Gaussian kernel and then extract the edge gradient map through dual-threshold Canny edge detection.

[0069] In this embodiment, the grayscale pixel matrix is ​​read from the image payload segment of the multimodal data cache as the input to the image preprocessing pipeline. The sub-block size for adaptive histogram equalization is determined based on the image resolution and the typical imaging size of the tray. The sub-block size covers an appropriate proportion of the tray area to capture local contrast differences within the tray caused by changes in illumination direction and color.

[0070] Sub-blocks slide across the entire image with a preset step size, achieving a balance between computational uniformity and computational complexity in the coverage between adjacent sub-blocks. Within each sub-block, the gray-level histogram of that sub-block is first calculated, and then the cumulative distribution function of the histogram is calculated as the basis for gray-level mapping.

[0071] To avoid excessive contrast amplification caused by histogram spikes due to local noise, contrast limiting is performed on the histogram. Pixel counts exceeding a preset limiting threshold in the histogram are truncated and evenly redistributed across all gray levels. The limiting coefficient is adjustable via a configuration interface. The adjusted cumulative distribution function maps the original gray values ​​to equalized gray values.

[0072] At the boundary pixels between sub-blocks, the mapping results of adjacent sub-blocks are fused using bilinear interpolation. The interpolation weights are taken as the normalized distance from the pixel to the center of each sub-block, thereby eliminating block artifacts between sub-blocks. The equalized image has a uniform overall contrast distribution, and the gray-level gradient in the tray edge region is enhanced.

[0073] The size and standard deviation parameters of the Gaussian filter kernel are selected to achieve appropriate smoothing radii in both the horizontal and vertical directions. The filter kernel removes pixel-level random noise from the CMOS sensor while preserving the gradient abrupt changes required for Canny edge detection. The Gaussian kernel function performs a two-dimensional discrete convolution operation with the equalized image at each pixel location, with boundary pixels processed using a mirror-fill method.

[0074] Canny edge detection first calculates the horizontal and vertical gradients of the filtered image using the Sobel operator. The gradient magnitude of each pixel is taken as the square root of the sum of the squares of the gradients in both directions, and the gradient direction is taken as the arctangent value. Then, non-maximum suppression is performed, checking whether the gradient magnitude of each pixel is a local maximum in the neighborhood along the gradient direction; pixels that are not local maximums have their gradient magnitudes set to zero.

[0075] The low and high thresholds for dual-threshold edge connectivity are set according to a preset ratio. Pixels with gradient magnitudes below the low threshold are classified as non-edges and directly suppressed, while pixels with gradient magnitudes above the high threshold are classified as strong edges and directly preserved. Pixels in between that threshold and connected to the preserved strong edges are classified as weak edges and preserved. A complete binary edge gradient map is output for subsequent Hough transform detection.

[0076] Step S320: Accumulate straight line segments on the edge gradient map using polar coordinate Hough transform and extract the longitudinal and transverse main edges of the tray by angle clustering. Determine the initial pixel positions of the four corner points by the intersection of the two pairs of main edges and iteratively solve the sub-pixel coordinates of the four corner points of the tray within their neighborhood window using the gray-level gradient weighting method.

[0077] This embodiment performs a Hough transform on the binary edge map output by edge detection. The discretization setting of the polar coordinate parameter space covers the entire angle range of 0 degrees to 180 degrees and the entire distance range of 0 to the length of the image diagonal. For each non-zero pixel in the edge map, the corresponding distance value is calculated by traversing all angle steps and incremented in the corresponding cell of the accumulation matrix.

[0078] An appropriate accumulator threshold is set to distinguish between genuine edge segments and false positive segments generated by random noise. Segments exceeding this threshold indicate that at least a certain number of edge pixels are collinear, corresponding to the shortest detectable straight line segment length of the tray edge. Parameters for minimum segment length and maximum segment gap are used to filter out short, fragmented segments and allow for edge breaks caused by local occlusion.

[0079] The detected line segments are clustered according to their orientation angles. Line segments with orientation angles within a preset horizontal range are grouped into a horizontal orientation cluster, corresponding to the horizontal edge of the tray; those with orientation angles within a preset vertical range are grouped into a vertical orientation cluster, corresponding to the vertical edge of the tray. Within each cluster, the two line segments are selected as the main edge line segments after being sorted in descending order by accumulator values.

[0080] The intersection points are obtained by pairwise intersecting the four main edge line segments in two groups. Intersection points whose coordinates exceed the image boundary are discarded. The remaining four valid intersection points are the initial pixel-level coordinates of the four corner points of the tray. The combined process of edge detection and line fitting ensures that the main edge lines can still be stably extracted even when there is local dirt or occlusion on the tray edge.

[0081] Subpixel refinement is performed within a preset neighborhood window of the initial coordinates of each corner point. First, the horizontal and vertical gradients of all pixels within the window are calculated using the Sobel operator. Then, the gradient outer product matrix is ​​constructed for each pixel and accumulated using the Gaussian weighted distance from the pixel to the initial position of the corner point.

[0082] The accumulated matrix is ​​decomposed into eigenvalues. The eigenvectors corresponding to the smaller eigenvalues ​​are tangents to the edges at the corners, and their orthogonal directions are the normals to the edges, i.e., the precise locations of the corners. The centroid of the weighted grayscale gradient is calculated along the normal direction to obtain the updated corner coordinates for this iteration.

[0083] Repeat the above iterative process until the update amount is less than the preset threshold or the maximum number of iterations is reached. After the iteration converges, the four sub-pixel corner coordinates are stored in a corner coordinate array for subsequent pose calculation and geometric dimension verification stages.

[0084] In one embodiment of the automatic pallet identification and correction method for an ASRS system in this application, see [link to relevant documentation]. Figure 4 It can also specifically include the following: Step S410: Read the three-dimensional point cloud from the point cloud portion of the multimodal data cache, traverse each point to calculate its average distance to K points in its neighborhood, and construct a statistical outlier criterion based on the global mean and standard deviation to filter out outliers. Perform downsampling on the filtered point cloud according to a preset voxel side length grid.

[0085] This embodiment reads a 3D point cloud frame from the point cloud payload segment of the multimodal data cache as input to the point cloud processing pipeline. The output point cloud of the structured light depth sensor contains a large number of spatial points, each composed of 3D coordinates. The first step in statistical outlier removal is to construct a spatial index structure for the entire point cloud to accelerate nearest neighbor search.

[0086] The value of the nearest neighbor search parameter K needs to be balanced between the local representativeness of the search radius and statistical robustness. If the value of K is too small, the variance of the mean estimation will be too large; if the value of K is too large, the neighborhood will no longer be local. After selecting the value of K, all points are traversed, and for each point, its K nearest neighbors are found and the average distance is calculated.

[0087] The set of average K-nearest neighbor distances for the entire point cloud constitutes a sample with an approximately Gaussian distribution. The global mean and global standard deviation of this sample are calculated, and points whose average K-nearest neighbor distance is greater than the mean plus a preset standard deviation multiple are marked as outliers. The value of the standard deviation multiple effectively filters out flypoints caused by specular reflection interference while retaining the vast majority of valid points.

[0088] After outlier removal, the point cloud is downsampled using a predefined voxel edge length. For each non-empty voxel, the arithmetic centroid of all its points is calculated, and the point closest to the centroid is retained as the representative point of that voxel; the rest are discarded. Downsampling significantly reduces the size of the point cloud while preserving key geometric features of the pallet corner region.

[0089] Step S420: Input the downsampled point cloud into RANSAC plane fitting, randomly select three non-collinear points to form a candidate plane and count the interior points according to the distance threshold condition, take the plane corresponding to the largest interior point set as the pallet bearing surface and solve its unit normal vector and distance, multiply the RFID confidence score by the first weight coefficient, add the visual matching score by the second weight coefficient, add the geometric verification score by the third weight coefficient to obtain the fusion confidence score.

[0090] In this embodiment, the downsampled point cloud is input into the RANSAC plane fitting algorithm. The distance threshold is determined by multiplying the depth accuracy of the structured light sensor at the standard working distance by a safety factor, ensuring that points caused by sensor measurement noise are still counted as inliers, while points caused by actual geometric offsets are excluded.

[0091] In each iteration, three points are randomly selected from the point cloud, and it is checked whether the three points are collinear. If they are not collinear, the equation of the plane determined by the three points is calculated. The entire point cloud is traversed, and points whose distance to the candidate plane is less than the distance threshold are counted as interior points. The iteration count is configured to ensure that at least one sample consisting entirely of interior points is drawn in the dataset containing non-planar points with a high probability.

[0092] After the iteration, the candidate plane with the largest number of interior points is selected. The final plane equation parameters are obtained by refitting all its interior points using the least squares method. The least squares estimate of the plane normal vector is obtained by singular value decomposition and then normalized to a unit vector. The directed distance is the signed distance from the origin to the plane.

[0093] RFID confidence scores are extracted from RFID read records and are normalized to a value between zero and one based on tag signal strength. The upper limit corresponds to the signal with no attenuation, and zero corresponds to the signal being unreadable. Visual matching scores are obtained from the matching process between corner coordinates and pre-stored standard corner coordinates in the template library, and are normalized and mapped to the zero-to-one range.

[0094] The geometric verification score is obtained from the point cloud fitting process, and is calculated based on the ratio of the number of points in the fitted plane to the total number of points in the tray area, as well as the deviation between the fitted tray size and the nominal size. The score is the upper limit when the deviation is within a preset range, and the score decays to zero when the deviation exceeds the acceptable range.

[0095] The three confidence components are weighted and summed according to preset weights to obtain the fusion confidence score. The weight coefficients are assigned based on the independent recognition performance and complementarity of each modality under typical operating conditions. The RFID confidence score is multiplied by the first weight coefficient, the visual matching score by the second weight coefficient, and the geometric verification score by the third weight coefficient; the sum of these three items is the fusion confidence score.

[0096] The system compares the confidence level with a preset threshold. If the confidence level is not lower than the threshold, the pallet identification is considered successful. A high confidence level is used to ensure that the material code, batch number, and pallet type of the current pallet are consistent with the task requirements, and the pallet identification identifier is output for use by the pose calculation module. If the confidence level is lower than the threshold, a re-acquisition or manual intervention process is triggered.

[0097] In one embodiment of the automatic pallet identification and correction method for an ASRS system in this application, see [link to relevant documentation]. Figure 5 It can also specifically include the following: Step S510: The sub-pixel coordinates of the four corner points of the tray and the plane normal vector are used as EPnP inputs. The control point coordinates are solved by constructing a linear equation through the correspondence between the four control points in the camera coordinate system and the world coordinate system. The initial pose values ​​of the tray corner points in the camera coordinate system are restored and then input into LM optimization. The six-degree-of-freedom pose is iteratively refined based on the cost function of reprojection error and plane constraint.

[0098] In this embodiment, the sub-pixel corner coordinates, plane normal vectors, and plane distances are read from the tray pose parameter set output in step S120 and used as inputs to the pose calculation pipeline. During the deployment phase, the system has completed camera intrinsic parameter calibration and extrinsic parameter calibration between the depth sensor and the camera. The camera intrinsic parameter matrix is ​​obtained using the checkerboard calibration method.

[0099] The EPnP algorithm takes multiple pairs of corresponding 3D and 2D points as input. The 3D reference points are the coordinates of the four corners of the tray in the tray's body coordinate system, with the tray's geometric center as the origin, the tray's length direction as the X-axis, and the width direction as the Y-axis. The 2D projection points are the corresponding sub-pixel corner coordinates extracted in steps S310 and S320.

[0100] EPnP represents a 3D reference point using a weighted average of four virtual control points. The control points are constructed based on the geometric distribution of the 3D reference point: the centroid of the 3D reference point is taken as the first control point, and the offset points on the three principal directions obtained through principal component analysis are taken as the other three control points.

[0101] After substituting the 3D reference points, weighted by control points, into the perspective projection equation, a system of linear equations concerning the coordinates of the control points in the camera coordinate system is constructed. Singular value decomposition is then used to find the least-squares solution. The coordinates of each 3D reference point in the camera coordinate system are recovered from the control point coordinates, and the initial rotation matrix and translation vector are decomposed from these to serve as initial values ​​for LM optimization.

[0102] The cost term of the LM optimization objective function includes reprojection error and plane constraint terms. The reprojection error term measures the consistency between the visual pose and the image observation. It projects each corner point onto the image plane using the current estimated pose and calculates the sum of squared Euclidean distances between the projected point and the corresponding sub-pixel corner point.

[0103] The planar constraint term introduces the planar normal vector obtained from the structured light depth sensor as a geometric constraint to correct the depth drift in visual pose. When the corner points of the tray are distributed in an approximately symmetrical rectangle in the image plane, the depth sensitivity of pure visual PnP is low. After adding the planar constraint, the depth drift is limited to the accuracy range of the depth sensor.

[0104] The LM algorithm iteratively solves for the refined pose over the total cost function, with the damping factor adaptively adjusted after each iteration based on the cost reduction. Upon convergence, the refined six-DOF pose is output, containing three translational components and three Euler angle components obtained from the transformation of the rotation matrix.

[0105] Step S520: Compare the refined six-degree-of-freedom pose with the standard picking pose, calculate the lateral offset as the difference between the current X coordinate and the standard X coordinate, the fork extension direction offset as the difference between the current Z coordinate and the standard Z coordinate, and the deflection angle as the rotation angle of the pallet normal vector around the Y axis, and obtain the three deviation components for selection of the correction strategy.

[0106] This embodiment starts the deviation calculation from the six-DOF pose after LM optimization convergence. The standard picking pose is obtained through the teaching and calibration process during the system deployment phase and stored in the system configuration database using the combined code of the location column number, layer number, and depth number as the index key. It supports recalibration and updates after equipment overhaul or rack structure adjustment.

[0107] The warehouse world coordinate system uses the aisle direction as the X-axis corresponding to the rack column direction, the vertical direction as the Y-axis corresponding to the rack layer direction, and the storage location depth direction as the Z-axis corresponding to the fork extension direction. Comparison between the current pose and the standard pose is performed according to this coordinate system.

[0108] When converting the rotation matrix output from LM optimization to Euler angles, a preset order is used. The lateral offset is the difference between the current pallet center X coordinate and the standard pose X coordinate; the fork extension direction offset is the difference between the current pallet center Z coordinate and the standard pose Z coordinate; and the deflection angle is the rotation angle of the pallet normal vector around the Y-axis.

[0109] These three deviation components constitute the core input for selecting the correction strategy. The lateral offset and the fork extension direction offset directly affect whether the forks can smoothly enter the pallet positioning slot, while the deflection angle affects the alignment accuracy between the fork posture and the pallet slot. The three together reflect the complete spatial deviation of the pallet in the current position relative to the ideal picking position.

[0110] After the deviation calculation is completed, cross-validation of the visual pose and point cloud geometric parameters can be optionally performed. Cross-validation checks the consistency between the depth component of the visual pose and the planar distance of the point cloud, as well as the consistency between the visual pose angle and the angle calculated from the inverse plane normal vector. If the consistency exceeds a preset threshold, pose recalculation or degradation processing is triggered.

[0111] In one embodiment of the automatic pallet identification and correction method for an ASRS system in this application, see [link to relevant documentation]. Figure 6 It can also specifically include the following: Step S610: The absolute values ​​of the three deviation components are weighted and summed according to a preset normalization coefficient to obtain the comprehensive offset amplitude. When the comprehensive offset amplitude is within a preset first threshold, a micro-deviation single compensation strategy is selected to directly superimpose each deviation component onto the servo position command of each axis and perform a single compensation motion at normal speed.

[0112] This embodiment begins the adaptation of the correction strategy from the three deviation components output in step S520. The overall offset magnitude is calculated by weighting and summing the deviation components according to the normalization coefficient. The normalization coefficient is set so that the contributions of the three components to the overall magnitude are comparable in magnitude, ultimately yielding a dimensionless deviation severity index.

[0113] A preset first threshold is used to distinguish between minor and moderate deviations. The first threshold is set based on the mechanical tolerance between the forks and the pallet positioning groove. The width of the pallet positioning groove is the width of the fork fingers plus the mating clearance. When the offset is within the tolerance range, the fork fingers can still enter the positioning groove without interference.

[0114] When the overall offset amplitude does not exceed a preset first threshold, the system determines the current deviation to be a minor deviation. The minor deviation mode adopts a single-shot fast compensation strategy, directly converting the three deviation components into the servo position correction amount for each axis. The system reads the current target position register value of each axis servo driver, adds the deviation component to it, and then rewrites it.

[0115] The three correction values ​​are simultaneously sent to the three servo drives via the process data object mapping of the industrial Ethernet bus within a single communication frame, with the instruction update cycle set to a typical value. The motion parameter configuration is set to an appropriate ratio of the rated acceleration and speed of each axis, achieving small-amplitude compensation at a relatively fast speed while maintaining sufficient mechanical safety margin.

[0116] The motion curve adopts an S-shaped velocity planning to reduce the mechanical shock caused by sudden acceleration changes. The compensation motion positioning judgment condition is that the position deviation of each axis is less than the corresponding positioning window and the stabilization time is not less than the preset duration. After positioning, the servo drive of each axis will transmit the positioning flag to the motion control module via the industrial Ethernet bus.

[0117] Step S620: When the overall offset amplitude is between the first threshold and the second threshold, a two-stage coarse and fine compensation strategy with medium deviation is selected. The coarse adjustment is performed at a high rate until the deviation is within the range of micro deviation, and the fine adjustment is performed at a low rate until convergence. When it exceeds the second threshold, a large deviation iterative compensation strategy is selected and pose retest is triggered after each round of compensation.

[0118] This embodiment begins with segmented determination of the overall offset magnitude to enter the compensation process for medium or large deviations. A preset second threshold corresponds to the extreme combination where the deviation in each direction exceeds the unilateral tolerance but is still within the compensable range. When the deviation exceeds the first threshold but not the second threshold, the magnitude of the deviation has exceeded the unilateral tolerance of the pallet positioning groove.

[0119] The target value in the coarse adjustment stage is calculated by multiplying each deviation component by a preset coarse adjustment ratio coefficient. This coefficient strikes a balance between the theoretically optimal compensation and the mechanical safety margin, aiming to quickly reduce the deviation to a micro-deviation range rather than striving for a perfect fit in one step. The coarse adjustment acceleration and the upper speed limit are set at an appropriate ratio to the rated values.

[0120] After the coarse adjustment is completed, the intermediate position verification process is triggered. The actual encoder position value is read back from each axis servo drive via the industrial Ethernet bus, and compared component by component with the coarse adjustment target value to calculate the coarse adjustment residual deviation. The residual deviation is usually within the range of minor deviations; if it unexpectedly exceeds the range, the full value is used as the fine adjustment target value.

[0121] In the fine-tuning phase, the residual deviation from the coarse-tuning is used as the target value, and the upper limits of acceleration and speed are further lowered. The low acceleration and low speed configuration makes the fine-tuning motion smoother, and the servo drive can use a higher position loop gain at low speeds to improve steady-state positioning accuracy. More stringent window conditions are used to determine when the fine-tuning is in place.

[0122] When the overall offset exceeds the preset second threshold, it is classified as a large deviation. Typical causes of large deviation scenarios include pallet slippage during the last loading, slight deformation of the rack structure due to long-term load deflection, or systematic bias in the stacker crane positioning system in a specific section.

[0123] The large deviation mode employs an iterative compensation strategy with remeasurement. After each round of compensation, a second multimodal acquisition with the same configuration as the initial acquisition is automatically triggered. During the iteration process, the motion acceleration and velocity upper limit are taken as an appropriate proportion of the rated value. After each compensation, the comprehensive offset amplitude and the three deviation components are recalculated. If the new comprehensive offset amplitude has dropped below a lower threshold, the next round of strategy automatically transitions to the corresponding mode, and the number of iterations does not exceed the preset upper limit.

[0124] In one embodiment of the automatic pallet identification and correction method for an ASRS system in this application, see [link to relevant documentation]. Figure 7 It can also specifically include the following: Step S710: After the correction compensation command is executed, a second multimodal acquisition with the same configuration as the initial acquisition is triggered. The acquisition range is reduced to a smaller area centered on the corrected coordinates to reduce processing latency. The grayscale image and 3D point cloud acquired by the second multimodal acquisition are processed in the same way as the initial acquisition to extract the corrected corner coordinates and planar parameters.

[0125] In this embodiment, a second multimodal acquisition is triggered after the position signals of all axes in the correction motion are valid, serving as the starting point for the iterative verification pipeline. The triggering mechanism for the second acquisition is the same as that for the first acquisition, whereby the motion control module outputs a hardware trigger pulse after detecting that the position flags of all axes are true and the stabilization time has reached a preset duration.

[0126] The data channel type, operating parameters of each sensor, and output data format for the secondary acquisition are consistent with those for the initial acquisition to ensure that the sensor characteristics of the two acquisitions are comparable. The key difference between the secondary and initial acquisitions lies in the optimization of the acquisition area.

[0127] Since the correction motion has moved the stacker crane's loading platform to a position closer to the standard picking pose, the region of interest (ROI) of the industrial camera is narrowed down to the estimated pallet center coordinates after correction. Pixels outside this region are not involved in subsequent processing, significantly reducing the number of input pixels to the image processing pipeline.

[0128] The projection area of ​​the structured light depth sensor also shrinks around the corrected coordinates, and the spatial frequency of the projected pattern maintains the same density as the initial measurement in the central region, while decreasing in density in the outer region. This significantly reduces the computation time for the overall depth map compared to the initial acquisition.

[0129] RFID identification uses the initial identification results in the secondary data acquisition without needing to be updated. Since the pallet's identity does not change within the same work task, the RFID data acquired in the secondary acquisition is only used to verify that the tag is still within the reading range and has not fallen off or shifted, reducing the time required to a small percentage of the initial acquisition.

[0130] The grayscale image frames acquired a second time are input through the same image preprocessing pipeline as the initial acquisition. Adaptive histogram equalization, Gaussian filtering, Canny edge detection, Hough transform line detection, and sub-pixel corner refinement are performed sequentially, outputting the corrected sub-pixel corner coordinates.

[0131] The second-time acquired 3D point cloud frame input uses the same point cloud preprocessing pipeline as the initial acquisition. Statistical outlier removal, voxel downsampling, and RANSAC plane fitting are performed sequentially, outputting the corrected plane normal vector and plane distance. The preprocessing results from the second acquisition are written to the pose calculation buffer.

[0132] Step S720: Solve the six-degree-of-freedom pose after correction based on the corrected corner coordinates and planar parameters, and compare it with the standard picking pose to obtain the residual deviation. When the absolute value of each residual deviation does not exceed the preset convergence threshold, the correction is determined to be successful and a completion signal is output. If the number of iterations does not reach the preset upper limit, the number of iterations is increased and a new round of correction is returned.

[0133] This embodiment begins with the calculation and convergence determination of residual deviations from the corrected corner coordinates and planar parameters output in step S710. The pose solution process is the same as the initial one, using the EPnP algorithm to construct a system of linear equations for the control points to solve for the initial values, and then using LM optimization iteration to solve for the refined pose.

[0134] The initial pose for LM optimization can be taken as the initial pose calculation result plus the compensation amount of this correction as an initial value closer to the true value, so as to accelerate convergence. After correction, the refined pose is compared with the standard picking pose component by component in the same process to calculate three residual deviation components: the lateral offset after correction, the fork extension direction offset, and the deflection angle.

[0135] Convergence is determined by comparing the absolute values ​​of the three residual deviation components one by one with the convergence threshold condition. The physical basis of the convergence threshold condition is the mechanical fit tolerance between the pallet positioning groove and the fork fingers. The width of the positioning groove is taken as the width of the fork fingers plus the fit clearance, and the convergence allowance of the translation residual deviation corresponds to an appropriate proportion of the clearance on one side.

[0136] The convergence thresholds for rotational residual deviation and translational residual deviation are matched in magnitude to ensure that the fork fingers can still enter the positioning slot without interference in the most unfavorable residual deviation direction. The correction is successful when the absolute values ​​of the three residual deviation components do not exceed the corresponding convergence threshold conditions.

[0137] The system sets a correction completion flag in shared memory. This flag triggers the fork access control state machine to transition from a waiting correction state to an access-allowed state, unlocking the fork extension, lifting, clamping, and retraction action sequence. A successful correction record is simultaneously written to the process database. Record fields include task identifier, initial deviation vector, selected correction strategy code, actual compensation amount for each axis, and final residual deviation component value.

[0138] When any residual bias component exceeds the convergence threshold, the iterative decision logic is initiated. The cumulative number of correction iterations for the current task is read and compared with a preset upper limit. If the current number of iterations is less than the upper limit, the number of iterations is incremented, and the current residual bias is used as the input for a new round of correction before returning to the correction strategy adaptation stage.

[0139] If the system fails to converge after reaching the upper limit, it determines that the correction has failed. The correction failure record is written to the process database, recording the strategy selection, compensation parameters, and residual deviation for each round. The storage location is marked as having a mechanical abnormality and requiring inspection with a warning icon on the operation interface. An equipment maintenance work order is automatically generated and pushed to the maintenance personnel's terminal. Subsequent automatic storage and retrieval operations skip this storage location until manual correction is completed.

[0140] As can be seen from the above description, the material pallet automatic identification and correction method for ASRS system provided in this application constructs a complete multimodal data cache by synchronously triggering multimodal sensor hardware and aligning with unified time base data. It improves the identification success rate of pallets under label damage and occlusion conditions by weighted fusion of RFID, vision and geometric three-modal confidence. It improves the pose detection accuracy by cross-validating EPnP and LM optimized pose estimation algorithms with joint constraints of two-dimensional corner points and three-dimensional planes. It achieves a balance between correction response time and success rate by deviation hierarchical adaptive correction strategy and pose retest closed-loop iteration.

[0141] To implement all or part of the aforementioned method, this application provides an embodiment of an apparatus for implementing the automatic pallet identification and correction method for an ASRS system, see [link to embodiment]. Figure 8 The device specifically includes the following components: The multimodal sensing trigger module is used to trigger multimodal sensors to collect pallet grayscale images, 3D point clouds, RFID tag material codes, and ultrasonic leading-edge distance values ​​after the stacker crane arrives at its position. The collected data is then aligned by timestamps, registered with channel acquisition marks, and compiled into a multimodal data buffer before being output. This module uses industrial cameras, structured light depth sensors, RFID readers, and ultrasonic sensor arrays as front-end sensing hardware, a programmable logic controller as the hardware triggering center, and shared memory as the data aggregation medium. It completes multi-sensor synchronous triggering and data acquisition after the stacker crane reaches the target cargo location and enters the positioning window.

[0142] The multimodal fusion recognition module reads the grayscale image from the multimodal data cache, extracts the sub-pixel coordinates of the four corner points of the tray through equalization and edge detection, obtains the plane normal vector and distance from the point cloud through filtering and plane fitting, and calculates the fusion confidence score by weighting the RFID confidence score, visual matching score, and geometric verification score according to preset weights, and outputs the tray pose parameter set. This module runs on the edge computing node with two parallel pipelines for image processing and point cloud processing. The pipeline outputs converge in the fusion recognition stage and output the fusion recognition conclusion based on hierarchical threshold conditions.

[0143] The pose solving and deviation calculation module is used to read the pallet pose parameter set, solve the six-DOF pose based on EPnP and LM optimization, and compare it with the standard picking pose to obtain the lateral offset, fork extension direction offset, and deflection angle. This module integrates three sub-units: EPnP linear initial value estimation, LM nonlinear optimization, and vision-point cloud cross-validation. In the optimization, a planar normal vector geometric constraint is introduced to correct the depth direction drift.

[0144] The correction strategy adaptation and execution module selects a correction strategy based on the combined offset magnitude of the three deviation components. It performs single-stage compensation for minor deviations, coarse-fine two-stage compensation for moderate deviations, and iterative compensation with weight measurement for large deviations. Each deviation component is mapped to a servo correction command for each axis, and the correction action is executed through the motion control system. This module is based on a deviation grading rule engine and an S-shaped speed curve generator, and uses an industrial Ethernet bus and servo driver to complete command issuance and status feedback.

[0145] The secondary verification and iterative convergence module is used to trigger a secondary acquisition to calculate the residual deviation after correction. If the residual deviation does not exceed the convergence threshold, the correction is considered successful; otherwise, iterative correction and remeasurement are performed, with the number of iterations not exceeding a preset upper limit. This module uses a finite state machine to manage the closed-loop logic of correction, verification, and iteration. The convergence criterion is determined based on the mechanical fit tolerance between the pallet positioning groove and the fork fingers.

[0146] The anomaly handling and alarm module is used to trigger re-collection or manual intervention when the fusion confidence level falls below a preset threshold, and to trigger an anomaly alarm when the correction iteration fails to converge after reaching a preset upper limit. This module pushes anomaly information to relevant personnel in a tiered alarm manner on the operation terminal interface.

[0147] As can be seen from the above description, the material pallet automatic identification and correction device for ASRS system provided in this application embodiment achieves full-process automation from multi-sensor hardware synchronous triggering to correction closed-loop verification through the coordination of six modules: multi-modal perception triggering, multi-modal fusion identification, pose solving and deviation calculation, correction strategy adaptation and execution, secondary verification and iterative convergence, and anomaly handling and alarm.

[0148] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic identification and correction of material pallets in an ASRS system, characterized in that, The method includes: After the stacker crane arrives, it triggers a multimodal sensor to collect a grayscale image of the pallet, a 3D point cloud, the material code on the label, and the ultrasonic leading edge distance value. This includes: an industrial camera collecting the grayscale image of the pallet for a preset exposure time after the trigger signal is valid, and writing it into the image buffer after low-pass filtering and noise reduction; a structured light depth sensor projecting an coded structured light pattern in the same time slot, and an infrared receiver collecting the reflected pattern and converting it into a 3D point cloud through triangulation; an RFID reader polling and reading the material code from the pallet label according to a preset transmission power; and an ultrasonic sensor array sequentially transmitting pulses according to a preset time sequence and collecting the echo flight time of each channel, converting it into a distance value. Using the system bus global clock as a reference, all sensor data is aligned to a unified timestamp. The completion and verification flags of each channel's data are detected and the results are registered as channel acquisition flags. If any channel is marked as failed, re-acquisition is triggered, and if continuous failures occur, it is marked as unavailable. All channel data is encapsulated into a multimodal data buffer and written to shared memory. The collected data is aligned by timestamp and registered with channel flags to form a multimodal data buffer, which is then output. The grayscale image is read from the multimodal data cache, and after adaptive equalization and Gaussian filtering, the sub-pixel coordinates of the four corner points of the tray are extracted by Canny edge detection and Hough transform. The three-dimensional point cloud is read from the multimodal data cache, and after outlier filtering and voxel downsampling, the normal vector and distance of the tray plane are obtained by fitting it with RANSAC. The RFID confidence score, visual matching and geometric verification score are weighted and summed according to preset weights to obtain the fusion confidence score, and the tray pose parameter set is output when it is not lower than the preset threshold condition. The pallet pose parameter set is read, and the six-DOF pose is solved based on EPnP and LM optimization. The six-DOF pose is compared with the standard picking pose to obtain the lateral offset, fork extension direction offset and deflection angle. Based on the comprehensive offset magnitude of the three deviation components, a correction strategy is selected and single compensation is performed for micro deviations, coarse and fine two-stage compensation is performed for medium deviations, and iterative compensation with weight measurement is performed for large deviations. After correction, a second acquisition is triggered to calculate the residual deviation. If each residual deviation does not exceed the convergence threshold, the correction is considered successful. If it exceeds the threshold, iterative correction is performed and the number of iterations does not exceed the preset upper limit.

2. The method for automatic identification and correction of material pallets in an ASRS system according to claim 1, characterized in that, The step of reading the grayscale image from the multimodal data cache, performing adaptive histogram equalization and Gaussian filtering, and then extracting the sub-pixel coordinates of the four corner points of the tray using Canny edge detection and Hough transform includes: The grayscale image is read from the image portion of the multimodal data cache, and adaptive histogram equalization is performed to compensate for uneven illumination. The image is then divided into blocks according to a preset number of sub-blocks and the contrast amplification factor is limited to not exceeding a preset upper limit. After equalization, the image is smoothed by a Gaussian kernel and then the edge gradient map is extracted by double-threshold Canny edge detection. On the edge gradient map, straight line segments are accumulated by polar coordinate Hough transform and the longitudinal and transverse main edges of the tray are extracted by angle clustering. The initial pixel positions of the four corner points are determined by the intersection points of the two pairs of main edges, and the sub-pixel coordinates of the four corner points of the tray are iteratively solved in their neighborhood window according to the gray-level gradient weighting method.

3. The method for automatic identification and correction of material pallets in an ASRS system according to claim 1, characterized in that, The step of obtaining the fused confidence score by weighting and summing the RFID confidence score, visual matching score, and geometric verification score according to preset weights includes: The three-dimensional point cloud is read from the point cloud portion of the multimodal data cache, each point is traversed to calculate its average distance to K neighboring points, and a statistical outlier criterion is constructed based on the global mean and standard deviation to filter out outliers. The filtered point cloud is downsampled according to a preset voxel side length grid. The downsampled point cloud is input into the RANSAC plane for fitting. Three non-collinear points are randomly selected to form a candidate plane, and the interior points are counted according to the distance threshold condition. The plane corresponding to the largest set of interior points is taken as the pallet bearing surface, and its unit normal vector and distance are calculated. The RFID confidence score is multiplied by the first weight coefficient, the visual matching score is multiplied by the second weight coefficient, the geometric verification score is multiplied by the third weight coefficient to obtain the fusion confidence score.

4. The method for automatic identification and correction of material pallets in an ASRS system according to claim 1, characterized in that, The method for solving the six-DOF pose based on EPnP and LM optimization includes: The sub-pixel coordinates of the four corner points of the tray and the plane normal vector are used as EPnP input. The linear equation is constructed by the correspondence between the four control points in the camera coordinate system and the world coordinate system to solve the control point coordinates. The initial pose values ​​of the tray corner points in the camera coordinate system are restored and then fed into LM optimization. The six-degree-of-freedom pose is iteratively refined based on the cost function of reprojection error and plane constraint. The refined six-degree-of-freedom pose is compared with the standard picking pose. The lateral offset is calculated as the difference between the current X coordinate and the standard X coordinate, the fork extension direction offset is calculated as the difference between the current Z coordinate and the standard Z coordinate, and the deflection angle is calculated as the rotation angle of the pallet normal vector around the Y axis. The three deviation components are obtained for the selection of the correction strategy.

5. The method for automatic identification and correction of material pallets in an ASRS system according to claim 1, characterized in that, The correction strategy is selected based on the combined offset magnitude of the three deviation components, including: The absolute values ​​of the three deviation components are weighted and summed according to a preset normalization coefficient to obtain the comprehensive offset amplitude. When the comprehensive offset amplitude is within a preset first threshold, a micro-deviation single compensation strategy is selected to directly superimpose each deviation component onto the servo position command of each axis and perform a single compensation motion at normal speed. When the overall offset is between the first threshold and the second threshold, a two-stage coarse and fine compensation strategy with moderate deviation is selected. The coarse adjustment is performed at a high rate until the deviation is within the range of micro deviation, and the fine adjustment is performed at a low rate until convergence. When the offset exceeds the second threshold, a large deviation iterative compensation strategy is selected, and pose retesting is triggered after each round of compensation.

6. The method for automatic identification and correction of material pallets in an ASRS system according to claim 1, characterized in that, The process of triggering a second data acquisition to calculate the residual deviation after correction includes: After the correction compensation command is executed, a second multimodal acquisition with the same configuration as the initial acquisition is triggered. The acquisition range is reduced to a smaller area centered on the corrected coordinates to reduce processing latency. The grayscale image and 3D point cloud acquired in the second multimodal acquisition are processed in the same way as the initial acquisition to extract the corrected corner coordinates and planar parameters. The six-degree-of-freedom pose after correction is calculated based on the corrected corner coordinates and planar parameters and compared with the standard picking pose to obtain the residual deviation. When the absolute value of each residual deviation does not exceed the preset convergence threshold, the correction is considered successful and a completion signal is output. If the number of iterations does not reach the preset upper limit, the number of iterations is increased and a new round of correction is returned.

7. An automatic pallet identification and correction device for an ASRS system, characterized in that, The device includes: The multimodal sensing trigger module is used to trigger multimodal sensors to collect grayscale images of the pallet, 3D point clouds, RFID tag material codes, and ultrasonic leading-edge distance values ​​after the stacker crane is in position. This includes: an industrial camera collecting the pallet grayscale image for a preset exposure time after the trigger signal is valid, and writing it to an image buffer after low-pass filtering and noise reduction; a structured light depth sensor projecting an coded structured light pattern in the same time slot, and an infrared receiver collecting the reflected pattern, converting it into a 3D point cloud via triangulation; an RFID reader polling and reading the pallet tag to extract the material code according to a preset transmission power; and an ultrasonic sensor array sequentially transmitting pulses according to a preset time sequence and collecting the echo flight time of each channel, converting it into a distance value. Using the system bus global clock as a reference, all sensor data is aligned to a unified timestamp. The completion and verification flags of each channel's data are detected and the results are registered as the channel acquisition flag. If any channel is marked as failed, re-acquisition is triggered, and if continuous failures occur, it is marked as unavailable. All channel data is encapsulated into a multimodal data buffer and written to shared memory. The collected data is aligned by timestamp and registered with channel acquisition flags to form a multimodal data buffer, which is then output. The multimodal fusion recognition module is used to read the grayscale image from the multimodal data cache, extract the sub-pixel coordinates of the four corner points of the tray through equalization and edge detection, obtain the plane normal vector and distance from the point cloud through filtering and plane fitting, and obtain the fusion confidence score by weighting the RFID confidence score, visual matching score and geometric verification score according to preset weights and output the tray pose parameter set. The pose solving and deviation calculation module is used to read the pallet pose parameter set, optimize the six-degree-of-freedom pose based on EPnP and LM, and compare it with the standard picking pose to obtain the lateral offset, fork extension direction offset and deflection angle. The correction strategy adaptation and execution module is used to select the correction strategy based on the comprehensive offset magnitude of the three deviation components. It performs single compensation for micro deviations, coarse and fine two-stage compensation for medium deviations, and iterative compensation with weight measurement for large deviations. It maps each deviation component into servo correction commands for each axis and executes the correction action through the motion control system. The secondary verification and iterative convergence module is used to trigger secondary acquisition to calculate the residual deviation after correction. If the residual deviation does not exceed the convergence threshold, the correction is considered successful. If it does, iterative correction and remeasurement are performed, and the number of iterations does not exceed the preset upper limit. The anomaly handling and alarm module is used to trigger re-acquisition or manual intervention when the fusion confidence level is lower than the preset threshold, and to trigger an anomaly alarm when the correction iteration reaches the preset upper limit but still fails to converge.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the automatic pallet identification and correction method for an ASRS system as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the automatic pallet identification and correction method for an ASRS system as described in any one of claims 1 to 6.

Images