Rack beam detection method for forklift, forklift, computer storage medium and computer program product

Abstract

The present application relates to a rack beam detection method for a forklift, comprising: acquiring depth data collected by a depth camera, and converting the depth data into point clouds in a preset coordinate system; acquiring a mast lifting height of the forklift, and extracting target point clouds from the point clouds based on the mast lifting height; orthogonally projecting the target point clouds to a vertical plane along a forking direction to generate a two-dimensional occupancy grid image; counting the number of occupancy grids of each row in the height direction of the occupancy grid image to generate a one-dimensional height-density histogram; performing difference calculation on the height-density histogram, and traversing from bottom to top along the height direction, and determining the first position meeting a preset negative peak value condition as an upper edge height value of a to-be-detected rack beam; and outputting a detection value based on the upper edge height value of the to-be-detected rack beam. The present application also relates to a forklift, a computer storage medium and a computer program product.

Description

Technical Field

[0001] This invention relates to the field of intelligent warehousing technology, and in particular to a method for detecting rack beams for forklifts, forklifts, computer storage media, and computer program products. Background Technology

[0002] In the field of automated warehousing and logistics, unmanned forklifts need to use onboard sensors to accurately locate the rack beams or the ground to ensure the safety and efficiency of picking up goods.

[0003] In existing technologies, a single-depth camera or LiDAR is typically installed in the middle of the forklift mast to collect 3D point cloud data of the environment and use algorithms such as 3D shape fitting, overall template matching, or finding the geometric center of the beam for target recognition. However, in actual working conditions, this approach has significant limitations: First, due to the limited field of view of a single camera, the beam is often not fully observed due to obstruction by the fork arms or goods; second, warehouse racks have diverse specifications, and beams of varying thicknesses make it difficult to universally apply algorithms based on center points or overall shapes; third, complex 3D point cloud processing algorithms place extremely high demands on the computing power of embedded controllers, making it difficult to balance detection accuracy and operational cycle time; finally, when the forklift mast is raised to a high position, the rigidity and inertia of the mechanical structure cause significant physical shaking of the mast, making it easy to capture extreme shaking values ​​in a single sampling, leading to deviations in height determination. Summary of the Invention

[0004] The purpose of this invention is to provide a method for detecting rack beams for forklifts, forklifts, media, and computer program products, to solve the technical problems of insufficient beam detection accuracy, poor stability, and low real-time performance caused by mechanical vibration, field of view obstruction, inconsistent target features, and limited computing power in complex operating environments of unmanned forklifts.

[0005] The first embodiment of the present invention discloses a method for detecting rack beams in forklifts, comprising:

[0006] Acquire depth data collected by a depth camera and convert the depth data into a point cloud in a preset coordinate system;

[0007] Obtain the mast lifting height of the forklift, and extract the target point cloud from the point cloud based on the mast lifting height;

[0008] The target point cloud is orthogonally projected onto a vertical plane along the fork-in direction to generate a two-dimensional occupancy grid image;

[0009] The number of occupied grid cells in each row of the occupied grid image in the height direction is counted to generate a one-dimensional height-density histogram;

[0010] The height-density histogram is differentially calculated and traversed from bottom to top along the height direction. The first position that meets the preset negative peak condition is determined as the upper edge height value of the shelf beam to be tested.

[0011] The detection value is output based on the height of the upper edge of the beam of the shelf to be tested.

[0012] According to a first embodiment of the present invention, converting the depth data into a point cloud in a preset coordinate system includes:

[0013] Obtain the extrinsic calibration matrix of the depth camera relative to the forklift mast coordinate system;

[0014] The depth data is converted to the forklift mast coordinate system using the extrinsic calibration matrix.

[0015] The converted data is spatially overlaid to obtain a fused point cloud covering the area in front of the fork arm.

[0016] According to a first embodiment of the present invention, before generating a two-dimensional occupancy raster image, the method further includes:

[0017] The fused point cloud is subjected to voxel downsampling filtering to divide the three-dimensional space into regular voxel grids, and a representative point is retained in each voxel grid.

[0018] According to a first embodiment of the present invention, obtaining the mast lifting height of the forklift and extracting the target point cloud from the point cloud based on the mast lifting height includes:

[0019] Obtain the mast lifting height of the forklift and compare the mast lifting height with a preset height threshold.

[0020] If the lifting height of the gantry is less than the preset height threshold, the target point cloud is extracted from the point cloud of a single frame;

[0021] If the lifting height of the gantry is greater than the preset height threshold, the target point cloud is extracted from the point cloud of multiple consecutive frames.

[0022] According to a first embodiment of the present invention, extracting the target point cloud includes:

[0023] Determine the theoretical height of the beam of the shelf to be inspected;

[0024] The point cloud whose spatial coordinates fall within the first, second, and third coordinate intervals of the preset coordinate system is used as the target point cloud. The first coordinate interval is set along the forklift entry direction based on the relative position of the forklift and the crossbeam of the rack to be inspected. The second coordinate interval is set along the forklift width direction based on the preset pallet width. The third coordinate interval is set along the height direction based on the theoretical height position.

[0025] According to a first embodiment of the present invention, the method further includes: if the lifting height of the gantry is less than the preset height threshold,

[0026] Given the height value of the upper edge of the shelf beam to be tested, calculate the deviation between the height value of the upper edge of the shelf beam to be tested and the theoretical height position of the shelf beam to be tested, and determine whether the deviation value exceeds the preset error range.

[0027] If there is no upper edge height value of the crossbeam to be tested, or if the deviation value exceeds the preset error range, the method is re-executed.

[0028] According to a first embodiment of the present invention, if the lifting height of the gantry is greater than the preset height threshold, the step of outputting a detection value based on the upper edge height value of the crossbeam of the shelf to be detected includes:

[0029] Determine the variance of the upper edge height values ​​of the multiple beams of the shelf to be detected corresponding to the consecutive multiple frames;

[0030] If the variance is less than the preset jitter threshold, then after removing outliers from the upper edge height values ​​of multiple shelf beams to be tested, the mean is calculated and the mean is output as the detection value.

[0031] According to a first embodiment of the present invention, the method further includes:

[0032] If the variance is greater than or equal to the preset jitter threshold, the sampling time is extended and the method is re-executed;

[0033] If the method is re-executed for more than a preset duration or more than a preset number of times and the variance is still greater than or equal to the preset jitter threshold, an error message is output.

[0034] According to a first embodiment of the present invention, generating a two-dimensional occupancy raster image includes:

[0035] The target point cloud is mapped onto a two-dimensional plane formed by the width and height directions of the forklift according to a preset projection ratio to form a pixel grid;

[0036] If at least one point cloud data exists within the pixel raster, it is marked as occupied; otherwise, it is marked as unoccupied.

[0037] According to a first embodiment of the present invention, the differential calculation of the height-density histogram includes:

[0038] The height-density histogram is smoothed using a Gaussian smoothing algorithm;

[0039] First-order difference calculation is performed on the smoothed height-density histogram to obtain the difference data sequence.

[0040] According to a first embodiment of the present invention, the preset negative peak condition is to simultaneously satisfy the following conditions:

[0041] The value at the current position in the differential data sequence is negative;

[0042] The absolute value of the value at the current position is greater than the judgment threshold;

[0043] The determination threshold is determined by multiplying the maximum number of occupied grid cells in the height-density histogram with a preset scaling factor.

[0044] A second embodiment of the present invention discloses a forklift, the forklift including a depth camera and an electronic device arranged on the two side forks and mast, the electronic device including a memory storing computer-executable instructions and a processor, the electronic device causing the electronic device to implement the rack beam detection method for a forklift according to the first embodiment of the present invention when the instructions are executed by the processor.

[0045] A third embodiment of the present invention discloses a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform a rack beam detection method for a forklift according to a first embodiment of the present invention.

[0046] A fourth embodiment of the present invention discloses a computer program product including computer-executable instructions, which are executed by a processor to implement a rack beam detection method for a forklift according to a first embodiment of the present invention.

[0047] The main differences and effects of the embodiments of the present invention compared with the prior art are as follows:

[0048] In this invention, depth data collected by a depth camera is acquired and converted into a point cloud in a preset coordinate system. The lifting height of the forklift mast is obtained, and a target point cloud is extracted from the point cloud based on the mast lifting height. The target point cloud is orthogonally projected onto a vertical plane along the fork entry direction to generate a two-dimensional occupancy grid image. The number of occupancy grids in each row of the occupancy grid image in the height direction is counted to generate a one-dimensional height-density histogram. The height-density histogram is differentially calculated and traversed from bottom to top along the height direction. The first position that meets the preset negative peak condition is determined as the upper edge height value of the rack beam to be detected. Based on the upper edge height value of the rack beam to be detected, the detection value is output. By reducing the three-dimensional point cloud to a two-dimensional occupancy grid and performing histogram statistics, the amount of data processing is greatly reduced, enabling the algorithm to run in real time on embedded controllers with limited computing power. At the same time, the density-based statistical features do not depend on the color, texture, or specific three-dimensional shape of the beam, and have strong versatility and anti-interference ability, effectively addressing the problem of partial point cloud loss due to changes in lighting or arm occlusion. Attached Figure Description

[0049] Figure 1 A flowchart illustrating a rack beam detection method for a forklift according to an embodiment of this application is shown.

[0050] Figure 2 A schematic diagram illustrating a forklift operation scenario and a multi-depth camera layout according to an embodiment of this application is shown.

[0051] Figure 3 A schematic diagram of multi-depth camera field-of-view fusion and generated fused point cloud is shown according to an embodiment of this application.

[0052] Figure 4 This diagram illustrates the fluctuation of the detection height over time caused by gantry swaying during high-altitude operations according to an embodiment of this application.

[0053] Figure 5 A schematic diagram of a target point cloud after voxel downsampling filtering and region of interest (ROI) screening is shown according to an embodiment of this application.

[0054] Figure 6 A two-dimensional occupancy raster image is shown according to an embodiment of this application.

[0055] Figure 7 The original height-density histogram is shown according to an embodiment of this application.

[0056] Figure 8 A flowchart illustrating the "high-speed / anti-shake" adaptive detection mode according to an embodiment of this application is shown.

[0057] Figure 9A schematic diagram illustrating height-density histogram smoothing processing according to an embodiment of this application is shown.

[0058] Figure 10 The diagram illustrates the differential data curves and upper edge positioning schematic according to an embodiment of this application.

[0059] Figure 11 This is a hardware structure block diagram of an electronic device implementing an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0061] This invention provides a method and system for detecting rack beams and the ground based on multi-depth camera fusion and a forklift height-low position adaptive strategy. It aims to address issues faced by unmanned forklifts in automated warehousing and logistics, such as poor stability during high-level operations, blind spot occlusion, inconsistent target features, and limited computing resources. The method utilizes multi-depth cameras distributed on the forks and mast to eliminate blind spots, unifies the identification standards for beams of different specifications by detecting "upper edge" features, and achieves near-millisecond, anti-shake high-precision detection on an embedded platform using a height-low position hierarchical strategy.

[0062] The first embodiment of the present invention discloses a method for detecting rack beams in forklifts, such as... Figure 1 As shown, the method includes the following steps:

[0063] S101: Acquire depth data collected by the depth camera and convert the depth data into a point cloud in a preset coordinate system.

[0064] In this embodiment, environmental data is acquired using a depth camera array deployed on a forklift. For example... Figure 2 As shown, the forklift is facing the rack beam 205 on which goods 207 are placed. Goods 207 are supported on pallets 206, or the ground 204 needs to be inspected. To address blind spots, a multi-camera fusion solution is employed. For example... Figure 3 As shown, the collected depth data, after coordinate transformation, forms point cloud data covering the area in front of the forklift. The dashed cones represent the field of view of each depth camera, and the white dots on the gray background schematically represent the high-density point cloud distribution after multi-camera data fusion. This point cloud data reflects the three-dimensional spatial structure in front of the forklift, including information on obstacles such as beams, goods, and columns.

[0065] S102, obtain the mast lifting height of the forklift, and extract the target point cloud from the point cloud based on the mast lifting height.

[0066] The mast lifting height of a forklift is typically obtained from feedback via encoders or cable sensors on the vehicle chassis. The system uses this height to determine whether the current operation is at a low (ground) or high position, and thus employs different extraction strategies. For example... Figure 4 As shown, when working at a high position, the gantry will experience significant physical shaking, causing the measurement data to fluctuate over time. Figure 4 The horizontal axis represents time, and the vertical axis represents the detected beam height. The curve shows that at high positions, due to factors such as the elasticity of the mechanical structure and motion inertia, the detected height value will exhibit periodic oscillations (i.e., jitter) within a certain range. This indicates that the reliability of single-frame data is significantly reduced when operating at high positions. Therefore, at high positions, data needs to be extracted from multiple consecutive frame point clouds for anti-jitter processing, while at low positions, data can be processed quickly directly based on a single frame point cloud.

[0067] S103, orthogonally project the target point cloud along the fork-in direction onto the vertical plane to generate a two-dimensional occupancy raster image.

[0068] To reduce computational complexity and accommodate the computing power limitations of embedded platforms, this embodiment does not employ complex 3D shape fitting. Instead, it performs dimensionality reduction processing on the 3D point cloud after region of interest (ROI) filtering. Specifically, the 3D point cloud is orthogonally projected along the X-axis (forklift entry direction) onto the YOZ plane (a plane perpendicular to the ground or a vertical coordinate plane parallel to the gantry lifting plane). Figure 5 The voxel filtering and pass-through filtering processes for point clouds are shown. The boxed area is the region of interest (ROI) extracted according to the preset coordinate interval. The point cloud outlines of the shelf beams and the bottom of the pallets are clearly visible, and the background noise has been filtered out. Figure 6 The image shows a two-dimensional occupied raster image generated after projection. Black areas represent unoccupied raster pixels, and white areas represent occupied raster pixels. For example, the height range indicated by bracket 601 is primarily filled with white pixels, indicating... Figure 2 The image shows a crossbeam 205. Arrow 602 points to the upper edge of the crossbeam 205. The black pixel area 603 represents the forklift space under the pallet 206. In this image, each pixel represents the presence of a point cloud at a corresponding location in space, thus transforming three-dimensional spatial features into two-dimensional image features.

[0069] S104: Count the number of occupied grid cells in each row of the occupied grid image in the height direction, and generate a one-dimensional height-density histogram.

[0070] based on Figure 6 The system counts the number of grid cells marked "occupied" row by row along the Z-axis (height direction) in the two-dimensional occupancy raster image shown. The statistical results form the following... Figure 7The height-density histogram shown here represents the height coordinate (Z(m)) on the horizontal axis and the number of occupied grid cells on the vertical axis. This histogram visually reflects the density of the object's point cloud at different height levels. For example, the histogram exhibits a clear stepped distribution; within the beam height range (e.g., -0.1m to 0.0m), the point cloud density is higher due to the presence of the beam's front surface, resulting in a larger number of occupied grid cells; while in the gaps above or below the beam, the number of occupied grid cells decreases sharply.

[0071] S105, perform differential calculation on the height-density histogram and traverse it from bottom to top along the height direction, and determine the first position that meets the preset negative peak condition as the upper edge height value of the shelf beam to be tested.

[0072] like Figure 10 As shown, differential calculations on the smoothed histogram data yield information on the density changes of the occupied grid. This invention utilizes the physical characteristic of the "upper edge step," meaning that regardless of the beam's thickness or whether its lower half is occluded, its upper surface always represents the boundary between "nothing" and "something" (viewed from top to bottom) or "something" and "nothing" (viewed from bottom to top) in the point cloud. In this step, by finding significant negative peaks in the differential curve, the upper edge of the beam can be precisely located. This characteristic is geometrically invariant for beams of different specifications.

[0073] S106, outputs the detection value based on the height of the upper edge of the beam of the shelf to be detected.

[0074] The determined height of the upper edge is the reference height for the forklift to pick up and place goods. The system outputs this detected value to the forklift's control system to adjust the height of the forks for precise docking.

[0075] This embodiment significantly reduces the amount of data processing by reducing the 3D point cloud to a 2D occupancy grid and performing histogram statistics, enabling the algorithm to run in real time on embedded controllers with limited computing power. Furthermore, the density-based statistical features are independent of the beam's color, texture, or specific 3D shape (such as I-beams or welded beams), exhibiting strong versatility and anti-interference capabilities, effectively addressing the problem of partial point cloud loss due to changes in lighting or fork arm occlusion.

[0076] According to some embodiments of this application, converting depth data into a point cloud in a preset coordinate system includes: obtaining the extrinsic calibration matrix of the depth camera relative to the forklift mast coordinate system; using the extrinsic calibration matrix to convert the depth data to the forklift mast coordinate system; and spatially overlaying the converted data to obtain a fused point cloud covering the area in front of the forklift arm.

[0077] Specifically, such as Figure 2 and Figure 3 As shown, this embodiment employs a distributed multi-depth camera field-of-view fusion scheme using a "forklift front end + gantry" configuration. The sensor layout includes a first depth camera 201 mounted on the front end of the right forklift, a second depth camera 202 mounted on the front end of the left forklift, and a third depth camera 203 mounted in the middle of the gantry. All three cameras are forward-facing. Using a pre-calibrated extrinsic parameter matrix, the point clouds from the three perspectives are uniformly transformed into the "forklift gantry coordinate system".

[0078] The first and second depth cameras 201 and 202 at the front of the forklift address near-field blind spots and cargo obstruction issues because they are located in front of the cargo; while the third depth camera 203 on the gantry provides a global view. The data from these three cameras, when superimposed, results in... Figure 3 As shown in the gray area with scattered white dots, a high-density fused point cloud is formed. It can be seen that the complementary fields of view of the third depth camera 203 on the gantry, the first depth camera 201 and the second depth camera 202 at the front of the fork arm, cover a wide area above and below the crossbeam, effectively eliminating blind spots from a single viewpoint and ensuring that the crossbeam point cloud can be acquired under any load condition (including empty and fully loaded). This solves the technical problem that existing single-gantry cameras are easily obstructed by cargo.

[0079] According to some embodiments of this application, before generating a two-dimensional occupied raster image, the method further includes: performing voxel downsampling filtering on the fused point cloud, dividing the three-dimensional space into a regular voxel grid, and retaining a representative point in each voxel grid.

[0080] like Figure 5 As shown, the original point cloud data is massive and contains noise. This step downsamples the fused point cloud using a voxel grid filter. Voxel filtering divides the entire 3D space into tiny 3D cuboid grids, traverses each voxel, finds all points falling within that voxel, and calculates the centroid (center of gravity) or center point of these points, replacing all points within the original voxel. For example, the 3D space can be divided into tiny cubes of 2mm × 2mm × 2mm, retaining only one centroid (center of gravity) or center point within each voxel as a representative. This significantly reduces the amount of data for subsequent processing, keeping computation time in the millisecond range; it also makes the point cloud distribution more uniform, avoiding interference from inconsistent point cloud density caused by different camera distances in subsequent histogram statistics.

[0081] According to some embodiments of this application, S102 obtains the mast lifting height of the forklift, and extracts the target point cloud from the point cloud based on the mast lifting height, such as... Figure 8 As shown in the flowchart, it specifically includes:

[0082] S1021, Obtain the forklift mast lifting height.

[0083] S1022, compare the gantry lifting height with the preset height threshold, and determine whether the gantry lifting height is less than or greater than the preset height threshold.

[0084] S1023, If the gantry lifting height is less than the preset height threshold, then extract the target point cloud from the point cloud of a single frame.

[0085] S1024, If the gantry lifting height is greater than the preset height threshold, then extract the target point cloud from the point cloud of multiple consecutive frames.

[0086] This embodiment provides an "Extreme Speed / Anti-Shake" adaptive detection mode. Preset height threshold ( The setting can be adjusted according to the mechanical characteristics of the forklift, for example, 4 meters.

[0087] When the gantry is lifted to a certain height At this point, it is determined to be in low-level mode (branch A). In this mode, mechanical vibration is minimal, and the system prioritizes maximum efficiency, collecting only a single frame of data for processing to meet the demands of a fast operational cycle.

[0088] When the gantry is lifted to a certain height When the system detects a high-level mode (branch B), it is prone to physical swaying due to the rigidity of the mechanical structure and the inertia of starting and stopping. If only a single frame is detected, extreme swaying values ​​can easily be captured, leading to height deviations. Therefore, the system automatically switches logic, continuously collecting N frames (e.g., 30 frames) of data for comprehensive analysis.

[0089] like Figure 4 The data waveforms shown indicate that if the single-frame extraction strategy used in low-level scenarios is applied in high-level scenes (i.e., when the gantry is shaking violently), it is highly likely that extreme values ​​of peaks or troughs will be captured (fluctuations in detection height), resulting in a large deviation in the calculated upper edge height. Therefore, entering multi-frame smoothing mode by thresholding is a necessary means to ensure the accuracy of high-level detection.

[0090] According to some embodiments of this application, extracting the target point cloud includes: determining the theoretical height position of the rack beam to be inspected. Point clouds whose spatial coordinates fall within a first coordinate interval, a second coordinate interval, and a third coordinate interval of a preset coordinate system are retained as the target point cloud; wherein, the first coordinate interval is set along the forklift entry direction based on the relative position of the forklift and the rack beam to be inspected; the second coordinate interval is set along the forklift width direction based on a preset pallet width; and the third coordinate interval is set along the height direction based on the theoretical height position.

[0091] The steps in this embodiment are called pass-through filtering. Pass-through filtering is a selection method based on coordinate range. It defines a Region of Interest (ROI) and retains only points within the defined range, deleting points outside the range. For example... Figure 5 As shown, the purpose of pass-through filtering is to define the ROI and filter out background interference (such as goods behind, ground debris, shelf columns, etc.).

[0092] Reference Figure 2 and Figure 5 The specific ROI setting logic is as follows:

[0093] The first coordinate interval (X-axis of fork entry direction): defined based on the approximate position of the forklift relative to the crossbeam 205 at the identification point. For example, if the surface of the crossbeam 205 is approximately at X=-1.5m in the vehicle coordinate system, then the interval is set to [-1.7m, -1.3m] to ensure that the surface of the crossbeam 205 is within the region.

[0094] The second coordinate interval (width direction Y-axis): is defined according to the standard pallet 206 width. For example, if the pallet 206 width is 1.0m or 1.2m, then the interval is set to [-0.7m, +0.7m], which can basically cover the left and right range of the pallet when it is placed on the shelf.

[0095] The third coordinate interval (Z-axis in the height direction): is defined according to the preset symmetry tolerance interval.

[0096] The unmanned forklift obtains the current lifting height information of the mast through an encoder installed on the mast lifting mechanism. Due to the certain measurement accuracy error of the encoder, there may be a deviation between the height value fed back by the encoder and the actual lifting height of the mast.

[0097] The multiple cameras mounted on the unmanned forklift have completed extrinsic parameter calibration, thereby establishing a unified spatial coordinate system. Based on this calibration relationship, an imaginary plane (imaginary rack plane or imaginary ground plane) can be constructed that changes synchronously with the lifting movement of the mast. The imaginary plane moves synchronously with the lifting of the mast and serves as a reference surface for height detection.

[0098] Ideally, when the lifting height of the gantry fed back by the encoder is consistent with the actual lifting height, the actual shelving plane or the actual ground should coincide with the imaginary plane. At this time, the theoretical height of the beam is defined as 0m.

[0099] In practical applications, considering the influence of encoder measurement errors and mechanical transmission errors, the actual shelf plane may deviate from the hypothetical plane. Therefore, a symmetrical tolerance range can be set on the Z-axis in the height direction to limit the effective detection range. For example, the third coordinate range is defined as [-0.3m, 0.3m], that is, with the hypothetical plane as the zero height reference, the detection target is allowed to float up or down by 0.3m in the height direction relative to this reference plane.

[0100] By pruning the ROI as described above, such as Figure 5 As shown in the box, only the effective point cloud of beam 205 and its vicinity is retained, greatly reducing environmental interference. This logic also applies to ground detection, where the ground is treated as an infinitely wide beam 205 with extremely small thickness.

[0101] According to some embodiments of this application, the method further includes: if the mast lifting height is less than a preset height threshold, and if there is a height value of the upper edge of the shelf beam to be tested, calculating the deviation between the height value of the upper edge of the shelf beam to be tested and the theoretical height position of the shelf beam to be tested, and determining whether the deviation exceeds a preset error range. If there is no height value of the upper edge of the shelf beam to be tested, or if the deviation exceeds the preset error range, the method is re-executed.

[0102] This is the closed-loop verification logic under the aforementioned low-level mode (branch A). The system compares the height value detected by vision with the theoretical height fed back by the encoder. If the deviation is within the allowable range (e.g., ±1cm), the detection is considered valid and output. If the deviation exceeds the limit, or the algorithm fails to extract valid upper edge features (e.g., due to severe occlusion), the system will prevent false detections (e.g., falsely detecting the edge of the next layer beam or goods) and enter a retry loop until the maximum number of retries is reached or the detection is successful.

[0103] According to some embodiments of this application, if the lifting height of the gantry is greater than a preset height threshold, a detection value is output based on the upper edge height value of the crossbeam to be detected, including: determining the variance of the upper edge height values ​​of multiple crossbeams to be detected corresponding to multiple consecutive frames. If the variance is less than a preset jitter threshold, outliers in the upper edge height values ​​of multiple crossbeams to be detected are removed, and the mean is calculated and output as the detection value.

[0104] This is the timing jitter immunity logic in the aforementioned high-level mode (branch B). For example... Figure 4As shown, the relative height of the high-level beam oscillates over time. The system continuously calculates the detection results of N frames of data and calculates the variance of this set of data. If the variance converges (less than the jitter threshold), it indicates that the jitter amplitude is within an acceptable range. At this point, to further improve accuracy, the system removes outliers from the sequence and takes the mean of the remaining valid data as the final output. This method achieves a dynamic balance between "efficiency" and "stability".

[0105] by Figure 4 For example, the system collects a data sequence on the time axis (e.g., data between 0.5s and 1.0s). It calculates the variance of all detected values ​​within this time window. If the variance is high (e.g., in the part of the curve with violent oscillations), it indicates that the gantry has not yet stabilized. As time progresses (e.g., after 1.5s), the oscillation amplitude decreases, and the variance drops below a preset threshold. At this point, the system determines that the data has converged and takes the mean of the data within this stable interval as the final detection result, thus effectively filtering out errors caused by mechanical shaking.

[0106] According to some embodiments of this application, the method further includes: if the variance is greater than or equal to a preset jitter threshold, extending the sampling time and re-executing the method. If the re-executing of the method exceeds a preset duration or a preset number of times and the variance is still greater than or equal to the jitter threshold, an error message is output.

[0107] If the variance remains large, it is determined to be "swaying". The system will automatically discard the current data and extend the waiting time until the data converges. If convergence is not achieved after a long waiting period (timeout), there may be extreme situations such as sensor failure, mechanical structure abnormality, or earthquake. The system will terminate the detection and report an error to avoid operating under high-risk conditions, thereby solving the safety hazards caused by high-level swaying.

[0108] According to some embodiments of this application, generating a two-dimensional occupancy grid image includes: mapping a target point cloud onto a two-dimensional plane formed by the width and height directions of a forklift according to a preset projection ratio, thereby forming a pixel grid. If at least one point cloud data exists within the pixel grid, it is marked as occupied; otherwise, it is marked as unoccupied.

[0109] like Figure 6As shown, this is a 3D to 2D binarization process: traversing each point in the 3D point cloud, taking its coordinates in the two directions other than the fork direction, and projecting them to obtain a binarized projected image. The projection ratio needs to be matched with the aforementioned voxel size. For example, if the voxel size is 2mm × 2mm × 2mm, and the projection ratio is set to 1 pixel corresponding to 2mm, then theoretically one pixel grid corresponds to one voxel. If there is a data point in the grid, it is set to 1 (white); otherwise, it is set to 0 (black). This binarization process further highlights the contour features of the object, ignoring the texture and color interference of the object's surface.

[0110] like Figure 6 As shown, the generated image visually reflects the cross-sectional structure in front of the forklift. Compared to Figure 5 The discrete point cloud is represented by white pixels, which are more coherent and clearly depict the space occupied by physical objects such as shelf uprights, shelf beams 205, pallets 206, and goods 207, while black pixels mainly correspond to the space where air is located. This binarized image is particularly suitable for subsequent edge detection algorithms, eliminating the complexity of 3D data processing.

[0111] According to some embodiments of this application, differential calculation of a height-density histogram includes: smoothing the height-density histogram using a Gaussian smoothing algorithm; and performing first-order differential calculation on the smoothed height-density histogram to obtain a differential data sequence.

[0112] Original histogram ( Figure 7 There may be spikes due to missing point clouds or noise. First, perform one-dimensional Gaussian smoothing on it. Figure 9 The line chart shows the... Figure 7 The Gaussian smoothing of the original histogram shows that the jagged abrupt changes caused by uneven point cloud sampling are smoothed into a continuously varying curve. This preserves the main density variation trend (i.e., the main outline of the beam) while filtering out high-frequency noise. Subsequently, the first difference of this curve is calculated. Figure 10 As shown in the figure, this diagram illustrates... Figure 9 The first-order difference results of the curve. The horizontal axis represents height Z (m), and the vertical axis represents the difference in occupied grid cells, i.e., the density change of the occupied grid cells. In the figure, the difference curve shows sharp fluctuations at a certain height, and these fluctuations accurately reflect the abrupt changes in point cloud density.

[0113] According to some embodiments of this application, the preset negative peak condition is that the following conditions are met simultaneously: the value at the current position in the differential data sequence is negative; and the absolute value of the value at the current position is greater than a judgment threshold. The judgment threshold is determined by multiplying the maximum number of grid cells occupied in the height-density histogram by a preset scaling factor.

[0114] This embodiment provides a "beam top edge" localization algorithm. In the differential sequence, positive peaks represent a surge in point cloud density (beam bottom edge or cargo bottom), while negative peaks represent a sharp drop in point cloud density (beam top edge). The system iterates from bottom to top, searching for the first significant negative peak.

[0115] The definition of "significant" is constrained by a threshold. Let N be the maximum number of grid cells in the histogram (representing the densest point on the beam surface), and let the preset scaling factor be... ( , which is an adjustable empirical parameter, for example, a value of 0.3).

[0116] The judgment criteria are:

[0117]

[0118] in Indicates the difference value. This represents the height value.

[0119] Figure 10 The dashed line indicates the location of the negative peak that meets the conditions. This location corresponds to... Figure 2 The physical scene shown depicts the upper surface of beam 205. The coefficient σ ensures the robustness of the detection: a setting that is too small makes it susceptible to local point cloud defects, while a setting that is too large may result in missed detections. This algorithm does not depend on the overall geometric center of the beam, so even if the beam thickness is uneven or the lower half of the beam is occluded, it can be accurately identified as long as the upper edge is clear.

[0120] Combination Figure 2 , Figure 6 and Figure 10 Let's take a look. Figure 2 The upper surface of the crossbeam 205 in the middle corresponds to Figure 6 The upper boundary of the white pixel area indicated by square brackets 601, i.e., the height position indicated by arrow 602. Figure 10 In this context, this boundary is transformed into the unique and most significant negative extremum point (the dashed line). A preset scaling factor serves as a noise threshold, ensuring that only when the density decrease is sufficiently large (i.e., it is indeed a subtle textural change at the edge of the beam rather than on the surface of the cargo) will it be considered a valid edge.

[0121] Figure 11 This is a hardware structure block diagram of an electronic device implementing an embodiment of this application.

[0122] like Figure 11As shown, the electronic device 1000 may include one or more processors 1002, a system motherboard 1008 connected to at least one of the processors 1002, a system memory 1004 connected to the system motherboard 1008, a non-volatile memory (NVM) 1006 connected to the system motherboard 1008, and a network interface 1010 connected to the system motherboard 1008.

[0123] Processor 1002 may include one or more single-core or multi-core processors. Processor 1002 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments of the present invention, processor 1002 may be configured to perform one or more embodiments according to various embodiments of this application.

[0124] In some embodiments, the system motherboard 1008 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1002 and / or any suitable device or component communicating with the system motherboard 1008.

[0125] In some embodiments, the system motherboard 1008 may include one or more memory controllers to provide an interface to the system memory 1004. The system memory 1004 may be used to load and store data and / or instructions. In some embodiments, the system memory 1004 of the electronic device 1000 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0126] The NVM 1006 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the NVM 1006 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of an HDD (Hard Disk Drive), a CD (Compact Disc) drive, or a DVD (Digital Versatile Disc) drive.

[0127] NVM 1006 may include a portion of the storage resources on the device installed on electronic device 1000, or it may be accessible by the device, but is not necessarily part of the device. For example, NVM 1006 may be accessed over a network via network interface 1010.

[0128] Specifically, system memory 1004 and NVM 1006 may each include a temporary copy and a permanent copy of instruction 1020. Instruction 1020 may include instructions that, when executed by at least one of processors 1002, cause electronic device 1000 to perform methods as described in any embodiment of this application. In some embodiments, instruction 1020, hardware, firmware, and / or its software components may additionally / alternatively be located in system motherboard 1008, network interface 1010, and / or processor 1002.

[0129] Network interface 1010 may include a transceiver for providing a radio interface to electronic device 1000, thereby enabling communication with any other suitable device (e.g., front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 1010 may be integrated into other components of electronic device 1000. For example, network interface 1010 may be integrated into at least one of processor 1002, system memory 1004, NVM 1006, and firmware device (not shown) with instructions, wherein when at least one of processor 1002 executes the instructions, electronic device 1000 implements one or more embodiments of various embodiments of this application.

[0130] The network interface 1010 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1010 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0131] In one embodiment, at least one of the processors 1002 may be packaged together with one or more controllers for the system motherboard 1008 to form a system-in-package (SiP). In another embodiment, at least one of the processors 1002 may be integrated on the same die with one or more controllers for the system motherboard 1008 to form a system-on-a-chip (SoC).

[0132] The electronic device 1000 may further include an input / output (I / O) device 1012 connected to the system motherboard 1008. The I / O device 1012 may include a user interface enabling a user to interact with the electronic device 1000; the peripheral component interface is designed to allow peripheral components to also interact with the electronic device 1000. In some embodiments, the electronic device 1000 may also include sensors for determining at least one of environmental conditions and location information related to the electronic device 1000.

[0133] In some embodiments, the I / O device 1012 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash), and a keyboard.

[0134] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0135] In some embodiments, the sensor may include, but is not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 1010 to communicate with components of the positioning network, such as Global Positioning System (GPS) satellites.

[0136] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0137] Program code can be applied to input instructions to perform the functions described in this invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions including processor 1002 includes any system having a processor such as a digital signal processor (DSP), microcontroller, application-specific integrated circuit (ASIC), or microprocessor.

[0138] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this invention are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0139] One or more aspects of at least one embodiment can be implemented by instructions stored on a computer-readable storage medium, which, when read and executed by a processor, enable an electronic device to implement the methods of the embodiments described in this invention.

[0140] According to some embodiments of this application, a computer storage medium is disclosed, on which instructions are stored, which, when executed on a computer, cause the computer to perform a rack beam detection method for a forklift according to embodiments of this application.

[0141] The method embodiments of this application correspond to this embodiment, and this embodiment can be implemented in conjunction with the method embodiments of this application. The relevant technical details mentioned in the method embodiments of this application are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the method embodiments of this application.

[0142] According to some embodiments of this application, a computer program product is disclosed, including computer-executable instructions that are executed by a processor to implement a rack beam detection method for a forklift according to embodiments of this application.

[0143] The method embodiments of this application correspond to this embodiment, and this embodiment can be implemented in conjunction with the method embodiments of this application. The relevant technical details mentioned in the method embodiments of this application are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the method embodiments of this application.

[0144] It is understood that the specific embodiments described herein are merely for illustrative purposes and not for limiting the scope of this application. Furthermore, for ease of description, the accompanying drawings show only the parts relevant to this application, and not all of the structures or processes. It should be noted that similar reference numerals and letters in the drawings denote similar items throughout this application.

[0145] It should be understood that although the terms "first," "second," etc., may be used herein to describe various features, these features should not be limited by these terms. The use of these terms is merely for distinction and should not be construed as indicating or implying relative importance. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0146] In the description of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this embodiment based on the specific circumstances.

[0147] The illustrative embodiments of this application include, but are not limited to, a method for detecting rack beams for forklifts, forklifts, computer storage media, and computer program products.

[0148] Various aspects of the illustrative embodiments will be described using terminology commonly employed by those skilled in the art to convey the essence of their work to others skilled in the art. However, it will be apparent to those skilled in the art that some alternative embodiments will be practiced using the features partially described. Specific figures and configurations are set forth for purposes of explanation in order to provide a more thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without specific details. In some other instances, well-known features have been omitted or simplified herein to avoid obscuring the illustrative embodiments of this application.

[0149] Furthermore, the various operations will be described as multiple separate operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order of description, and many of these operations may be performed in parallel, concurrently, or simultaneously. Moreover, the order of the operations may also be rearranged. The process may be terminated when the described operations are completed, but may also include additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0150] References to "an embodiment," "embodiment," "illustrative embodiment," etc., in this application indicate that the described embodiment may include specific features, structures, or properties; however, each embodiment may or may not necessarily include specific features, structures, or properties. Furthermore, these phrases are not necessarily directed at the same embodiment. Moreover, when specific features are described in conjunction with specific embodiments, the knowledge of those skilled in the art can influence the combination of these features with other embodiments, whether or not those embodiments are explicitly described.

[0151] Unless the context otherwise specifies, the terms “comprising,” “having,” and “including” are synonyms. The phrase “A and / or B” means “(A), (B), or (A and B).”

[0152] As used herein, the term "module" may refer to, as part of, or include: a memory (shared, dedicated, or grouped), an application-specific integrated circuit (ASIC), electronic circuitry and / or a processor (shared, dedicated, or grouped), combinational logic circuitry, and / or other suitable components that provide the said functionality for running one or more software or firmware programs.

[0153] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Rather, in some embodiments, these features may be illustrated in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular drawing does not mean that all embodiments need to include such features; in some embodiments, these features may be omitted or may be combined with other features.

[0154] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions or programs carried or stored on one or more transient or non-transient machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors, etc. When the instructions or program are run by a machine, the machine may perform the various methods described above. For example, the instructions may be distributed via a network or other computer-readable media. Therefore, machine-readable media may include, but are not limited to, any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, such as floppy disks, optical disks, optical disc read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electronically erasable programmable read-only memories (EEPROMs), magnetic cards or optical cards, or flash memory or tangible machine-readable storage for transmitting network information via electrical, optical, acoustic, or other forms of signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media includes any form of machine-readable medium suitable for storing or transmitting electronic instructions or machine-readable (e.g., computer-readable) information.

[0155] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, the use of the technical solutions of this application is not limited to the various applications mentioned in the embodiments of this application. Various structures and modifications can be easily implemented with reference to the technical solutions of this application to achieve the various beneficial effects mentioned herein. Within the scope of knowledge possessed by those skilled in the art, all changes made without departing from the spirit of this application should be considered within the scope of this patent application.

Claims

1. A method for detecting rack beams in forklifts, characterized in that, include: Acquire depth data collected by a depth camera and convert the depth data into a point cloud in a preset coordinate system; Obtain the mast lifting height of the forklift, and extract the target point cloud from the point cloud based on the mast lifting height; The target point cloud is orthogonally projected onto a vertical plane along the fork-in direction to generate a two-dimensional occupancy grid image; The number of occupied grid cells in each row of the occupied grid image in the height direction is counted to generate a one-dimensional height-density histogram; The height-density histogram is differentially calculated and traversed from bottom to top along the height direction. The first position that meets the preset negative peak condition is determined as the upper edge height value of the shelf beam to be tested. The detection value is output based on the height of the upper edge of the beam of the shelf to be tested.

2. The method according to claim 1, characterized in that, The step of converting the depth data into a point cloud in a preset coordinate system includes: Obtain the extrinsic calibration matrix of the depth camera relative to the forklift mast coordinate system; The depth data is converted to the forklift mast coordinate system using the extrinsic calibration matrix. The converted data is spatially overlaid to obtain a fused point cloud covering the area in front of the fork arm.

3. The method according to claim 2, characterized in that, Before generating a two-dimensional occupancy raster image, the method further includes: The fused point cloud is subjected to voxel downsampling filtering to divide the three-dimensional space into a regular voxel grid, and a representative point is retained in each voxel grid.

4. The method according to claim 3, characterized in that, The process of obtaining the mast lifting height of the forklift and extracting the target point cloud from the point cloud based on the mast lifting height includes: Obtain the mast lifting height of the forklift and compare the mast lifting height with a preset height threshold. If the lifting height of the gantry is less than the preset height threshold, the target point cloud is extracted from the point cloud of a single frame; If the lifting height of the gantry is greater than the preset height threshold, the target point cloud is extracted from the point cloud of multiple consecutive frames.

5. The method according to claim 4, characterized in that, Extracting the target point cloud includes: Determine the theoretical height of the beam of the shelf to be inspected; The point cloud whose spatial coordinates fall within the first, second, and third coordinate intervals of the preset coordinate system is used as the target point cloud. The first coordinate interval is set along the forklift entry direction based on the relative position of the forklift and the crossbeam of the rack to be inspected. The second coordinate interval is set along the forklift width direction based on the preset pallet width. The third coordinate interval is set along the height direction based on the theoretical height position.

6. The method according to claim 4, characterized in that, The method further includes: if the lifting height of the gantry is less than the preset height threshold... Given the height value of the upper edge of the shelf beam to be tested, calculate the deviation between the height value of the upper edge of the shelf beam to be tested and the theoretical height position of the shelf beam to be tested, and determine whether the deviation value exceeds the preset error range. If there is no upper edge height value of the crossbeam to be tested, or if the deviation value exceeds the preset error range, the method is re-executed.

7. The method according to claim 4, characterized in that, If the lifting height of the mast is greater than the preset height threshold, the detection value is output based on the upper edge height value of the beam to be detected, including: Determine the variance of the upper edge height values ​​of the multiple beams of the shelf to be detected corresponding to the consecutive multiple frames; If the variance is less than the preset jitter threshold, then after removing outliers from the upper edge height values ​​of multiple shelf beams to be tested, the mean is calculated and the mean is output as the detection value.

8. The method according to claim 7, characterized in that, The method further includes: If the variance is greater than or equal to the preset jitter threshold, the sampling time is extended and the method is re-executed; If the method is re-executed for more than a preset duration or more than a preset number of times and the variance is still greater than or equal to the preset jitter threshold, an error message is output.

9. The method according to any one of claims 1-8, characterized in that, The generation of the two-dimensional occupancy raster image includes: The target point cloud is mapped onto a two-dimensional plane formed by the width and height directions of the forklift according to a preset projection ratio to form a pixel grid; If at least one point cloud data exists within the pixel raster, it is marked as occupied; otherwise, it is marked as unoccupied.

10. The method according to claim 9, characterized in that, The differential calculation of the height-density histogram includes: The height-density histogram is smoothed using a Gaussian smoothing algorithm; First-order difference calculation is performed on the smoothed height-density histogram to obtain the difference data sequence.

11. The method according to claim 10, characterized in that, The preset negative peak value condition is to simultaneously satisfy the following conditions: The value at the current position in the differential data sequence is negative; The absolute value of the value at the current position is greater than the judgment threshold; The determination threshold is determined by multiplying the maximum number of occupied grid cells in the height-density histogram with a preset scaling factor.

12. A forklift, characterized in that, The forklift includes depth cameras and electronic devices arranged on the side forks and mast, the electronic devices including a memory storing computer-executable instructions and a processor, which, when executed by the processor, causes the electronic devices to perform the method according to any one of claims 1-11.

13. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method according to any one of claims 1-11.

14. A computer program product, characterized in that, It includes computer-executable instructions that are executed by a processor to implement the method according to any one of claims 1-11.

Images