Forklift operation method and device based on intelligent recognition of forklift point

Abstract

The application discloses a forklift operation method and device based on forklift point intelligent identification, relates to the technical field of image recognition, and comprises the following steps: acquiring multi-modal environment data and forklift operation state information of an operation area, identifying a restricted space obstacle in a bottom operation space of a conveying line based on the multi-modal environment data, generating three-dimensional information of the obstacle containing an obstacle height occupation state, forming an operation area map containing height constraint information, determining an optimal driving path in combination with a real-time pose of the forklift, a pallet fork height and a passing height range, and controlling the forklift to perform a fork picking operation on a device to be replaced. The application can simultaneously consider plane obstacle avoidance and height passing constraint in a low, narrow and obstacle-dense bottom space of the conveying line, improve the coordination of pallet hole positioning, path planning and fork picking control, and reduce the risk of collision and equipment damage.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a forklift operation method and apparatus based on intelligent forklift location recognition. Background Technology

[0002] In the automobile assembly workshop, the overhead conveyor line adopts an elevated steel structure design. The core power components, such as the drive motor and reducer, are installed below the bottom support of the conveyor line, typically 0.5-1.2 meters above the ground. The working area is less than 2 meters wide laterally and up to 5 meters deep longitudinally. The surrounding area is densely covered with fixed obstacles such as conveyor line columns, cable trays, pneumatic pipelines, and sensor junction boxes. The forklift position specifically refers to the pre-fabricated standard fork insertion holes at the bottom of the equipment to be replaced, which is the only benchmark for the forklift to accurately pick up and install the equipment.

[0003] Currently, the replacement of equipment at the bottom of conveyor lines generally relies on manually driven forklifts. Due to the low and narrow working space, the forklift driver's view is easily obstructed by the conveyor support, making it difficult to fully observe surrounding obstacles. This may lead to collisions, resulting in conveyor structural deformation, electrical cable breakage, or equipment damage. In intelligent path planning, the obstacle avoidance algorithms of automated forklifts are mostly designed for open workshop environments. They have insufficient accuracy in recognizing low obstacles less than 1.5 meters high and suspended pipelines less than 50 mm in diameter. Furthermore, they do not take into account the height limitations of the space at the bottom of the conveyor line when planning paths, which cannot support the safe conduct of intelligent maintenance operations. Summary of the Invention

[0004] This invention provides a forklift operation method and device based on intelligent forklift location recognition, which provides a forklift operation control scheme suitable for low and narrow spaces at the bottom of a conveyor line, enabling the forklift to complete the recognition of obstacles in the confined space, the construction of a height constraint map, the positioning of the target forklift location, and path control before performing the picking operation of the equipment to be replaced.

[0005] In a first aspect, the present invention provides a forklift operation method based on intelligent forklift location recognition. This method can be executed by the main control device of the forklift operation device. In this method, the main control device acquires multimodal environmental data and forklift operation status information of the operation area where the forklift is located. Based on the multimodal environmental data, it identifies confined space obstacles in the operation area and generates three-dimensional obstacle information containing the space occupancy status of the obstacles. The confined space obstacles are used to characterize obstacles that affect the passage height or passage gap of the operation space at the bottom of the conveyor line. Based on the obstacle height occupancy status in the three-dimensional obstacle information, it generates an operation area map containing height constraint information. Based on the forklift operation status information and the operation area map containing height constraint information, it determines the optimal travel path of the forklift and controls the forklift to perform the picking operation of the equipment to be replaced based on the optimal travel path.

[0006] In the above method, the main control device does not only determine the two-dimensional distance between the forklift and the obstacle in the planar grid, but also establishes a correspondence between the occupancy of the obstacle in different height ranges and the current posture of the forklift, the current height of the forks and the height range of the vehicle body. In this way, it can screen candidate driving areas that meet the height passage conditions in the area where the bottom support of the conveyor line, the foundation of low-level equipment, the suspended pipeline and the cable tray coexist, and form the optimal driving path corresponding to the target forklift position in the candidate driving area.

[0007] Further preferably, the confined space obstacles include low-position candidate obstacles and suspended candidate obstacles corresponding to the working space at the bottom of the conveyor line. The multimodal environmental data includes LiDAR point clouds, corrected depth images, and a forklift real-time pose matrix. The main control device determines a low-position recognition region and a near-end supplementary recognition region corresponding to the working space at the bottom of the conveyor line based on the LiDAR point clouds and corrected depth images. Low-position candidate obstacles are identified in the low-position recognition region, and suspended candidate obstacles are identified in the near-end supplementary recognition region. The low-position candidate obstacles and the suspended candidate obstacles are fused to generate the three-dimensional information of the obstacles.

[0008] Through this design, LiDAR point clouds can be used to provide the three-dimensional spatial location of pillars, bases, equipment edges, and fixed structures in the work area. Depth images can be used to supplement the identification of near-end structures such as small cables, pipelines, and junction boxes, enabling low-lying obstacles and suspended obstacles to be merged in the same work coordinate system, reducing missed detections caused by a single sensing method in low-light, occluded, or small structure scenarios.

[0009] More preferably, when identifying low-position candidate obstacles within the low-position identification area, the main control device extracts point cloud data corresponding to the low-position identification area from the lidar point cloud, removes points corresponding to the ground area and the far-end background area based on the height distribution, density distribution and neighborhood continuity of the points in the point cloud data, and aggregates the remaining points to obtain the low-position candidate obstacles.

[0010] With this design, the low-position recognition area can be set around the actual passage range of the forklift body and forks at the bottom of the conveyor line. The main control equipment can remove ground reflection points, far-end wall background points and sparse outliers from the obstacle candidates, and then form low-position candidate obstacles according to the neighborhood connectivity relationship, so that structures such as low equipment bases, bracket feet and column bases can be stably recognized.

[0011] More preferably, when identifying suspended candidate obstacles in the near-end supplementary identification area, the main control device extracts point cloud projection information corresponding to the near-end supplementary identification area from the lidar point cloud, identifies image objects with suspended distribution characteristics and slender contour characteristics in the depth image, and spatially correlates the image objects with the point cloud projection information to obtain the suspended candidate obstacles.

[0012] Through this design, the near-end supplementary identification area can cover the local space between the front end of the fork and the bottom of the equipment to be replaced. Targets with slender outlines, local depth abrupt changes, and discontinuities with the ground in the depth image can be identified as suspended candidate obstacles. Then, the three-dimensional height and spatial range are supplemented by point cloud projection information, so that the edges of suspended pipelines and junction boxes can be included in the subsequent height constraint map.

[0013] More preferably, when fusing the low-position candidate obstacles and the suspended candidate obstacles, the main control device merges the repeatedly identified obstacle candidates based on the spatial correspondence between the two in the working area, determines the obstacle position and obstacle height occupancy status of the corresponding obstacle based on the spatial range of the merged obstacle candidates, determines the obstacle category of the corresponding obstacle based on the structural features of the merged obstacle candidates, and associates the obstacle position, the obstacle category, and the obstacle height occupancy status as the obstacle's three-dimensional information.

[0014] With this design, when the same cable tray or the same pipeline is repeatedly identified by LiDAR and depth camera, it can be merged into a single obstacle object. The obstacle location is used to characterize its coordinates in the work area map, the obstacle category is used to distinguish between columns, pipelines, cable trays, junction boxes or equipment housings, and the obstacle height occupancy status is used to characterize whether the corresponding obstacle occupies passage space in one or more height layers.

[0015] More preferably, when the main control device determines the optimal travel path of the forklift, it determines a passable height area from the work area map containing height constraint information based on the current height of the forklift's forks and the forklift's passing height range. Based on the grid occupancy status corresponding to the passable height area, it determines candidate travel areas that meet the height passage conditions. Based on the spatial position of the effective fork holes of the device to be replaced, it generates target forklift location information. Taking the real-time pose of the forklift as the path start point and the target position in the pre-determined target forklift location information as the path end point, it determines the optimal travel path within the candidate travel area.

[0016] With this design, the path start point comes from the real-time pose of the forklift, the path end point comes from the effective fork hole of the equipment to be replaced, and the path search area comes from the height constraint map. All three are kept consistent in the same working coordinate system, so that the path planning result can not only satisfy the passage of the forklift body, but also allow the forks to approach the equipment to be replaced in an executable posture.

[0017] More preferably, during the process of controlling the forklift to travel along the optimal travel path, the main control device updates the work area map containing height constraint information based on the updated obstacle 3D information, and when the updated work area map containing height constraint information indicates that there is a collision risk on the optimal travel path, the optimal travel path is re-determined.

[0018] With this design, when operators, mobile tool vehicles, or temporary pipelines enter the bottom passage of the conveyor line, the main control equipment can write the new obstacles into the work area map and determine whether the current path still meets the height passage conditions and safe distance conditions. When the conditions are not met, replanning is triggered, thereby enabling the forklift to have dynamic obstacle avoidance capabilities during operation.

[0019] Secondly, the present invention provides a forklift operation device based on intelligent forklift location recognition. The device includes a data acquisition module, a perception and recognition module, a map generation module, and a path control module. The data acquisition module is used to acquire multimodal environmental data of the work area where the forklift is located and forklift operation status information. The perception and recognition module is used to identify confined space obstacles in the work area based on the multimodal environmental data and generate three-dimensional obstacle information including the obstacle space occupancy status. The map generation module is used to generate a work area map including height constraint information based on the obstacle height occupancy status in the obstacle three-dimensional information. The path control module is used to determine the optimal driving path of the forklift based on the forklift operation status information and the work area map including height constraint information, and control the forklift to perform the forking operation of the equipment to be replaced based on the optimal driving path.

[0020] The technical effects that can be achieved in the second aspect described above can be referred to the description of the technical effects that can be achieved by the corresponding design scheme in the first aspect described above, and will not be repeated here.

[0021] The technical effects and advantages provided by the present invention in the above technical solution are as follows: Compared to manually driving forklifts into the bottom area of ​​the conveyor line for equipment replacement, this application integrates confined space obstacle identification, obstacle height occupancy status, and height constraint map into the same control process. This allows the main control equipment to not only determine whether the forklift can bypass obstacles in a planar direction, but also to filter passable areas by combining fork height, vehicle body clearance range, and obstacle occupancy at different height levels. Therefore, in scenarios where low obstacles, suspended pipelines, and support structures coexist, the path planning process can reduce misjudgments caused by relying solely on two-dimensional planar obstacle avoidance, thereby lowering the probability of collisions between the forklift body or forks and the bottom structure of the conveyor line. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0023] Figure 1 This is a schematic diagram of a forklift operation scenario at the bottom of a conveyor line, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of a forklift operation device system architecture based on intelligent forklift location recognition, provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of a forklift operation method based on intelligent forklift location recognition, provided by an embodiment of the present invention. Figure 4 A schematic diagram of coordinate transformation provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of obstacle zone recognition in a confined space provided by an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the fusion of low-lying obstacles and suspended obstacles to generate three-dimensional obstacle information, as provided in an embodiment of the present invention. Figure 7 A schematic diagram of a layered work area map containing height constraint information is provided for an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating an effective forkhole recognition and target forklift location generation method provided in an embodiment of the present invention. Figure 9 This is a schematic diagram of highly constrained path planning and dynamic replanning provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the functional module structure of a forklift operating device provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the objectives, technical solutions, and implementation effects of this invention clearer, the embodiments of this invention are described in detail below. The described embodiments are used to illustrate the implementation of this invention and are not intended to limit the scope of protection of this invention. Where there is no conflict, the technical features in the various embodiments of this invention can be combined or substituted with each other.

[0026] like Figure 2 As shown, in this embodiment of the invention, the main control device is located at the center of the forklift operation device system architecture. Its input side includes LiDAR point cloud data, depth camera depth images, real-time pose formed by IMU and wheel encoder, fork height and pressure collected by fork sensors, and equipment identification in maintenance work orders. Its internal processing side includes at least data acquisition, perception and recognition, map generation, and path control. Data acquisition is used to collect and synchronize sensor inputs, perception and recognition is used to obtain information on obstacles and effective fork holes in confined spaces, map generation is used to form a work area map containing height constraints, and path control is used to output the optimal travel path and actuator control strategy. Its output side is connected to the travel drive and steering drive, fork lifting and fork tilting, display warning and human-machine interaction, and CMMS / WMS data interaction, respectively, so that task data, environmental perception results, path planning results, and forklift actuators form a closed-loop control.

[0027] In this embodiment of the invention, the working area can be the equipment replacement area at the bottom of the conveyor line in an automobile assembly workshop, such as... Figure 1 As shown in the image, this area typically contains structures such as conveyor pillars, cable trays, pneumatic pipelines, sensor junction boxes, equipment bases, support legs, and temporary tool carts. Some of these structures are close to the ground, while others are suspended near the passage height of the forks or the forklift body. The main control equipment treats these structures that obstruct the passage of the forklift body, forks, or equipment to be replaced as confined space obstacles.

[0028] The multimodal environmental data in this embodiment of the invention may include LiDAR point clouds, depth images, real-time forklift pose data, and external work order data. The LiDAR can be installed on the top or front of the forklift cab to collect a large area of ​​three-dimensional point clouds at the bottom of the conveyor line; the depth camera can be installed in the middle of the fork front end to supplement the collection of depth images of the near-field area in front of the forks; the inertial measurement unit and wheel encoder can be used to obtain the real-time pose of the forklift in the work area; and the work order data can provide equipment identification and maintenance task information for the equipment to be replaced.

[0029] Example 1 Please see Figure 3 As shown, this embodiment provides a forklift operation method based on intelligent forklift location recognition, which can be executed by a main control device. The main control device first acquires multimodal environmental data of the work area where the forklift is located and forklift operation status information; then, it identifies obstacles in confined spaces based on the multimodal environmental data and generates 3D obstacle information; next, it generates a work area map containing height constraint information based on the 3D obstacle information; subsequently, it determines the optimal travel path by combining the forklift operation status information and the work area map; finally, it controls the forklift to perform the forking operation of the equipment to be replaced according to the optimal travel path.

[0030] S101, the main control device acquires multimodal environmental data of the work area where the forklift is located and forklift operation status information.

[0031] In this step, the main control device can receive raw point clouds from the LiDAR, raw images from the depth camera, attitude data from the inertial measurement unit, odometer data from the wheel encoder, and fork status data according to a preset sampling period. Fork status data can include current fork height, fork tilt angle, fork root pressure, and fork insertion status. Forklift operation status information can include real-time forklift position, real-time forklift heading, vehicle body dimensions, current fork height, vehicle body clearance height range, fork width, fork length, and current motion state.

[0032] When the main control equipment preprocesses the raw point cloud from the lidar, it can first retain valid point clouds based on the height and distance range of the work area. For example, in a scenario operating at the bottom of a conveyor line, the point cloud between the ground and the bottom support of the conveyor line, as well as the point cloud within the working distance in front of the forklift, can be prioritized for retention, while removing point clouds that are significantly beyond the work space. Subsequently, the main control equipment removes outliers based on the distance relationship between point clouds and neighboring points. Specifically, when the average distance between a point and several surrounding neighboring points is significantly greater than the average distance level of the normal point cloud in that area, that point can be identified as an outlier and deleted from the point cloud.

[0033] like Figure 4As shown, when the main control device performs coordinate transformation, it uses the LiDAR coordinate system, depth camera coordinate system, forklift coordinate system, and work area coordinate system as the transformation objects. First, it establishes the spatial correspondence between the LiDAR coordinate system and the depth camera coordinate system through calibration transformation. Then, it uses time alignment to ensure that the depth image and the forklift's real-time pose matrix are at the same acquisition time. Subsequently, it uses coordinate fusion to map the real-time pose in the forklift coordinate system to the work area coordinate system. After the above processing, the point cloud data, depth image, and forklift's real-time pose matrix are unified into the same work coordinate reference, which is used for subsequent obstacle recognition, forkhole localization, and path planning.

[0034] When the main control device acquires the real-time pose of the forklift, it can fuse the attitude change output by the inertial measurement unit (IMU) with the mileage output by the wheel encoder. Specifically, the wheel encoder provides the displacement changes of the forklift on the ground, while the IMU provides the forklift's heading and attitude changes. The main control device aligns these two data points according to timestamps to form the forklift's real-time pose. When slippage or jumps are detected in the wheel encoder, the main control device can reduce the reliability of the corresponding mileage data and make corrections using the IMU and the continuity of the pose over time.

[0035] S102, the main control equipment identifies obstacles in confined spaces within the work area based on multimodal environmental data, and generates three-dimensional information of the obstacles, including their space occupancy status.

[0036] like Figure 5 As shown, in this step, the main control equipment can first determine the low-level identification area and the near-end supplementary identification area based on the forklift's current position, vehicle outline, fork position, and the working range at the bottom of the conveyor line. The low-level identification area is used to identify obstacles close to the ground or below the vehicle's clearance height, such as the bottom of the conveyor line columns, equipment bases, support feet, and ground protrusions. The near-end supplementary identification area is used to identify small obstacles near the fork tips and in the suspended area at the bottom of the conveyor line, such as thin cables, pneumatic lines, junction box edges, and pipe fasteners.

[0037] When the main control device identifies low-level candidate obstacles within the low-level recognition area, it can extract the corresponding point cloud data from the LiDAR point cloud and filter them based on the height distribution, density distribution, and neighborhood continuity of the points. Points located on the ground plane with minimal height variation can be identified as ground areas and eliminated; points located in the distant background and not intersecting with the forklift passage space can be identified as background areas and eliminated; the remaining points, if spatially continuous and with stable outlines, can be aggregated into low-level candidate obstacles.

[0038] Aggregation of low-level candidate obstacles can be performed based on spatial proximity. The master control device can group points that are close together and have continuous height into the same obstacle candidate, and points that are far apart or have discontinuous height into different obstacle candidates. For long supports, cable trays, or pipe supports, the master control device can preserve their elongated outline and height range during aggregation, avoiding the misclassification of the same structure into multiple discontinuous obstacles.

[0039] When the main control device identifies suspended candidate obstacles within the near-end supplementary identification area, it can utilize the corrected depth image to identify image objects with suspended distribution characteristics and slender contour features. Suspended distribution characteristics indicate that the object is discontinuous with the ground and its depth position is near the fork front or the passable space above the vehicle body; slender contour features indicate that the object is linear, strip-shaped, or tubular. The main control device can determine image objects satisfying these characteristics as suspended candidate objects and then obtain their three-dimensional position and height occupancy range through point cloud projection information.

[0040] When identifying the edges of small pipes, cables, and junction boxes, a single LiDAR may produce discontinuous points due to sparse point clouds, and a single depth image may show unstable edges due to reflections, shadows, or low texture. The master control device can map discrete 3D points in the point cloud to continuous contours in the depth image, and when the two have a consistent relationship in spatial position and orientation, they are treated as the same suspended candidate obstacle.

[0041] The fusion processing takes the location information, planar range, lower height limit, and upper height limit of low-lying candidate obstacles as one type of input, and the image contour and point cloud projection range of suspended candidate obstacles as another type of input. When determining spatial correspondence, the main control device comprehensively compares the projection position, height range, and category features of candidate objects. If they are determined to be the same entity, duplicate candidates are merged, and only one obstacle record is retained. If they are determined to be different entities, they are retained separately. The final output of obstacle 3D information includes obstacle location, obstacle category, height occupancy status, and recognition confidence, for subsequent writing into the height-constrained map, specifically as follows: Figure 6 As shown.

[0042] After fusion, the main control device generates 3D obstacle information for each obstacle object. This 3D information can include obstacle location, obstacle category, planar occupancy range, lower height limit, upper height limit, height occupancy status, and recognition confidence level. Obstacle categories can include columns, equipment bases, cable trays, pneumatic pipelines, sensor junction boxes, temporary tool carts, personnel, or other unknown obstacles. The height occupancy status indicates whether the obstacle occupies the passage space of a forklift or forks within one or more height levels.

[0043] S103, the main control device generates a work area map containing height constraint information based on the obstacle height occupancy status in the obstacle 3D information.

[0044] like Figure 7 As shown, in this step, the work area map can be divided into height layers H1, H2, and H3. Height layer H1 represents the near-ground space, height layer H2 represents the forklift passage space, and height layer H3 represents the vehicle body or mast passage space. Each height layer includes occupied grids and passable grids. After integrating multiple height layers, the main control device obtains the passable height range of each planar grid and marks areas near obstacles or within the safety margin as safety buffer zones or constraint areas. Candidate driving areas need to be unoccupied in all height layers involved in the forklift passage and meet the safety buffer distance requirements.

[0045] When the main control equipment writes the 3D information of obstacles into the work area map, it can determine the grid cells covered by the obstacle based on the plane occupancy range of the obstacle, and then determine the height layer covered by the obstacle based on the lower and upper height limits of the obstacle. If an obstacle spatially overlaps with a grid cell within a certain height layer, the corresponding grid cell in that height layer is marked as occupied; if the location is not occupied by an obstacle in that height layer, it remains either passable or unknown.

[0046] To improve traffic safety, the main control equipment can perform safety expansion on occupied grids. Safety expansion refers to adding a safety buffer zone outside the actual occupied area of ​​the obstacle, corresponding to the forklift positioning error, vehicle body margin, and fork swing margin. After safety expansion, grids near the obstacle edge can also be marked as impassable or low-confidence passable areas to reduce the risk of forklifts traveling close to pipelines, supports, or equipment edges.

[0047] The height constraint information in the work area map can include the grid occupancy status of each height level, the passable height range of each grid, the safe distance between each grid and the nearest obstacle, and the passability confidence of each grid. Using this height constraint information, the main control equipment can determine whether the same horizontal location is passable under different forklift heights or different vehicle body heights.

[0048] S104, the main control device determines the optimal travel path of the forklift based on the forklift's operating status information and the operating area map containing height constraint information.

[0049] In this step, the main control equipment first determines the passable height area based on the forklift's current fork height, the vehicle's clearance height range, and the action to be performed. The vehicle's clearance height range can be determined by the vehicle's upper height, mast height, fork height, and safety margin. When the forklift approaches the equipment to be replaced unloaded, the main control equipment can focus on determining the required clearance height for the vehicle's outer profile and the fork tips; when the forklift needs to insert into the fork holes or lift the equipment, the main control equipment can also determine whether the fork insertion height and the space under the equipment meet the clearance requirements.

[0050] After determining the passable height area, the main control equipment can comprehensively judge the occupancy status of the grids in the corresponding height layers. If a certain planar grid is occupied by an obstacle in any height layer required for the forklift to pass through, or if its safety buffer zone overlaps with the forklift's outline, then the planar grid is determined to be impassable; if the planar grid is not occupied in all necessary height layers and sufficient safety distance is maintained, then the planar grid is determined to be a candidate driving area.

[0051] When the main control equipment generates target forklift location information, it can identify the valid fork holes on the bottom of the equipment to be replaced. The equipment to be replaced can be a conveyor drive motor, reducer, or other equipment with prefabricated fork insertion holes. The main control equipment can read the standard 3D model of the equipment to be replaced based on the equipment identifier in the maintenance work order, and extract the number of prefabricated fork holes, fork hole size, fork hole center spacing, fork hole offset relative to the geometric center of the equipment, and fork hole depth from the standard 3D model.

[0052] When a standard fork hole template is configured, the main control device can extract the bottom plane region of the device to be replaced based on the LiDAR point cloud, and detect candidate fork holes within this bottom plane region. The bottom plane region can be determined through plane fitting and comparison with the standard device contour. Specifically, the main control device can find a plane point cloud that matches the bottom of the device within a preset height range, and register this plane point cloud with the standard device model; when the registration error is lower than a preset allowable range, this plane is determined as the bottom plane of the device to be replaced, and point cloud and image regions for fork hole detection are reserved in its vicinity.

[0053] Candidate fork hole detection can be performed by combining point cloud edges and depth image edges. The main control device can extract edge points from the point cloud at the bottom of the device, and combine edges whose length, direction, and perpendicularity relationships satisfy the fork hole contour features to form point cloud candidate fork holes. The main control device can also determine the region of interest at the bottom of the device in the depth image, and obtain image candidate fork holes through edge detection and rectangular contour filtering. If the point cloud candidate fork holes and the image candidate fork holes have a high degree of overlap in spatial position and contour range, the two are merged into the final candidate fork holes.

[0054] When the main control device performs template similarity matching on candidate fork holes, it does not need to use complex formulas. It can judge based on the degree of consistency between the local image of the candidate fork hole and the standard fork hole template in terms of contour shape, opening ratio, edge direction, and size. When the consistency between the candidate fork hole and the standard fork hole template reaches a preset threshold, the main control device considers it a fork hole to be verified. Subsequently, the main control device continues to verify the geometric relationships between the fork holes to be verified, such as whether the number of valid fork holes is consistent with the number of standard fork holes, whether the center distance between two fork holes is within the allowable deviation range, and whether the offset of the two fork holes relative to the geometric center of the device is consistent with the standard model. After template matching and geometric relationship verification, the fork holes to be verified are determined as valid fork holes.

[0055] Without a standard fork hole template, the main control device can determine valid fork holes based on the outline integrity, depth abrupt change features, and paired spatial relationships of candidate fork holes. Outline integrity indicates whether a candidate fork hole has a continuous open edge; depth abrupt change features indicate whether there is a significant inward depth change in the fork hole area relative to the bottom plane of the device; paired spatial relationships indicate whether two candidate fork holes are paired and distributed on the left and right sides of the bottom of the device and have similar dimensions.

[0056] The target forklift location is not directly determined by the center points of the two fork openings. Instead, candidate fork openings are first obtained based on point cloud edges and image edges in the bottom plane area of ​​the equipment to be replaced. Then, template similarity matching is performed on the candidate fork openings. The matching content includes the fork opening outline, size scale, and opening ratio. Further geometric relationship verification is performed, including the number of fork openings, the center-to-center distance between fork openings, and the offset relative to the geometric center of the equipment. Figure 8 As shown; after the candidate forkholes pass the above verification, the main control device determines them as the first effective forkhole and the second effective forkhole. Then, it determines the direction of the forkhole center line and the fork-taking posture angle based on the direction of the center line connecting the two effective forkholes. It determines the fork-taking center point and target position based on the midpoint of the center line and the fork insertion depth based on the effective forkhole depth. Finally, it outputs the target forklift position information containing the target position, fork-taking posture angle and fork insertion depth.

[0057] When determining the optimal travel path within a candidate travel area, the main control device can use the forklift's real-time pose as the path starting point and the target position in the target forklift's location information as the path ending point. The path search process can comprehensively consider travel distance, number of turns, risk of approaching obstacles, difficulty of fork posture adjustment, and height clearance constraints. If the grid through which the candidate path passes is close to overhead pipelines or low-level supports, the risk cost of the candidate path increases; if the candidate path requires frequent sharp turns or frequent fork height adjustments, the execution cost of the candidate path increases. The main control device prioritizes the path with the shortest distance, smoothest steering, safe distance from obstacles, and satisfactory height clearance conditions as the optimal travel path.

[0058] The main control unit can smooth the optimal travel path, converting discrete path points into a continuous travel trajectory that the forklift can execute. Smoothing can include removing unnecessary turnaround points, limiting angle changes between adjacent path segments, and ensuring continuous fork height adjustment within permissible speed ranges. Through smoothing, the path planning results are better suited to the actual motion control of the forklift entering the conveyor belt at low speeds.

[0059] S105, the main control equipment controls the forklift to perform the picking operation of the equipment to be replaced based on the optimal travel path.

[0060] In this step, the main control device can decompose the optimal travel path into multiple executable control segments. Each control segment can include the target position, target heading, suggested travel speed, target fork height, and safety distance requirements. The main control device sends control commands to the motion controller, which then controls the travel drive motor, steering drive motor, fork lifting hydraulic cylinder, and fork tilting hydraulic cylinder to move the forklift along the planned trajectory.

[0061] During forklift operation, the main control unit continuously collects data on the forklift's operating status and the environment. The forklift's operating status can include current speed, steering angle, fork height, fork tilt angle, fork root pressure, and vehicle posture; the environmental status can include newly acquired point cloud data, depth images, and obstacle distances. When the deviation between the real-time pose and the planned path exceeds a preset allowable range, the main control unit can reduce the travel speed and recalculate subsequent control segments based on the current position.

[0062] When the forklift approaches the target forklift position, the main control device can enter a low-speed precision positioning mode. In this mode, the main control device uses the current identified position of the valid forkholes as a reference to fine-tune the forklift's lateral position, longitudinal position, heading angle, and fork height, ensuring that the two forks are aligned with the first and second valid forkholes, respectively. If the depth camera detects that the deviation between the fork tip and the edge of the forkhole exceeds a preset range, the main control device will correct the posture before continuing insertion.

[0063] During fork insertion, the main control unit uses the pressure sensor at the fork root and displacement feedback to determine the insertion status. If the fork pushing resistance suddenly increases and the depth image shows the fork has not entered the fork hole, the main control unit can stop pushing and retract a certain distance for realignment. When the fork pushing distance reaches the preset insertion depth and the pressure change is within the normal range, the main control unit determines that the fork has been effectively inserted. Subsequently, the main control unit controls the fork lifting mechanism to raise the equipment to be replaced at a preset speed and continuously monitors the stability of the equipment posture and fork pressure.

[0064] During the process of controlling the forklift to travel along the optimal path and pick up items, the main control equipment can update the 3D information of obstacles and the work area map including height constraint information in real time, such as... Figure 9 As shown. If workers, temporary tool vehicles, or temporary pipelines enter the vicinity of the current path, the main control equipment can add the new obstacle to the work area map and determine whether the current path is still safe in terms of horizontal distance and height clearance. If there is a collision risk on the current path, the main control equipment controls the forklift to slow down or stop and re-determines the optimal travel path.

[0065] When determining whether a collision risk exists, the main control device can comprehensively consider the minimum distance between the current path and the obstacle, the obstacle's height occupancy status, the forklift's clearance height range, the current fork height, and the dynamic obstacle's movement trend. When a dynamic obstacle is expected to enter the safe area in front of the forklift, the main control device can immediately trigger braking or path replanning. If the obstacle is only located within a non-accessible height level and does not affect the passage of the forklift or forks, the main control device can maintain the current path and continue monitoring.

[0066] Example 2 This embodiment provides an optional method for creating a standard fork hole template. The main control device can obtain the equipment identifier from the maintenance work order through the communication module, and read the standard 3D model of the equipment to be replaced from the workshop equipment management system based on the equipment identifier. The standard 3D model can be the equipment design model or a digital model after on-site calibration. The main control device extracts the geometric parameters of the prefabricated fork holes from the standard 3D model. The geometric parameters may include the number of fork holes, the size of the fork holes, the center distance between the fork holes, the offset of the fork holes relative to the geometric center of the equipment, and the depth of the fork holes.

[0067] The main control equipment can generate a standard fork hole template based on the above geometric parameters. The standard fork hole template may include a fork hole contour template, a fork hole spatial position template, and a fork hole pairing template. The fork hole contour template is used to determine whether candidate fork holes have an opening shape consistent with the standard fork hole; the fork hole spatial position template is used to determine the relative position of candidate fork holes in the bottom plane of the equipment; the fork hole pairing template is used to determine whether multiple candidate fork holes satisfy the pairing relationship of the equipment's standard fork holes. This standard fork hole template is used for optional verification of candidate fork holes. Even without a standard fork hole template, valid fork holes can still be determined through contour integrity, depth abrupt change features, and pairing spatial relationships.

[0068] Example 3 This embodiment provides a method for determining the occupancy status of obstacles at different heights. The main control device can determine the occupancy status of an obstacle at each height level based on the lowest point height, highest point height, and outer contour of the merged obstacle object. For example, a low-level equipment base may only occupy the height level closest to the ground, a suspended cable may only occupy the middle height level, and a cable tray may simultaneously occupy higher height levels and part of the lateral space. The main control device updates the work area map based on this occupancy status, enabling path planning to distinguish between passage states with the same horizontal position but different heights.

[0069] Example 4 This embodiment provides a path replanning method. In each update cycle, the main control device compares the newly acquired 3D obstacle information with the work area map from the previous cycle. When a grid cell changes from passable to occupied, or when the height of an obstacle enters the forklift's current passage height range, the main control device marks that area as a risk zone. If the risk zone intersects with the current optimal driving path or its safety buffer zone, the main control device can pause the forklift's movement and re-search for a path within the updated candidate driving area; if the risk zone does not affect the current path, the main control device continues to execute the current path and maintains monitoring.

[0070] Example 5 Please see Figure 10 As shown, this embodiment provides a forklift operating device. The device may include a data acquisition module, a perception and recognition module, a map generation module, and a path control module. The data acquisition module acquires multimodal environmental data of the forklift's operating area and forklift operating status information; the perception and recognition module identifies confined space obstacles within the operating area based on the multimodal environmental data, generating three-dimensional obstacle information including the obstacle's space occupancy status; the map generation module generates a map of the operating area containing height constraint information based on the obstacle's height occupancy status in the obstacle's three-dimensional information; and the path control module determines the optimal driving path for the forklift based on the forklift operating status information and the operating area map containing height constraint information, and controls the forklift to perform the picking operation of the equipment to be replaced.

[0071] The data acquisition module may include a sensor data receiving unit, a time synchronization unit, and a preprocessing unit. The sensor data receiving unit is used to receive LiDAR point clouds, depth images, inertial measurement data, wheel encoder data, and fork status data; the time synchronization unit is used to align data from different sources according to the acquisition time; the preprocessing unit is used to perform effective range filtering, outlier removal, and coordinate transformation on the point cloud, and to perform distortion correction and depth noise processing on the depth image.

[0072] The perception and recognition module may include a low-level obstacle recognition unit, a suspended obstacle recognition unit, and a fusion unit. The low-level obstacle recognition unit is used to identify low-level candidate obstacles within the low-level recognition area; the suspended obstacle recognition unit is used to identify suspended candidate obstacles within the near-end supplementary recognition area; and the fusion unit is used to merge the low-level candidate obstacles and the suspended candidate obstacles into three-dimensional obstacle information and determine the obstacle's position, category, and height occupancy status.

[0073] The map generation module may include a height layer division unit, a grid status update unit, and a safety buffer unit. The height layer division unit is used to divide the working space at the bottom of the conveyor line into multiple height layers; the grid status update unit is used to write the three-dimensional information of obstacles into the grid map of the corresponding height layer; and the safety buffer unit is used to expand the occupied grid according to the forklift outline, positioning error, and fork swing margin.

[0074] The path control module may include a target point generation unit, a candidate area determination unit, a path search unit, a path replanning unit, and a motion control unit. The target point generation unit generates target forklift location information based on the spatial location of the available fork openings; the candidate area determination unit determines candidate driving areas based on the accessible height area; the path search unit determines the optimal driving path within the candidate driving areas; the path replanning unit re-determines the optimal driving path when the updated work area map indicates a collision risk in the current path; and the motion control unit converts the path results into control commands for the forklift actuators.

[0075] This invention can also provide an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the forklift operation method based on intelligent forklift position recognition in any of the above embodiments. This electronic device can be an on-board industrial control computer or an edge computing device communicatively connected to a forklift motion controller.

[0076] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the forklift operation method based on intelligent forklift location recognition as described in any of the above embodiments. The computer-readable storage medium can be a solid-state drive, memory card, read-only memory, random access memory, or other media capable of storing program instructions.

[0077] It should be noted that the preset range, preset threshold, sampling period, safety margin, grid size, and height layer division method described in this specification can be set according to the forklift model, sensor resolution, bottom structure height of the conveyor line, size of the equipment to be replaced, and workshop safety regulations. The specific values ​​of the above parameters should not be construed as limiting the scope of protection of this invention. Any implementation of this invention that enables the coordinated processing of obstacle recognition in confined spaces, height-constrained map construction, target forklift location determination, and forklift path control is considered an embodiment of this invention.

[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. For those skilled in the art, equivalent substitutions or combinations of sensor types, obstacle recognition methods, map update methods, path search strategies, forklift location verification methods, and motion control methods can be made without departing from the concept of the present invention, and such substitutions or adjustments should all fall within the protection scope of the present invention.

[0079] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A forklift operation method based on intelligent identification of forklift point, characterized in that, Applied to a main control device, the method includes: Acquire multimodal environmental data of the work area where the forklift is located and forklift operation status information; Based on the multimodal environmental data, obstacles in the confined space within the work area are identified, and three-dimensional information of the obstacles, including their space occupancy status, is generated. These obstacles are used to characterize obstacles that affect the passage height or passage gap of the work space at the bottom of the conveyor line. A work area map containing height constraint information is generated based on the obstacle height occupancy status in the three-dimensional obstacle information. Based on the forklift operation status information and the operation area map containing height constraint information, the optimal driving path of the forklift is determined. Based on the optimal travel path, the forklift is controlled to perform the picking operation of the equipment to be replaced.

2. The method of claim 1, wherein, The confined space obstacles include low-lying candidate obstacles and suspended candidate obstacles corresponding to the working space at the bottom of the conveyor line. The multimodal environmental data includes lidar point clouds, corrected depth images, and forklift real-time pose matrix. Based on the multimodal environmental data, the confined space obstacles within the working area are identified, and three-dimensional obstacle information including the obstacle's space occupancy status is generated, including: Based on the lidar point cloud and the corrected depth image, determine the low-level identification area and the near-end supplementary identification area corresponding to the working space at the bottom of the conveyor line. Low-position candidate obstacles are identified in the low-position identification area, and suspended candidate obstacles are identified in the near-end supplementary identification area; The low-lying candidate obstacles and the suspended candidate obstacles are fused to generate the three-dimensional information of the obstacles.

3. The method of claim 2, wherein, The step of identifying low-position candidate obstacles within the low-position recognition area includes: Extract point cloud data corresponding to the low-level identification area from the lidar point cloud; Based on the height distribution, density distribution, and neighborhood continuity of the points in the point cloud data, points corresponding to the ground area and the distant background area are removed. The remaining points are aggregated to obtain the low-position candidate obstacles.

4. The method of claim 2, wherein, The process of identifying suspended candidate obstacles within the proximal supplementary identification area includes: Extract the point cloud projection information corresponding to the near-end supplementary identification area from the lidar point cloud; Identify image objects with suspended distribution features and slender contour features in the depth image; The image object is spatially mapped to the point cloud projection information to obtain the suspended candidate obstacle.

5. The method of claim 2, wherein, The process of fusing the low-lying candidate obstacles and the suspended candidate obstacles to generate the three-dimensional information of the obstacles includes: Based on the spatial correspondence between the low-position candidate obstacles and the suspended candidate obstacles in the work area, the repeatedly identified obstacle candidates are merged. Based on the spatial range of the merged obstacle candidates, determine the obstacle position and obstacle height occupancy status of the corresponding obstacle; Based on the structural features of the merged obstacle candidates, the obstacle category of the corresponding obstacle is determined; The obstacle's location, obstacle type, and obstacle height occupancy status are associated with the obstacle's three-dimensional information.

6. The method of claim 4, wherein, Determining the optimal travel path of the forklift includes: Based on the current height of the forklift forks and the forklift's pass-through height range, determine the passable height area from the work area map containing height constraint information; Based on the grid occupancy status corresponding to the passable height area, candidate driving areas that meet the height passability conditions are determined; Target forklift location information is generated based on the spatial position of the effective forkhole of the equipment to be replaced. The optimal driving path is determined within the candidate driving area, with the real-time pose of the forklift as the starting point and the target position in the pre-selected target forklift position information as the ending point. During the process of controlling the forklift to travel along the optimal travel path, the work area map containing height constraint information is updated based on the updated obstacle 3D information, and when the updated work area map containing height constraint information indicates that there is a collision risk on the optimal travel path, the optimal travel path is re-determined.

7. The method of claim 6, wherein, Before determining the optimal driving path within the candidate driving area, using the real-time pose of the forklift as the path starting point and the target position in the pre-determined target forklift location information as the path ending point, the method further includes: Target forklift location information is generated based on the spatial position of the effective forkhole of the equipment to be replaced. The effective forkholes include a first effective forkhole and a second effective forkhole. Generating target forklift location information based on the spatial position of the effective forkholes of the equipment to be replaced includes: The forking center point is determined based on the three-dimensional coordinates of the centers of the first and second valid forks, and the forking center point is taken as the target position. The forking posture angle is determined based on the direction of the center line connecting the first effective fork hole and the second effective fork hole; The fork insertion depth is determined based on the depth information of the effective fork holes; The target forklift location information is generated based on the target location, the forklift posture angle, and the fork insertion depth.

8. The method as described in claim 7, characterized in that, Before determining the fork center point based on the three-dimensional coordinates of the centers of the first and second valid fork holes, the method further includes: The effective fork hole of the device to be replaced is determined based on the lidar point cloud. The step of determining the effective fork hole of the device to be replaced based on the lidar point cloud includes: Based on the lidar point cloud, the bottom plane region of the device to be replaced is extracted, and candidate fork holes are detected from the bottom plane region; When a standard fork hole template is configured, the candidate fork hole is matched with the standard fork hole template to obtain the fork hole to be verified; and, the fork hole to be verified is verified based on geometric parameters, and when the fork hole to be verified satisfies the template similarity matching and the geometric relationship verification, the fork hole to be verified is determined as the valid fork hole; When the standard fork hole template is not configured, the effective fork hole is determined based on the outline integrity, depth abruptness features and paired spatial relationships of the candidate fork holes.

9. The method as described in claim 8, characterized in that, Before performing template similarity matching between the candidate fork hole and the standard fork hole template to obtain the fork hole to be verified when a standard fork hole template is configured, the method further includes: Create the standard fork hole template; Creating the standard fork hole template includes: Obtain the device identifier of the device to be replaced, and read the standard 3D model of the device to be replaced based on the device identifier; Geometric parameters of the prefabricated fork holes are extracted from the standard three-dimensional model. The geometric parameters include at least one of the following: number of fork holes, size of fork holes, center-to-center distance of fork holes, offset of fork holes relative to the geometric center of the equipment, and depth of fork holes. The standard fork hole template is generated based on the geometric parameters, and the standard fork hole template is used to perform optional verification of candidate fork holes.

10. A forklift operation device based on intelligent forklift location recognition, used to implement the method of claim 6, characterized in that, include: The data acquisition module is used to acquire multimodal environmental data of the forklift's operating area and forklift operating status information; The perception and recognition module is used to identify obstacles in the confined space within the work area based on the multimodal environmental data, and generate three-dimensional information of the obstacles, including the space occupancy status of the obstacles. The map generation module is used to generate a work area map containing height constraint information based on the obstacle height occupancy status in the obstacle 3D information; The path control module is used to determine the optimal travel path of the forklift based on the forklift operation status information and the operation area map containing height constraint information, and to control the forklift to perform the picking operation of the equipment to be replaced based on the optimal travel path.

Images