Automatic container grabbing and releasing method for straddle carrier

Abstract

The invention discloses an automatic container grabbing and releasing method for a straddle carrier. The method comprises the following steps: acquiring field conditions around a container to be grabbed and released; whether visual detection or laser perception is used for assisting in grabbing the container is judged; acquiring the position of the characteristic part of the container by utilizing visual inspection or laser perception; moving the straddle carrier to align a sling of the straddle carrier with the container by means of the position of the feature; and the lifting appliance is moved to grab the container. According to the automatic grabbing and releasing method for the container, the stable effect can be achieved, and good working efficiency is obtained.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and more particularly to an automatic container grasping and placing method for a straddle carrier. Background Art

[0002] Generally, straddle carriers are used to handle containers at cargo concentration sites such as docks or warehouses. A straddle carrier is a heavy engineering vehicle with relatively slender legs that reaches a height of more than ten meters, and a spreader is installed above the vehicle to perform operations such as grasping, placing, and stacking containers. Conventional spreaders have connection mechanisms such as hooks that align with the lock holes on the upper surface of the container. The spreader can drop after the hook and the lock hole are aligned to connect the connection mechanism with the lock hole, thereby grasping the container. Then the straddle carrier can move to the container placement position to place the container.

[0003] In the above process, if there are deviations in the positions of grasping and placing the container, it may be necessary to spend additional effort and time to re-operate, which significantly reduces the efficiency. Therefore, how to accurately grasp and place the container has always been a concern. Currently, the above alignment of the straddle carrier is mainly manually operated, which relies on the estimation of the human eye and the judgment of experience, with low efficiency. Moreover, manual operation is prone to not strictly following the regulations, resulting in inconvenience or inability to operate other containers at the same or adjacent positions, and causing the accumulation of stacking position errors.

[0004] Therefore, in order to solve the problem of low efficiency of manual operation, various designs for automatically grasping and placing containers by straddle carriers have been proposed currently to try to improve the efficiency. In the prior art, the alignment and grasping and placing operations are usually achieved by methods based on vision or laser.

[0005] The vision-based method (also known as "vision detection") usually uses a camera to capture on-site images and detect the position of the container in the image, and performs the grasping and placing of the container based on the detected position. However, this method has the following deficiencies: First, for situations of light changes or extreme rain and snow weather, the vision detection method is unstable or even completely unable to work because it is difficult to detect the container; Second, most vision-based methods also rely on the calibration of the internal and external parameters of the camera. For a straddle carrier operating for a long time with high intensity, it is easy for the sensor to be deformed and displaced due to jolts, resulting in inaccurate calibration parameters and detected positions. Frequent re-calibration is also very burdensome for the site, thereby reducing the efficiency advantage of automatic container grasping and placing over manual operation.

[0006] Laser-based methods (also known as "laser perception" methods) typically establish a template at the position where the container is picked up and placed, then use laser scanning to obtain the point cloud of the container to be picked up and infer the predicted pose of the container, and then compare the predicted pose of the container with... However, this method is troubled by the following serious problems: First, the method based on template matching depends on the accuracy of the template first, and recording the template is extremely time-consuming. Second, the inference speed of this method is slow, and the motion planning and control module is prone to overshoot due to delays. Finally, the straddle carrier has relatively slender legs up to more than ten meters high, and the laser sensor can only be installed at a height of 1-2.5 meters on the legs to see the container. When the container is under heavy loads of different weights, the legs will tilt inward or outward (outer fork, inner fork), resulting in inaccurate positions of the original laser point cloud, and existing detection methods are difficult to solve this error.

[0007] Currently, there is still a desire to develop an automatic container picking and placing method for straddle carriers that can improve work efficiency while ensuring accurate picking and placing of containers. Summary of the Invention

[0008] To solve the problem of low work efficiency of existing automatic container picking methods, the present invention proposes an automatic container picking and placing method for straddle carriers, which has achieved extremely stable effects in all seasons and all scenarios and obtained good work efficiency.

[0009] Specifically, this automatic container picking and placing method for straddle carriers includes the following steps: determining whether to use visual detection or laser perception to assist in picking up the container; using visual detection or laser perception to obtain the position of the feature part of the container; moving the straddle carrier to align the spreader of the straddle carrier with the container by means of the position of the feature part; and moving the spreader to pick up the container. Further, after picking up the container, the automatic container picking and placing method for straddle carriers includes the following steps: determining whether to use visual detection or laser perception to assist in placing the container based on the target placement position; using visual detection or laser perception to obtain the target placement position; moving the straddle carrier and its spreader to align the container with the target placement position by means of the target placement position; and moving the spreader to place the container at the target placement position. The picking and placing method of the present invention preferably combines two detection methods, visual detection and laser perception, compared with the existing methods, combines the advantages of both, and optimizes each of them, so that the handling of containers can be efficiently completed in various situations, avoiding problems such as environmental constraints or poor accuracy encountered due to a single detection method.

[0010] In an embodiment of the present invention, the step of determining whether to use visual detection or laser perception to assist in picking up the container includes determining whether visual detection can be used to detect the feature part of the container.

[0011] In an exemplary method of the present invention for gripping a container, the steps of using visual detection to obtain the position of the feature portion of the container include obtaining an image including the feature portion of the container; detecting the pixel image of the feature portion from the image; and determining the position of the center of the pixel image of the feature portion. Further, the steps of aligning the spreader with the container include determining a preset gripping position in the image; and moving the straddle carrier so that the position of the center of the pixel image of the feature portion coincides with the preset gripping position. The visual detection of the automatic container gripping and placing method for the straddle carrier according to the present invention does not rely on the calibration of the internal and external parameters of the camera, continuously outputs the pixel difference directly for the planning and control module to use, has a more stable effect compared with the existing visual detection, and greatly reduces the proportion of manual intervention, thereby improving the efficiency.

[0012] In another exemplary method of the present invention for gripping a container, the steps of using laser sensing to obtain the position of the feature portion of the container include using a laser to scan the container to obtain a point cloud of at least a part of the horizontal cross-section of the container; calculating the central coordinates of the horizontal cross-section based on the point cloud; and calculating the coordinate position of the feature portion based on the central coordinates. Further, the steps of aligning the spreader with the container include determining a preset coordinate position in the point cloud image; and moving the straddle carrier so that the coordinate position of the feature portion reaches the preset coordinate position. The laser sensing of the automatic container gripping and placing method for the straddle carrier according to the present invention has an extremely fast inference speed, greatly shortening the adjustment time of the control module. At the same time, it significantly offsets the sensor error caused by the deformation of the vehicle legs.

[0013] Preferably, the feature portion is a lock hole, and wherein aligning the spreader of the straddle carrier with the container includes aligning the hook of the spreader with the lock hole.

[0014] In an embodiment of the present invention, the target placement position includes the ground with a bay line and the top of another container.

[0015] In one embodiment, the steps of determining whether to use visual detection or laser sensing to place the container based on the target placement position include using visual detection if the target placement position is the ground; and using laser sensing if the target placement position is the top of another container. The method of the present invention can grip and place the container in various situations, avoiding external factors from reducing the work efficiency.

[0016] In an exemplary method for placing a container according to the present invention, the steps of using visual detection to obtain a target placement position include obtaining an image of a ground including bay lines; detecting a pixel image of the bay lines from the image; and determining the positions of the intersections of the pixel images of the bay lines. Further, the steps of aligning the container with the bay lines include determining a preset placement position in the image of the ground having the bay lines; and moving the straddle carrier so that the positions of the intersections of the pixel images of the bay lines coincide with the preset placement position.

[0017] In another exemplary method for placing a container according to the present invention, the steps of using laser sensing to obtain a target placement position include using a laser to scan another container to obtain a point cloud of at least a part of the horizontal cross-section of the other container; and calculating the central coordinates of the horizontal cross-section of the other container based on the point cloud. Further, the steps of aligning the container with the other container include determining a preset central coordinate position in the point cloud image; and moving the straddle carrier so that the central coordinates of the horizontal cross-section of the other container reach the preset central coordinate position.

[0018] Additional features and advantages of the described automatic container gripping and placing method for a straddle carrier will be set forth in the detailed description which follows, and will be apparent to those of ordinary skill in the art from the detailed description or recognized by practicing the embodiments described herein, which description includes the detailed description below and the drawings. Description of the Drawings

[0019] For the above purposes, the technical features of the present invention are clearly described in the following claims, and its advantages are apparent from the following detailed description with reference to the drawings, which illustrate by way of example the preferred embodiments of the present invention without limiting the scope of the inventive concept.

[0020] Figure 1 Shows an overall flowchart of an automatic container gripping and placing method for a straddle carrier according to the present invention;

[0021] Figure 2 Shows a front view of a straddle carrier according to the present invention;

[0022] Figure 3 Shows a schematic view of a straddle carrier according to the present invention from another perspective;

[0023] Figure 4 Shows a top view of a spreader of a straddle carrier according to the present invention;

[0024] Figure 5 Shows a flowchart of a gripping step based on visual detection of an automatic container gripping and placing method for a straddle carrier according to the present invention;

[0025] Figure 6 Shows an image captured by a camera on the spreader of a straddle carrier according to the present invention, where the spreader is not aligned with the container;

[0026] Figure 7 Shows an image captured by a camera on the spreader of a straddle carrier according to the present invention, where the spreader is already aligned with the container;

[0027] Figure 8 Shows a flowchart of the grasping step based on laser perception of the automatic container picking and placing method for a straddle carrier according to the present invention;

[0028] Figure 9 Shows the schematic principle of laser perception of the automatic container picking and placing method for a straddle carrier according to the present invention;

[0029] Figure 10 Shows an image scanned by a lidar sensor on the outrigger of a straddle carrier according to the present invention;

[0030] Figure 11 Shows a flowchart of the placing step based on visual detection of the automatic container picking and placing method for a straddle carrier according to the present invention;

[0031] Figure 12 Shows an image captured by a camera on the spreader of a straddle carrier according to the present invention, where the spreader has grasped the container and is ready to place it on the ground; and

[0032] Figure 13 Shows a flowchart of the placing step based on laser perception of the automatic container picking and placing method for a straddle carrier according to the present invention. Detailed implementation mode

[0033] The present invention will be further described in detail below in conjunction with the drawings and embodiments, but it shall not be used as a basis for any limitation to the present invention.

[0034] The terms "upper", "lower", "top", "bottom", "front", "rear", "vertical", "horizontal" and their derivatives used herein refer to the orientation of the components during actual use. For example, the "rear" of the vehicle or the rear of the vehicle body refers to the rear part of the vehicle during actual normal driving, and the "horizontal plane" refers to a plane parallel to the ground (assuming the ground is flat), and so on. However, it should be understood that unless explicitly indicated to the contrary, the components may adopt various alternative orientations.

[0035] The term "alignment" used herein means that the centers of two objects are generally in a straight line in a certain direction, and it is not necessary for each point to be aligned.

[0036] For clarity, not all identical parts in the figures are marked with reference numerals, but only some of them are.

[0037] This application relates to an unmanned straddle carrier, and specifically to a straddle carrier for handling containers. More specifically, a method for handling containers is provided for such a straddle carrier. The method mainly includes a judgment step before grasping, a grasping step, a judgment step before placing, and a placing step. The automatic container grasping and placing method used can achieve stable effects and obtain good working efficiency. However, it should be understood that this application is not limited thereto.

[0038] Figure 1 A general flowchart of an automatic container grasping and placing method for a straddle carrier according to the present invention is shown. The method includes a grasping step: obtaining on-site conditions around the container to be grasped and placed (step 1001); judging whether to use visual detection or laser sensing to grasp the container (step 1002); using visual detection or laser sensing to obtain the position of the feature part of the container (step 1003); moving the straddle carrier so that the spreader of the straddle carrier is aligned with the container by means of the position of the feature part (step 1004); and moving the spreader to grasp the container (step 1005). Further, the method includes a placing step: judging whether to use visual detection or laser sensing to place the container based on the target placement position (step 1006); using visual detection or laser sensing to obtain the target placement position (step 1007); moving the straddle carrier and its spreader so that the container is aligned with the target placement position by means of the target placement position (step 1008); and moving the spreader to place the container at the target placement position (step 1009).

[0039] The implementation manners of these steps will be described separately below.

[0040] Figure 2 A straddle carrier 1 according to the present invention is shown. Figure 3The straddle carrier 1 is shown from another perspective. The straddle carrier includes a container automatic grasping and releasing system for grasping and releasing containers. The straddle carrier 1 includes outriggers 2, a spreader 3 supported by the outriggers 2 above, and a chassis 4 below. A lidar sensor 5 is installed on the chassis 4. The height at which the lidar sensor 5 is located is lower than the heights of various containers, and laser scanning in the horizontal direction is implemented (which will be described in detail below). In this embodiment, there may be six lidar sensors 5, two of which are installed inside on the cross beam of the chassis 4, and four are installed at the edges of the chassis 4 facing outward. Of course, the number of lidar sensors 5 is variable. For example, it may be four or eight. Note that the lidar sensors may not necessarily be set at the same horizontal height. When the spreader 3 grasps a container, the weight of the container is transmitted to the outriggers 2, which may cause the outriggers to deform slightly (for example, bend inward or outward), thereby possibly causing a change in the position of the lidar sensor 5, resulting in inaccurate data that may be obtained by the laser sensing method.

[0041] Figure 4 A top view of the spreader 3 of the straddle carrier according to the present invention is shown. The spreader 3 has a main beam 6 and end beams 7 on both sides of the main beam 6. Visual sensors 8 such as cameras are installed at both ends of each end beam 7 (that is, a total of four visual sensors 8). The cameras take continuous images downward to obtain the targets to be detected (such as the keyhole and the bay line to be described below). The movements or operations of the straddle carrier 1, the spreader 3, the lidar sensor 5, and the visual sensor 8 are all controlled by the motion control module of the straddle carrier.

[0042] "Field conditions" are the key to determining whether visual detection can be used to grasp containers. It includes natural environmental factors, mainly including at least one of light conditions, rain and snow coverage, and sand and dust conditions. Optionally, the field conditions may also include factors such as geographical location, air quality, and human activities. Light conditions are an important factor affecting camera shooting. Too low light intensity will cause the contrast of the captured image to decrease, affecting image clarity. Similarly, rain, snow, and sand and dust may also cause the captured image to be affected in terms of brightness, color, and authenticity. In the subsequent image target detection stage of using the trained model, the above factors may all lead to problems such as difficult feature extraction and background confusion in the image, making it difficult to detect the target and resulting in the failure of grasping and releasing based on visual detection.

[0043] The motion control module can determine whether to use visual detection or laser sensing to assist in grasping containers based on the field conditions. Specifically, the "field conditions" can be quantified by parameters such as the light intensity, rainfall and snowfall intensity, and visibility at the site, and compared with a pre-determined threshold. If the values of these parameters are greater than the pre-determined threshold, then visual detection is adopted; otherwise, laser sensing is adopted.

[0044] In addition, the judgment basis of the path planning and control module can also be the noise standard deviation threshold in the color photo. When the noise standard deviation of the captured picture is lower than this threshold, visual detection is adopted; otherwise, laser perception is adopted. The noise standard deviation threshold can be lower than 5 gray levels, 10 gray levels or 15 gray levels, which can be adjusted according to actual application requirements.

[0045] Furthermore, in addition to judging whether to use visual detection or laser perception based on on-site conditions, it is also possible to judge whether to adopt visual detection by judging whether the characteristic part of the target can be detected by visual detection. If the characteristic part of the target can be detected (for example, the keyhole in this article), then visual detection is continued. If the characteristic part of the target cannot be detected due to environmental reasons, the path planning and control module switches to laser perception to detect the target. In other words, step 1001 can be omitted, and it can be judged whether to use visual detection or laser perception by whether the characteristic part can be detected after the transporter approaches the container.

[0046] In any case, visual detection is preferably adopted because visual detection will obtain more data beneficial to positioning. When visual detection cannot be used for grasping and placing, laser perception can be adopted.

[0047] The automatic container grasping and placing method for a straddle carrier provided by the present invention combines visual detection and laser perception. Compared with the method that only uses visual detection, the method of the present invention can perform the grasping and placing of the container in various situations, avoiding the "jamming" of the process caused by external factors and reducing work efficiency; compared with the method that only uses visual perception, the method of the present invention preferentially adopts more accurate and faster visual detection when conditions permit, and the accuracy is more easily achieved to the successful alignment accuracy of the keyhole of 5 cm.

[0048] The following describes the automatic container grasping method for a straddle carrier of the present invention.

[0049] Refer to Figures 5 - 7 , and the following describes how to grasp the container by means of visual detection. Figure 5 Fig. shows an automatic container grasping method for a straddle carrier according to the present invention, which includes:

[0050] 1) The step of using visual detection to obtain the position of the characteristic part of the container, specifically including: obtaining an image including the characteristic part of the container (step 2001); detecting the pixel image of the characteristic part from the image (step 2002); determining the position of the center of the pixel image of the characteristic part (step 2003).

[0051] 2) Move the straddle carrier to align the spreader of the straddle carrier with the container by means of the position of the feature part, specifically including: determining a preset grasping position in the image (step 2004); moving the straddle carrier to make the position of the center of the pixel image of the feature part coincide with the preset grasping position (step 2005).

[0052] 3) The step of moving the spreader to grasp the container (step 2006).

[0053] First, the planning and control module controls the vision sensor to obtain the position of the feature part of the container.

[0054] As Figure 6 shown, an image captured by the camera is shown, and this image shows the feature part 11 of the container 10. Preferably, the feature part 11 is a keyhole, and wherein, aligning the spreader of the straddle carrier (not shown, see Figure 4 ) with the container 10 includes aligning the hook of the spreader with the keyhole. However, in other embodiments, other obvious signs such as labels may also be added to the container 10 as the feature part. As shown in the figure, when the feature part 11 is a keyhole, the vision sensor can transmit the captured image including the keyhole to the detection module, and then the detection module detects the pixel image of the keyhole based on a learning model such as yolo, and determines the pixel position 101 of the center of the pixel image of the keyhole. This pixel position 101 is shown as a red dot in the Figures 6 - 7 shown image. When the container 10 is not grasped, the pixel position 101 may change its position in the image as the straddle carrier moves (i.e., the position change of the camera).

[0055] Secondly, after completing the step of obtaining the position of the feature part of the container by means of vision detection, the planning and control module can control the straddle carrier to move to align the spreader of the straddle carrier with the container by means of the position of the feature part. In order to move the straddle carrier to the accurate position where it can grasp the container, two parameters need to be input to the planning and control module: the distance offset on the horizontal plane (i.e., the coordinate distance in the two-dimensional plane) and the heading angle offset (rotation offset).

[0056] In order to obtain the above two parameters, another pixel position, that is, the preset grasping position, also needs to be determined in the image, and the preset grasping position is compared with the pixel position 101 to obtain the coordinate distance between the current position and the preset grasping position, so as to convert it into the actual distance. Note that when the straddle carrier does not move, the actual position of the camera is fixed, so the coordinate distance can be directly converted into the current actual distance through the calibration coefficient.

[0057] In Figures 6 - 7In the shown image, the preset grasping position 102 is shown as a green dot, which is regarded as the true value. A conventional method for collecting the preset grasping position 102 is to control the vehicle to perform in-situ grasping and releasing of the container at the center position of the spreader, and set the pixel position at the center of the lock hole image of the container captured at this time as the preset grasping position 102. Each pixel position 101 determines a preset grasping position 102. Only when the pixel position 101 at each center coincides with or is at least substantially coincident with the preset grasping position 102 can the spreader be aligned with the container 10.

[0058] It should be noted that since the actual position of the camera relative to the spreader is fixed, in fact, in the corresponding image coordinate system, the preset grasping position 102 is also almost determined, that is, the preset grasping position 102 does not change its position in the image with the movement of the straddle carrier (i.e., the movement of the camera), that is, it is at a fixed coordinate in all images. Therefore, as long as the vehicle and the spreader are controlled so that the preset grasping positions 102 at the four viewing angles are all substantially aligned with the pixel position 101 at the center, the alignment of the spreader with the lock hole can be ensured. In other words, for the two unknown quantities greater than zero, namely the central coordinate distance and the rotational offset between the current container and the spreader on the horizontal plane, there are four sets of binary quadratic equations, so the solutions of the central coordinate distance and the rotational offset can surely be obtained. Therefore, this method does not need to rely on the internal and external parameter calibration of the camera.

[0059] It should be understood that in this embodiment, four vision sensors are used to detect the corresponding four lock holes, but the number of vision sensors is only an example rather than a limitation, and in other embodiments, only two or three vision sensors can be used to detect some of the lock holes, such as the lock holes on the diagonal. And in other embodiments, six or more vision sensors can be used to detect the lock holes.

[0060] Finally, after the spreader of the straddle carrier is aligned with the container, the motion control module can control the spreader to move to grasp the container. At this time, the step of grasping the container is completed, and then the step of placing the container will be carried out.

[0061] The vision detection of the automatic container grasping and releasing method for the straddle carrier according to the present invention adopts vision segmentation based on the YOLO model, does not need to rely on the internal and external parameter calibration of the camera, continuously outputs the pixel gap directly for the motion control module to use, has a more stable effect compared with the existing vision detection, greatly reduces the proportion of manual intervention, and thus improves the efficiency.

[0062] Referring to Figures 8 - 10 , the following describes how to grasp the container with the aid of laser sensing. Figure 8 Fig. shows another automatic container grasping method for the straddle carrier according to the present invention, which includes:

[0063] 1) Steps of obtaining the position of the feature part of the container using laser sensing, specifically including: scanning the container with a laser to obtain the point cloud of at least a part of the horizontal cross-section of the container (step 3001); calculating the central coordinates of the horizontal cross-section based on the point cloud (step 3002); and calculating the coordinate position of the feature part based on the central coordinates (step 3003).

[0064] 2) Steps of aligning the spreader with the container, specifically including: determining a preset coordinate position in the image (step 3004); and moving the straddle carrier so that the coordinate position of the feature part reaches the preset coordinate position (step 3005).

[0065] 3) Steps of moving the spreader to grab the container (step 3006).

[0066] Generally, template matching is used to obtain the feature part of the container. The template matching algorithm is used to study where the pattern of a specific object is located in the image, and then identify the object. The matching process is to compare the input image with the template images in the library one by one to find the part of the input image that is most similar to the template image. For example, if the input image is a point cloud pattern obtained by laser scanning and denoising, and the template images are various template point clouds, each template point cloud is traversed through the input point cloud pattern and evaluated to obtain the most similar template point cloud, so as to identify the object (such as a container) based on the most similar template point cloud.

[0067] As described above, because the legs may be deformed, resulting in the deviation of the lidar sensor from the initial position, the input point cloud obtained by laser scanning has errors. However, the laser sensing method provided by the present invention can weaken the influence brought by these errors to a great extent.

[0068] As Figure 9 shown, assuming that the container 10 has a substantially cuboid shape and ignoring the uneven parts on its surface, then its horizontal cross-section also has a substantially rectangular shape. The lidar sensors 5 on both sides and in the middle of the straddle carrier respectively horizontally scan a part of the horizontal cross-section of the container 10 at their respective horizontal heights. When the lidar sensors 5 are at the same height, the scanned part can be the same horizontal cross-section of the rectangle, so that the four vertices of the horizontal cross-section can be included. When the lidar sensors 5 are at different heights, from the top view, the scanning results of each lidar sensor 5 can be projected onto a horizontal cross-section of the container 10, so as to equivalently scan a part of the same horizontal cross-section. As Figure 10As shown, it is the contour of the container obtained by single-line laser scanning. Specifically, it is four point clouds 201 of at least a part of the horizontal cross-section of the container. These point clouds are obtained through the filtering of rain, snow and noise points, as well as noise filtering and preliminary clustering according to the distance and quantity of neighboring points. The laser points are sparser the farther they are. Based on these four point clouds 201, the vertices 202 of the four corners of the horizontal cross-section of the container can be obtained. Thus, based on these four vertices 202, the central coordinates of the horizontal cross-section can be calculated. Through the above method, the positions of the four vertices 202 and the center of the horizontal cross-section of the container 10 are obtained.

[0069] Secondly, after completing the step of using visual detection to obtain the position of the feature part of the container, the planning and control module can control the movement of the straddle carrier to align the spreader of the straddle carrier with the container by means of the position of the feature part. As described above, in order to move the straddle carrier to the accurate position where it can grab the container, the coordinate distance and the course angle offset on the horizontal plane need to be input to the planning and control module. In order to calculate the coordinate distance and the course angle deflection, a preset coordinate position can be determined in the coordinate system as Figure 10 shown. This is because for a known container, the grabbing positions of the straddle carrier are fixed. Therefore, the coordinate positions for grabbing can also be preset in the coordinate system. Then, based on the coordinate data of the above four vertices 202 and the center, the coordinate distance and the course angle deflection associated with the matching preset coordinate position are calculated. As described above, after obtaining these two parameters, the planning and control module can move the straddle carrier to the accurate position where it can grab the container.

[0070] The central coordinates of the horizontal cross-section calculated by the laser sensing method provided by the present invention can greatly weaken the influence of the error caused by the deformation of the legs. When dealing with deformation, the calculated central position is more stable and it is not easy to have matching problems. Specifically, in this laser sensing method, it is assumed that the legs are exactly the same. Then, the error caused by the deformation of the legs is evenly distributed on both sides. Therefore, the errors on both sides can be considered to be basically the same. On this basis, calculating the central coordinates of the horizontal cross-section based on the point clouds can cancel the errors on both sides to the greatest extent, so that the obtained central coordinates have as small an error as possible. After obtaining the central coordinates, the transverse and longitudinal offsets, that is, the coordinate distance, can be easily obtained. As for the course angle error, based on the above assumption, the four vertices 202 obtained from the four point clouds 201 still form a rectangle, and each side of this rectangle is parallel to the corresponding side of the rectangle in the case of no error at all. Therefore, the calculation of the course angle deflection is not affected.

[0071] Finally, after the spreader of the straddle carrier is aligned with the container, the planning and control module controls the spreader to move to grab the container. At this time, the step of grabbing the container is completed, and then the step of placing the container will be carried out.

[0072] The laser sensing used in the automatic container grasping and placing method for straddle carriers according to the present invention employs a detection method based on the geometric features of manual point clouds, with extremely fast inference speed, significantly shortening the adjustment time of the control module. At the same time, a mathematical model is established for the deformation of the vehicle legs, offsetting the sensor error caused by the deformation of the vehicle legs.

[0073] The following describes an automatic container placing method for straddle carriers according to the present invention.

[0074] In various embodiments, the target placement positions of the container may include the ground with the bay line 9 and the top of another container, that is, the container to be placed can be placed on the desired ground or on one layer or two layers of containers. The steps of determining whether to use visual detection or laser sensing to place the container based on the target placement position include: if the target placement position is the ground, visual detection is used; if the target placement position is the top of another container, since the spreader has grasped a container, a part of the field of view of the visual sensor is blocked by the container to be placed, and the characteristic parts (such as the lock hole) of the other container below cannot be seen, so visual detection cannot be used to assist in placing the container, and thus laser sensing is required. The bay line 9 is the ground marking, which is usually arranged in a cross shape on the ground in bright colors, so it is usually not easily affected by environmental factors and cannot be visually detected.

[0075] Refer to Figures 11 - 12 and the following describes how to place the container with the aid of visual detection. Figure 11 Fig. shows an automatic container placing method for straddle carriers according to the present invention, which includes:

[0076] 1) The step of obtaining the target placement position by using visual detection, specifically including: obtaining an image including the ground with the bay line (step 4001); detecting the pixel image of the bay line from the image (step 4002); and determining the position of the intersection of the pixel images of the bay line (step 4003).

[0077] 2) The step of aligning the container with the bay line, specifically including: determining a preset placement position in the image of the ground with the bay line (step 4004); and moving the straddle carrier so that the position of the intersection of the pixel images of the bay line coincides with the preset placement position (step 2005).

[0078] 3) The step of moving the spreader to place the container at the target placement position (step 4006).

[0079] Refer to Figure 12, similar to the above-mentioned use of visual detection to grab containers, as long as the pixel position 301 of the intersection of the Bay Line in the pixel image and the preset placement position 302 are determined, the pixel distance between each pixel position 301 and the corresponding preset placement position 302 can be calculated for use by the regulation and control module. As shown in the figure, the preset placement position 302, that is, the red dot is the target position recorded in advance. The acquisition method can also use the above-mentioned control vehicle to grab and release the center position of the spreader in situ, and the red dot is regarded as the true value. Pixel position 301, that is, the green dot is the current position obtained by first performing image segmentation of the Bay Line and then obtaining the intersection of the straight lines.

[0080] Note that since the position of the camera relative to the sling is fixed, in fact, the pixel position of the preset placement position 302 that can be placed at the intersection of the Bay Line 9 is almost fixed, so as long as the vehicle and the sling are controlled so that the preset placement positions of the four perspectives are basically aligned with the intersection position of the pixel image of the Bay Line 9, the placement position alignment can be ensured, and there is no need to consider rotation deviations such as heading angles. Therefore, this method does not need to rely on camera internal and external parameter calibration.

[0081] Figure 13 Another method for automatically placing containers on a straddle carrier according to the present invention is shown, which comprises:

[0082] 1) The step of using laser sensing to obtain the position of the characteristic part of the container specifically includes: using a laser to scan another container to obtain a point cloud of at least a portion of the horizontal cross-section of the other container (step 5001); calculating the center coordinates of the horizontal cross-section based on the point cloud (step 5002); and calculating the coordinate position of the characteristic part based on the center coordinates (step 5003).

[0083] 2) The step of aligning the spreader with the container specifically includes: determining a preset coordinate position in the image (step 5004); and moving the straddle carrier so that the coordinate position of the feature portion reaches the preset coordinate position (step 5005).

[0084] 3) The step of moving the spreader to place the container at the target placement position (step 5006).

[0085] In this embodiment, the steps of obtaining the position of the characteristic part of the container by laser sensing and aligning the spreader with the container are substantially the same as the aforementioned grabbing step, except that the spreader is finally lowered to place the container instead of grabbing the container.

[0086] The automatic container grabbing and placing method for straddle carriers of the present invention uses a method that uses both laser and visual methods, combining the advantages of both methods and optimizing each of them. For visual detection, this proposal adopts yolo-based visual segmentation, which does not rely on internal and external parameters, and continuously outputs pixel differences directly for regulation and control. For laser perception, this proposal uses a detection method based on manual point cloud geometric features, which has an extremely fast inference speed and greatly shortens the adjustment time of the control module. At the same time, mathematical modeling is done for the deformation of the vehicle legs to offset the sensor error caused by the deformation of the vehicle legs. The automatic container grabbing and placing method for straddle carriers of the present invention has achieved extremely stable detection effects in all seasons and all scenarios, greatly reducing the proportion of manual intervention and improving efficiency.

[0087] Although the structure and operation method of the present invention are described above in conjunction with the preferred embodiments, it should be recognized by those skilled in the art that the above examples are only for illustration and cannot be used as limitations of the present invention. Therefore, the present invention may be modified and varied, and these modifications and variations will fall within the scope defined by the appended claims of the present application.

Claims

1. A method for automatically picking up and placing containers for a straddle carrier, comprising the following steps: Determine the use of visual inspection or laser sensing to help grasp the container; Acquiring the position of the characteristic part of the container by using the visual detection or the laser sensing; moving the straddle carrier to align the spreader of the straddle carrier with the container by virtue of the position of the feature; as well as The spreader is moved to grab the container.

2. The automatic container grabbing and placing method for a straddle carrier according to claim 1, characterized in that: After grabbing the container, the following steps are also included: Using visual detection or laser sensing to assist in placing the container based on target placement location determination; Determining the use of visual detection or laser sensing to assist in placing the container; Using the visual detection or the laser sensing to obtain the target placement position; moving the straddle carrier and the spreader thereof to align the container with the target placement location by means of the target placement location; and The spreader is moved to place the container at the target placement location.

3. The automatic container grabbing and placing method for a straddle carrier according to claim 1 or 2, characterized in that: The step of determining whether to use the visual detection or laser sensing to assist in grasping the container includes determining whether the characteristic portion of the container can be detected using the visual detection.

4. The automatic container grabbing and placing method for a straddle carrier according to claim 1 or 2, characterized in that: The step of using the visual detection to obtain the position of the characteristic part of the container includes: acquiring an image including the characteristic portion of the container; Detecting a pixel image of the feature portion from the image; and The location of the center of the pixel image of the feature is determined.

5. The automatic container grabbing and placing method for a straddle carrier according to claim 4, characterized in that: The steps of aligning the spreader with the container include: determining a predetermined grab position in the image; and The straddle carrier is moved so that the position of the center of the pixel image of the feature coincides with the predetermined grasping position.

6. The automatic container grabbing and placing method for a straddle carrier according to claim 4, characterized in that: The step of using the laser sensing to obtain the position of the characteristic part of the container includes: Scanning the container using a laser to obtain a point cloud of at least a portion of a horizontal cross-section of the container; Calculating the center coordinates of the horizontal cross section based on the point cloud; and The coordinate position of the feature portion is calculated based on the center coordinate.

7. The automatic container grabbing and placing method for a straddle carrier according to claim 6, characterized in that: The step of aligning the spreader with the container comprises: Determining a preset coordinate position in the point cloud image; and The straddle carrier is moved so that the coordinate position of the feature portion reaches the preset coordinate position.

8. The automatic container grabbing and placing method for a straddle carrier according to claim 1 or 2, characterized in that: The feature is a keyhole, and wherein aligning a spreader of the straddle carrier with the container comprises aligning a hook of the spreader with the keyhole.

9. The automatic container grabbing and placing method for a straddle carrier according to claim 1 or 2, characterized in that: The target placement location includes the ground having a bay line and a top of another container.

10. The automatic container grabbing and placing method for a straddle carrier according to claim 9, characterized in that: The step of placing the container using visual detection or laser sensing based on the target placement position comprises: If the target placement location is the ground, using visual detection; and If the target placement position is the top of the other container, laser sensing is used.

11. The automatic container grabbing and placing method for a straddle carrier according to claim 10, characterized in that: The step of using the visual detection to obtain the target placement position includes: acquiring an image including the ground surface having the bay line; Detecting a pixel image of the bay line from the image; and The locations of the intersections of the pixel images of the bay lines are determined.

12. The automatic container grabbing and placing method for a straddle carrier according to claim 11, characterized in that: The step of aligning the container with the bay line comprises: determining a preset placement position in an image of the ground having the bay line; and The straddle carrier is moved so that the position of the intersection point of the pixel image of the Bay Line coincides with the preset placement position.

13. The automatic container grabbing and placing method for a straddle carrier according to claim 10, characterized in that: The step of using the laser sensing to obtain the target placement position includes: Scanning the other container using a laser to obtain a point cloud of at least a portion of a horizontal cross-section of the other container; and The center coordinates of the horizontal cross section of the other container are calculated based on the point cloud.

14. The automatic container grabbing and placing method for a straddle carrier according to claim 13, characterized in that: The step of aligning the container with the other container comprises: Determining a preset center coordinate position in the point cloud image; and The straddle carrier is moved so that the center coordinate of the horizontal cross section of the other container reaches the preset center coordinate position.

15. A vehicle-mounted unit, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for automatically grabbing and placing containers for a straddle carrier according to claim 1 or 2.

16. A computer program product storing a program, wherein when the program is executed by a computer, the program causes the computer to execute the steps of the method for automatically grabbing and placing a container for a straddle carrier according to claim 1 or 2.

17. A non-transitory computer-readable storage medium storing a program executed by a computer, the program causing the computer to execute the steps of the method for automatically grabbing and placing a container for a straddle carrier according to claim 1 or 2.

Images