Container lockhole identification method and system and engineering vehicle
By using a pre-trained alignment model and a binocular camera system, the three-dimensional spatial coordinates of container lock holes were acquired and accurately positioned, solving the problem of low lock hole positioning accuracy and improving the container unloading efficiency of the stacker crane.
Patent Information
- Application Number
- CN202411021079.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
Existing technology cannot effectively identify the lower locking holes during container unloading operations, resulting in low locking hole positioning accuracy and reduced stacker operation efficiency.
A pre-trained alignment model is used in conjunction with a binocular camera to acquire the three-dimensional spatial coordinates of the keyhole through image acquisition, and the camera position is adjusted according to the execution command to control the forklift spreader to move to the keyhole.
It improves the positioning accuracy of the lock hole and enhances the operating efficiency of the stacker crane in the container placement operation.
Smart Images

Figure CN118992839A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of container lifting, and in particular to a container keyhole identification method, system and engineering vehicle. Background Art
[0002] Currently, the technology for container keyhole recognition and alignment mainly uses cameras combined with deep learning, which is trained based on the collected keyhole information and uses the trained model to recognize the keyhole of the container.
[0003] However, the existing technology generally only considers the recognition and alignment of the key holes when grabbing the container, that is, only the four key holes above the container are recognized, and there is a lack of recognition training for the key holes below when placing the container, resulting in the inability to recognize the key holes when performing the container placing operation, which in turn leads to low accuracy in positioning the key holes of the container, thereby reducing work efficiency. Summary of the invention
[0004] In view of this, the present application provides a container lock hole identification method, system and engineering vehicle, which solve the technical problem of low positioning accuracy of the container lock hole and reduced operation efficiency of the forklift.
[0005] As a first aspect of the present application, the present application provides a method for identifying a keyhole of a container, comprising: in response to receiving an execution instruction for a container, using a pre-trained alignment model to identify a collected container image to obtain the keyhole of the container; capturing an image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustably installed on a sling of a forklift, and the position of capturing the image of the keyhole changes based on the execution instruction; based on the execution instruction and the three-dimensional spatial coordinates, controlling the sling of the forklift to move to the keyhole.
[0006] In a possible implementation of the present application, the position of the key hole for image capture changes based on the execution instruction, including: in response to the execution instruction being a container grabbing instruction, the position of the key hole for image capture is the lower lock hole of the container; in response to the execution instruction being a container placing instruction on the ground, the position of the key hole for image capture is the lower lock hole of the container grabbed by the sling and the upper lock hole of the placed container; in response to the execution instruction being a container placing instruction on a container truck, the position of the key hole for image capture is the lock hole of the container grabbed by the sling and the lock hole of the container truck.
[0007] In a possible implementation of the present application, in response to an execution instruction being a container grabbing instruction, the sling of the forklift is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: based on the three-dimensional space coordinates, determining whether the offset between the sling and the lower locking hole of the container is less than a preset offset value; if the offset between the sling and the lower locking hole of the container is not less than the preset offset value, moving the sling laterally, and determining whether the offset between the sling and the lower locking hole of the container after the movement is less than the preset offset value; if the offset between the sling and the lower locking hole of the container is less than the preset offset value, controlling the forklift to advance to a preset distance from the container, and controlling the sling to move to the top locking hole of the container for locking.
[0008] In a possible implementation of the present application, if the offset between the sling and the lower locking hole of the container is not less than a preset offset value, the sling is moved laterally, including: determining whether the lower locking hole of the container is biased to the left; if the lower locking hole of the container is biased to the left, controlling the sling to move to the left; if the lower locking hole of the container is biased to the right, controlling the sling to move to the right.
[0009] In a possible implementation of the present application, in response to an execution instruction being a ground container placement instruction, the spreader of the forklift is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: determining whether the container grasped by the spreader is the first container; if the container grasped by the spreader is not the first container, aligning and identifying the lower locking hole of the container grasped by the spreader and the upper locking hole of the placed container to obtain the spreader offset distance; based on the spreader offset position, controlling the spreader to move to the upper locking hole of the placed container; if the container grasped by the spreader is the first container, placing the container in a preset area.
[0010] In a possible implementation of the present application, in response to an execution instruction being an instruction to place a container on a container truck, the spreader of the forklift is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: obtaining the position of the container truck by collecting the position of the container truck; obtaining the relative offset distance between the locking hole of the container truck and the locking hole of the container grasped by the spreader by aligning and identifying the locking hole of the container truck and the locking hole of the container grasped by the spreader; based on the position of the container truck, the forklift is controlled to advance to the container truck, and based on the relative offset distance, the spreader is controlled to place the grasped container.
[0011] In a possible implementation of the present application, the location of the container truck is obtained by collecting the position of the container truck, including: collecting the longitudinal position of the container truck based on the longitudinal line laser radar to obtain the longitudinal position of the container truck; collecting the transverse position of the container truck based on the transverse line laser radar to obtain the transverse position of the container truck; integrating the longitudinal position and the transverse position to obtain the location of the container truck.
[0012] In a possible implementation of the present application, before capturing an image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, the method includes: using a light sensor to capture light from the environment where the binocular camera is located to obtain light intensity; using a humidity sensor to capture humidity from the environment where the binocular camera is located to obtain humidity information; obtaining environmental correction parameters by analyzing and processing the light intensity and humidity information; and performing parameter correction on the binocular camera based on the environmental correction parameters.
[0013] As a second aspect of the present application, the present application also provides a container keyhole recognition device, including: an identification module, used to respond to receiving an execution instruction for the container, use a pre-trained alignment model to recognize the collected container image, and obtain the keyhole of the container; an acquisition module, used to capture the image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera can be adjustably installed on the sling of the forklift, and the position of the image capture of the keyhole changes based on the execution instruction; a control module, used to control the sling of the forklift to move to the keyhole based on the execution instruction and the three-dimensional spatial coordinates.
[0014] As the third aspect of the present application, the present application also provides a container keyhole identification system, comprising: a controller and at least one sensor, at least one laser radar and a binocular camera, wherein the controller is connected to the sensor, the laser radar and the binocular camera respectively; the controller is used to execute the container keyhole identification method as any of the above items.
[0015] As a fourth aspect of the present application, the present application further provides an engineering vehicle, comprising: a keyhole identification system for a container as described above.
[0016] As a fifth aspect of the present application, the present application further provides a computer-readable storage medium, the storage medium storing a computer program, and the computer program is used to execute any of the above-mentioned container keyhole identification methods.
[0017] The present application provides a method for identifying the keyhole of a container. In response to receiving an execution instruction for a container, a pre-trained alignment model is used to identify the collected container image to obtain the keyhole of the container; the keyhole is imaged by a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustably installed on the spreader of a stacker, and the position of the image acquisition of the keyhole is changed based on the execution instruction; based on the execution instruction and the three-dimensional spatial coordinates, the spreader of the stacker is controlled to move to the keyhole. It is easy to notice that by performing binocular vision acquisition on the identified container keyhole, the three-dimensional spatial coordinates of different keyhole positions can be obtained based on different execution instructions, thereby achieving the purpose of being able to identify the lower keyhole position when performing a container placement operation, facilitating the execution of the container placement operation, and achieving the technical effect of improving the positioning accuracy of the container keyhole, thereby improving the operating efficiency of the stacker, thereby solving the technical problem of low positioning accuracy of the container keyhole and reducing the operating efficiency of the stacker. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0019] Figure 1 Shown is a flow chart of a container lock hole identification method provided by an embodiment of the present application;
[0020] Figure 2 Shown is a flow chart of model training provided by an embodiment of the present application;
[0021] Figure 3 The figure shows a flow chart of model detection provided by an embodiment of the present application;
[0022] Figure 4 Shown is a flowchart of keyhole positioning provided by an embodiment of the present application;
[0023] Figure 5 The figure shows a schematic diagram of a binocular camera identifying a keyhole during a box grabbing operation provided by an embodiment of the present application;
[0024] Figure 6 The figure shows a schematic diagram of a binocular camera identifying a lock hole during a box placing operation provided by an embodiment of the present application;
[0025] Figure 7 Shown is a flow chart of a box grabbing process of a forklift provided by an embodiment of the present application;
[0026] Figure 8 The figure is a flow chart of a process of placing boxes on the ground by a forklift provided by an embodiment of the present application;
[0027] Fig. 9 The figure is a flow chart of a container placing process of a forklift provided by an embodiment of the present application;
[0028] Fig.10 Shown is a schematic diagram of the structure of each sensor provided in an embodiment of the present application;
[0029] Fig.11 Shown is a schematic diagram of a lock hole identification device for a container provided by an embodiment of the present application;
[0030] Fig.12 Shown is a schematic diagram of a keyhole identification system for a container provided by an embodiment of the present application;
[0031] Fig.13 Shown is a device block diagram of an engineering vehicle provided in one embodiment of the present application. DETAILED DESCRIPTION
[0032] In the description of the application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically limited. In the embodiment of the present application, all directional indications (such as up, down, left, right, front, back, top, bottom ...) are only used to explain the relative position relationship, motion conditions, etc. between the components under a certain specific posture (as shown in the drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their deformations are intended to cover non-exclusive inclusions. For example, the process, method, system, product or equipment comprising a series of steps or units are not limited to the steps or units listed, but optionally also include the steps or units not listed, or optionally also include other steps or units inherent to these processes, methods, products or equipment.
[0033] In addition, the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0034] Exemplary Methods
[0035] As a first aspect of the present application, the present application provides a container keyhole identification method. Figure 1 FIG. 1 is a flow chart of a container keyhole identification method provided by an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0036] S101, in response to receiving an execution instruction for a container, using a pre-trained alignment model to recognize a collected container image to obtain a lock hole of the container;
[0037] Specifically, the execution instruction can be used to indicate the instruction received by the forklift to operate the container. Generally, the execution instruction can be a container grabbing instruction, a container placement instruction, a container truck placement instruction, etc. The execution instruction is not specifically limited here.
[0038] The above-mentioned pre-trained alignment model can be used to represent the pre-trained model for alignment recognition of the lock hole of the container. Since the lock hole and the container pin need to be aligned in this application, there are high requirements for recognition accuracy, but at the same time, real-time recognition is required, and there are also high requirements for recognition speed, so YOLOv8s is selected as the training model.
[0039] Generally, the lock hole of a container usually refers to the installation position of the lock on the container door. During the operation of the container, the lock hole on the container is identified and aligned, and the container is grabbed based on the lock hole position, so that the operation accuracy of the container can be improved.
[0040] Specifically, using YOLOv8s as a training model can be performed through the following steps: select the model and dataset, that is, select the YOLOv8 model configuration file suitable for the task (such as yolov8n.yaml), and prepare the dataset, including the partition files of the training set and the validation set, and the category list; download the pre-trained model, that is, download the corresponding version of the pre-trained model from the YOLOv8 GitHub open source website; train the model, that is, use the command line tool or Python API to train the model; export the model, that is, after the training is completed, the model can be exported to the ONNX format for deployment and use on different platforms, and the model can be exported using Python scripts or command line tools; model reasoning, that is, load the exported ONNX model in the application and use it for real-time target detection or other computer vision tasks. Reasoning.
[0041] Exemplary, used in this application, can be as follows Figure 2 As shown, Figure 2The flowchart of the model training provided by an embodiment of the present application includes: data collection and standardization, that is, it is necessary to collect pictures of the side lock holes of the container and standardize the collected pictures; network framework construction, that is, to build a preliminary training network; parameter setting, that is, to set the initial parameters of the training network; model training, that is, after setting the parameters, the model is trained using the standardized pictures of the side lock holes of the container; actual detection, that is, to perform model detection on the trained model; whether the accuracy is satisfactory, that is, to determine whether the model training results are consistent with the actual calibration results; if so, it means that the accuracy of the model training reaches the preset accuracy, and the trained model is output. Otherwise, if it is inconsistent, it is necessary to re-set the parameters of the training network and re-perform the model training process.
[0042] Furthermore, the above-mentioned actual detection process can be achieved by Figure 3 For example, Figure 3 FIG. 1 is a flow chart of a model detection provided by an embodiment of the present application. Figure 3 As shown in the figure, the model detection process includes: inputting an image, that is, inputting a collected image of the keyhole on the side of the container; performing data enhancement on the image and unifying the size, that is, standardizing the input image; dividing the network, that is, dividing the training network into a backbone network and a neck network; extracting image features through the backbone network, and performing feature fusion through the neck network to obtain the final feature map; predicting the target bounding box, that is, obtaining a bounding box containing the keyhole by performing keyhole recognition on the feature map; predicting the target category, the target category and the corresponding keyhole; performing confidence scoring, that is, scoring the confidence of the prediction results of the target bounding box and the target category. The higher the score, the more accurate the model training; judging whether the confidence scoring result is greater than the threshold; if it is greater than, it means that the model training is relatively accurate, and the target bounding box and target category information are output. If it is not greater than, it means that the model training result has not reached the preset accuracy and the bounding box needs to be discarded.
[0043] The above confidence score can use BCE (binary cross entropy) as the classification loss, judge "whether it is this category" for each category, and output the confidence.
[0044] In addition, applied to the embodiments of the present application, in the actual detection process, the detection program can be divided into four main parts according to the left and right cameras and the instructions received to grab or put the box, grabbing the box on the left side, putting the box on the left side, grabbing the box on the right side, putting the box on the right side, etc.
[0045] S102, capturing an image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustably mounted on a spreader of the forklift, and a position at which the image of the keyhole is captured changes based on an execution instruction;
[0046] Specifically, the above-mentioned binocular camera, also known as a stereo camera or dual-camera system, is a device that uses two cameras to capture the same scene. This system obtains three-dimensional spatial information by simulating the principle of human binocular vision. Each camera captures images from a different angle, and then by calculating the difference between the two images (called disparity), the depth information of the scene can be reconstructed.
[0047] Applied in the embodiment of the present application, by installing the binocular camera on the spreader of the forklift and by adjusting the installation position of the binocular camera, the forklift can identify the lock hole of the container when grabbing or placing the container, ensuring that the field of view is not blocked.
[0048] In addition, different execution instructions for the container correspond to different positions for collecting images of the keyholes, that is, the present application solves the problem of only identifying the four keyholes above the container in the prior art.
[0049] In an optional embodiment, Figure 4 FIG. 1 is a flowchart of keyhole positioning provided by an embodiment of the present application. Figure 4 As shown in the figure, in the process of acquiring the image of the keyhole through the binocular camera and obtaining the three-dimensional space coordinates of the keyhole, after the binocular camera detects the position of the keyhole, the two-dimensional pixel points of the corresponding area can be mapped to the three-dimensional space to obtain the depth information of the keyhole, thereby calculating the position of the keyhole relative to the binocular camera. By calibrating the binocular camera, the coordinates of the keyhole in the world coordinate system can be obtained, thereby controlling the lifting device to lock and realize the grabbing operation of the container.
[0050] S103, based on the execution instruction and the three-dimensional space coordinates, controlling the spreader of the forklift to move to the lock hole.
[0051] Specifically, after the three-dimensional spatial coordinates of the keyhole are obtained, the spreader of the forklift can be controlled to move to the keyhole based on different execution instructions.
[0052] Exemplarily, if the above-mentioned execution instruction is a container grabbing instruction, the forklift's sling is controlled to move to the locking hole position of the container to be grabbed, and the sling is controlled to lock; if the above-mentioned execution instruction is a container releasing instruction, the lock hole of the container grabbed by the forklift's sling is controlled to align with the upper locking hole of the placed container, and then the sling is controlled to descend to release the container, etc.
[0053] The container keyhole recognition method provided by the present application, in response to receiving an execution instruction for the container, uses a pre-trained alignment model to recognize the collected container image to obtain the container keyhole; uses a binocular camera to collect images of the keyhole to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera can be adjusted and installed on the spreader of the stacker, and the position of the keyhole image collection is changed based on the execution instruction; based on the execution instruction and the three-dimensional spatial coordinates, the stacker spreader is controlled to move to the keyhole. It is easy to notice that by performing binocular vision collection on the identified container keyhole, the three-dimensional spatial coordinates of different keyhole positions can be obtained based on different execution instructions, thereby realizing the purpose of being able to identify the lower keyhole position when performing the container placement operation, facilitating the execution of the container placement operation, achieving the technical effect of improving the positioning accuracy of the container keyhole, and then improving the operating efficiency of the stacker, thereby solving the technical problem of low positioning accuracy of the container keyhole and reducing the operating efficiency of the stacker.
[0054] In a possible implementation of the present application, the position of the key hole for image capture changes based on the execution instruction, including: in response to the execution instruction being a container grabbing instruction, the position of the key hole for image capture is the lower lock hole of the container; in response to the execution instruction being a container placing instruction on the ground, the position of the key hole for image capture is the lower lock hole of the container grabbed by the sling and the upper lock hole of the placed container; in response to the execution instruction being a container placing instruction on a container truck, the position of the key hole for image capture is the lock hole of the container grabbed by the sling and the lock hole of the container truck.
[0055] Specifically, in an optional embodiment, if the execution instruction is a container grabbing instruction, that is, the spreader of the forklift is required to grab the container, the position for capturing the image of the lock hole is the lower lock hole of the container. Figure 5 FIG. 1 is a schematic diagram of a binocular camera identifying a keyhole during a box grabbing operation provided by an embodiment of the present application. Figure 5 As shown, the binocular camera captures images of the lock holes on both sides directly below the container to be grabbed. At the same time, by calibrating the current position of the binocular camera, the three-dimensional coordinates of the lock holes on both sides directly below the container to be grabbed can be obtained. Based on the three-dimensional coordinates, the sling can be controlled to lock and the container can be grabbed.
[0056] In another optional embodiment, if the execution instruction is a ground container placement instruction, that is, the container grasped by the forklift's sling needs to be placed in a designated area, and the positions for capturing images of the lock holes are the lower lock hole of the container grasped by the sling and the upper lock hole of the placed container. Figure 6 FIG. 1 is a schematic diagram of a binocular camera identifying a keyhole during a box placing operation provided by an embodiment of the present application. Figure 6As shown, the binocular camera collects images of the lower lock hole of the container grasped by the spreader and the upper lock hole of the placed container. At the same time, by calibrating the current position of the binocular camera and identifying the three-dimensional coordinates of the lower lock hole of the grasped container and the upper lock hole of the placed container, the container can be placed on the ground.
[0057] In the third optional embodiment, if the execution instruction is a container truck release instruction, that is, the container grabbed by the spreader of the forklift needs to be placed on the container truck, and the position of the lock hole image acquisition is the lock hole of the container grabbed by the spreader and the lock hole of the container truck. Generally, container truck release is different from ground release, that is, after the forklift completes the container grabbing operation, it needs to retreat a certain distance, and wait for the truck driver to drive over before placing the container on the container truck.
[0058] In a possible implementation of the present application, in response to an execution instruction being a container grabbing instruction, the sling of the forklift is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: based on the three-dimensional space coordinates, determining whether the offset between the sling and the lower locking hole of the container is less than a preset offset value; if the offset between the sling and the lower locking hole of the container is not less than the preset offset value, moving the sling laterally, and determining whether the offset between the sling and the lower locking hole of the container after the movement is less than the preset offset value; if the offset between the sling and the lower locking hole of the container is less than the preset offset value, controlling the forklift to advance to a preset distance from the container, and controlling the sling to move to the top locking hole of the container for locking.
[0059] In a possible implementation of the present application, if the offset between the sling and the lower locking hole of the container is not less than a preset offset value, the sling is moved laterally, including: determining whether the lower locking hole of the container is biased to the left; if the lower locking hole of the container is biased to the left, controlling the sling to move to the left; if the lower locking hole of the container is biased to the right, controlling the sling to move to the right.
[0060] Specifically, the above-mentioned preset offset value can be used to indicate the offset distance of the preset lock hole center relative to the hanger. The preset offset value is not specifically limited here and can be adjusted according to actual conditions.
[0061] The above-mentioned preset distance can be used to represent a preset distance between the forklift and the container. Under this preset distance, the spreader can be moved to the container. The preset distance is not specifically limited here and can be adjusted according to actual conditions.
[0062] In an optional embodiment, in response to the execution instruction being a container grabbing instruction, based on the execution instruction and the three-dimensional spatial coordinates, in the process of controlling the forklift's sling to move to the locking hole, it can be determined whether the offset between the sling and the lower locking hole of the container to be grabbed is less than a preset offset value based on the three-dimensional coordinates of the locking holes on both sides directly below the container to be grabbed. If it is less than, it means that the sling and the lower locking hole of the container to be grabbed are accurately aligned, and the forklift can be controlled to advance to the preset distance from the container, and the sling is controlled to move to the lower locking hole of the container; conversely, if it is not less than, it means that the sling and the lower locking hole of the container to be grabbed are not accurately aligned, and the sling needs to be adjusted, that is, the sling is moved laterally, and it is continued to be determined whether the offset between the sling after the move and the lower locking hole of the container is less than the preset offset value, and this iterative cycle is performed until the offset between the sling and the lower locking hole of the container is less than the preset offset value, the forklift is controlled to advance, and the sling is controlled to move to the top locking hole of the container to lock it, thereby completing the container grabbing action.
[0063] For example, Figure 7 FIG. 1 is a flow chart of a process of grabbing a box by a forklift provided in an embodiment of the present application. Figure 7 As shown in the figure, the box grabbing process includes the following operations:
[0064] The forklift completes the alignment, that is, the forklift and the container to be grabbed are aligned;
[0065] The laser radar detects the height and number of layers N of the container, that is, the longitudinal laser radar installed on the forklift scans the container to obtain the height and number of layers of the container;
[0066] Calculate the actual height of the container according to the number of layers: H = N * 2.6 (m);
[0067] The forklift spreader is lifted to a height above H;
[0068] The camera performs keyhole recognition, that is, the binocular camera is used to identify the lower keyhole of the container to be grabbed;
[0069] Determine whether the center offset of the lock hole B≤B0, that is, determine whether the offset between the spreader and the lower lock hole of the container to be grabbed is less than the preset offset value;
[0070] If the center offset of the lock hole is B≤B0, the lock hole is controlled to align left and right, that is, the spreader is controlled to align with the lock holes on both sides directly below the container to be grabbed, and the forklift moves forward to the distance D=D0 from the container, that is, the forklift is controlled to move forward to the preset distance from the container, and the forklift is locked when it is empty;
[0071] If the lock hole center offset B>0, determine whether the lock hole is biased to the left;
[0072] If the lock hole is tilted to the left, the lifting device is controlled to move left by B. If the lock hole is tilted to the right, the lifting device is controlled to move right by B.
[0073] Continue the above-mentioned keyhole identification operation, that is, determine whether the keyhole center offset B≤B0, and iterate in this way until the keyhole center offset B≤B0, and complete the container grabbing operation.
[0074] In a possible implementation of the present application, in response to an execution instruction being a ground container placement instruction, the spreader of the forklift is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: determining whether the container grasped by the spreader is the first container; if the container grasped by the spreader is not the first container, aligning and identifying the lower locking hole of the container grasped by the spreader and the upper locking hole of the placed container to obtain the spreader offset distance; based on the spreader offset position, controlling the spreader to move to the upper locking hole of the placed container; if the container grasped by the spreader is the first container, placing the container in a preset area.
[0075] Specifically, the above-mentioned spreader offset distance can be used to indicate the offset distance of the lower locking hole of the container grasped by the spreader relative to the upper locking hole of the placed container.
[0076] In an optional embodiment, in response to an execution instruction being an instruction for placing a container on the ground, based on the execution instruction and the three-dimensional space coordinates, in the process of controlling the forklift's sling to move to the locking hole, it can be determined whether the container currently grasped by the sling is the first container. If so, the currently grasped container can be placed in a designated area; conversely, if the currently grasped container is not the first container, it is necessary to align the lower locking hole of the currently grasped container and the upper locking hole of the placed container to obtain the offset distance of the lower locking hole of the container grasped by the sling relative to the upper locking hole of the placed container, and adjust the sling according to the sling offset distance until the lower locking hole of the grasped container and the upper locking hole of the placed container are aligned left and right, and then control the sling to descend and place the container.
[0077] For example, Figure 8 FIG. 1 is a flow chart of a process of placing boxes on the ground by a forklift according to an embodiment of the present application. Figure 8 As shown in the figure, the process of placing boxes on the ground includes the following operations:
[0078] After the forklift completes the box grabbing operation, the path is planned by back-stepping method, and the forklift is controlled to drive along the path to the box placement point;
[0079] Determine whether the container grabbed by the spreader is the first container;
[0080] If yes, the seized container is placed in the designated area, and the vehicle returns along the path, grabs the next container, and drives along the path to the direction point;
[0081] If the container grabbed by the spreader is not the first container, the laser radar detects the bottom container for alignment, that is, the horizontal line laser radar on the spreader is used to identify and align the placed bottom container;
[0082] The laser radar detects the height H of the container, that is, the height of the placed container is collected through the longitudinal laser radar installed on the spreader;
[0083] Move the spreader up and down to a position higher than the height H of the detected container, and control the forklift to move forward to a position aligned with the front and rear of the placed container;
[0084] The binocular camera on the spreader is used to identify the lower lock hole of the currently grasped container and the upper lock hole of the placed container. After ensuring that the lower lock hole of the grasped container and the upper lock hole of the placed container are aligned left and right, the spreader is controlled to descend and release the container.
[0085] In a possible implementation of the present application, in response to an execution instruction being an instruction to place a container on a container truck, the spreader of the forklift is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: obtaining the position of the container truck by collecting the position of the container truck; obtaining the relative offset distance between the locking hole of the container truck and the locking hole of the container grasped by the spreader by aligning and identifying the locking hole of the container truck and the locking hole of the container grasped by the spreader; based on the position of the container truck, the forklift is controlled to advance to the container truck, and based on the relative offset distance, the spreader is controlled to place the grasped container.
[0086] In a possible implementation of the present application, the location of the container truck is obtained by collecting the position of the container truck, including: collecting the longitudinal position of the container truck based on the longitudinal line laser radar to obtain the longitudinal position of the container truck; collecting the transverse position of the container truck based on the transverse line laser radar to obtain the transverse position of the container truck; integrating the longitudinal position and the transverse position to obtain the location of the container truck.
[0087] Specifically, the above relative offset distance can be used to indicate the offset distance of the locking hole of the container grasped by the spreader relative to the locking hole of the container truck.
[0088] In an optional embodiment, in response to an execution instruction being an instruction to release a container by a truck, based on the execution instruction and the three-dimensional space coordinates, in the process of controlling the forklift's spreader to move to the locking hole, the position of the truck can be collected by a laser radar installed on the forklift's spreader to obtain the position of the truck. After obtaining the position of the truck, the forklift is controlled to advance to the truck and align it front and back with the container. Then, the lock hole of the truck and the lock hole of the container grasped by the spreader are identified by a binocular camera to obtain the offset distance of the lock hole of the container grasped by the spreader relative to the lock hole of the truck. The spreader is adjusted by the relative offset distance until the lower lock hole of the container grasped by the spreader is aligned left and right with the lock hole of the truck, and the spreader is controlled to descend to release the container.
[0089] For example, Fig. 9 FIG. 1 is a flow chart of a container placing process of a forklift provided by an embodiment of the present application, as shown in FIG. Fig. 9 As shown in the figure, the container truck unloading process includes the following operations:
[0090] After the forklift completes the container grabbing operation, the forklift is controlled to move backward, and the truck driver drives to the front of the forklift;
[0091] The laser radar detects the position and distance of the container truck. That is, the position and distance of the container truck are collected through the transverse line laser radar installed on the spreader, and then the path is planned by the backstepping method. That is, after the position and distance of the container truck are determined, the path for the forklift to move forward to the container truck is planned;
[0092] Laser radar detects the height of the container truck, that is, the longitudinal laser radar installed on the spreader collects the height of the container truck and controls the spreader to move up and down to a position above the height of the container truck;
[0093] Control the forklift to drive along the planned path to the front of the container truck, and control the spreader to align the front and back of the container on the container truck;
[0094] The camera recognizes the left and right alignment of the upper and lower lock holes, that is, the binocular camera is used to align the lock hole of the container truck and the lock hole of the container grasped by the spreader, so that the lower lock hole of the container grasped by the spreader is aligned with the lock hole of the container truck, and then the spreader is controlled to descend to complete the container release operation.
[0095] It should be noted that when the laser radar detects the position of the container truck, it first completes the alignment and height detection. The two laser radars on both sides of the spreader scan vertically to determine the height, deflection and distance of the container truck, the laser radar in front scans horizontally to determine the distance, and the binocular cameras on both sides scan whether the container lock hole and the container truck lock hole are aligned left and right. Then, as the forklift moves forward, the left and right position and height of the spreader are adjusted, and finally the front and rear distance is adjusted to complete the lock hole alignment, and the spreader is moved down to complete the container placement.
[0096] In a possible implementation of the present application, before capturing an image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, the method includes: using a light sensor to capture light from the environment where the binocular camera is located to obtain light intensity; using a humidity sensor to capture humidity from the environment where the binocular camera is located to obtain humidity information; obtaining environmental correction parameters by analyzing and processing the light intensity and humidity information; and performing parameter correction on the binocular camera based on the environmental correction parameters.
[0097] Specifically, the above-mentioned environmental correction parameters can be used to represent parameters for correcting the parameters of the binocular camera based on the light intensity and humidity information of the environment.
[0098] In an optional embodiment, considering the container operation scenario, it is necessary to test the detection effect in different environments. Therefore, the embodiment of the present application exemplarily gives the detection effect in daytime and night, as well as the detection effect in sunny and rainy days, and adjusts the camera parameters to make it adaptive to different environments.
[0099] Specifically, the light environment of the binocular camera is collected based on the light sensor to obtain the light intensity, thereby distinguishing the detection effects during the day and at night. The humidity environment of the binocular camera is collected based on the humidity sensor to obtain humidity information, thereby distinguishing the detection effects between sunny days and rainy days. By analyzing and processing the light intensity and humidity information, environmental correction parameters can be obtained, and then the binocular camera parameters are calibrated based on the environmental correction parameters, thereby adjusting the camera parameters to improve the keyhole recognition effect of the container.
[0100] For example, Fig.10 FIG. 1 is a schematic diagram of the structure of each sensor provided in an embodiment of the present application. Fig.10 As shown, the industrial computer is connected to the humidity sensor and the light sensor respectively, and is used to receive the humidity information collected by the humidity sensor and the light intensity collected by the light sensor. The industrial computer obtains environmental correction parameters by analyzing and processing the humidity information and the light intensity, and performs parameter correction on the binocular camera through the environmental correction parameters.
[0101] Exemplary Devices
[0102] As a second aspect of the present application, the present application also provides a keyhole identification device for a container. Fig.11 FIG. 1 is a schematic diagram of a container keyhole identification device provided by an embodiment of the present application. Fig.11 As shown, the device comprises:
[0103] The recognition module 1101 is used to recognize the collected container image using the pre-trained alignment model in response to receiving an execution instruction for the container, and obtain the lock hole of the container;
[0104] The acquisition module 1102 is used to acquire an image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustably mounted on the spreader of the forklift, and the position of acquiring the image of the keyhole is changed based on the execution instruction;
[0105] The control module 1103 is used to control the spreader of the forklift to move to the lock hole based on the execution instruction and the three-dimensional space coordinates.
[0106] Optionally, the control module includes: a first judgment module, used to judge whether the offset between the spreader and the lower locking hole of the container is less than a preset offset value based on the three-dimensional space coordinates; a moving module, used to move the spreader laterally if the offset between the spreader and the lower locking hole of the container is not less than the preset offset value, and judge whether the offset between the spreader and the lower locking hole of the container after the movement is less than the preset offset value; a control forward module, used to control the forklift to advance to a preset distance from the container if the offset between the spreader and the lower locking hole of the container is less than the preset offset value, and control the spreader to move to the top locking hole of the container for locking.
[0107] Optionally, the moving module includes: a left-biased judgment module, used to judge whether the lower locking hole of the container is biased to the left; a left-moving module, used to control the spreader to move to the left if the lower locking hole of the container is biased to the left; and a right-moving control module, used to control the spreader to move to the right if the lower locking hole of the container is biased to the right.
[0108] Optionally, the control module also includes: a second judgment module, used to judge whether the container grasped by the spreader is the first container; a first alignment identification module, used to align and identify the lower locking hole of the container grasped by the spreader and the upper locking hole of the placed container to obtain the spreader offset distance if the container grasped by the spreader is not the first container; a control movement module, used to control the spreader to move to the upper locking hole of the placed container based on the spreader offset position; and a control placement module, used to place the container in a preset area if the container grasped by the spreader is the first container.
[0109] Optionally, the control module also includes: a position acquisition module, which is used to obtain the position of the container truck by acquiring the position of the container truck; a second alignment and identification module, which is used to obtain the relative offset distance between the lock hole of the container truck and the lock hole of the container grasped by the spreader by aligning and identifying the lock hole of the container truck and the lock hole of the container grasped by the spreader; and a control advance module, which is used to control the forklift to advance to the container truck based on the position of the container truck, and control the spreader to place the grasped container based on the relative offset distance.
[0110] Optionally, the position acquisition module includes: a longitudinal acquisition module, which is used to collect the longitudinal position of the container truck based on the longitudinal line laser radar to obtain the longitudinal position of the container truck; a transverse acquisition module, which is used to collect the transverse position of the container truck based on the transverse line laser radar to obtain the transverse position of the container truck; and an integration module, which is used to integrate the longitudinal position and the transverse position to obtain the position of the container truck.
[0111] Optionally, the device also includes: a light collection module, which is used to use a light sensor to collect light from the environment where the binocular camera is located to obtain light intensity; a humidity collection module, which is used to use a humidity sensor to collect humidity from the environment where the binocular camera is located to obtain humidity information; an analysis module, which is used to obtain environmental correction parameters by analyzing and processing the light intensity and humidity information; and a correction module, which is used to perform parameter correction on the binocular camera based on the environmental correction parameters.
[0112] The container lock hole recognition device provided in this embodiment includes: a lock hole recognition module, which is used to respond to receiving an execution instruction for the container, use a pre-trained alignment model to recognize the collected container image, and obtain the lock hole of the container; a lock hole coordinate acquisition module, which is used to acquire images of the lock hole through a binocular camera to obtain the three-dimensional spatial coordinates of the lock hole, wherein the binocular camera can be adjusted and installed on the spreader of the stacker, and the position of the lock hole acquisition is changed based on the execution instruction; a control module, which is used to control the stacker to move to the lock hole based on the execution instruction and the three-dimensional spatial coordinates. It is easy to notice that by performing binocular vision acquisition on the identified container lock hole, the three-dimensional spatial coordinates of different lock hole positions can be obtained based on different execution instructions, thereby realizing the purpose of being able to identify the lower lock hole position when performing the container release operation, facilitating the execution of the container release operation, and achieving the technical effect of improving the positioning accuracy of the container lock hole, thereby improving the operating efficiency of the stacker, thereby solving the technical problem of low positioning accuracy of the container lock hole and reducing the operating efficiency of the stacker.
[0113] Exemplary Systems
[0114] As a third aspect of the present application, the present application also provides a keyhole identification system for a container. Fig.12 FIG. 1 is a schematic diagram of a container keyhole identification system provided by an embodiment of the present application. Fig.12 As shown, the keyhole recognition system includes: a controller and at least one sensor, at least one laser radar and a binocular camera, wherein the controller is connected to the sensor, the laser radar and the binocular camera respectively; the controller is used to execute the keyhole recognition method of the container as any of the above items.
[0115] Exemplary Construction Vehicles
[0116] As a fourth aspect of the present application, the present application further provides an engineering vehicle, comprising: a keyhole identification system for a container as described above.
[0117] Fig.13 Shown is a device block diagram of an engineering vehicle provided in one embodiment of the present application.
[0118] like Fig.13As shown, the engineering vehicle includes one or more processors 131 and memory 132 .
[0119] The processor 131 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the engineering vehicle to perform desired functions.
[0120] The memory 132 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement the above container keyhole identification method of each embodiment of the present application and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0121] In one example, the engineering vehicle may further include: an input device 133 and an output device 134 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0122] When the engineering vehicle is a stand-alone device, the input device 133 may be a communication network connector for receiving collected input signals from the first device and the second device.
[0123] In addition, the input device 133 may also include, for example, a keyboard, a mouse, etc.
[0124] The output device 134 can output various information to the outside, including the determined distance information, direction information, etc. The output device 134 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0125] Of course, to simplify, Fig.13 Only some of the components in the engineering vehicle related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the engineering vehicle may further include any other appropriate components according to specific application conditions.
[0126] Exemplary computer-readable storage media
[0127] As a fifth aspect of the present application, the present application provides a computer-readable storage medium, the storage medium storing a computer program, and the computer program is used to perform the following steps:
[0128] S1, in response to receiving an execution instruction for a container, using a pre-trained alignment model to recognize the collected container image to obtain a lock hole of the container;
[0129] S2, collecting images of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustably installed on the spreader of the forklift, and the position of collecting images of the keyhole changes based on the execution instruction;
[0130] S3, based on the execution instruction and the three-dimensional space coordinates, controls the spreader of the forklift to move to the lock hole.
[0131] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program information, which, when executed by a processor, enables the processor to execute the steps of the simulation and optimization method of the manufacturing model according to various embodiments of the present application described in this specification.
[0132] The computer program product may be written in any combination of one or more programming languages to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0133] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program information is stored. When the computer program information is executed by a processor, the processor executes the steps in the simulation and optimization method of the manufacturing model according to various embodiments of the present application described in this specification.
[0134] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0135] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.
[0136] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.
[0137] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.
Claims
1. A method for identifying a keyhole of a container, characterized in that: include: In response to receiving an execution instruction for a container, using a pre-trained alignment model to recognize the collected container image to obtain a lock hole of the container; Capturing an image of the keyhole through a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustably mounted on a spreader of a forklift, and a position at which the image of the keyhole is captured is changed based on the execution instruction; Based on the execution instruction and the three-dimensional space coordinates, the spreader of the forklift is controlled to move to the lock hole.
2. The container keyhole identification method according to claim 1, characterized in that: The position of the keyhole for image acquisition is changed based on the execution instruction, including: In response to the execution instruction being a container grabbing instruction, the position for capturing the image of the lock hole is the lower lock hole of the container; In response to the execution instruction being a ground container placement instruction, the positions for capturing images of the lock holes are the lower lock holes of the container grasped by the spreader and the upper lock holes of the placed containers; In response to the execution instruction being a container truck release instruction, the positions for capturing images of the lock holes are the lock holes of the container grasped by the spreader and the lock holes of the container truck.
3. The container keyhole identification method according to claim 2, in response to the execution instruction being a container grabbing instruction, characterized in that: Based on the execution instruction and the three-dimensional space coordinates, controlling the spreader of the forklift to move to the lock hole includes: Based on the three-dimensional space coordinates, determining whether the offset between the spreader and the lower locking hole of the container is less than a preset offset value; If the offset between the spreader and the lower locking hole of the container is not less than the preset offset value, the spreader is moved laterally, and it is determined whether the offset between the spreader and the lower locking hole of the container after the movement is less than the preset offset value; If the offset between the spreader and the lower locking hole of the container is less than the preset offset value, the forklift is controlled to move forward to a preset distance from the container, and the spreader is controlled to move to the top locking hole of the container for locking.
4. The container keyhole identification method according to claim 3, characterized in that: If the offset between the spreader and the lower locking hole of the container is not less than the preset offset value, the spreader is moved laterally, including: Determining whether the lower lock hole of the container is biased to the left; If the lower lock hole of the container is biased to the left, the spreader is controlled to move to the left; If the lower locking hole of the container is biased to the right, the spreader is controlled to move to the right.
5. The container keyhole identification method according to claim 2, in response to the execution instruction being a ground-laying container instruction, characterized in that: Based on the execution instruction and the three-dimensional space coordinates, controlling the spreader of the forklift to move to the lock hole includes: Determining whether the container grabbed by the spreader is the first container; If the container grasped by the spreader is not the first container, the lower locking hole of the container grasped by the spreader and the upper locking hole of the placed container are aligned and identified to obtain the spreader offset distance; Based on the offset position of the spreader, controlling the spreader to move to the upper locking hole of the placed container; If the container grabbed by the spreader is the first container, the container is placed in a preset area.
6. The container keyhole identification method according to claim 2, in response to the execution instruction being a container truck release instruction, characterized in that: Based on the execution instruction and the three-dimensional space coordinates, controlling the spreader of the forklift to move to the lock hole includes: By collecting the position of the container truck, the location of the container truck is obtained; By performing alignment identification on the locking hole of the container truck and the locking hole of the container grasped by the spreader, a relative offset distance between the locking hole of the container truck and the locking hole of the container grasped by the spreader is obtained; Based on the location of the container truck, the forklift is controlled to move toward the container truck, and based on the relative offset distance, the spreader is controlled to place the grabbed container.
7. The method for identifying the keyhole of a container according to claim 6, wherein the location of the container truck is obtained by collecting the location of the container truck, comprising: The longitudinal position of the container truck is collected based on the longitudinal line laser radar to obtain the longitudinal position of the container truck; The lateral position of the container truck is collected based on a lateral line laser radar to obtain the lateral position of the container truck; The longitudinal position and the transverse position are integrated to obtain the location of the container truck.
8. The container keyhole identification method according to claim 1, characterized in that: Before acquiring an image of the keyhole by a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, the method includes: Using a light sensor to collect light from the environment where the binocular camera is located to obtain light intensity; Using a humidity sensor to collect humidity information from the environment where the binocular camera is located; Obtaining environmental correction parameters by analyzing and processing the light intensity and humidity information; Parameter correction is performed on the binocular camera based on the environmental correction parameters.
9. A container keyhole identification system, characterized in that: include: A controller and at least one sensor, at least one laser radar and a binocular camera, wherein: The controller is respectively connected to the sensor, the laser radar and the binocular camera; The controller is used to execute the lock hole identification method of a container as described in any one of claims 1 to 8.
10. An engineering vehicle, characterized in that: include: The keyhole identification system for a container as claimed in claim 9.
