Lock hole identification method and system for container, and engineering vehicle

Abstract

Provided in the present application are a lock hole identification method and system for a container, and an engineering vehicle. The method comprises: in response to having received an execution instruction for a container, using a pre-trained alignment model to perform identification on a container picture that is acquired, so as to obtain a lock hole of the container; performing image collection on the lock hole by means of a binocular camera, so as to obtain three-dimensional spatial coordinates of the lock hole, wherein the binocular camera is adjustably mounted on a spreader of a stacker, and the position where the image collection of the lock hole is performed changes on the basis of the execution instruction; and on the basis of the execution instruction and the three-dimensional spatial coordinates, controlling the spreader of the stacker to move to the lock hole. The present application solves the technical problem of the positioning accuracy of lock holes of containers being relatively low thus reducing the operating efficiency of stackers.

Description

Lock hole identification method and system for container and engineering vehicle TECHNICAL FIELD

[0001] The present application relates to the field of container hoisting, in particular to a lock hole identification method and system for container and an engineering vehicle.

[0002] BACKGROUND

[0003] Currently, the technology for identifying and aligning the lock holes of a container mainly uses a camera combined with deep learning, trains based on the collected lock hole information, and uses the trained model to identify the lock holes of the container.

[0004] However, the existing technology generally only considers the identification and alignment of the lock holes when grabbing the container, that is, only the four lock holes above the container are identified, and the identification and training of the lock holes below the container are lacking, which leads to the inability to identify the lock holes when performing the container dropping operation, and further leads to low positioning accuracy of the lock holes of the container and reduced work efficiency of the container handler.

[0005] SUMMARY

[0006] Therefore, the present application provides a lock hole identification method and system for container and an engineering vehicle, which solves the technical problem of low positioning accuracy of the lock holes of the container and reduced work efficiency of the container handler.

[0007] As a first aspect of the present application, the present application provides a lock hole identification method for container, comprising: in response to receiving an execution instruction for a container, identifying a collected container picture by using a pre-trained alignment model to obtain the lock holes of the container; collecting images of the lock holes by a binocular camera to obtain three-dimensional space coordinates of the lock holes, wherein the binocular camera is adjustably installed on a spreader of a container handler, and the position for collecting images of the lock holes changes based on the execution instruction; based on the execution instruction and the three-dimensional space coordinates, controlling the spreader of the container handler to move to the lock holes.

[0008] In a possible implementation manner of the present application, the position for collecting images of the lock holes changes based on the execution instruction, comprising: in response to the execution instruction being a container grabbing instruction, the position for collecting images of the lock holes is the lower lock holes of the container; or, in response to the execution instruction being a container dropping on flat ground instruction, the position for collecting images of the lock holes is the lower lock holes of the container grabbed by the spreader and the upper lock holes of the dropped container; or, in response to the execution instruction being a container dropping by a truck instruction, the position for collecting images of the lock holes is the lock holes of the container grabbed by the spreader and the lock holes of the truck.

[0009] In a possible implementation of the present application, in response to the execution instruction being a grab box instruction, the spreader of the straddle carrier is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: determining whether the distance between the spreader and the lower locking hole of the container is less than a preset distance value based on the three-dimensional space coordinates; if the distance between the spreader and the lower locking hole of the container is not less than the preset distance value, the spreader is moved horizontally, and it is determined whether the distance between the moved spreader and the lower locking hole of the container is less than the preset distance value; if the distance between the spreader and the lower locking hole of the container is less than the preset distance value, the straddle carrier is controlled to advance to a preset distance from the container, and the spreader is controlled to move to the top locking hole of the container to be locked.

[0010] In a possible implementation of the present application, if the distance between the spreader and the lower locking hole of the container is not less than the preset distance value, the spreader is moved horizontally, including: determining whether the lower locking hole of the container is left-biased; if the lower locking hole of the container is left-biased, the spreader is controlled to move leftward.

[0011] In a possible implementation of the present application, if the lower locking hole of the container is right-biased, the spreader is controlled to move rightward.

[0012] In a possible implementation of the present application, in response to the execution instruction being a flat ground box placing instruction, the spreader of the straddle carrier is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: determining whether the container grabbed by the spreader is the first container; if the container grabbed by the spreader is not the first container, the lower locking hole of the container grabbed by the spreader and the upper locking hole of the placed container are identified to obtain the offset distance of the spreader; based on the offset position of the spreader, the spreader is controlled to move to the upper locking hole of the placed container.

[0013] In a possible implementation of the present application, if the container grabbed by the spreader is the first container, the container is placed in a preset area.

[0014] In a possible implementation of the present application, in response to the execution instruction being a container truck box placing instruction, the spreader of the straddle carrier is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: obtaining the position of the container truck by collecting the position of the container truck; obtaining the relative offset distance between the locking hole of the container truck and the locking hole of the container grabbed by the spreader by identifying the positions of the locking holes; based on the position of the container truck, the straddle carrier is controlled to advance to the container truck, and based on the relative offset distance, the spreader is controlled to place the grabbed container.

[0015] In a possible implementation of the present application, the position of the container truck is obtained by collecting the position of the container truck, including: collecting the longitudinal position of the container truck based on the longitudinal line laser radar to obtain the longitudinal position of the container truck; collecting the transverse position of the container truck based on the transverse line laser radar to obtain the transverse position of the container truck; and integrating the longitudinal position and the transverse position to obtain the position of the container truck.

[0016] In one possible implementation of this application, before acquiring the three-dimensional spatial coordinates of the keyhole by capturing an image of the keyhole using a binocular camera, the process includes: acquiring light intensity from the environment where the binocular camera is located using a light sensor; acquiring humidity information from the environment where the binocular camera is located using a humidity sensor; obtaining environmental correction parameters by analyzing and processing the light intensity and humidity information; and performing parameter correction on the binocular camera based on the environmental correction parameters.

[0017] As a second aspect of this application, this application also provides a container keyhole recognition device, comprising: a recognition module, configured to, in response to receiving an execution command for the container, recognize the acquired container image using a pre-trained alignment model to obtain the container keyhole; a acquisition module, configured to acquire images of the keyhole using a binocular camera to obtain the three-dimensional spatial coordinates of the keyhole, wherein the binocular camera is adjustablely mounted on the spreader of the forklift, and the position for acquiring images of the keyhole changes based on the execution command; and a control module, configured to, based on the execution command and the three-dimensional spatial coordinates, control the spreader of the forklift to move to the keyhole.

[0018] In one possible implementation of this application, the acquisition module is configured to, in response to an execution instruction of grabbing a container, acquire an image of the lock hole at the lower lock hole of the container; or, in response to an execution instruction of placing a container on flat ground, acquire an image of the lock hole at the lower lock hole of the container grabbed by the spreader and the upper lock hole of the placed container; or, in response to an execution instruction of placing a container on a truck, acquire an image of the lock hole at the lock hole of the container grabbed by the spreader and the lock hole of the truck.

[0019] As a third aspect of this application, this application also provides a container keyhole recognition system, comprising: a controller and at least one sensor, at least one lidar and a binocular camera, wherein the controller is connected to the sensor, lidar and binocular camera respectively; the controller is used to perform the container keyhole recognition method as described in any of the above.

[0020] As a fourth aspect of this application, this application also provides an engineering vehicle, including: a lock hole recognition system for a container as described above.

[0021] As a fifth aspect of this application, this application also provides a computer-readable storage medium storing a computer program for performing the container keyhole identification method described above.

[0022] As a sixth aspect of this application, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the container keyhole recognition method described above.

[0023] The application provides a container lock hole identification method. In response to receiving an execution instruction for a container, a pre-trained positioning model is used to identify a collected container picture to obtain a lock hole of the container. A binocular camera is used to collect an image of the lock hole to obtain three-dimensional space coordinates of the lock hole. The binocular camera is adjustably mounted on a spreader of a straddle carrier, and the position for collecting the image of the lock hole changes based on the execution instruction. Based on the execution instruction and the three-dimensional space coordinates, the spreader of the straddle carrier is controlled to move to the lock hole. It is easy to note that the binocular vision collection is performed on the identified container lock hole, the three-dimensional space coordinates of different lock hole positions can be obtained based on different execution instructions, the lock hole position below the container can be identified when the container is placed, the placing operation is facilitated, the positioning accuracy of the lock hole of the container is improved, and the working efficiency of the straddle carrier is improved.

[0024] BRIEF DESCRIPTION OF DRAWINGS

[0025] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. The drawings provided in the specification and the embodiments of the application together with the detailed description serve to provide a thorough understanding of the application. The drawings included are for illustrative purposes and are not intended to limit the application. The same reference numerals in different drawings refer to the same or similar elements.

[0026] FIG. 1 is a flowchart of a container lock hole identification method provided by an embodiment of the present application.

[0027] FIG. 2 is a flowchart of model training provided by an embodiment of the present application.

[0028] FIG. 3 is a flowchart of model detection provided by an embodiment of the present application.

[0029] FIG. 4 is a flowchart of lock hole positioning provided by an embodiment of the present application.

[0030] FIG. 5 is a schematic diagram of binocular camera identification of a lock hole during a grabbing operation provided by an embodiment of the present application.

[0031] FIG. 6 is a schematic diagram of binocular camera identification of a lock hole during a placing operation provided by an embodiment of the present application.

[0032] FIG. 7 is a flowchart of a grabbing process of a straddle carrier provided by an embodiment of the present application.

[0033] FIG. 8 is a flowchart of a flat placing process of a straddle carrier provided by an embodiment of the present application.

[0034] Fig. 9 is a flow chart of a container placing process of the stacker crane according to an embodiment of the present application.

[0035] Fig. 10 is a structural diagram of the sensors according to an embodiment of the present application.

[0036] Fig. 11 is a schematic diagram of a lock hole identification device of the container according to an embodiment of the present application.

[0037] Fig. 12 is a schematic diagram of a lock hole identification system of the container according to an embodiment of the present application.

[0038] Fig. 13 is a device block diagram of the engineering vehicle according to an embodiment of the present application.

[0039] Implementation of the present application

[0040] In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited. All directional indications (such as upper, lower, left, right, front, back, top, bottom, etc.) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between the components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0041] In addition, the reference to "embodiments" in this document means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it mutually exclusive or alternative to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0042] Exemplary method

[0043] As a first aspect of the present application, the present application provides a lock hole identification method of a container. Fig. 1 is a flow chart of a lock hole identification method of a container according to an embodiment of the present application. As shown in Fig. 1, the method comprises the following steps:

[0044] S101, in response to receiving an execution instruction of the container, identifying the collected container picture by using a pre-trained positioning model to obtain the lock hole of the container.

[0045] Specifically, the execution instruction can be used to represent an instruction received by the stacker crane for handling the container. Generally, the execution instruction can be a box grabbing instruction, a box placing on the ground instruction, a truck box placing instruction, etc. The execution instruction is not specifically limited herein.

[0046] The pre-trained alignment model can be used to represent a model pre-trained for alignment recognition of the lock hole of the container. Since the lock hole alignment and the truck lock hole alignment are required in the present application, the recognition accuracy is required to be high. However, real-time recognition is also required, and the recognition speed is also required to be high. Therefore, YOLOv8s is selected as the training model.

[0047] Generally, the lock hole of the container usually refers to the lock installation position on the door of the container. During the handling of the container, the lock hole of the container is recognized and aligned, and the container is grabbed based on the position of the lock hole, which can improve the handling accuracy of the container.

[0048] Specifically, using YOLOv8s as the training model can be executed through the following steps: selecting a model and a data set, that is, selecting a YOLOv8 model configuration file (such as yolov8n.yaml) suitable for the task, and preparing a data set, including a division file of a training set and a validation set, and a category list; downloading a pre-trained model, that is, downloading a pre-trained model of a corresponding version from the GitHub open source website of YOLOv8; training the model, that is, using a command line tool or a Python API to train the model; model export, that is, after training is completed, the model can be exported to an ONNX format for deployment on different platforms, and the export of the model can be realized by using a Python script or a command line tool; model inference, that is, loading the exported ONNX model in the application, and using it for real-time target detection or inference of other computer vision tasks.

[0049] For example, as applied in the present application, as shown in FIG. 2, FIG. 2 is a flowchart of model training provided by an embodiment of the present application. The process of the model training includes: S201: data collection and standardization. Specifically, the side lock hole pictures of the container need to be collected, and the collected pictures need to be standardized; S202: network framework construction. Specifically, a preliminary training network is built; S203: parameter setting. Specifically, the initial parameters of the training network are set; S204: model training. Specifically, after the parameters are set, the model is trained using the standardized side lock hole pictures of the container; S205: actual detection. Specifically, the trained model is detected; S206: whether the accuracy is satisfied is judged. Specifically, whether the model training result is consistent with the actual calibration result is judged; if yes, it means that the accuracy of the model training reaches the preset accuracy, and the trained model is output (step S207); otherwise, if no, the parameters of the training network need to be set again (step S203), and the model training process is performed again.

[0050] Further, the actual detection process described above can be exemplarily illustrated by FIG. 3, FIG. 3 is a flowchart of model detection provided by an embodiment of the present application. As shown in FIG. 3, the process of model detection includes: S301: input image. Specifically, the collected side lock hole pictures of the container are input; S302: data enhancement and uniform size are performed on the image. Specifically, the input image is standardized; S303: network division. Specifically, the training network is divided into a backbone network and a neck network; S304: image features are extracted through the backbone network, and feature fusion is performed through the neck network to obtain a final feature map; S305: target bounding box is predicted. Specifically, the bounding box containing the lock hole is obtained by performing lock hole recognition on the feature map; S306: target class is predicted. Specifically, the target class and the corresponding lock hole are predicted; S307: confidence score is evaluated. Specifically, the prediction results of the target bounding box and the target class are evaluated in terms of confidence score, and the higher the score is, the more accurate the model training is; S308: whether the confidence score result is greater than a threshold is judged; if yes, it means that the model training is relatively accurate, and the target bounding box and the target class information are output (step S309); if no, it means that the model training result has not reached the preset accuracy, and the bounding box needs to be discarded (step S310).

[0051] The confidence score described above can use binary cross entropy (BCE) as a classification loss, judge "whether it is this class" for each class, and output the confidence score.

[0052] In addition, in the actual detection process, the detection program can be divided into four main parts according to the left and right cameras and the received instructions for grabbing or placing the container, i.e., left container grabbing, left container placing, right container grabbing, and right container placing.

[0053] In S102, the lock hole is imaged by a binocular camera to obtain three-dimensional spatial coordinates of the lock hole, wherein the binocular camera is adjustably mounted on a spreader of the reach stacker, and the position for imaging the lock hole changes based on an execution instruction.

[0054] Specifically, the binocular camera mentioned above, also known as a stereo camera or a dual-camera system, is a device that captures the same scene using two cameras. This system simulates the principle of human binocular vision to obtain three-dimensional spatial information. Each camera captures an image from a different angle, and then the depth information of the scene can be reconstructed by calculating the difference between the two images, called parallax.

[0055] In the embodiments of the present application, by mounting the binocular camera on the spreader of the reach stacker and adjusting the mounting position of the binocular camera, the reach stacker can recognize the lock hole of the container during the container grabbing or placing operation, ensuring that the view is not blocked.

[0056] In addition, the position for imaging the lock hole is different corresponding to different execution instructions for the container, i.e., the present application solves the problem of only recognizing the four lock holes above the container in the prior art.

[0057] In an optional embodiment, FIG. 4 shows a flowchart of lock hole positioning provided by an embodiment of the present application. As shown in FIG. 4, in the process of obtaining the three-dimensional spatial coordinates of the lock hole by imaging the lock hole with the binocular camera, after the binocular camera detects the position of the lock hole (step S401), the two-dimensional pixel points of the lock hole region can be mapped to the three-dimensional space (step S402) to obtain the lock hole depth information (step S403), so as to calculate the position of the lock hole relative to the binocular camera (step S404). By calibrating the binocular camera, the coordinates of the lock hole in the world coordinate system can be obtained (step S405), so as to control the spreader to drop the lock and realize the grabbing operation of the container.

[0058] In S103, based on the execution instruction and the three-dimensional spatial coordinates, the spreader of the reach stacker is controlled to move to the lock hole.

[0059] Specifically, after obtaining the three-dimensional spatial coordinates of the lock hole, the spreader of the reach stacker can be controlled to move to the lock hole based on different execution instructions.

[0060] For example, if the execution instruction is a grab box instruction, the position of the container lock hole is controlled to be the lower lock hole of the container to be grabbed, and the spreader of the container handler is controlled to be locked; if the execution instruction is a drop box instruction, the position of the container lock hole is controlled to be the lower lock hole of the container to be grabbed and the upper lock hole of the container to be dropped, and the spreader of the container handler is controlled to be lowered to drop the container, and the like.

[0061] The container lock hole identification method provided in the application comprises the following steps: in response to receiving an execution instruction for a container, a pre-trained alignment model is used to identify a container picture collected to obtain a container lock hole; a binocular camera is used to collect an image of the container lock hole to obtain a three-dimensional space coordinate of the container lock hole, wherein the binocular camera is adjustably installed on a spreader of a container handler, and the position for collecting the image of the container lock hole changes based on the execution instruction; and the spreader of the container handler is controlled to move to the container lock hole based on the execution instruction and the three-dimensional space coordinate. It is easy to note that, by using binocular vision to collect the identified container lock hole, the three-dimensional space coordinates of different lock hole positions can be obtained based on different execution instructions, so that the lower lock hole position can be identified when the container is dropped, which facilitates the execution of the drop box operation, improves the positioning accuracy of the container lock hole, and improves the work efficiency of the container handler. Thus, the technical problem of low positioning accuracy of the container lock hole and low work efficiency of the container handler is solved.

[0062] In a possible implementation of the application, the position for collecting the image of the container lock hole changes based on the execution instruction, which comprises the following steps: in response to the execution instruction being a grab box instruction, the position for collecting the image of the container lock hole is the lower lock hole of the container; or in response to the execution instruction being a drop box instruction on the ground, the position for collecting the image of the container lock hole is the lower lock hole of the container to be grabbed and the upper lock hole of the container to be dropped; or in response to the execution instruction being a container truck drop box instruction, the position for collecting the image of the container lock hole is the lock hole of the container to be grabbed and the lock hole of the container truck.

[0063] Specifically, in an optional embodiment, if the execution instruction is a grab box instruction, i.e., the spreader of the container handler needs to grab the container, the position for collecting the image of the container lock hole is the lower lock hole of the container. As shown in FIG. 5, the binocular camera collects the images of the two lock holes directly below the container to be grabbed, and at the same time, by calibrating the current position of the binocular camera, the three-dimensional coordinates of the two lock holes directly below the container to be grabbed can be obtained, and based on the three-dimensional coordinates, the spreader can be controlled to be locked, i.e., the grab box operation can be performed.

[0064] In another alternative embodiment, if the execution instruction is a flat ground placing instruction, i.e., the container gripped by the spreader of the stacker crane needs to be placed to a designated area, the positions for image acquisition of the locking holes are the lower locking hole of the gripped container and the upper locking hole of the placed container. FIG. 6 shows a schematic diagram of the binocular camera identifying the locking holes during the placing operation according to an embodiment of the present application. As shown in FIG. 6, the binocular camera acquires images of the lower locking hole of the gripped container and the upper locking hole of the placed container, and at the same time, the current position of the binocular camera is calibrated, and the three-dimensional coordinates of the lower locking hole of the gripped container and the upper locking hole of the placed container are identified, so that the flat ground placing operation of the container can be performed.

[0065] In a third alternative embodiment, if the execution instruction is a truck placing instruction, i.e., the container gripped by the spreader of the stacker crane needs to be placed on the truck, the positions for image acquisition of the locking holes are the locking hole of the gripped container and the locking hole of the truck. Generally, the truck placing is different from the flat ground placing, i.e., after the stacker crane completes the gripping operation, it needs to retreat a certain distance, and then the truck driver drives the truck to the place, and then the container is placed on the truck.

[0066] In a possible implementation manner of the present application, in response to the execution instruction being the gripping instruction, the spreader of the stacker crane is controlled to move to the locking hole based on the execution instruction and the three-dimensional space coordinates, including: judging whether the offset distance between the spreader and the lower locking hole of the container is less than a preset offset distance value based on the three-dimensional space coordinates; if the offset distance between the spreader and the lower locking hole of the container is not less than the preset offset distance value, the spreader is moved horizontally, and it is judged whether the offset distance between the moved spreader and the lower locking hole of the container is less than the preset offset distance value; if the offset distance between the spreader and the lower locking hole of the container is less than the preset offset distance value, the stacker crane is controlled to advance to a preset distance from the container, and the spreader is controlled to move to the top locking hole of the container for locking.

[0067] In a possible implementation manner of the present application, if the offset distance between the spreader and the lower locking hole of the container is not less than the preset offset distance value, the spreader is moved horizontally, including: judging whether the lower locking hole of the container is left-biased; if the lower locking hole of the container is left-biased, the spreader is controlled to move left; and if the lower locking hole of the container is right-biased, the spreader is controlled to move right.

[0068] Specifically, the above-mentioned preset offset distance value can be used to represent the preset offset distance of the center of the locking hole relative to the spreader, and the preset offset distance value is not specifically limited herein and can be adjusted according to actual conditions.

[0069] The above-mentioned preset distance can be used to represent the preset distance between the stacker crane and the container, and the spreader can be moved to the container at the preset distance, and the preset distance is not specifically limited herein and can be adjusted according to actual conditions.

[0070] In an optional embodiment, in response to the execution instruction being a container grabbing instruction, based on the execution instruction and the three-dimensional space coordinates, in the process of controlling the spreader of the straddle carrier to move to the locking hole, it can be determined whether the offset distance between the spreader and the lower locking hole of the container to be grabbed is less than a preset offset distance value based on the three-dimensional coordinates of the two side locking holes directly below the container to be grabbed. If it is less than the preset offset distance value, it indicates that the spreader is accurately aligned with the lower locking hole of the container to be grabbed, the straddle carrier can be controlled to advance to a preset distance from the container, and the spreader can be controlled to move to the lower locking hole of the container. On the contrary, if it is not less than the preset offset distance value, it indicates that the spreader is not accurately aligned with the lower locking hole of the container to be grabbed, and the spreader needs to be adjusted, that is, the spreader needs to be moved horizontally, and it is continuously determined whether the offset distance between the spreader after moving and the lower locking hole of the container is less than the preset offset distance value, and the iteration is continued until the offset distance between the spreader and the lower locking hole of the container is less than the preset offset distance value, the straddle carrier is controlled to advance, and the spreader is controlled to move to the top locking hole of the container to perform locking, and the container grabbing operation is completed.

[0071] For example, FIG. 7 shows a flowchart of the container grabbing process of the straddle carrier provided by an embodiment of the present application. As shown in FIG. 7, the container grabbing process includes the following operations:

[0072] S701: The straddle carrier completes alignment. Specifically, the straddle carrier and the container to be grabbed are aligned; S702: The laser radar detects the height and the number N of layers of the container. Specifically, the container is scanned by the longitudinal line laser radar installed on the straddle carrier, and the height and the number N of layers of the container are obtained; S703: The actual height H of the container is calculated according to the number N of layers, H=N*2.6 (m); S704: The spreader of the straddle carrier is lifted above the height H; S705: The camera performs locking hole recognition. Specifically, the lower locking hole of the container to be grabbed is identified by the binocular camera.

[0073] S706: It is determined whether the locking hole center offset distance B≤B0. Specifically, it is determined whether the offset distance between the spreader and the lower locking hole of the container to be grabbed is less than a preset offset distance value; wherein if the locking hole center offset distance B≤B0, the locking hole is controlled to be aligned left and right (step S707), that is, the spreader is controlled to be aligned with the two side locking holes directly below the container to be grabbed, the straddle carrier is controlled to advance to a distance D=D0 from the container (step S708), that is, the straddle carrier is controlled to advance to a preset distance from the container, and the straddle carrier is controlled to lock (step S709).

[0074] If the locking hole center offset distance B>0, step S710 is performed: it is determined whether the locking hole is left-biased; if the locking hole is left-biased, the spreader is controlled to move left by B (step S711); if the locking hole is right-biased, the spreader is controlled to move right by B (step S712). S713: The above locking hole recognition operation is continuously performed. Specifically, it is determined whether the locking hole center offset distance B≤B0, and the iteration is continued until the locking hole center offset distance B≤B0, and the container grabbing operation is completed.

[0075] In a possible implementation of the present application, in response to the execution instruction being a flat ground box placing instruction, based on the execution instruction and the three-dimensional space coordinates, the spreader of the straddle carrier is controlled to move to the locking hole, including: determining whether the container grasped by the spreader is the first container; if the container grasped by the spreader is not the first container, the lower locking hole of the container grasped by the spreader and the upper locking hole of the placed container are identified, and the offset distance of the spreader is obtained; based on the offset position of the spreader, the spreader is controlled to move to the upper locking hole of the placed container; and if the container grasped by the spreader is the first container, the container is placed in a preset area.

[0076] Specifically, the offset distance of the spreader can be used to represent the offset distance of the lower locking hole of the container grasped by the spreader relative to the upper locking hole of the placed container.

[0077] In an optional embodiment, in the process of controlling the spreader of the straddle carrier to move to the locking hole based on the execution instruction and the three-dimensional space coordinates in response to the execution instruction being a flat ground box placing instruction, it can be determined whether the container currently grasped by the spreader is the first container, if so, the container currently grasped can be placed in a designated area; otherwise, if the container currently grasped is not the first container, the lower locking hole of the container currently grasped and the upper locking hole of the placed container need to be identified, and the offset distance of the lower locking hole of the container grasped by the spreader relative to the upper locking hole of the placed container is obtained, the spreader is adjusted through the offset distance of the spreader, and after the lower locking hole of the container grasped and the upper locking hole of the placed container are aligned, the spreader is controlled to drop and place the container.

[0078] For example, FIG. 8 shows a flowchart of a flat ground box placing process of a straddle carrier provided by an embodiment of the present application. As shown in FIG. 8, the flat ground box placing process includes the following operations:

[0079] After the straddle carrier completes the container grasping operation (step S801), the path is planned by the back-stepping method (step S802), and the straddle carrier is controlled to travel along the path to the placing point (step S803); S804: determining whether the container grasped by the spreader is the first container; if so, the container grasped is placed in a designated area (step S805), and the path is returned to grasp the next container (step S806) and travel along the path to the direction point; if the container grasped by the spreader is not the first container, step S807 is executed: the laser radar detects the bottom container for alignment. Specifically, the placed bottom container is identified and aligned by the transverse line laser radar on the spreader; S808: the laser radar detects the container height H. Specifically, the height of the placed container is collected by the longitudinal line laser radar installed on the spreader.

[0080] S809: move the spreader up and down to a position higher than the height H of the detected container, control the reach stacker to advance to a position aligned with the front and back of the placed container (step S810); identify the lower locking hole of the current grabbed container and the upper locking hole of the placed container through the binocular camera on the spreader (step S811), and ensure that the lower locking hole of the grabbed container and the upper locking hole of the placed container are aligned left and right, then control the spreader to lower and place the container (step S812).

[0081] In a possible implementation of the present application, in response to the execution instruction being the container truck container placing instruction, based on the execution instruction and the three-dimensional space coordinates, the spreader of the reach stacker is controlled to move to the locking hole, including: obtaining the position of the container truck by collecting the position of the container truck; obtaining the relative offset distance of the locking hole of the container truck and the locking hole of the container grabbed by the spreader by identifying the locking hole of the container truck and the locking hole of the container grabbed by the spreader; based on the position of the container truck, controlling the reach stacker to advance to the container truck, and based on the relative offset distance, controlling the spreader to place the grabbed container.

[0082] In a possible implementation of the present application, the position of the container truck is obtained by collecting the position of the container truck, including: obtaining the longitudinal position of the container truck by collecting the longitudinal position of the container truck based on the longitudinal line laser radar; obtaining the transverse position of the container truck by collecting the transverse position of the container truck based on the transverse line laser radar; and integrating the longitudinal position and the transverse position to obtain the position of the container truck.

[0083] Specifically, the relative offset distance described above can be used to represent the offset distance of the locking hole of the container grabbed by the spreader relative to the locking hole of the container truck.

[0084] In an optional embodiment, in the process of controlling the spreader of the reach stacker to move to the locking hole in response to the execution instruction being the container truck container placing instruction, the position of the container truck can be collected by the laser radar installed on the spreader of the reach stacker to obtain the position of the container truck, after obtaining the position of the container truck, the reach stacker is controlled to advance to the container truck and align with the front and back of the container, and then the binocular camera is used to identify the locking hole of the container truck and the locking hole of the container grabbed by the spreader to obtain the offset distance of the locking hole of the container grabbed by the spreader relative to the locking hole of the container truck, the spreader is adjusted through the relative offset distance until the lower locking hole of the container grabbed by the spreader is aligned left and right with the locking hole of the container truck, and then the spreader is controlled to lower and place the container.

[0085] For example, FIG. 9 shows a flowchart of the container truck container placing process of the reach stacker according to an embodiment of the present application. As shown in FIG. 9, the container truck container placing process includes the following operations:

[0086] After the stacker completes the box grabbing operation (step S901), the stacker is controlled to retreat (step S902), and the truck driver drives to the front of the stacker (step S903); S904: the laser radar detects the position and distance of the truck. Specifically, the position and distance of the truck are collected through the lateral line laser radar installed on the spreader, and then the path is planned through the backstepping method (step S905), that is, after the position and distance of the truck are determined, the path for the stacker to advance to the truck is planned.

[0087] S906: the laser radar detects the height of the truck. Specifically, the height of the truck is collected through the longitudinal line laser radar installed on the spreader, and the spreader is controlled to move up and down to a position above the height of the truck (step S907); the stacker is controlled to drive along the planned path to the front of the truck (step S908), and the spreader is controlled to align with the front and back of the container on the truck (step S909); S910: the camera identifies the left-right alignment of the upper and lower lock holes. Specifically, the lock holes of the truck and the container grabbed by the spreader are identified by the binocular camera to make the lower lock hole of the container grabbed by the spreader left-right aligned with the lock hole of the truck, and then the spreader is controlled to descend to complete the box placing operation (step S911).

[0088] It should be noted that when the laser radar detects the position of the truck, the alignment and height detection are first completed. The two laser radars on both sides of the spreader vertically scan to determine the height, angle and distance of the truck, the laser radar in front horizontally scans to determine the distance, and the binocular cameras on both sides scan whether the left and right lock holes of the container and the truck are aligned. Then the left and right positions and the height of the spreader are adjusted during the advance of the stacker, and finally the lock hole alignment is completed after adjusting the front and back distance, and the spreader is moved downward to complete the box placing.

[0089] In a possible implementation of the present application, before the three-dimensional space coordinates of the lock hole are obtained through the binocular camera, the method comprises: collecting light intensity of an environment in which the binocular camera is located by using a light sensor; collecting humidity information of the environment in which the binocular camera is located by using a humidity sensor; obtaining an environment correction parameter by analyzing and processing the light intensity and the humidity information; and correcting parameters of the binocular camera based on the environment correction parameter.

[0090] Specifically, the above-mentioned environment correction parameter can be used to correct the parameters of the binocular camera based on the light intensity and the humidity information of the environment.

[0091] In an optional embodiment, considering the container operation scene, it is necessary to test the detection effect in different environments. Therefore, the present application exemplarily gives the detection effect in the daytime and at night, and the detection effect in sunny and rainy days, so that the camera parameters can be adjusted to adapt to different environments.

[0092] Specifically, the light sensor is used to collect light in the environment where the binocular camera is located, and light intensity is obtained to distinguish the detection effect in the daytime and the nighttime. The humidity sensor is used to collect humidity in the environment where the binocular camera is located, and humidity information is obtained to distinguish the detection effect in the sunny day and the rainy day. The light intensity and the humidity information are analyzed and processed to obtain the environment correction parameter, and then the binocular camera is corrected based on the environment correction parameter, so that the camera parameter is adjusted to improve the lock hole recognition effect of the container.

[0093] For example, FIG. 10 shows a structure diagram of the sensors according to an embodiment of the present application. As shown in FIG. 10, the industrial computer 105 is connected with the humidity sensor 106 and the light sensor 107, respectively, to receive the humidity information collected by the humidity sensor 106 and the light intensity collected by the light sensor 107. The industrial computer 105 analyzes and processes the humidity information and the light intensity to obtain the environment correction parameter, and corrects the binocular camera 108 based on the environment correction parameter.

[0094] Exemplary device

[0095] As a second aspect of the present application, the present application also provides a lock hole recognition device for a container. FIG. 11 shows a schematic diagram of a lock hole recognition device for a container according to an embodiment of the present application. As shown in FIG. 11, the device includes:

[0096] The recognition module 1101 is configured to recognize the collected container picture by using the pre-trained alignment model to obtain the lock hole of the container in response to receiving the execution instruction for the container. The collection module 1102 is configured to collect the image of the lock hole by using the binocular camera to obtain the three-dimensional space coordinates of the lock hole, wherein the binocular camera is adjustably installed on the spreader of the container crane, and the position for collecting the image of the lock hole changes based on the execution instruction. The control module 1103 is configured to control the spreader of the container crane to move to the lock hole based on the execution instruction and the three-dimensional space coordinates.

[0097] Optionally, the control module 1103 includes a first judgment module configured to judge whether the offset distance between the spreader and the lower lock hole of the container is less than a preset offset distance value based on the three-dimensional space coordinates, a moving module configured to move the spreader horizontally if the offset distance between the spreader and the lower lock hole of the container is not less than the preset offset distance value, and judge whether the offset distance between the moved spreader and the lower lock hole of the container is less than the preset offset distance value, and a control advancing module configured to control the container crane to advance to a preset distance from the container and control the spreader to move to the top lock hole of the container for locking if the offset distance between the spreader and the lower lock hole of the container is less than the preset offset distance value.

[0098] Optionally, the moving module 1103 comprises: a left deviation judging module, configured to judge whether the lower locking hole of the container is deviated to the left; a left moving module, configured to control the spreader to move to the left if the lower locking hole of the container is deviated to the left; and a right moving module, configured to control the spreader to move to the right if the lower locking hole of the container is deviated to the right.

[0099] Optionally, the control module 1103 further comprises: a second judging module, configured to judge whether the container gripped by the spreader is the first container; a first alignment identifying module, configured to, if the container gripped by the spreader is not the first container, identify the lower locking hole of the container gripped by the spreader and the upper locking hole of the placed container to obtain the offset distance of the spreader; a control moving module, configured to control the spreader to move to the upper locking hole of the placed container based on the offset position of the spreader; and a control placing module, configured to, if the container gripped by the spreader is the first container, place the container to the preset area.

[0100] Optionally, the control module 1103 further comprises: a position collecting module, configured to obtain the position of the truck by collecting the position of the truck; a second alignment identifying module, configured to identify the relative offset distance between the locking hole of the truck and the locking hole of the container gripped by the spreader by identifying the locking hole of the truck and the locking hole of the container gripped by the spreader; a control advancing module, configured to control the reach stacker to advance to the truck based on the position of the truck, and control the spreader to place the gripped container based on the relative offset distance.

[0101] Optionally, the position collecting module comprises: a longitudinal collecting module, configured to obtain the longitudinal position of the truck by collecting the position of the truck in the longitudinal direction based on the longitudinal line laser radar; a transverse collecting module, configured to obtain the transverse position of the truck by collecting the position of the truck in the transverse direction based on the transverse line laser radar; and an integrating module, configured to integrate the longitudinal position and the transverse position to obtain the position of the truck.

[0102] Optionally, the device further comprises: a light collecting module, configured to collect the light intensity of the environment in which the binocular camera is located by using the light sensor; a humidity collecting module, configured to collect the humidity information of the environment in which the binocular camera is located by using the humidity sensor; an analyzing module, configured to analyze and process the light intensity and the humidity information to obtain the environmental correction parameter; and a correction module, configured to correct the parameters of the binocular camera based on the environmental correction parameter.

[0103] The container lock hole identification device provided by the embodiment comprises a lock hole identification module, a lock hole coordinate collection module, and a control module.

[0104] Exemplary system

[0105] As a third aspect of the present application, the present application also provides a container lock hole identification system. FIG. 12 shows a schematic diagram of the container lock hole identification system provided by an embodiment of the present application. As shown in FIG. 12, the container lock hole identification system comprises a controller 121 and at least one sensor 122, at least one laser radar 123, and a binocular camera 124, wherein the controller 121 is connected with the sensor 122, the laser radar 123, and the binocular camera 124 respectively; the controller 121 is configured to execute the container lock hole identification method according to any one of the above.

[0106] Exemplary engineering vehicle

[0107] As a fourth aspect of the present application, the present application also provides an engineering vehicle, comprising the container lock hole identification system according to the above.

[0108] FIG. 13 shows a device block diagram of the engineering vehicle provided by an embodiment of the present application.

[0109] As shown in FIG. 13, the engineering vehicle comprises one or more processors 131 and a memory 132.

[0110] The processor 131 can be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction executing capabilities, and can control other components in the engineering vehicle to perform desired functions.

[0111] The memory 132 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor can execute the program instructions to implement the container hole identification method of the container of the various embodiments of the present application and / or other desired functions. Various contents such as input signals, signal components, noise components, and the like can also be stored in the computer-readable storage media.

[0112] In one example, the engineering vehicle can further include an input device 133 and an output device 134, which are interconnected through a bus system and / or other forms of connection mechanism (not shown).

[0113] When the engineering vehicle is a single machine device, the input device 133 can be a communication network connector for receiving the collected input signals from the first device and the second device.

[0114] In addition, the input device 133 can also include, for example, a keyboard, a mouse, and the like.

[0115] The output device 134 can output various information to the outside, including the determined distance information, direction information, and the like. The output device 134 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0116] Of course, for simplicity, only some of the components in the engineering vehicle related to the present application are shown in FIG. 13, and components such as buses, input / output interfaces, and the like are omitted. In addition, the engineering vehicle can also include any other appropriate components according to specific application cases.

[0117] Exemplary computer-readable storage medium

[0118] As a fifth aspect of the present application, the present application provides a computer-readable storage medium, which stores a computer program for performing the following steps:

[0119] S1, in response to receiving an execution instruction for the container, identifying the collected container picture using a pre-trained positioning model to obtain the lock hole of the container;

[0120] S2, image acquisition of the lock hole is performed by a binocular camera, and three-dimensional space coordinates of the lock hole are obtained, wherein the binocular camera is adjustably mounted on a spreader of the stacker, and a position for image acquisition of the lock hole is changed based on an execution instruction;

[0121] S3, based on the execution instruction and the three-dimensional space coordinates, the spreader of the stacker is controlled to move to the lock hole.

[0122] In addition to the above method and device, an embodiment of the present application can also be a computer program product, which includes computer program information, and the computer program information causes a processor to execute steps in the container lock hole identification method according to various embodiments of the present application described in the specification when the computer program information is run by the processor.

[0123] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including an object-oriented programming language, such as Java, C++, etc., and a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] In addition, an embodiment of the present application can also be a computer readable storage medium, which stores computer program information, and the computer program information causes a processor to execute steps in the container lock hole identification method according to various embodiments of the present application described in the specification when the computer program information is run by the processor.

[0125] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or instrument, or any combination of the above. More specific examples (non-exhaustive list) of readable storage medium include: electrical connection with one or more conductive wires, portable disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0126] The basic principles of the application are described above in connection with specific embodiments, but it should be noted that the advantages, benefits, effects and the like mentioned in the present application are only examples and are not limiting, and it cannot be considered that these advantages, benefits, effects and the like are necessarily possessed by each embodiment of the present application. In addition, the specific details of the above disclosure are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present application to necessarily adopt the above specific details to realize.

[0127] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration as shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, mean "include but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0128] It should also be noted that in the devices, equipment and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions of the present application.

Claims

1. A method of identifying a locking hole of a container, characterized by, The method comprises the following steps: In response to receiving the execution instruction of the container, a pre-trained alignment model is used to identify the collected container picture to obtain the lock hole of the container; An image of the lock hole is collected by a binocular camera to obtain the three-dimensional spatial coordinates of the lock hole, wherein the binocular camera is adjustably mounted on a spreader of a straddle carrier, and the position for collecting the image of the lock hole changes based on the execution instruction; Based on the execution instruction and the three-dimensional spatial coordinates, the spreader of the straddle carrier is controlled to move to the lock hole.

2. The container hole identification method according to claim 1, wherein The position for collecting the image of the lock hole changes based on the execution instruction, which comprises: In response to the execution instruction being a grab box instruction, the position for collecting the image of the lock hole is the lower lock hole of the container; or, In response to the execution instruction being a flat ground box instruction, the position for collecting the image of the lock hole is the lower lock hole of the container grabbed by the spreader and the upper lock hole of the placed container; or, In response to the execution instruction being a truck box instruction, the position for collecting the image of the lock hole is the lock hole of the container grabbed by the spreader and the truck lock hole.

3. The container's lock hole identification method according to claim 2, in response to the execution instruction being a grab box instruction, characterized in that, Based on the execution instruction and the three-dimensional spatial coordinates, the spreader of the straddle carrier is controlled to move to the lock hole, which comprises: Based on the three-dimensional spatial coordinates, it is determined whether the offset distance between the spreader and the lower lock hole of the container is less than a preset offset distance value; If the offset distance between the spreader and the lower lock hole of the container is not less than the preset offset distance value, the spreader is moved horizontally, and it is determined whether the offset distance between the moved spreader and the lower lock hole of the container is less than the preset offset distance value; If the offset distance between the spreader and the lower lock hole of the container is less than the preset offset distance value, the straddle carrier is controlled to advance to a preset distance from the container, and the spreader is controlled to move to the top lock hole of the container for locking.

4. The container hole identification method according to claim 3, wherein If the offset distance between the spreader and the lower lock hole of the container is not less than the preset offset distance value, the spreader is moved horizontally, which comprises: It is determined whether the lower lock hole of the container is offset to the left; If the lower lock hole of the container is offset to the left, the spreader is controlled to move to the left.

5. The container hole identification method according to claim 4, wherein Further comprising: If the lower lock hole of the container is offset to the right, the spreader is controlled to move to the right.

6. The container hole identification method according to claim 2, in response to the execution instruction being a flat ground drop instruction, characterized in that, Based on the execution instruction and the three-dimensional spatial coordinates, the spreader of the straddle carrier is controlled to move to the lock hole, which comprises: It is determined whether the container grabbed by the spreader is the first container; If the container grabbed by the spreader is not the first container, the lower lock hole of the container grabbed by the spreader and the upper lock hole of the placed container are identified to obtain the offset distance of the spreader; Based on the offset position of the spreader, the spreader is controlled to move to the upper lock hole of the placed container.

7. The container hole identification method according to claim 6, wherein Further comprising: If the container grabbed by the spreader is the first container, the container is placed in a preset area.

8. The container's lock hole identification method according to claim 2, in response to the execution instruction being a truck drop-off container instruction, characterized in that, Based on the execution instruction and the three-dimensional spatial coordinates, the spreader of the straddle carrier is controlled to move to the lock hole, which comprises: The position of the truck is collected to obtain the position of the truck; The relative offset distance of the container lock hole of the truck and the container lock hole of the container gripped by the spreader is obtained by aligning and identifying the container lock hole of the truck and the container lock hole of the container gripped by the spreader; Based on the position of the truck, the reacher of the reach stacker is controlled to advance to the truck, and based on the relative offset distance, the spreader is controlled to place the gripped container.

9. The container lock hole identification method of claim 8, wherein the position of the truck is obtained by collecting the position of the truck, comprising: The longitudinal position of the truck is obtained by collecting the longitudinal position of the truck based on the longitudinal line laser radar; The lateral position of the truck is obtained by collecting the lateral position of the truck based on the lateral line laser radar; The position of the truck is obtained by integrating the longitudinal position and the lateral position.

10. The container hole identification method according to any one of claims 1 to 9, characterized by, Before the three-dimensional space coordinates of the container lock hole are obtained by collecting the image of the container lock hole by the binocular camera, comprising: The light intensity is obtained by collecting the light of the environment where the binocular camera is located by the light sensor; The humidity information is obtained by collecting the humidity of the environment where the binocular camera is located by the humidity sensor; The environment correction parameter is obtained by analyzing and processing the light intensity and the humidity information; The binocular camera is parameter corrected based on the environment correction parameter.

11. A lock hole recognition system of a container, characterized by, Comprising: A controller and at least one sensor, at least one laser radar and a binocular camera, wherein, The controller is connected with the sensor, the laser radar and the binocular camera respectively; The controller is used to execute the container lock hole identification method of any one of claims 1 to 10.

12. An engineering vehicle characterized by, Comprising: The container lock hole identification system of claim 11.

13. A locking hole recognition device of a container, characterized by, Comprising: An identification module, configured to identify the container picture collected by using the pre-trained alignment model in response to receiving the execution instruction of the container, to obtain the container lock hole of the container; A collection module, configured to collect the image of the container lock hole by the binocular camera to obtain the three-dimensional space coordinates of the container lock hole, wherein the binocular camera is adjustably installed on the spreader of the reach stacker, and the position for collecting the image of the container lock hole is changed based on the execution instruction; A control module, configured to control the spreader of the reach stacker to move to the container lock hole based on the execution instruction and the three-dimensional space coordinates.

14. The container hole identification method according to claim 13, wherein The collection module is configured to, in response to the execution instruction being a container gripping instruction, the position for collecting the image of the container lock hole being the lower container lock hole of the container; Or, in response to the execution instruction being a container placing instruction on the ground, the position for collecting the image of the container lock hole being the lower container lock hole of the container gripped by the spreader and the upper container lock hole of the placed container; or in response to the execution instruction being a truck container placing instruction, the position for collecting the image of the container lock hole being the container lock hole of the container gripped by the spreader and the truck container lock hole.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the container lock hole identification method of any one of claims 1 to 10.

Images