Box grab control method, device, system and yard bridge automation system, crane

Abstract

The present application relates to a container grabbing control method, device, system, yard bridge automation system, and crane. The container grabbing control method is applied to a container grabbing control system, which includes multiple cameras evenly distributed around the spreader. The multiple cameras are used to collect sub-images of all the keyholes of the container, and the keyholes corresponding to each camera are different. The method includes: obtaining a target image; the target image is obtained by splicing the sub-images; identifying each target keyhole in the target image, and detecting whether the target keyhole meets the preset position; the preset position includes: the perpendicular bisector of the line connecting the designated positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image; if the target keyhole does not meet the preset position, adjust the position of the spreader so that the target keyhole meets the preset position. In this way, the influence of the changing environment at the work site is avoided, and there is no need for repeated calibration, ensuring that the spreader can accurately and efficiently grab the container in the yard.

Description

Technical Field

[0001] The present application relates to the technical field of operating machinery, and in particular to a box grabbing control method, device, system, field bridge automation system, and crane. Background Art

[0002] With the rapid growth of container throughput at ports, large-scale mechanized loading and unloading operations are becoming more frequent. During these mechanized loading and unloading operations, how to accurately grab containers in the yard has always been a key concern.

[0003] However, existing cranes often struggle with low efficiency and accuracy when grasping containers in storage yards due to the ever-changing operating environment. For example, radar-based container calibration and grasping require repeated calibration whenever the hardware environment changes, resulting in low grasping efficiency and accuracy. Therefore, finding a way to accurately and efficiently grasp target containers in storage yards is an urgent issue. Summary of the Invention

[0004] In view of this, the present application provides a container grabbing control method, device, system, yard crane automation system, and crane, which can accurately and efficiently grab containers in the yard.

[0005] To achieve the above objectives, this application adopts the following technical solutions:

[0006] A first aspect of the present application provides a container grabbing control method, which is applied to a container grabbing control system. The container grabbing control system includes a plurality of cameras uniformly distributed around a spreader, the plurality of cameras being used to capture sub-images of all keyholes of a container, and the keyholes corresponding to the cameras are different. The method includes:

[0007] Acquire a target image; the target image is obtained by stitching together the sub-images;

[0008] Identify each target keyhole in the target image and detect whether the target keyhole meets a preset position; the preset position includes: a perpendicular bisector of a line connecting designated positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image;

[0009] If the target lock hole does not meet the preset position, the position of the sling is adjusted so that the target lock hole meets the preset position.

[0010] Optionally, adjusting the position of the spreader includes:

[0011] In the same coordinate system, respectively determining the coordinate positions of the target keyhole in the target image and the center point of the target image;

[0012] determining adjustment information based on a relative positional relationship between the target lock hole and the center point, the adjustment information including an adjustment distance and an adjustment direction;

[0013] The position of the spreader is adjusted according to the adjustment direction and the adjustment distance.

[0014] Optionally, identifying each target keyhole in the target image includes:

[0015] Inputting the target image into a pre-built keyhole detection model to obtain all keyholes in the target image; wherein the keyhole detection model takes the keyhole image as input, the keyholes as detection targets, and the corresponding keyhole images with labeled boxes as output, and is obtained by training a neural network;

[0016] The target keyhole is determined from all keyholes in the target image.

[0017] Optionally, determining the target keyhole from all keyholes in the target image includes:

[0018] Determining the distance between each keyhole in the target image and the center point of the target image;

[0019] In each sub-image of the target image, the keyhole with the shortest distance from the center point is determined as the target keyhole.

[0020] Optionally, the method for constructing the keyhole detection model includes:

[0021] Obtaining an original data set and annotating the keyholes in the original data set to obtain a training data set;

[0022] The deep learning neural network model is trained using the training data in the training data set to obtain the keyhole detection model.

[0023] Optionally, after detecting whether the target keyhole meets a preset position, the method further includes:

[0024] If the target lock hole meets the preset position, the container is grabbed based on the current position of the spreader.

[0025] A second aspect of the present application provides a box grabbing control device, comprising:

[0026] An acquisition module, configured to acquire a target image; the target image is obtained by splicing the sub-images;

[0027] an identification and detection module, configured to identify each target keyhole in the target image and detect whether the target keyhole satisfies a preset position; the preset position comprising: a perpendicular bisector of a line connecting the designated positions of the target keyholes in any two adjacent sub-images passing through the center point of the target image;

[0028] The adjustment module is used to adjust the position of the sling if the target lock hole does not meet the preset position, so that the target lock hole meets the preset position.

[0029] A third aspect of the present application provides a box grabbing control system, comprising a control device and a plurality of cameras;

[0030] The multiple cameras are used to be evenly distributed around the spreader to capture sub-images of all the keyholes of the container, and the keyholes corresponding to the cameras are different from each other;

[0031] The control device is used to: obtain a target image; the target image is obtained by splicing the sub-images; identify each target keyhole in the target image, and detect whether the target keyhole meets a preset position; the preset position includes: the perpendicular bisector of the line connecting the specified positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image; if the target keyhole does not meet the preset position, adjust the position of the sling so that the target keyhole meets the preset position.

[0032] A fourth aspect of the present application provides a field crane automation system, comprising: a box grab control system as described in the third aspect of the present application.

[0033] The fifth aspect of the present application provides a crane, comprising: a crane body and the field crane automation system as described in the fourth aspect of the present application.

[0034] The technical solution provided by this application may have the following beneficial effects:

[0035] In the solution of the present application, a container grabbing control method is applied to a container grabbing control system, which includes a plurality of cameras evenly distributed around a spreader, and the plurality of cameras are used to capture sub-images of all the keyholes of the container, and the keyholes corresponding to the cameras are different from each other. Based on this, a target image can be first obtained, wherein the target image is obtained by splicing the sub-images, that is, the target image includes all the keyholes of the container. Then, each target keyhole in the target image is identified, and by detecting whether the target keyhole meets a preset position, it can be detected whether the current position of the spreader accurately corresponds to the position of the container, wherein the preset position includes: the perpendicular bisector of the line connecting the specified positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image. If the position of the target keyhole does not meet the preset position, it means that the current position of the spreader does not correspond to the position of the container, and the position of the spreader can be adjusted so that the target keyhole meets the preset position, so that the adjusted position of the spreader accurately corresponds to the position of the container. In this way, the influence of the changing environment at the work site can be avoided. During the process of grabbing the container, the position of the spreader can be adjusted in real time based on the position of the lock hole on the container. In addition, there is no need to repeatedly calibrate the container, which can effectively save grabbing time and ensure that the spreader can grab the container accurately and efficiently in the yard. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 This is a flowchart of a box grabbing control method provided by an embodiment of the present application.

[0038] Figure 2 This is a top view of a container provided in one embodiment of the present application.

[0039] Figure 3 yes Figure 2 The target image corresponding to the container shown.

[0040] Figure 4 This is an output result diagram obtained by inputting a target image into a keyhole detection model provided by an embodiment of the present application.

[0041] Figure 5 This is an output result diagram obtained by inputting another target image into a keyhole detection model provided by an embodiment of the present application.

[0042] Figure 6This is a schematic diagram of a coordinate system of a target keyhole provided in one embodiment of the present application.

[0043] Figure 7 This is a structural diagram of a box grabbing control device provided in one embodiment of the present application.

[0044] Figure 8 This is a schematic structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be described in detail below. Obviously, the embodiments described are only some of the embodiments of this application, rather than all of them. Based on the embodiments in this application, all other implementation methods obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0046] In existing container handling conditions, how to accurately grab containers from a storage yard has always attracted much attention. To this end, an embodiment of the present application provides a container grabbing control method that can accurately grab containers in a storage yard by adjusting the position of a spreader.

[0047] During implementation, the container grabbing control method can be applied to a container grabbing control system, which may include multiple cameras evenly distributed around the spreader. The multiple cameras are used to collect sub-images of all keyholes of the container, and the keyholes corresponding to each camera are different.

[0048] Specifically, the number of cameras can be four. When evenly distributed around the spreader, one camera can be placed at the same position at each of the four corners of the spreader's ground-facing side, with each camera's shooting angle perpendicular to the ground. This way, when the spreader is moved over a container to grab it, each camera's shooting area will contain one of the container's keyholes. In other words, the four sub-images captured by the four cameras will capture all four of the container's keyholes. Furthermore, when the spreader's four corners align with the container's four keyholes, the spreader's position accurately corresponds to the container's position. At this point, lowering the spreader to grab the container allows for accurate and rapid capture.

[0049] The following takes the case where there are four cameras, that is, the number of sub-images acquired by the box grabbing control system each time is four, as an example to explain the box grabbing control method in detail:

[0050] like Figure 1 As shown, the box grabbing control method may include the following steps:

[0051] S101, obtaining a target image; the target image is obtained by stitching together the sub-images.

[0052] In practice, before acquiring the target image, the sub-images of the target image must first be acquired, that is, the sub-images of the keyholes of the container. After acquiring the sub-images, the sub-images can be spliced ​​according to the positions of the keyholes on the container to obtain the target image.

[0053] For example, the four lock holes of a container are A, B, C and D, and their corresponding sub-images are a, b, c and d respectively. Figure 2 As shown in the figure, the four lock holes A, B, C and D of the container are arranged in a clockwise direction. When stitching the target image, Figure 3 As shown, sub-images a, b, c, and d are also arranged in a clockwise order. Thus, the arrangement relationship between the lock holes in the acquired target image is the same as the arrangement relationship between the lock holes in the container.

[0054] It's important to understand that the target image reflects the container as seen by the spreader. In other words, the positional relationship between the keyholes in the target image reflects the positional relationship between the spreader and the container. When the four corners of the spreader correspond to the four keyholes of the container, connecting the designated positions of the four keyholes of the container in the target image will form a rectangle, with the center point of the rectangle also being the center point of the target image.

[0055] The designated position of the keyhole can be set according to actual needs. For example, the designated position can be the center point position, or the position of the keyhole vertex in the target image, etc., which is not limited here. However, it should be noted that the designated position of each keyhole is a unified designated position.

[0056] S102, identifying each target keyhole in the target image, and detecting whether the target keyhole meets a preset position; the preset position includes: the perpendicular bisector of the line connecting the designated positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image.

[0057] The target keyhole is the keyhole of the container to be captured.

[0058] Since a large number of containers are usually stored in a yard and the containers are stored densely, it is easy for the collected sub-image to contain not only the keyhole of the container to be grabbed, but also the keyholes of the surrounding containers. Therefore, after obtaining the target image, it is necessary not only to identify the keyhole in the target image, but also to identify the target keyhole of the container to be grabbed from the identified keyholes to ensure the accuracy of the container grabbing operation.

[0059] After identifying all target lock holes of the container to be grabbed in the target image, it is possible to determine whether the position of the spreader accurately corresponds to the position of the container to be grabbed by detecting the positional relationship between the target lock holes in the target image, that is, detecting whether the target lock holes meet the preset positions.

[0060] During implementation, if the target keyhole meets the preset position—that is, if the perpendicular bisector of the line connecting the designated positions of the target keyholes in any two sub-images passes through the center point of the target image—then connecting the designated positions of each target keyhole in the target image sequentially forms a rectangle whose center point is the center point of the target image. This means that the current spreader position accurately corresponds to the position of the container to be grasped. Thus, the container can be grasped based on the current spreader position.

[0061] S103: If the target lock hole does not meet the preset position, adjust the position of the spreader so that the target lock hole meets the preset position.

[0062] If the target lock hole does not meet the preset position, it means that the current position of the spreader does not accurately correspond to the position of the container to be grabbed. Then, based on the positional relationship between the target lock holes in the target image, the position of the spreader can be adjusted so that the target lock hole meets the preset position.

[0063] Specifically, after adjusting the spreader's position, the process returns to step S101 to recapture the target image and inspect the target keyhole within the target image. If the target keyhole meets the preset position, the process ceases adjusting the spreader's position; if the target keyhole still does not meet the preset position, the process returns to step S101. This cyclic adjustment method allows for real-time adjustment of the spreader's position, ensuring that the spreader's position always accurately corresponds to the container's position during the container grabbing process, ensuring accurate grabbing while avoiding additional work caused by misaligned positions and improving container grabbing efficiency.

[0064] In this embodiment, a container grabbing control method is applied to a container grabbing control system, which includes multiple cameras evenly distributed around a spreader. The multiple cameras are used to capture sub-images of all keyholes of a container, and the keyholes corresponding to the cameras are different from each other. Based on this, a target image can be first obtained, wherein the target image is obtained by splicing the sub-images, that is, the target image includes all keyholes of the container. Then, each target keyhole in the target image is identified, and by detecting whether the target keyhole meets a preset position, it can be detected whether the current position of the spreader and the position of the container accurately correspond. The preset position includes: the perpendicular bisector of the line connecting the specified positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image. If the position of the target keyhole does not meet the preset position, it means that the current position of the spreader does not correspond to the position of the container. The position of the spreader can then be adjusted so that the target keyhole meets the preset position, so that the adjusted position of the spreader accurately corresponds to the position of the container. In this way, the influence of the changing environment at the work site can be avoided. During the process of grabbing the container, the position of the spreader can be adjusted in real time based on the position of the lock hole on the container. In addition, there is no need to repeatedly calibrate the container, which can effectively save grabbing time and ensure that the spreader can grab the container accurately and efficiently in the yard.

[0065] In some embodiments, when identifying each target keyhole in a target image, the target image can be first input into a pre-built keyhole detection model to obtain all keyholes in the target image; wherein the keyhole detection model uses the keyhole image as input and the corresponding keyhole image with annotated boxes as output, and is obtained by training a neural network; then, the target keyhole is determined from all keyholes in the target image.

[0066] Specifically, a deep learning neural network model is trained with a keyhole image as input, a keyhole as the detection target, and a corresponding keyhole image with an annotated box as output to obtain a keyhole detection model. Based on this keyhole detection model, keyhole image features can be extracted from the target image to accurately locate the keyhole. This effectively improves the accuracy of detection without the need for feature engineering. For example, Figure 4 As shown in the figure, it is the output result after inputting a target image into the keyhole detection model. Figure 4 As can be seen from the figure, there are four keyholes of one type detected, one keyhole in each sub-image, and the identified keyholes are marked with a box. Figure 5 As shown in the figure, it is the output result after inputting another target image into the keyhole detection model. Figure 5 As can be seen from the figure, there are a total of ten keyholes detected, and all ten keyholes are marked with boxes. Among them, there are two keyholes in each of the two sub-images on the left, and three keyholes in each of the two sub-images on the right.

[0067] During implementation, the method for constructing a keyhole detection model may specifically include: obtaining an original data set and marking the keyholes in the original data set to obtain a training data set; using the training data in the training data set to train a deep learning neural network model to obtain a keyhole detection model.

[0068] In practice, a video of container grabbing at a container yard can be recorded and then parsed frame by frame to obtain an initial dataset. To improve training accuracy, this initial dataset can be cleaned to remove images not showing container grabbing conditions. This yields an original dataset without keyhole annotations. The keyholes in the original dataset are then annotated to create a training dataset. The neural network model is trained using the data in the training dataset to obtain a keyhole detection model.

[0069] like Figure 5 As shown, the target image also includes the keyholes of containers surrounding the container to be grasped. To prevent the keyholes of containers surrounding the container to be grasped in the sub-image from interfering with the container grasping process, in some embodiments, after identifying all keyholes in the target image, it is necessary to further determine the target keyhole from among all keyholes. Specifically, when determining the target keyhole from among all keyholes in the target image, the distance between each keyhole in the target image and the center point of the target image can be determined. Then, in each sub-image of the target image, the keyhole with the shortest distance to the center point is determined as the target keyhole. This effectively eliminates keyholes in the target image that are not containers to be grasped, providing an effective reference for accurate container grasping.

[0070] In some embodiments, in order to ensure that the position of the spreader accurately corresponds to the position of the container, when adjusting the position of the spreader, the coordinate positions of the target keyhole in the target image and the center point of the target image can be determined separately in the same coordinate system; then, based on the relative position relationship between the target keyhole and the center point, the adjustment information is determined, and the adjustment information includes the adjustment distance and the adjustment direction; the position of the spreader is adjusted according to the adjustment direction and the adjustment distance.

[0071] During implementation, the correspondence between pixels and actual distances can be pre-set, and a plane coordinate system can be constructed based on the X-axis and Y-axis being parallel to the stitching line of two mutually perpendicular sub-images in the target image, such as Figure 6 As shown, where P0(x o ,y o ), P1(x1,y1), P2(x2,y2), P3(x3,y3) are the center points of the four target keyholes respectively, M(m x ,m y ) is the center point of the target image, the direction of the X-axis is consistent with the long side direction of the top view of the container, and the direction of the Y-axis is consistent with the short side direction of the top view of the container.

[0072] Based on this, by comparing the relationship between P0 and P2 and the center point M, the adjustment distance and adjustment direction of one side of the spreader corresponding to P0 and P2 in the Y-axis direction can be determined; by comparing the relationship between P1 and P3 and the center point M, the adjustment distance and adjustment direction of one side of the spreader corresponding to P1 and P3 in the Y-axis direction can be determined; by comparing the relationship between P0 and P1 and the center point M, the adjustment distance and adjustment direction of the spreader in the X-axis direction can be determined.

[0073] Specifically, if 2m y If -(y0+y2)>0, the adjustment direction of the side of the spreader corresponding to P0 and P2 in the Y-axis direction is to adjust in the positive direction of the Y-axis; otherwise, the adjustment direction of the side of the spreader corresponding to P0 and P2 in the Y-axis direction is to adjust in the negative direction of the Y-axis. In order to make the perpendicular bisector of the line connecting P0 and P2 pass through the center point M, the target image needs to be adjusted in the negative direction of the Y-axis by a distance of |m y -(y0+y2) / 2|, accordingly, the adjustment distance of the spreader should be |m y -(y0+y2) / 2| corresponds to the actual distance.

[0074] If 2m y -(y1+y3)>0, then the adjustment direction of the side of the spreader corresponding to P1 and P3 in the Y-axis direction is adjusted towards the positive direction of the Y-axis; otherwise, the adjustment direction of the side of the spreader corresponding to P1 and P3 in the Y-axis direction is adjusted towards the negative direction of the Y-axis. Among them, the adjustment distance of the spreader is |m y -(y1+y3) / 2| corresponds to the actual distance.

[0075] If 2m x -(x0+x1)>0, the adjustment direction of the spreader in the X-axis direction is to adjust in the positive direction of the X-axis; otherwise, the adjustment direction of the spreader in the X-axis direction is to adjust in the negative direction of the X-axis. x -(x0+x1) / 2| corresponds to the actual distance.

[0076] By adjusting the position of the spreader in the above manner, the target lock hole in the target image can be aligned with the preset position, so that the spreader can be quickly and accurately aligned with the container, thereby achieving accurate grasping of the container, effectively improving the operating efficiency of the container grasping condition.

[0077] The embodiment of the present application also provides a box grabbing control device, such as Figure 7As shown, the device may include: an acquisition module 701, used to acquire a target image; the target image is obtained by splicing the sub-images; an identification and detection module 702, used to identify each target keyhole in the target image and detect whether the target keyhole meets a preset position; the preset position includes: the perpendicular bisector of the line connecting the specified positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image; and an adjustment module 703, used to adjust the position of the sling if the target keyhole does not meet the preset position so that the target keyhole meets the preset position.

[0078] Optionally, when adjusting the position of the sling, the adjustment module 703 can be specifically used to: determine the coordinate positions of the target keyhole in the target image and the center point of the target image in the same coordinate system; determine the adjustment information based on the relative position relationship between the target keyhole and the center point, the adjustment information including the adjustment distance and the adjustment direction; and adjust the position of the sling according to the adjustment direction and the adjustment distance.

[0079] Optionally, when identifying each target keyhole in the target image, the recognition and detection module 702 is specifically used to: input the target image into a pre-built keyhole detection model to obtain all keyholes in the target image; wherein the keyhole detection model uses the keyhole image as input, the keyhole as the detection target, and the corresponding keyhole image with the labeled box as output, and is obtained by training a neural network; and determine the target keyhole from all keyholes in the target image.

[0080] Optionally, when determining the target keyhole from all keyholes in the target image, the identification and detection module 702 can be specifically used to: determine the distance between each keyhole in the target image and the center point of the target image; and determine the keyhole with the shortest distance to the center point in each sub-image of the target image as the target keyhole.

[0081] Optionally, the box grabbing control device may further include a construction module, which may be used to: obtain an original data set, and mark the keyholes in the original data set to obtain a training data set; and use the training data in the training data set to train a deep learning neural network model to obtain a keyhole detection model.

[0082] Optionally, the container grabbing control device may further include a grabbing module, which may be configured to grab the container based on the current position of the spreader if the target lock hole meets a preset position.

[0083] It should be understood that the specific implementation of the box grabbing control device provided in the embodiments of the present application can refer to the specific implementation of the box grabbing control method described in any of the above embodiments, and will not be repeated here.

[0084] The present application also provides an electronic device, wherein: Figure 8As shown, the electronic device may include: a memory 801 and a processor 802; wherein the memory 801 is connected to the processor 802 and is used to store programs; the processor 802 is used to implement the box grabbing control method disclosed in any of the above embodiments by running the program stored in the memory 801.

[0085] Specifically, the electronic device may further include: a bus, a communication interface 803 , an input device 804 and an output device 805 .

[0086] The processor 802, the memory 801, the communication interface 803, the input device 804 and the output device 805 are interconnected via a bus.

[0087] A bus may include a pathway that transfers information between components of a computer system.

[0088] Processor 802 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, or the like, or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. Alternatively, it can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware components.

[0089] The processor 802 may include a main processor, and may also include a baseband chip, a modem, etc.

[0090] The memory 801 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include program code, which may include computer operating instructions. More specifically, the memory 801 may include read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), other types of dynamic storage devices that can store information and instructions, disk storage, flash memory, etc.

[0091] The input device 804 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0092] Output device 805 may include a device that allows information to be output to a user, such as a display screen, a speaker, etc.

[0093] The communication interface 803 may include any transceiver or similar device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0094] The processor 802 executes the program stored in the memory 801 and calls other devices to implement each step of the box grabbing control method provided in the embodiment of the present application.

[0095] An embodiment of the present application further provides a box grabbing control system, comprising a control device and multiple cameras.

[0096] Multiple cameras are evenly distributed around the spreader to capture sub-images of all the container's keyholes, with each camera corresponding to a different keyhole. The control device is configured to: acquire a target image; the target image is constructed by stitching together the sub-images; identify each target keyhole in the target image; and detect whether the target keyhole meets a preset position; the preset position includes the perpendicular bisector of the line connecting the designated positions of the target keyholes in any two adjacent sub-images passing through the center point of the target image; and if the target keyhole does not meet the preset position, adjust the spreader's position to ensure that the target keyhole meets the preset position.

[0097] An embodiment of the present application further provides a field crane automation system, which may include the box grab control system as described in any of the above embodiments.

[0098] An embodiment of the present application further provides a crane, comprising: a crane body and a box grab control system as described in any of the above embodiments.

[0099] An embodiment of the present application further provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, each step of the box grabbing control method provided in any of the above embodiments is implemented.

[0100] For the sake of simplicity, the aforementioned method embodiments are described as a series of action combinations. However, those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0101] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similarities between the various embodiments can be referred to in conjunction with each other. For device embodiments, since they are generally similar to method embodiments, their description is relatively simple, and for relevant details, reference can be made to the description of the method embodiments.

[0102] The steps in the methods of the various embodiments of the present application can be adjusted in sequence, combined, and deleted according to actual needs.

[0103] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be merged, divided, and deleted according to actual needs.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or submodules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0105] The modules or submodules described as separate components may or may not be physically separate, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules may be selected to achieve the purpose of this embodiment according to actual needs.

[0106] In addition, each functional module or submodule in each embodiment of the present application may be integrated into a processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into a single module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or software functional modules or submodules.

[0107] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software units executed by a processor, or a combination of the two. The software units may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0110] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A box grabbing control method, characterized in that: Applied to a container grabbing control system, the container grabbing control system includes multiple cameras evenly distributed around a spreader, the multiple cameras are used to capture sub-images of container lock holes, and the lock holes corresponding to the cameras are different. The method includes: Acquire a target image; the target image is obtained by stitching together the sub-images; Identifying each target keyhole in the target image and detecting whether the target keyhole meets a preset position; the preset position includes: a perpendicular bisector of a line connecting designated positions of target keyholes in any two adjacent sub-images passes through the center point of the target image; identifying each target keyhole in the target image includes: inputting the target image into a pre-built keyhole detection model to obtain all keyholes in the target image; wherein the keyhole detection model uses the keyhole image as input, the keyhole as a detection target, and the corresponding keyhole image with a marked box as output, and is obtained by training a neural network; determining the target keyhole from all keyholes in the target image; determining the target keyhole from all keyholes in the target image includes: determining the distance between each keyhole in the target image and the center point of the target image; and determining the keyhole with the shortest distance to the center point in each sub-image of the target image as the target keyhole; If the target lock hole does not meet the preset position, adjust the position of the sling so that the target lock hole meets the preset position; adjusting the position of the sling includes: determining the coordinate positions of the target lock hole in the target image and the center point of the target image in the same coordinate system; determining adjustment information based on the relative position relationship between the target lock hole and the center point, the adjustment information including an adjustment distance and an adjustment direction; and adjusting the position of the sling according to the adjustment direction and the adjustment distance.

2. The method according to claim 1, characterized in that The method for constructing the keyhole detection model includes: Obtaining an original data set and annotating the keyholes in the original data set to obtain a training data set; The deep learning neural network model is trained using the training data in the training data set to obtain the keyhole detection model.

3. The method according to claim 1, characterized in that After detecting whether the target lock hole meets the preset position, the method further includes: If the target lock hole meets the preset position, the container is grabbed based on the current position of the spreader.

4. A box grabbing control device, characterized in that: include: An acquisition module, used to acquire a target image; The target image is obtained by splicing the sub-images; An identification and detection module is configured to identify each target keyhole in the target image and detect whether the target keyhole satisfies a preset position; the preset position comprising: a perpendicular bisector of a line connecting designated positions of target keyholes in any two adjacent sub-images passing through the center point of the target image; identifying each target keyhole in the target image comprises: inputting the target image into a pre-built keyhole detection model to obtain all keyholes in the target image; wherein the keyhole detection model is trained using a neural network with a keyhole image as input, keyholes as detection targets, and corresponding keyhole images with labeled frames as output; determining the target keyhole from all keyholes in the target image; determining the target keyhole from all keyholes in the target image comprises: determining the distance between each keyhole in the target image and the center point of the target image; and determining the keyhole with the shortest distance from the center point in each sub-image of the target image as the target keyhole; An adjustment module is configured to adjust the position of a sling if the target lockhole does not meet the preset position so that the target lockhole meets the preset position; adjusting the position of the sling includes: determining the coordinate positions of the target lockhole in the target image and the center point of the target image in the same coordinate system; determining adjustment information based on the relative positional relationship between the target lockhole and the center point, the adjustment information including an adjustment distance and an adjustment direction; and adjusting the position of the sling according to the adjustment direction and the adjustment distance.

5. A box grabbing control system, characterized in that: Includes control equipment and multiple cameras; The plurality of cameras are used to be evenly distributed around the spreader to capture sub-images of all the keyholes of the container, and the keyholes corresponding to the cameras are different from each other; The control device is used to: acquire a target image; The target image is obtained by splicing the sub-images; Identify each target keyhole in the target image and detect whether the target keyhole meets a preset position; the preset position includes: the perpendicular bisector of the line connecting the designated positions of the target keyholes in any two adjacent sub-images passes through the center point of the target image; the identifying of each target keyhole in the target image includes: inputting the target image into a pre-built keyhole detection model to obtain all keyholes in the target image; wherein the keyhole detection model uses the keyhole image as input, the keyhole as the detection target, and the corresponding keyhole image with a marked box as output, and is obtained by training a neural network; determine the target keyhole from all keyholes in the target image; determine the target keyhole from all keyholes in the target image The keyhole comprises: determining the distance between each keyhole in the target image and the center point of the target image; determining the keyhole with the shortest distance to the center point in each sub-image of the target image as the target keyhole; if the target keyhole does not meet the preset position, adjusting the position of the sling so that the target keyhole meets the preset position; adjusting the position of the sling comprises: determining the coordinate positions of the target keyhole in the target image and the center point of the target image in the same coordinate system respectively; determining adjustment information based on the relative position relationship between the target keyhole and the center point, the adjustment information including an adjustment distance and an adjustment direction; and adjusting the position of the sling according to the adjustment direction and the adjustment distance.

6. A yard crane automation system, comprising the box grab control system according to claim 5.

7. A crane, characterized in that: include: A crane body and a yard crane automation system as claimed in claim 6.

Images