Container placing method and device based on vision, field bridge and medium

Abstract

The invention provides a vision-based container placing method and equipment, a field bridge and a medium, and relates to the technical field of automation. The method comprises the steps that after it is detected that a lifting appliance is in a locked state and begins to descend, a container position image including the container corner point position is obtained in real time; determining a current angular point pixel coordinate corresponding to the angular point position in the container position image at the current moment by using a computer vision technology; determining whether the container is in a suspended state or not according to the current angular point pixel coordinate at the current moment and each historical angular point pixel coordinate at the historical moment; and if the container is not in the suspended state, the lifting appliance is controlled to be unlocked, and the container placing process of the container is completed. According to the method, the risk of safety accidents caused by unlocking of the lifting appliance when the container is in the suspended state can be avoided, and therefore the safety of the container placing process is improved.

Description

Technical Field

[0001] The present application relates to the field of automation technology, and in particular to a vision-based container placement method, equipment, field crane and medium. Background Art

[0002] With the improvement of port automation level, more and more port operations are being automated, such as automatic loading and unloading of containers.

[0003] In the related technology, when loading and unloading containers automatically, radar or machine vision detection technology is usually used to determine the relative position relationship between the container and the ground, trucks and other supports, so as to place the container at the correct position of the support by adjusting the position and posture of the spreader to complete the container release operation. However, during the container release process, if the container has not fallen on the support, that is, the spreader is controlled to unlock when it is still in a suspended state, the container will fall, which may damage the container and the cargo at the least, or even cause personal safety accidents at the worst.

[0004] Therefore, there is a need for a container release solution that can improve the safety of the container release process and avoid safety accidents caused by the lifting device being unlocked when the container is in a suspended state. Summary of the invention

[0005] The embodiments of the present application provide a vision-based container release method, equipment, yard crane and medium, which can avoid the risk of safety accidents caused by unlocking the lifting device when the container is in a suspended state, thereby improving the safety of the container release process.

[0006] In a first aspect, an embodiment of the present application provides a method for placing a container based on vision, comprising:

[0007] After detecting that the spreader is in a locked state and starts to descend, acquiring a container position image including the container corner point positions in real time, wherein the container position image includes the container corner point positions;

[0008] Determine the pixel coordinates of the current corner point corresponding to the corner point position in the container position image at the current moment by using computer vision technology;

[0009] Determine whether the container is in a suspended state according to the pixel coordinates of the current corner point at the current moment and the pixel coordinates of each historical corner point at the historical moment;

[0010] If the container is not in a suspended state, the spreader is controlled to unlock, completing the container placement process.

[0011] In a possible implementation manner, the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment determined by using computer vision technology include:

[0012] Input the container position image at the current moment into the preset key point detection model to obtain the current corner point pixel coordinates corresponding to the corner point position of the container, wherein the corner point position includes any one of the following: the bottom corner point position of the left front of the container, the bottom corner point position of the left rear of the container, the bottom corner point position of the right front of the container, and the bottom corner point position of the right rear of the container;

[0013] The key point detection model is obtained by pre-training using container sample images including corner point positions of the container, and the container sample images include sample labels, and the sample labels are corner point pixel coordinates corresponding to the corner point positions of the container.

[0014] In a possible implementation, determining whether the container is in a suspended state according to the pixel coordinates of the current corner point at the current moment and the pixel coordinates of each historical corner point at a historical moment includes:

[0015] Determine the mean value of the pixel coordinates of each historical corner point;

[0016] Determining whether the position of the container relative to the spreader at the current moment has changed according to the current corner point pixel coordinates and the mean of the corner point pixel coordinates;

[0017] If the position of the container relative to the spreader changes, determining that the container is not in a suspended state;

[0018] If the position of the container relative to the spreader does not change, it is determined that the container is in a suspended state.

[0019] In a possible implementation, determining whether the position of the container relative to the spreader at the current moment has changed according to the current corner point pixel coordinates and the mean of the corner point pixel coordinates includes:

[0020] Determine the absolute value of the difference between the current corner point pixel coordinates and the mean of the corner point pixel coordinates;

[0021] Determine whether the absolute value of the difference is greater than a preset pixel value threshold;

[0022] If it is greater than a preset pixel value threshold, it is determined that the position of the container relative to the spreader at the current moment has changed;

[0023] If it is not greater than the preset pixel value threshold, it is determined that the position of the container relative to the spreader has not changed at the current moment.

[0024] In a possible implementation, it further includes:

[0025] If the container is in a suspended state, continue to perform the container placement operation, and put the current corner point pixel coordinates into the queue of historical corner point pixel coordinates;

[0026] The container position image at the next moment is obtained, and the container position image at the next moment is used as the container position image at the current moment, and the step of using computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment is re-executed.

[0027] In a possible implementation, the spreader is provided with a plurality of cameras, each camera respectively collects a container position image including a container corner point position, and the container corner point positions in the images collected by each camera are different. After the container position image including the container corner point position is obtained in real time, the method further includes:

[0028] For each container position image captured by a camera, computer vision technology and the historical corner pixel coordinates corresponding to the camera are used to determine whether the container is in a suspended state;

[0029] According to the container status recognition results corresponding to each camera, it is determined whether to control the spreader to unlock.

[0030] In a possible implementation, determining whether to control the spreader to unlock according to the container status recognition results corresponding to each camera respectively includes:

[0031] Determine whether the container status recognition results corresponding to each camera are all that the container is not in a suspended state;

[0032] If the container is not in a suspended state, the spreader is controlled to unlock, and the container placing process is completed;

[0033] If the container is not in a suspended state, the container placing operation is continued to reacquire the container position image.

[0034] In a second aspect, an embodiment of the present application provides a computing device, including:

[0035] A processor, and a memory communicatively connected to the processor;

[0036] Memory is used to store computer executable instructions;

[0037] The processor is used to execute the computer-executable instructions stored in the memory, so that the processor executes the above-mentioned first aspect and / or various possible implementations of the first aspect.

[0038] In a third aspect, an embodiment of the present application provides a field bridge, comprising: a spreader, one or more cameras arranged on the spreader, and a computing device as described in the second aspect;

[0039] The camera is in communication with the computing device and is used to collect a container position image including container corner point positions in real time during container placement;

[0040] The computing device is used to acquire a container position image including the container corner point positions in real time after detecting that the spreader is in a locked state and starts to descend; determine the current corner point pixel coordinates corresponding to the corner point positions in the container position image at the current moment by using computer vision technology; determine whether the container is in a suspended state based on the current corner point pixel coordinates at the current moment and the historical corner point pixel coordinates at historical moments; if the container is not in a suspended state, control the spreader to unlock and complete the container placement process.

[0041] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above-mentioned first aspect and / or various possible implementation methods of the first aspect.

[0042] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the above-mentioned first aspect and / or various possible implementation methods of the first aspect.

[0043] The embodiment of the present application provides a vision-based container release method, equipment, yard bridge and medium, which can obtain the container position image including the container corner point position in real time during the container release process, use computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment, and compare the current corner point pixel coordinates at the current moment with the historical corner point pixel coordinates at the historical moment, thereby determining whether the container is in a suspended state at the current moment. If the container is not in a suspended state, the sling will be controlled to unlock and complete the container release process. Through such a setting, the corner point pixel coordinates of the container corner point position can be used to determine the current container position during the container release process, and the current container position can be compared with the historical container position during the release process to determine whether the container is in a suspended state. Only when it is determined that the container is not in a suspended state will the sling be controlled to unlock, avoiding the risk of safety accidents caused by the sling unlocking when the container is in a suspended state, thereby improving the safety of the container release process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0045] Figure 1 A system architecture diagram of an embodiment of the present application;

[0046] Figure 2 This is a flow chart of a method for placing a container based on vision according to an embodiment of the present application;

[0047] Figure 3 This is a diagram showing the relationship between the positions of the spreader and the container during the container placement process;

[0048] Figure 4 This is a schematic diagram of the container position change before and after the container is loaded;

[0049] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of a computing device according to another embodiment of the present application.

[0051] Figure numerals: 1. trolley frame; 2. sling; 3. container; 4. camera; 5. lock hole.

[0052] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0053] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of information such as financial data or user data involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0056] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0057] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0058] The vision-based container release method, equipment, field bridge and medium of the present application can be used in the field of automation technology, and can also be used in any field other than the field of automation technology, such as the container field, etc. The application field of the vision-based container release method, equipment, field bridge and medium of the present application is not limited.

[0059] The vision-based container release method, equipment, yard bridge and medium of the present application can be applied to scenarios where container release operations are performed. Any scenario involving container release can apply the vision-based container release method, equipment, yard bridge and medium of the present application.

[0060] In the related technology, radar or machine vision detection technology is usually used for automatic loading and unloading of containers to determine the relative position relationship between the container and the ground, container trucks and other supports, so as to place the container in the exact position of the support by adjusting the position and posture of the spreader to complete the container placement operation.

[0061] The container unloading process mainly includes: the spreader grabs the container and moves it down; the container is placed on a support (container, container truck, ground, etc.); the spreader is unlocked; and the spreader moves up.

[0062] However, during the container placement process, if the container has not fallen onto the support, that is, if the spreader is controlled to unlock while it is still in a suspended state, the container will fall, which may damage the container and the cargo at the least, or even cause personal safety accidents at the worst.

[0063] Based on the above technical problems, the invention concept of the present application is: how to provide a container release solution that can improve the safety of the container release process and avoid the lifting equipment from unlocking when the container is in a suspended state.

[0064] The embodiment of the present application provides a vision-based container release method, equipment, yard bridge and medium, which can obtain a container position image including the container corner point position in real time after receiving a locking signal between the spreader and the container lock hole, that is, after determining that the container is in the process of releasing the container, and use computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment, that is, determine the current container position. After that, the current container position is compared with the historical container position during the release process, so as to accurately determine whether the container is in a suspended state. Only when it is determined that the container is not in a suspended state will the spreader be controlled to unlock, avoiding the risk of safety accidents caused by the spreader unlocking when the container is in a suspended state, thereby improving the safety of the container release process.

[0065] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 This is a system architecture diagram of an embodiment of the present application, such as Figure 1As shown, 1 is a trolley frame, 2 is a spreader, 3 is a container, 4 is a camera, and 5 is a lock hole above the container. The camera 4 is arranged on the spreader 2, so that the camera 4 can collect the container position image including the corner position of the container 3 below. The camera 4 is also connected to the PLC for communication, and sends the collected container position image to the PLC. When the lock of the spreader 2 is inserted into the lock hole 5 of the container 3, a locking signal will be issued. After the PLC detects that the spreader is in a locked state and starts to descend, the camera 4 arranged on the spreader 2 is used to obtain the container position image including the corner position of the container in real time; the computer vision technology is used to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment; according to the current corner point pixel coordinates at the current moment and the pixel coordinates of each historical corner point at the historical moment, it is determined whether the container is in a suspended state; if the container is not in a suspended state, the spreader is controlled to unlock and the container release process is completed.

[0067] Figure 2 This is a flow chart of a method for placing containers based on vision according to an embodiment of the present application. This embodiment uses a computing device as an execution subject to illustrate the method for placing containers based on vision. Figure 2 As shown, the vision-based container placement method may include the following steps:

[0068] S201: After detecting that the spreader is in a locked state and starts to descend, acquiring a container position image including container corner point positions in real time.

[0069] In this embodiment, the computing device may be a programmable logic controller (PLC) or other devices, as long as the computing device can control the spreader to complete the automatic loading and unloading of the container, and no limitation is made here. The following takes the computing device as an example to explain the vision-based container placement method of the present application.

[0070] In this embodiment, after the lock of the spreader is inserted into the lock hole of the container, a locking signal is issued. After the PLC detects the locking signal issued by the spreader and determines that the spreader is in a locked state and starts to descend, it will obtain the container position image including the container corner point position in real time through the camera set on the spreader.

[0071] In this embodiment, one or more cameras may be provided on the spreader, and each camera may capture an image including the corner position of the container below. The specific position of the camera on the spreader may be flexibly set by a person skilled in the art, as long as the camera can capture an image of the container position including a corner position of the container.

[0072] In this embodiment, since the camera is set on the spreader above the container, the corner positions of the top of the container cannot be detected, and one camera can only capture a certain corner position of the container. Therefore, the corner position of the container can be any one of the bottom corner position of the left front of the container, the bottom corner position of the left rear of the container, the bottom corner position of the right front of the container, and the bottom corner position of the right rear of the container.

[0073] S202: Using computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment.

[0074] In this embodiment, computer vision technologies such as image segmentation algorithms or target detection algorithms may be used to determine the current corner point pixel coordinates corresponding to the corner point positions in the container position image at the current moment.

[0075] S203: Determine whether the container is in a suspended state according to the pixel coordinates of the current corner point at the current moment and the pixel coordinates of each historical corner point at the historical moment.

[0076] In this embodiment, after acquiring the container position image in real time, computer vision technology can be used to determine the corner point pixel coordinates corresponding to each container position image, and the corner point pixel coordinates corresponding to the container position image at the current moment are used as the current corner point pixel coordinates, and the corner point pixel coordinates corresponding to the container position image acquired before the current moment are used as the historical corner point pixel coordinates, and the historical corner point pixel coordinates are organized into a historical coordinate queue in the order of acquisition time.

[0077] Figure 3 This is the position relationship diagram between the spreader and the container during the container placement process. Figure 3 (a) is the position relationship diagram between the spreader and the container when the container is suspended. Figure 3 (b) is the position relationship diagram between the spreader and the container when the container is in the loaded state. Figure 3 It can be seen that when the container is suspended, due to gravity, the lock of the spreader is close to the upper edge of the inner side of the container lock hole, resulting in a gap between the spreader and the container, and the camera on the spreader is far away from the container. After the container lands, the container will land first, and then the lock inside the spreader will be close to the lower edge of the inner side of the container lock hole, resulting in a smaller gap between the spreader and the container, and the spreader will be closer to the container, so the camera will also be closer to the container.

[0078] Figure 4 This is a schematic diagram of the container position change before and after the container is loaded. Figure 4 Area A in the middle is the position of the container before loading (suspended state), and area B is the position of the container after loading. Figure 4It can be seen that after the container is loaded, the container will appear closer and larger from the camera's image, and the position of the container's corners will also change accordingly.

[0079] Therefore, when the container is in a suspended state, the pixel coordinates of the corner points of the container will not change significantly relative to the historical data. When the container is converted from a suspended state to a landed state, the pixel coordinates of the corner points of the container will change significantly relative to the historical data.

[0080] In this embodiment, after determining the current corner point pixel coordinates, the current corner point pixel coordinates can be compared with the historical corner point pixel coordinates in the historical coordinate queue to determine whether the position of the container relative to the spreader has changed significantly, and based on this, determine whether the container is in a suspended state.

[0081] S204: If the container is not in a suspended state, the spreader is controlled to unlock, completing the container release process.

[0082] In this embodiment, when it is determined that the container is not in a suspended state, the PLC will control the spreader to unlock and complete the container release process, thereby avoiding safety accidents caused by the container falling due to the spreader unlocking when the container is in a suspended state.

[0083] In this embodiment, during the container placement process, the container position image including the container corner position can be obtained in real time, and the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment can be determined by using computer vision technology, and the current corner point pixel coordinates at the current moment can be compared with the historical corner point pixel coordinates at the historical moment, thereby determining whether the container is in a suspended state at the current moment. If the container is not in a suspended state, the spreader will be controlled to unlock and complete the container placement process. Through such a setting, the current container position during the container placement process can be determined by using the corner point pixel coordinates of the container corner position, and the current container position can be compared with the historical container position during the container placement process to determine whether the container is in a suspended state. Only when it is determined that the container is not in a suspended state will the spreader be controlled to unlock, thereby avoiding the risk of safety accidents caused by the spreader unlocking when the container is in a suspended state, thereby improving the safety of the container placement process.

[0084] In a possible implementation, the above step S202 uses computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment, which may include:

[0085] The container position image at the current moment is input into the preset key point detection model to obtain the current corner point pixel coordinates corresponding to the corner point position of the container. The corner point position includes any of the following: the bottom corner point position of the left front of the container, the bottom corner point position of the left rear of the container, the bottom corner point position of the right front of the container, and the bottom corner point position of the right rear of the container.

[0086] Among them, the key point detection model is obtained by pre-training using container sample images including the corner point positions of the container. The container sample images include sample labels, and the sample labels are the corner point pixel coordinates corresponding to the corner point positions of the container.

[0087] In this embodiment, the preset key point detection model can be obtained by training using an image segmentation algorithm and deep learning technology, or by training using a target detection algorithm and deep learning technology, or by training using an image segmentation algorithm, a target detection algorithm and deep learning technology, without any limitation here.

[0088] In this embodiment, a container sample image including the corner point positions and sample labels of the container can be used, and a key point detection model can be pre-trained using computer vision technology, deep learning technology, etc., so that the key point detection model has the ability to detect the corner point pixel coordinates corresponding to the corner point positions of the container in the image. After obtaining the container position image at the current moment, the container position image at the current moment is input into the preset key point detection model, and the current corner point pixel coordinates corresponding to the corner point positions of the container can be accurately obtained.

[0089] In a possible implementation, the above step S203 determines whether the container is in a suspended state according to the pixel coordinates of the current corner point at the current moment and the pixel coordinates of each historical corner point at a historical moment, and may include:

[0090] S11: Determine the mean value of the corner point pixel coordinates of each historical corner point pixel coordinate.

[0091] S12: Determine whether the position of the container relative to the spreader at the current moment has changed according to the current corner point pixel coordinates and the mean of the corner point pixel coordinates.

[0092] S13: If the position of the container relative to the spreader changes, it is determined that the container is not in a suspended state.

[0093] S14: If the position of the container relative to the spreader does not change, it is determined that the container is in a suspended state.

[0094] In this embodiment, the corner point pixel coordinates corresponding to the box position image acquired before the current moment are the historical corner point pixel coordinates, and the historical corner point pixel coordinates are grouped into a historical coordinate queue in the order of acquisition time.

[0095] In this implementation, after the current corner point pixel coordinates at the current moment are determined, the corner point pixel coordinate average of each historical corner point pixel coordinate in the historical coordinate queue may be calculated.

[0096] In this embodiment, by comparing the current corner pixel coordinates with the average corner pixel coordinates of each historical corner pixel coordinate, it is possible to determine whether the corner pixel coordinates have changed significantly, thereby determining whether the position of the container relative to the spreader has changed. If the position of the container relative to the spreader has changed, it can be said that the container has been transformed from a suspended state to a landed state (i.e., not in a suspended state).

[0097] In a possible implementation, the above step S12 determines whether the position of the container relative to the spreader at the current moment has changed according to the current corner point pixel coordinates and the mean of the corner point pixel coordinates, and may include:

[0098] S21: Determine the absolute value of the difference between the current corner point pixel coordinates and the mean of the corner point pixel coordinates.

[0099] S22: Determine whether the absolute value of the difference is greater than a preset pixel value threshold.

[0100] S23: If it is greater than a preset pixel value threshold, it is determined that the position of the container relative to the spreader at the current moment has changed.

[0101] S24: If it is not greater than the preset pixel value threshold, it is determined that the position of the container relative to the spreader at the current moment has not changed.

[0102] In this implementation, those skilled in the art can flexibly set the specific pixel value threshold according to actual conditions, and no limitation is made here.

[0103] For example, the current corner pixel coordinate is 100, and the corner pixel coordinate mean is 80, and the absolute value of the difference between the two is 20. If the preset pixel value threshold is 10, the absolute value of the difference between the two is greater than the pixel value threshold, and the position of the container relative to the spreader at the current moment has changed.

[0104] In this embodiment, by comparing the absolute value of the difference between the current corner point pixel coordinates and the mean of the corner point pixel coordinates with the preset pixel value threshold, it is possible to simply and accurately determine whether the corner point pixel coordinates have changed significantly at the current moment, thereby determining whether the position of the container relative to the spreader has changed.

[0105] In a possible implementation, the method may further include:

[0106] S31: If the container is in a suspended state, continue to perform the container placement operation, and put the current corner point pixel coordinates into the queue of historical corner point pixel coordinates;

[0107] S32: Acquire the container position image at the next moment, and use the container position image at the next moment as the container position image at the current moment, and re-execute the above step S202: use computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment.

[0108] Exemplarily, after determining the first corner pixel coordinates corresponding to the first container location image, the first corner pixel coordinates can be placed in the historical coordinate queue. After determining the second corner pixel coordinates corresponding to the second container location image, the second corner pixel coordinates can be compared with the first corner pixel coordinates in the historical coordinate queue. If the absolute value of the difference between the two is less than the pixel value threshold, the second corner pixel coordinates are placed in the historical coordinate queue. After determining the third corner pixel coordinates corresponding to the third container location image, the average of the first corner pixel coordinates and the second corner pixel coordinates in the historical coordinate queue can be determined, and the third corner pixel coordinates can be compared with the average of the corner pixel coordinates. If the absolute value of the difference between the two is less than the pixel value threshold, the third corner pixel coordinates are placed in the historical coordinate queue. By analogy, if the target corner pixel coordinates corresponding to the container location image at a certain moment are obtained, and the absolute value of the difference between the target corner pixel coordinates and the average of the corner pixel coordinates is greater than the pixel value threshold, it is determined that the position of the container relative to the spreader at that moment has changed, and has been converted from a suspended state to a landed state (i.e., not in a suspended state).

[0109] In this embodiment, if the container is still in a suspended state, the container position image can continue to be acquired, and the current corner point pixel coordinates corresponding to the container position image can continue to be determined using computer vision technology until it is determined that the container is not in a suspended state based on the current corner point pixel coordinates.

[0110] In a possible implementation, a plurality of cameras may be provided on the spreader, and each camera respectively collects a container position image including a container corner point position, and the container corner point positions in the images collected by each camera are different. After the container position image including the container corner point position is obtained in real time in the above step S201, the following may also be included:

[0111] S41: For each container position image captured by the camera, use computer vision technology and the historical corner point pixel coordinates corresponding to the camera to execute the above steps S202-S203 to determine whether the container is in a suspended state.

[0112] S42: Determine whether to control the spreader to unlock according to the container status recognition results corresponding to each camera.

[0113] In this embodiment, those skilled in the art can flexibly set the specific number of cameras according to actual conditions. For example, the number of cameras can be 2, 3, or 4, and no limitation is made herein.

[0114] For example, a camera is provided on each of the two long sides and the two short sides of the spreader, and the PLC receives the container position image including the container corner position collected by each camera in real time, and determines the current corner pixel coordinates corresponding to the container position image of each camera at the current moment. Afterwards, the PLC determines the container status recognition results corresponding to each camera according to the current corner pixel coordinates and historical corner pixel coordinates corresponding to each camera.

[0115] In this embodiment, a plurality of cameras may be provided on the spreader, and the corner positions of the container in the images captured by each camera are not the same. After the current corner pixel coordinates corresponding to each camera are detected by computer vision technology, the container status recognition results corresponding to each camera may be determined according to the current corner pixel coordinates and the historical corner pixel coordinates corresponding to each camera. Comprehensive analysis is performed based on the container status recognition results corresponding to each camera, which can improve the accuracy of container status determination, avoid detection errors, and thus improve the accuracy of spreader unlocking control, further avoiding the risk of spreader unlocking when the container is in a suspended state.

[0116] In a possible implementation, the above step S4 determines whether to control the spreader to unlock according to the container status recognition results corresponding to each camera, which may include:

[0117] S51: Determine whether the container status recognition results corresponding to each camera are all that the container is not in a suspended state.

[0118] S52: If all containers are not in a suspended state, the spreader is controlled to unlock, and the container placing process is completed.

[0119] S53: If all containers are not in the suspended state, continue to perform the container placement operation and reacquire the container position image.

[0120] In this embodiment, only when the container status recognition results corresponding to each camera are all that the container is not in the suspended state, will the PLC finally determine that the container is not in the suspended state, and can control the spreader to unlock and complete the container release process. As long as the container status recognition result of one camera is that the container is in the suspended state, the spreader will not be controlled to unlock, further avoiding the risk of safety accidents caused by the spreader unlocking when the container is in the suspended state, and further improving the safety of the container release process.

[0121] The vision-based container placement method of the present application is described below with a specific embodiment.

[0122] In a specific embodiment, a port uses a yard crane to automatically load and unload containers. A camera is set in front of the short sides of the yard crane to collect container images including the bottom corner position of the left front of the container (first camera) and the bottom corner position of the right front of the container (second camera). The specific container unloading process is as follows:

[0123] In the first step, the lock of the spreader is inserted into the lock hole of the container and a locking signal is sent. After the PLC detects the locking signal sent by the spreader and determines that the spreader is in a locked state and starts to descend, the camera set on the spreader is used to obtain in real time the first container position image including the left front bottom corner position of the container and the second container position image including the right front bottom corner position of the container.

[0124] In the second step, the PLC inputs the first container position image at the first moment into the preset key point detection model to obtain the first corner point pixel coordinates corresponding to the bottom left front corner point position of the container; the second container position image at the first moment is input into the preset key point detection model to obtain the second corner point pixel coordinates corresponding to the bottom right front corner point position of the container.

[0125] In the third step, the PLC determines the mean of the first corner point pixel coordinates of the historical corner point pixel coordinates corresponding to the first camera, and determines that the absolute value of the difference between the first corner point pixel coordinates and the mean of the first corner point pixel coordinates is less than the pixel value threshold, then the container state recognition result corresponding to the first camera is determined to be that the container is in a suspended state, and the PLC controls the hoist to continue to perform the container release operation, and puts the first corner point pixel coordinates into the historical coordinate queue corresponding to the first camera, and puts the second corner point pixel coordinates into the historical coordinate queue corresponding to the second camera.

[0126] In the fourth step, the PLC inputs the first container position image at the second moment into the preset key point detection model to obtain the first current corner point pixel coordinates corresponding to the bottom left front corner point position of the container; the second container position image at the second moment is input into the preset key point detection model to obtain the second current corner point pixel coordinates corresponding to the bottom right front corner point position of the container.

[0127] In the fifth step, the PLC determines the mean of the first corner point pixel coordinates of the historical corner point pixel coordinates corresponding to the first camera, and determines that the absolute value of the difference between the first current corner point pixel coordinates and the mean of the first corner point pixel coordinates is greater than the pixel value threshold, then determines that the container state recognition result corresponding to the first camera is that the container is not in a suspended state.

[0128] In the sixth step, the PLC determines the mean of the second corner point pixel coordinates of the historical corner point pixel coordinates corresponding to the second camera, and determines that the absolute value of the difference between the second current corner point pixel coordinates and the mean of the second corner point pixel coordinates is greater than the pixel value threshold, then determines that the container state recognition result corresponding to the second camera is that the container is not in a suspended state.

[0129] In the seventh step, the PLC determines that the container is not in a suspended state, controls the lifting device to unlock, and completes the container release process.

[0130] Figure 5 This is a schematic diagram of the structure of a computing device according to an embodiment of the present application. Figure 5 As shown, the computing device includes: an acquisition module 51, which is used to acquire a container position image including the container corner point position in real time after detecting that the spreader is in a locked state and starts to descend; a processing module 52, which is used to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment by using computer vision technology; determine whether the container is in a suspended state based on the current corner point pixel coordinates at the current moment and the historical corner point pixel coordinates at historical moments; if the container is not in a suspended state, control the spreader to unlock and complete the container release process.

[0131] The computing device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0132] Figure 6 This is a schematic diagram of the structure of a computing device according to an embodiment of the present application. Figure 6 As shown, the computing device includes: a processor 601, and a memory 602 that is communicatively connected to the processor 601; the memory 602 stores computer-executable instructions; the processor 601 executes the computer-executable instructions stored in the memory 602 to implement the steps of the vision-based container placement method in the above-mentioned method embodiments.

[0133] In the above computing device, the memory 602 and the processor 601 are electrically connected directly or indirectly to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines, such as through a bus connection. The memory 602 stores computer-executable instructions for implementing the data access control method, including at least one software function module that can be stored in the memory 602 in the form of software or firmware. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602.

[0134] The memory 602 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 602 is used to store programs, and the processor 601 executes the programs after receiving the execution instruction. Furthermore, the software programs and modules in the above-mentioned memory 602 may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0135] The processor 601 may be an integrated circuit chip having the ability to process signals. The processor 601 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0136] An embodiment of the present application also provides a field bridge, such as Figure 1 As shown, it includes: a sling, one or more cameras arranged on the sling, and Figure 6 The computing device shown.

[0137] Each camera is connected to the computing device for real-time acquisition of container position images including container corner point positions during container placement.

[0138] A computing device is used to obtain a container position image including the container corner point positions in real time after detecting that the spreader is in a locked state and starts to descend; determine the current corner point pixel coordinates corresponding to the corner point positions in the container position image at the current moment by using computer vision technology; determine whether the container is in a suspended state based on the current corner point pixel coordinates at the current moment and the historical corner point pixel coordinates at historical moments; if the container is not in a suspended state, control the spreader to unlock and complete the container release process.

[0139] In this implementation, during the container placement process, the container position image including the container corner point position can be obtained in real time, and the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment can be determined by using computer vision technology, and the current corner point pixel coordinates at the current moment can be compared with the historical corner point pixel coordinates at the historical moment, thereby determining whether the container is in a suspended state at the current moment. If the container is not in a suspended state, the sling will be controlled to unlock and complete the container placement process. Through such a setting, the corner point pixel coordinates of the container corner point position can be used to determine the current container position during the container placement process, and the current container position can be compared with the historical container position during the placement process to determine whether the container is in a suspended state. The sling will only be controlled to unlock when it is determined that the container is not in a suspended state, thereby avoiding the risk of safety accidents caused by the sling unlocking when the container is in a suspended state, thereby improving the safety of the container placement process.

[0140] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the steps of each method embodiment of the present application.

[0141] An embodiment of the present application further provides a computer program product, including a computer program, which implements the steps of each method embodiment of the present application when executed by a processor.

[0142] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0143] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0144] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0145] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0146] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the appended claims.

[0148] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for placing containers based on vision, characterized in that: include: After detecting that the spreader is in a locked state and starts to descend, a container position image including container corner point positions is acquired in real time; Determine the pixel coordinates of the current corner point corresponding to the corner point position in the container position image at the current moment by using computer vision technology; Determine whether the container is in a suspended state according to the pixel coordinates of the current corner point at the current moment and the pixel coordinates of each historical corner point at the historical moment; If the container is not in a suspended state, the spreader is controlled to unlock, completing the container placement process.

2. The method for placing containers based on vision according to claim 1, characterized in that: The method of using computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment includes: Input the container position image at the current moment into the preset key point detection model to obtain the current corner point pixel coordinates corresponding to the corner point position of the container, wherein the corner point position includes any one of the following: the bottom corner point position of the left front of the container, the bottom corner point position of the left rear of the container, the bottom corner point position of the right front of the container, and the bottom corner point position of the right rear of the container; The key point detection model is obtained by pre-training using container sample images including corner point positions of the container, and the container sample images include sample labels, and the sample labels are corner point pixel coordinates corresponding to the corner point positions of the container.

3. The method for placing containers based on vision according to claim 2, characterized in that: The determining whether the container is in a suspended state according to the pixel coordinates of the current corner point at the current moment and the pixel coordinates of each historical corner point at the historical moment includes: Determine the mean value of the pixel coordinates of each historical corner point; Determining whether the position of the container relative to the spreader at the current moment has changed according to the current corner point pixel coordinates and the mean of the corner point pixel coordinates; If the position of the container relative to the spreader changes, determining that the container is not in a suspended state; If the position of the container relative to the spreader does not change, it is determined that the container is in a suspended state.

4. The method for placing containers based on vision according to claim 3, characterized in that: The determining, based on the current corner point pixel coordinates and the corner point pixel coordinate mean, whether the position of the container relative to the spreader at the current moment has changed includes: Determine the absolute value of the difference between the current corner point pixel coordinates and the mean of the corner point pixel coordinates; Determine whether the absolute value of the difference is greater than a preset pixel value threshold; If it is greater than a preset pixel value threshold, it is determined that the position of the container relative to the spreader at the current moment has changed; If it is not greater than the preset pixel value threshold, it is determined that the position of the container relative to the spreader has not changed at the current moment.

5. The method for placing containers based on vision according to claim 1, characterized in that: Also includes: If the container is in a suspended state, continue to perform the container placement operation, and put the current corner point pixel coordinates into the queue of historical corner point pixel coordinates; The container position image at the next moment is obtained, and the container position image at the next moment is used as the container position image at the current moment, and the step of using computer vision technology to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment is re-executed.

6. The method for placing a container based on vision according to any one of claims 1 to 5, characterized in that: The spreader is provided with a plurality of cameras, each camera respectively collects a container position image including a container corner point position, and the container corner point positions in the images collected by each camera are different. After the container position image including the container corner point position is obtained in real time, the method further includes: For each container position image captured by a camera, computer vision technology and the historical corner pixel coordinates corresponding to the camera are used to determine whether the container is in a suspended state; According to the container status recognition results corresponding to each camera, it is determined whether to control the spreader to unlock.

7. The method for placing containers based on vision according to claim 6, characterized in that: The determining whether to control the spreader to unlock according to the container status recognition results corresponding to each camera respectively includes: Determine whether the container status recognition results corresponding to each camera are all that the container is not in a suspended state; If the container is not in a suspended state, the spreader is controlled to unlock, and the container placing process is completed; If the container is not in a suspended state, the container placing operation is continued to reacquire the container position image.

8. A computing device, characterized in that include: A processor, and a memory communicatively connected to the processor; The memory is used to store computer-executable instructions; The processor is configured to execute the computer-executable instructions stored in the memory, so that the processor executes the vision-based container placement method according to any one of claims 1 to 7.

9. A field bridge, characterized in that: include: A sling, one or more cameras disposed on the sling, and a computing device as claimed in claim 8; The camera is in communication with the computing device and is used to collect a container position image including container corner point positions in real time during container placement; The computing device is used to acquire a container position image including container corner point positions in real time after detecting that the spreader is in a locked state and starts to descend; Computer vision technology is used to determine the current corner point pixel coordinates corresponding to the corner point position in the container position image at the current moment; based on the current corner point pixel coordinates at the current moment and the historical corner point pixel coordinates at historical moments, it is determined whether the container is in a suspended state; if the container is not in a suspended state, the sling is controlled to unlock and the container placement process is completed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the vision-based container placement method according to any one of claims 1 to 7.

Images