Collision warning method and device, lifting equipment and storage medium

Abstract

The present application provides a collision warning method and device, lifting equipment and storage medium, the collision warning method comprising: collecting a first image of a first container being lifted and a second container fixed on a container truck; the first image at least includes a first marker located at a bottom corner of a side of the first container and close to the second container and a second marker located at a top corner of a side of the second container and close to one side of the first container; according to the first image, determining a first distance between the first container and the second container in a vertical direction; when the first distance is less than a first distance threshold, collecting a second image, the second image at least including a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container; determining a second distance between the first container and the second container in a horizontal direction according to the second image; when the second distance is less than a second distance threshold, sending a collision warning signal.

Description

Technical Field

[0001] The present application relates to the technical field of lifting machinery, and more specifically, to a collision warning method and device, lifting equipment and storage medium. Background Art

[0002] With the rapid growth of port container throughput, large-scale mechanized loading and unloading operations have become more frequent. In the case of double-container unloading operations on container trucks, it is necessary to avoid collisions between the hoisted containers and the containers fixed on the container trucks.

[0003] At present, the anti-collision warning of double-box unloading operation is usually carried out by the operator of the lifting equipment, that is, the unloading is carried out by manual observation. When it is determined that there is a risk of collision, the corresponding operation is performed to ensure safe operation. This manual observation method is inefficient and requires a lot of manpower and material resources. Summary of the invention

[0004] The present application provides a collision warning method and device, lifting equipment and storage medium. The following introduces various aspects involved in the embodiments of the present application.

[0005] In a first aspect, a collision warning method is provided, the method comprising: collecting a first image of a first container being lifted and a second container fixed on a container truck; wherein the first image includes at least a first marker and a second marker, the first marker is located at a bottom corner of a side surface of the first container and close to the second container, and the second marker is located at a top corner of a side surface of the second container and close to one side of the first container; according to the first image, determining a first distance between the first container and the second container in a vertical direction; when the first distance is less than a first distance threshold, collecting a second image, the second image at least including a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container; determining a second distance between the first container and the second container in a horizontal direction according to the second image; and sending a collision warning signal when the second distance is less than a second distance threshold.

[0006] Optionally, the method further includes: determining a third distance between the first container and the second container in the horizontal direction according to the first image; and sending the collision warning signal when the third distance is less than a third distance threshold.

[0007] Optionally, determining a first distance in the vertical direction between the first container and the second container based on the first image includes: extracting a plurality of first contour points of the first marker and a plurality of second contour points of the second marker in the first image using a deep learning model; determining a nearest vertical distance between the plurality of first contour points and the plurality of second contour points; and using the nearest vertical distance as the first distance.

[0008] Optionally, determining a third distance in the horizontal direction between the first container and the second container based on the first image includes: extracting a plurality of first contour points of the first marker and a plurality of second contour points of the second marker in the first image using a deep learning model; determining a nearest horizontal distance between the plurality of first contour points and the plurality of second contour points; and using the nearest horizontal distance as the third distance.

[0009] Optionally, the second image includes a first sub-image and a second sub-image, the first sub-image includes at least the third marker and the first edge, and the third image includes at least the fourth marker and the first edge; determining the second distance in the horizontal direction between the first container and the second container based on the second image includes: determining a fourth distance between the third marker and the first edge based on the first sub-image; determining a fifth distance between the fourth marker and the first edge based on the second sub-image; and taking the smaller value between the fourth distance and the fifth distance as the second distance.

[0010] Optionally, based on the first sub-image, determining the fourth distance between the third marker and the first edge includes: using a deep learning model to extract multiple third contour points of the third marker in the first sub-image and multiple fourth contour points of the first edge; determining a first closest distance between the multiple third contour points and the multiple fourth contour points; using the first closest distance as the fourth distance; based on the second sub-image, determining the fifth distance between the second marker and the first edge includes: using the deep learning model to extract multiple fifth contour points of the fourth marker in the second sub-image and multiple sixth contour points of the first edge; determining a second closest distance between the multiple fifth contour points and the multiple sixth contour points; using the second closest distance as the fifth distance.

[0011] Optionally, the first marker is a first lock hole at the bottom corner of the first container; the second marker is a second lock hole at the top corner of the second container; the third marker and the fourth marker are a third lock hole and a fourth lock hole on the top surface of the first container, and the third lock hole and the fourth lock hole are located on the side of the first container close to the second container.

[0012] In a second aspect, a collision warning device is provided, comprising: a first image acquisition unit, used to acquire a first image of a first container being lifted and a second container fixed on a container truck; wherein the first image includes at least a first marker and a second marker, the first marker is located at a bottom corner of the first container and close to the second container, and the second marker is located at a top corner of the second container and close to one side of the first container; a processing unit, used to determine a first distance between the first container and the second container in a vertical direction according to the first image; a second image acquisition unit, used to acquire a second image when the first distance is less than a first distance threshold, the second image at least including a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container; the processing unit is also used to determine a second distance between the first container and the second container in a horizontal direction according to the second image; and a warning unit, used to send a collision warning signal when the second distance is less than a second distance threshold.

[0013] Optionally, the processing unit is further used to determine a third distance between the first container and the second container in the horizontal direction according to the first image; and the warning unit is further used to send the collision warning signal when the third distance is less than a third distance threshold.

[0014] Optionally, the processing unit is also used to: extract multiple first contour points of the first marker and multiple second contour points of the second marker in the first image using a deep learning model; determine the closest vertical distance between the multiple first contour points and the multiple second contour points; and use the closest vertical distance as the first distance.

[0015] Optionally, the processing unit is also used to: extract multiple first contour points of the first marker and multiple second contour points of the second marker in the first image using a deep learning model; determine the closest horizontal distance between the multiple first contour points and the multiple second contour points; and use the closest horizontal distance as the third distance.

[0016] Optionally, the second image includes a first sub-image and a second sub-image, the first sub-image includes at least the third marker and the first edge, and the third image includes at least the fourth marker and the first edge; the processing unit is also used to: determine a fourth distance between the third marker and the first edge based on the first sub-image; determine a fifth distance between the fourth marker and the first edge based on the second sub-image; and use the smaller value of the fourth distance and the fifth distance as the second distance.

[0017] Optionally, the processing unit is also used to: extract multiple third contour points of the third marker in the first sub-image and multiple fourth contour points of the first edge using a deep learning model; determine a first closest distance between the multiple third contour points and the multiple fourth contour points; use the first closest distance as the fourth distance; and, extract multiple fifth contour points of the fourth marker in the second sub-image and multiple sixth contour points of the first edge using the deep learning model; determine a second closest distance between the multiple fifth contour points and the multiple sixth contour points; use the second closest distance as the fifth distance.

[0018] Optionally, the first marker is a first lock hole at the bottom corner of the first container; the second marker is a second lock hole at the top corner of the second container; the third marker and the fourth marker are a third lock hole and a fourth lock hole on the top surface of the first container, and the third lock hole and the fourth lock hole are located on the side of the first container close to the second container.

[0019] In a third aspect, a lifting device is provided, comprising: a body, a sling and a collision warning device as described in the second aspect.

[0020] Optionally, the first image acquisition unit is arranged on a roadside door leg of the machine body, and the second image acquisition unit is arranged on the hoist.

[0021] According to a fourth aspect, a computer-readable storage medium is provided, wherein a program is stored on the storage medium, and the program enables a computer to execute the method according to the first aspect.

[0022] In the collision warning method provided in the present application, the side images of the lifted container and the container fixed on the container truck are used to determine the relative position between the two containers. When the containers move to a height where there is a risk of collision, the image of the top of the container is used to determine the horizontal distance between the two containers. When the horizontal distance between the two containers is small, it is determined that there is a risk of collision at this time, and a collision warning signal is issued so that the operator of the lifting equipment can take appropriate measures to avoid collision. This method can save human resources and has higher efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an application scenario of the technical solution provided in this application.

[0024] Figure 2 It is a schematic flow chart of a collision warning method provided in one embodiment of the present application.

[0025] Figure 3 It is a schematic flow chart of a collision warning method provided in another embodiment of the present application.

[0026] Figure 4 It is an example of the first image in the embodiment of the present application.

[0027] Figure 5 It is an example of the first sub-image in the embodiment of the present application.

[0028] Figure 6 It is an example of the second sub-image in the embodiment of the present application.

[0029] Figure 7 It is a schematic flow chart of a collision warning method provided in yet another embodiment of the present application.

[0030] Figure 8 It is a schematic structural diagram of the collision warning device provided in an embodiment of the present application.

[0031] Fig. 9 It is a schematic structural diagram of the lifting equipment provided in the embodiment of the present application.

[0032] Fig.10 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] With the rapid growth of container logistics transportation, the use of container transportation for cargo has the advantages of speed and convenience, cargo safety, low loss, transportation standardization, and a wide range of applicable transportation methods, and is becoming increasingly popular.

[0036] Currently, the size of containers is usually 20 feet or 40 feet. The transportation of containers on land requires the use of container trucks (container trucks). A container truck can usually transport one 40-foot container or two 20-foot containers.

[0037] There are usually multiple lock holes on the container body. When using a shore container crane or a gantry crane for lifting, the lock on the lifting equipment cooperates with the multiple holes on the top of the container. After the container is fixed on the lifting equipment, it is lifted and moved to the container truck. The lock on the container truck cooperates with the lock holes on the ground of the container to fix the container on the container truck.

[0038] When lifting a 40-foot standard container, it is necessary to ensure that the lock holes on the top of the container and the locks on the lifting equipment can be locked smoothly, and when the container falls onto the container truck, the lock holes on the floor of the container can be correctly aligned with the locks on the container truck; when lifting a 20-foot container, in addition to ensuring the alignment between the lock holes and the locks on the container, it is also necessary to avoid collisions between the lifted container and the container already fixed on the container truck. Figure 1 This issue is further explained.

[0039] Figure 1 Schematic diagram of the application scenario of the technical solution of the present application is shown in FIG. Among them, the lifting device 110 is used to lift the first container 120 onto the container truck 130. At the same time, the second container 140 is also arranged on the container truck 130. When the first container 120 is lowered to a height close to the second container 140, there is a risk of collision between the two containers. In the related art, in order to prevent the collision between the two containers, manual observation is usually adopted.

[0040] In one implementation of the related technology, the operator of the lifting device 110 can estimate the distance between the two containers based on the image of the top of the container captured by the image acquisition unit such as the camera under the spreader, and timely adjust the position of the first container 120 according to the distance, so as to avoid collision between the two containers. This method requires manual participation, resulting in low efficiency and requiring a large amount of manpower and material resources.

[0041] In another implementation of the related technology, it can also be carried out based on radar scanning technology. Specifically, a radar is set under the hoist to collect the distance between the two boxes in the vertical and horizontal directions, and a collision warning is performed accordingly; the problem with this method is that during the hoisting process, there is a risk that the radar signal will be blocked, resulting in data loss, so that the distance accuracy obtained is low.

[0042] The embodiments of the present application provide a collision warning method and device, lifting equipment and storage medium to solve the above problems.

[0043] Figure 2 is a schematic flow chart of a collision warning method provided in an embodiment of the present application, Figure 2 The method includes steps S210-S250.

[0044] In step S210, first images of the first container being lifted and the second container fixed on the container truck are collected.

[0045] The method provided in the embodiment of the present application can be applied to Figure 1 In the scenario shown, the container truck has adjacent first and second positions, which are respectively used to place two containers; wherein the second container that has been fixed on the container truck is located at the first position, and the first container is suspended above the second position by a lifting device.

[0046] The first image can be acquired by using an image acquisition unit such as a camera or a still camera.

[0047] The first image is an image captured from the side of the first container and the second container, and the first image includes at least a first marker and a second marker, wherein the first marker is located at the bottom corner of the side of the first container and close to the second container, and the second marker is located at the top corner of the side of the second container and close to one side of the first container.

[0048] The first marker may be a first lock hole on the first container, which is arranged at the bottom corner of the side of the first container and close to the second container; the second marker may be a second lock hole on the second container, which is arranged at the top corner of the side of the second container and close to the first container.

[0049] The acquisition device for acquiring the first image can be arranged on the body of the lifting device, for example, on the roadside door leg. The specific location of the acquisition device is not specifically limited in the embodiment of the present application, and it is only necessary to ensure that the image acquired by the acquisition device can meet the above conditions.

[0050] In step S220, a first distance between the first container and the second container in the vertical direction is determined according to the first image.

[0051] There are many methods to determine the first distance based on the first image. As an example, the first image can be segmented using a deep learning algorithm to determine the edges of the first marker and the second marker in the first image. The first distance between the two boxes in the vertical direction can be obtained based on the edge expansion information of the two markers.

[0052] It should also be noted that, in this embodiment, the reason why the minimum vertical distance between the first lock hole at the bottom corner of the first container and the second lock hole at the top corner of the second container is used as the first distance is that the camera used to capture the first image is usually set on the door leg of the lifting equipment, which is close to the container truck, and the camera can usually only capture part of the side area of ​​the container. For example, when the key features for determining the first distance are the lock hole at the top corner of the first container and the lock hole at the bottom corner of the second container, the camera cannot capture the above two lock holes in the same picture. Selecting the two lock holes closest to each other in the two boxes as the key features for determining the first distance can ensure that the two key features can appear in the first image at the same time.

[0053] In step S230, when the first distance is less than the first distance threshold, a second image is acquired.

[0054] The first distance threshold is set to: when a first distance between the first container and the second container in the vertical direction is less than the threshold, there is a collision risk between the two containers.

[0055] Taking the first distance being the distance between the bottom surface of the first container and the top surface of the second container as an example, when the distance is small (for example, less than 10 cm), there is a risk of collision between the two containers, and the second distance threshold can be set to 10 cm.

[0056] The second image is an image collected from the top surfaces of the first container and the second container, and includes at least the third marker and the fourth marker on the top surface of the first container and a first edge of the second container close to the first container.

[0057] The third marker and the fourth marker may be a third lock hole and a fourth lock hole on the top surface of the second container, and the third lock hole and the fourth lock hole are located on a side close to the second container.

[0058] The acquisition device for acquiring the second image may be arranged on a beam, a trolley or a sling of the equipment. The embodiment of the present application does not impose any specific limitation on this. It is only necessary to ensure that the acquired image meets the above conditions.

[0059] In step S240, a second distance between the first container and the second container in the horizontal direction is determined according to the second image.

[0060] There are many methods to determine the first distance based on the second image. As an example, the second image can be segmented using a deep learning algorithm to determine the third marker, the fourth marker and the first edge in the second image, and the contour information of the third marker, the fourth marker and the first edge can be extracted. The second distance between the two boxes in the horizontal direction can be obtained using the contour information.

[0061] In step S250, when the second distance is less than the second distance threshold, a collision warning signal is sent.

[0062] The second distance threshold is the minimum safe distance that can ensure that the two boxes do not collide; when the distance between the two boxes is small, if the sling shakes, it will cause a collision between the two boxes. Therefore, it is necessary to send a collision warning signal when the horizontal distance between the two boxes is less than the second distance threshold, so that the operator of the lifting equipment or the automatic control system of the lifting equipment can make appropriate adjustments to eliminate the risk of collision.

[0063] According to the above-mentioned collision warning method provided in the embodiment of the present application, the side images of the lifted container and the container fixed on the container truck are used to determine the relative position between the two containers. When the containers move to a height where there is a risk of collision, the image of the top of the container is used to determine the horizontal distance between the two containers. When the horizontal distance between the two containers is small, it is determined that there is a risk of collision at this time, and a collision warning signal is issued to enable the operator of the lifting equipment to take corresponding measures to avoid collision. This method can save human resources and has higher efficiency.

[0064] In some embodiments, when the lifting equipment is a manually operated device, the collision warning signal can be presented to the operator in any form such as sound, light, electricity, etc., for example, by notifying the operator of the current collision risk by flashing a warning light in the cockpit of the lifting equipment, sounding a buzzer, or displaying a notification message on the central control panel.

[0065] In some embodiments, when the lifting equipment is a device with automatic control capabilities, the collision warning signal can be a signal sent by the collision warning to the automatic control system, so that the automatic control system can control the trolley or sling of the lifting equipment to stop the current movement; or, the automatic control system of the equipment can adjust the movement of the trolley or sling according to the current first distance to control the distance between the two boxes within a safe range.

[0066] like Figure 3 As shown, in some embodiments, after acquiring the first image in step S210, the method provided in the embodiment of the present application further includes steps S260 and S270.

[0067] In step S260, a third distance between the first container and the second container in the horizontal direction is determined according to the first image.

[0068] See also Figure 1 When the first container is suspended by the spreader and moves toward the container truck, if the distance between the two containers in the horizontal direction is close, there will be a risk of collision during the descent of the first container.

[0069] Therefore, in the embodiment of the present application, the first image of the side of the container can be used to roughly determine its third distance in the horizontal direction, and then determine whether there is a risk of collision.

[0070] In step S270, when the third distance is less than the third distance threshold, a collision warning signal is sent.

[0071] In some embodiments, the above Figure 3 Step S220 in which a first distance between the first container and the second container in a vertical direction is determined according to the first image further comprises:

[0072] Step 1: extracting a first contour point of a first marker and a plurality of second contour points of a second marker in a first image using a deep learning model. Figure 4 Given an example of this first image, Figure 4 The following is an example where the landmark is a keyhole. Figure 4 As shown, the first locking hole 121 is located at the bottom corner of the first container 120 and is close to the second container 140 , and the second locking hole 141 is located at the top corner of the second container and is close to the first container 120 .

[0073] Step 2: Determine the shortest vertical distance between the plurality of first contour points and the plurality of second contour points. As an implementation method, the plurality of first contour points and the plurality of second contour points may be traversed to determine the vertical distance between each first contour point and each second contour point, and the minimum value D among the plurality of vertical distances may be used as the vertical distance. min1 as the closest vertical distance.

[0074] Step 4: Use the shortest vertical distance as the first distance.

[0075] In some embodiments, the above Figure 3 Step S260 in which, according to the first image, a third distance between the first container and the second container in the horizontal direction is determined further comprises:

[0076] Step 1: extracting a first contour point of a first marker and a plurality of second contour points of a second marker in a first image using a deep learning model. Figure 4 An example of this first image is given.

[0077] Step 2: Determine the shortest horizontal distance between the first contour points and the second contour points. As an implementation method, the first contour points and the second contour points can be traversed to determine the horizontal distance between each first contour point and each second contour point, and the minimum value D among the multiple horizontal distances can be used as the minimum value D. min2 as the closest horizontal distance.

[0078] Step 4: Use the shortest vertical distance as the second distance.

[0079] The second distance mentioned above is preferably the shortest distance between the two containers, which is the smaller value of the distance between the two container corners of the first container close to the second container and the edge of the second container.

[0080] Therefore, determining the second distance includes: respectively determining the distances between the two corners of the first container and the second container, and taking the smaller value thereof as the first distance.

[0081] In an embodiment of the present application, the second image can be collected by a graphics acquisition device such as a camera. In practical applications, the camera needs to be set at a higher position, such as fixing it on a trolley of a lifting device, to ensure that the two corners of the first container and the edge of the second container can be included in the second image at the same time. By processing the second image, the above-mentioned first distance can be obtained. However, there is a risk of obstruction for the camera set on the trolley. For example, at some angles, the corners and edges of the container may be obstructed by the sling, resulting in the inability to obtain key features from the collected image.

[0082] In some embodiments, the second image includes a first sub-image and a second sub-image. The first sub-image includes at least a third marker of the first container and a first edge of the second container, and the second sub-image includes at least a fourth marker and the first edge. The third marker and the fourth marker are located on the top surface of the first container and close to the second container, and the first edge is located on the top surface of the second container and close to the first container. The first sub-image and the second sub-image can be collected using cameras set at the corners of the spreader, respectively, so as to avoid blocking key features in the image.

[0083] In some embodiments, the camera disposed at the corner of the sling is a gun camera.

[0084] Figure 5 and Figure 6 An example of the first sub-image and the second sub-image is given, which is similar to the above Figure 4 similar, Figure 5 and Figure 6 The description is also made by taking the lock hole as an example, wherein the first sub-image includes the third lock hole 122 of the first container 120 and the first edge 142 of the second container 140 , and the second sub-image includes the fourth lock hole 123 and the first edge 142 of the first container 120 .

[0085] like Figure 7 As shown, step S240 in the foregoing text further includes steps S241 to S243.

[0086] In step S241, a fourth distance between the third marker and the first edge is determined according to the first sub-image.

[0087] In step S242, a fifth distance between the fourth marker and the first edge is determined according to the second sub-image.

[0088] In step S243, the smaller value between the fourth distance and the fifth distance is determined as the second distance.

[0089] In some embodiments, see also Figure 5 , determining a fourth distance between the third keyhole and the first edge according to the first sub-image includes:

[0090] Step 1: Use a deep learning model to extract multiple third contour points of the third keyhole 122 and multiple fourth contour points of the first edge 142 in the first sub-image.

[0091] Step 2: determine the first shortest distance between the plurality of third contour points and the plurality of fourth contour points.

[0092] As an implementation manner, the plurality of third contour points and the plurality of fourth contour points may be traversed to determine the distance between each third contour point and each fourth contour point, and the minimum value of the plurality of distances may be used as the first closest distance.

[0093] For example, Figure 5 The distance D between the third contour point and the fourth contour point shown in min3 is the first closest distance.

[0094] Step 3: Take the first closest distance as the fourth distance, i.e. D4 = D min3 .

[0095] In some embodiments, see Figure 6 , determining a fifth distance between the fourth marker and the first edge according to the second sub-image, comprising:

[0096] Step 1, extracting a plurality of fifth contour points of the fourth keyhole 123 and a plurality of sixth contour points of the first edge 143 in the second sub-image using a deep learning model;

[0097] Step 2: Determine the second closest distance D between the plurality of fifth contour points and the plurality of sixth contour points min4 This step is the same as the method for determining the first closest distance and will not be repeated here.

[0098] Step 3: Use the second closest distance as the fifth distance, i.e. D5 = D min4 .

[0099] After determining the fourth distance and the fifth distance, the smaller value thereof is used as the second distance, and when the second distance is less than the second distance threshold, a collision warning signal is sent.

[0100] Combination of the above Figure 1-Figure 7 The method embodiment of the present application is described in detail. Figures 8 to 10 , describes the device embodiment of the present disclosure in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0101] Figure 8 is a schematic structural diagram of a collision warning device provided in an embodiment of the present application, wherein the collision warning device is applied to a lifting device, which may be, for example, a lifting device Figure 1 The lifting device 110 is shown in FIG.

[0102] Figure 8 The collision warning device 800 includes:

[0103] The first image acquisition unit 810 is used to acquire a first image of the first container being lifted and the second container fixed on the container truck. The first image includes at least a first marker and a second marker, the first marker is located at the bottom corner of the first container and close to the second container, and the second marker is located at the top corner of the second container and close to one side of the first container.

[0104] The processing unit 820 is configured to determine a first distance between the first container and the second container in a vertical direction according to the first image.

[0105] The second image acquisition unit 830 is used to acquire a second image when the first distance is less than a first distance threshold, wherein the second image at least includes a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container.

[0106] The processing unit 820 is further configured to determine a second distance between the first container and the second container in the horizontal direction according to the second image.

[0107] The warning unit 840 is configured to send a collision warning signal when the second distance is less than a second distance threshold.

[0108] In some embodiments, the processing unit is further used to determine a third distance between the first container and the second container in the horizontal direction based on the first image; the warning unit is further used to send the collision warning signal when the third distance is less than a third distance threshold.

[0109] In some embodiments, the processing unit is also used to: use a deep learning model to extract multiple first contour points of the first marker and multiple second contour points of the second marker in the first image; determine the closest vertical distance between the multiple first contour points and the multiple second contour points; and use the closest vertical distance as the first distance.

[0110] In some embodiments, the processing unit is also used to: use a deep learning model to extract multiple first contour points of the first marker and multiple second contour points of the second marker in the first image; determine the nearest horizontal distance between the multiple first contour points and the multiple second contour points; and use the nearest horizontal distance as the third distance.

[0111] In some embodiments, the second image includes a first sub-image and a second sub-image, the first sub-image includes at least the third marker and the first edge, and the third image includes at least the fourth marker and the first edge; the processing unit is also used to: determine a fourth distance between the third marker and the first edge based on the first sub-image; determine a fifth distance between the fourth marker and the first edge based on the second sub-image; and use the smaller value of the fourth distance and the fifth distance as the second distance.

[0112] In some embodiments, the processing unit is also used to: extract multiple third contour points of the third marker in the first sub-image and multiple fourth contour points of the first edge using a deep learning model; determine a first closest distance between the multiple third contour points and the multiple fourth contour points; use the first closest distance as the fourth distance; and extract multiple fifth contour points of the fourth marker in the second sub-image and multiple sixth contour points of the first edge using the deep learning model; determine a second closest distance between the multiple fifth contour points and the multiple sixth contour points; use the second closest distance as the fifth distance.

[0113] In some embodiments, the first marker is a first lock hole at the bottom corner of the first container; the second marker is a second lock hole at the top corner of the second container; the third marker and the fourth marker are the third lock hole and the fourth lock hole on the top surface of the first container, and the third lock hole and the fourth lock hole are located on the side of the first container close to the second container.

[0114] When the collision warning device 800 provided in the embodiment of the present application is applied to a lifting device, it can determine whether there is a collision risk between the lifted container and the container on the container truck based on the collected image, and send a collision warning signal when there is a collision risk, so that the operator of the lifting device or the automatic control system of the lifting device can make appropriate adjustments to eliminate the collision risk. The image recognition-based approach can improve the accuracy of the collision warning, save a lot of manpower and material resources, and improve the work efficiency.

[0115] Fig. 9 is a schematic structural diagram of a lifting device provided in an embodiment of the present application, Fig. 9 The lifting equipment 900 includes a body 910, a sling 920 and a collision warning device 930. The collision warning device 930 can be the collision warning device described in any of the above embodiments.

[0116] In some embodiments, the collision warning device includes: a first image acquisition unit, a second image acquisition unit, a processing unit and a warning unit.

[0117] The first image acquisition unit is used to acquire a first image of the first container being hoisted and the second container fixed on the container truck. The first image includes at least a first marker and a second marker, the first marker is located at the bottom corner of the first container and close to the second container, and the second marker is located at the top corner of the second container and close to one side of the first container.

[0118] In some embodiments, the first image acquisition unit is disposed on the body 910 .

[0119] In some embodiments, the first image acquisition unit includes a first camera.

[0120] In some embodiments, the first camera is disposed on a roadside door leg.

[0121] The processing unit is used to determine a first distance between the first container and the second container in a vertical direction according to the first image.

[0122] The second image acquisition unit is used to acquire a second image when the first distance is less than a first distance threshold, wherein the second image at least includes a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container.

[0123] In some embodiments, the second image acquisition unit is disposed on the sling 920 .

[0124] In some embodiments, the first image acquisition unit includes at least a second camera and a third camera, which are used to acquire the first sub-image and the second sub-image respectively.

[0125] In some embodiments, the first camera and the second camera are respectively disposed at a first corner and a second corner corresponding to the third marker and the fourth marker.

[0126] In some embodiments, the second camera and the third camera are gun cameras.

[0127] The processing unit is further configured to determine a second distance between the first container and the second container in a horizontal direction according to the second image.

[0128] The warning unit is used to send a collision warning signal when the second distance is less than a second distance threshold.

[0129] In some embodiments, the processing unit is further used to determine a third distance between the first container and the second container in the horizontal direction based on the first image; the warning unit is further used to send the collision warning signal when the third distance is less than a third distance threshold.

[0130] In some embodiments, the processing unit is also used to: use a deep learning model to extract multiple first contour points of the first marker and multiple second contour points of the second marker in the first image; determine the closest vertical distance between the multiple first contour points and the multiple second contour points; and use the closest vertical distance as the first distance.

[0131] In some embodiments, the processing unit is also used to: use a deep learning model to extract multiple first contour points of the first marker and multiple second contour points of the second marker in the first image; determine the nearest horizontal distance between the multiple first contour points and the multiple second contour points; and use the nearest horizontal distance as the third distance.

[0132] In some embodiments, the second image includes a first sub-image and a second sub-image, the first sub-image includes at least the third marker and the first edge, and the third image includes at least the fourth marker and the first edge; the processing unit is also used to: determine a fourth distance between the third marker and the first edge based on the first sub-image; determine a fifth distance between the fourth marker and the first edge based on the second sub-image; and use the smaller value of the fourth distance and the fifth distance as the second distance.

[0133] In some embodiments, the processing unit is also used to: extract multiple third contour points of the third marker in the first sub-image and multiple fourth contour points of the first edge using a deep learning model; determine a first closest distance between the multiple third contour points and the multiple fourth contour points; use the first closest distance as the fourth distance; and extract multiple fifth contour points of the fourth marker in the second sub-image and multiple sixth contour points of the first edge using the deep learning model; determine a second closest distance between the multiple fifth contour points and the multiple sixth contour points; use the second closest distance as the fifth distance.

[0134] In some embodiments, the first marker is a first lock hole at the bottom corner of the first container; the second marker is a second lock hole at the top corner of the second container; the third marker and the fourth marker are the third lock hole and the fourth lock hole on the top surface of the first container, and the third lock hole and the fourth lock hole are located on the side of the first container close to the second container.

[0135] In the technical solution provided in the present application, by providing the above-mentioned collision warning device on the lifting equipment, the accuracy of the collision warning can be improved, manpower and material resources can be saved, and the operating efficiency of the dock or port can be improved.

[0136] Fig.10 It is a schematic structural diagram of an electronic device 1000 provided in an embodiment of the present application. Fig.10 The dotted line in the figure indicates that the unit or module is optional. The electronic device 1000 can be used to implement the method described in the above method embodiment. The electronic device 1000 can be, for example, the terminal device or network device mentioned above; or, the electronic device 1000 can also be a chip or integrated circuit capable of implementing the method of the above embodiment.

[0137] The electronic device 1000 may include one or more processors 1010. The processor 1010 may support the electronic device 1000 to implement the method described in the foregoing method embodiment. The processor 1010 may be a general-purpose processor or a special-purpose processor. For example, the processor may be a central processing unit (CPU). Alternatively, the processor 1010 may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0138] The electronic device 1000 may further include one or more memories 1020. The memory 1020 stores a program, which can be executed by the processor 1010, so that the processor 1010 executes the method described in the above method embodiment. The memory 1020 may be independent of the processor 1010 or integrated in the processor 1010.

[0139] The electronic device 1000 may further include a transceiver 1030. The processor 1010 may communicate with other devices or chips through the transceiver 1030. For example, the processor 1010 may transmit and receive data with other devices or chips through the transceiver 1030.

[0140] The embodiments of the present application also provide a computer-readable storage medium on which executable codes are stored. When the executable codes are executed, the methods in the embodiments of the present application can be implemented.

[0141] The embodiment of the present application also provides a computer program product. The computer program product includes a program, which enables a computer to execute the method in each embodiment of the present application.

[0142] The present application also provides a computer program that enables a computer to execute the methods in the various embodiments of the present application.

[0143] In the several embodiments provided in the present application, it should be understood that the disclosed systems and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A collision warning method, characterized in that: The method comprises: Capturing a first image of a first container being hoisted and a second container fixed on a container truck; wherein the first image at least includes a first marker and a second marker, the first marker being located at a bottom corner of the first container and close to the second container, and the second marker being located at a top corner of the second container and close to the first container; determining a first distance between the first container and the second container in a vertical direction according to the first image; When the first distance is less than a first distance threshold, acquiring a second image, wherein the second image at least includes a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container; determining a second distance between the first container and the second container in a horizontal direction according to the second image; When the second distance is less than a second distance threshold, sending a collision warning signal; The second image includes a first sub-image and a second sub-image, the first sub-image includes at least the third marker and the first edge, and the second sub-image includes at least the fourth marker and the first edge, wherein the third marker and the fourth marker are a third lock hole and a fourth lock hole on the top surface of the first container, and the third lock hole and the fourth lock hole are located on a side of the first container close to the second container; The determining a second distance between the first container and the second container in the horizontal direction according to the second image includes: determining, according to the first sub-image, a fourth distance between the third marker and the first edge; determining a fifth distance between the fourth marker and the first edge according to the second sub-image; The smaller value between the fourth distance and the fifth distance is used as the second distance.

2. The method according to claim 1, characterized in that The method further comprises: determining, according to the first image, a third distance between the first container and the second container in a horizontal direction; When the third distance is less than a third distance threshold, the collision warning signal is sent.

3. The method according to claim 1, characterized in that The determining, according to the first image, a first distance between the first container and the second container in a vertical direction includes: Extracting a plurality of first contour points of the first marker and a plurality of second contour points of the second marker in the first image using a deep learning model; determining the shortest vertical distances between the plurality of first contour points and the plurality of second contour points; The shortest vertical distance is taken as the first distance.

4. The method according to claim 2, characterized in that: The determining, according to the first image, a third distance between the first container and the second container in the horizontal direction includes: Extracting a plurality of first contour points of the first marker and a plurality of second contour points of the second marker in the first image using a deep learning model; Determining the shortest horizontal distances between the plurality of first contour points and the plurality of second contour points; The closest horizontal distance is used as the third distance.

5. The method according to claim 1, characterized in that Determining a fourth distance between the third marker and the first edge according to the first sub-image includes: Extracting a plurality of third contour points of the third marker and a plurality of fourth contour points of the first edge in the first sub-image by using a deep learning model; determining a first closest distance between the plurality of third contour points and the plurality of fourth contour points; Taking the first shortest distance as the fourth distance; Determining a fifth distance between the second marker and the first edge according to the second sub-image includes: Extracting a plurality of fifth contour points of the fourth marker and a plurality of sixth contour points of the first edge in the second sub-image using the deep learning model; determining a second closest distance between the plurality of fifth contour points and the plurality of sixth contour points; The second shortest distance is used as the fifth distance.

6. The method according to any one of claims 1 to 5, characterized in that The first marker is a first lock hole at the bottom corner of the first container; The second marker is a second lock hole at the top corner of the second container.

7. A collision warning device, characterized in that: include: A first image acquisition unit is used to acquire a first image of the first container being hoisted and the second container fixed on the container truck; wherein the first image includes at least a first marker and a second marker, the first marker is located at a bottom corner of the first container and close to the second container, and the second marker is located at a top corner of the second container and close to one side of the first container; a processing unit, configured to determine a first distance between the first container and the second container in a vertical direction according to the first image; a second image acquisition unit, configured to acquire a second image when the first distance is less than a first distance threshold, wherein the second image at least includes a third marker and a fourth marker located on the top surface of the first container and a first edge of the second container close to the first container; The processing unit is further used to: determine a second distance between the first container and the second container in the horizontal direction according to the second image; an early warning unit, configured to send a collision early warning signal when the second distance is less than a second distance threshold; The second image includes a first sub-image and a second sub-image, the first sub-image includes at least the third marker and the first edge, and the second sub-image includes at least the fourth marker and the first edge, wherein the third marker and the fourth marker are a third lock hole and a fourth lock hole on the top surface of the first container, and the third lock hole and the fourth lock hole are located on a side of the first container close to the second container; the processing unit is further used for: determining, according to the first sub-image, a fourth distance between the third marker and the first edge; determining a fifth distance between the fourth marker and the first edge according to the second sub-image; The smaller value between the fourth distance and the fifth distance is used as the second distance.

8. The device according to claim 7, characterized in that The processing unit is further configured to determine, based on the first image, a third distance between the first container and the second container in a horizontal direction; The warning unit is further configured to send the collision warning signal when the third distance is less than a third distance threshold.

9. The device according to claim 7, characterized in that The processing unit is also used for: Extracting a plurality of first contour points of the first marker and a plurality of second contour points of the second marker in the first image using a deep learning model; determining the shortest vertical distances between the plurality of first contour points and the plurality of second contour points; The shortest vertical distance is taken as the first distance.

10. The device according to claim 8, characterized in that The processing unit is also used for: Extracting a plurality of first contour points of the first marker and a plurality of second contour points of the second marker in the first image using a deep learning model; Determining the shortest horizontal distances between the plurality of first contour points and the plurality of second contour points; The closest horizontal distance is used as the third distance.

11. The device according to claim 7, characterized in that The processing unit is also used for: Extracting a plurality of third contour points of the third marker and a plurality of fourth contour points of the first edge in the first sub-image by using a deep learning model; determining a first closest distance between the plurality of third contour points and the plurality of fourth contour points; taking the first shortest distance as the fourth distance; and, Extracting a plurality of fifth contour points of the fourth marker and a plurality of sixth contour points of the first edge in the second sub-image using the deep learning model; determining a second closest distance between the plurality of fifth contour points and the plurality of sixth contour points; The second shortest distance is used as the fifth distance.

12. The device according to any one of claims 7 to 11, characterized in that The first marker is a first lock hole at the bottom corner of the first container; The second marker is a second lock hole at the top corner of the second container.

13. A lifting device, characterized in that: The invention comprises: a machine body, a sling and a collision warning device as claimed in any one of claims 7 to 12.

14. The lifting device according to claim 13, characterized in that The first image acquisition unit is arranged on the roadside door leg of the machine body, and the second image acquisition unit is arranged on the hoist.

15. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program enables a computer to execute the method according to any one of claims 1 to 6.

Images