Container door abnormity identification method and system and container crane
By integrating the data of the lidar and camera, it is possible to determine whether the container door is open, which solves the problem of low judgment accuracy in the existing technology, and achieves more efficient and reliable door detection.
Patent Information
- Application Number
- CN202311701675.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-11
- Publication Date
- 2025-06-13
AI Technical Summary
The prior art is not accurate enough when judging whether the container door is open, which can easily lead to serious accidents.
By combining the data of lidar and cameras, integrating radar point cloud data and image data, extracting the angle between the box door point cloud and the side point cloud, and determining whether the box door is open through a pre-trained classification model, comprehensively determining the results to improve accuracy.
It improves the accuracy of container door opening judgment, can remotely identify box door abnormalities, reduces the impact of dropping and shaking of spreaders on recognition accuracy, and improves operating efficiency and robustness.
Smart Images
Figure CN120143175A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of automated yards, and specifically relates to a method and system for identifying abnormal container doors and a container crane. Background Art
[0002] With the continuous popularization of automated yards, a large number of container cranes have been introduced in automated yards to grab or stack containers. In an automated yard, whether the container door is properly closed has a significant impact on the entire workflow of grabbing and releasing containers. Grabbing or releasing a container with the door open may cause serious accidents. Currently, image recognition methods are mostly used to determine whether the container door is open, but the discrimination results of such methods are not precise enough. In summary, how to improve the accuracy of judging whether the container door is open is a technical problem to be solved in this field. Summary of the Invention
[0003] In view of this, this application provides a method and system for identifying abnormal container doors and a container crane, which can comprehensively use radar data and image data to determine whether the door is open, improving the judgment accuracy.
[0004] In a first aspect, a method for identifying abnormal container doors provided by this application is applied to a container crane. The container crane includes a trolley platform, the trolley platform is connected with a spreader, and a lidar and a camera are arranged at the bottom of the trolley platform. The lidar is used to detect radar point cloud data including a container, and the camera is used to detect image data including the container. The method for identifying abnormal container doors includes: respectively obtaining the radar point cloud data and the image data detected by the lidar and the camera facing downward; fusing the radar point cloud data and the image data to obtain point cloud fusion data of a target container; extracting the door point cloud and the side point cloud of the target container from the point cloud fusion data; if the angle between the door point cloud and the side point cloud is greater than a preset angle, generating a first judgment result; inputting the image data into a pre-trained classification model to obtain a second judgment result through discrimination; and obtaining an abnormal door judgment result according to the first judgment result, the second judgment result, and a preset weight parameter.
[0005] This aspect can fuse radar point cloud data and image data to obtain more accurate radar point cloud. Among them, based on the image coordinates, the radar point cloud of the target container can be accurately screened based on the box contour, and the shape contour of the target container can be accurately obtained through the radar point cloud in the point cloud fusion data. Extract the door point cloud and side point cloud of the container from the point cloud fusion data corresponding to the target container, calculate the included angle between the door point cloud and the side point cloud, and then the first judgment result calculated by the radar point cloud can be obtained. Then, detect whether the door is open through image recognition to obtain the second judgment result. Finally, combine the weight parameter, the first judgment result and the second judgment result to finally obtain the door abnormality judgment result to know whether the door of the target container is opened. This aspect can comprehensively judge whether the container door is opened based on radar data and image data, and the comprehensive judgment results in multiple aspects can improve the judgment accuracy.
[0006] Combined with the first aspect, in a possible implementation manner, the fusing of the radar point cloud data and the image data to obtain the point cloud fusion data of the target container includes: obtaining the external parameter matrix of the lidar and the camera; based on the image recognition algorithm, recognizing the image data to obtain the container image coordinates of the target container; according to the external parameter matrix, the internal parameter matrix of the camera, the radar point cloud data and the image data, projecting the radar point cloud data onto the image data; and according to the container image coordinates, screening the target point cloud data of the target container from the radar point cloud data; the point cloud fusion data includes the target point cloud data; wherein, the extracting of the door point cloud and the side point cloud of the target container from the point cloud fusion data includes: extracting the door point cloud and the side point cloud from the target point cloud data.
[0007] Combined with the first aspect, in a possible implementation manner, the extracting of the door point cloud and the side point cloud of the target container from the point cloud fusion data includes: according to the preset door features, recognizing the door edge coordinates of the target container from the image data, and the first side coordinates and the second side coordinates adjacent to the door edge coordinates; extracting the door edge point cloud corresponding to the door edge coordinates from the point cloud fusion data, and extracting the first side point cloud and the second side point cloud corresponding to the first side coordinates and the second side coordinates respectively; and dividing the door edge point cloud into two halves to obtain the first door point cloud and the second door point cloud; the first door point cloud is adjacent to the first side point cloud, and the second door point cloud is adjacent to the second side point cloud.
[0008] In combination with the first aspect, in a possible implementation manner, the generating the first judgment result when the included angle between the door point cloud and the side point cloud is greater than a preset included angle includes: calculating a first included angle between the first door point cloud and the first side point cloud; calculating a second included angle between the second door point cloud and the second side point cloud; and generating the first judgment result if the first included angle and / or the second included angle is greater than the preset included angle.
[0009] In combination with the first aspect, in a possible implementation manner, the extracting the door point cloud and the side point cloud of the target container from the point cloud fusion data includes: extracting two pairs of adjacent length and width edge point clouds of the target container from the point cloud fusion data; wherein, the generating the first judgment result when the included angle between the door point cloud and the side point cloud is greater than a preset included angle includes: calculating an edge point cloud included angle of the adjacent length and width edge point clouds; and generating the first judgment result if one of the edge point cloud included angles is greater than the preset included angle.
[0010] In combination with the first aspect, in a possible implementation manner, the extracting two pairs of adjacent length and width edge point clouds of the target container from the point cloud fusion data includes: determining a first long side coordinate, a second long side coordinate, a first short side coordinate, and a second short side coordinate of the target container according to the image data; the second long side coordinate is on the opposite side of the first long side coordinate, and the second short side coordinate is on the opposite side of the first short side coordinate; and respectively extracting a first long side point cloud, a second long side point cloud, a first short side point cloud, and a second short side point cloud corresponding to the first long side coordinate, the second long side coordinate, the first short side coordinate, and the second short side coordinate from the point cloud fusion data.
[0011] In combination with the first aspect, in a possible implementation manner, the calculating the edge point cloud included angle of the adjacent length and width edge point clouds includes: calculating a third included angle between the first long side edge point cloud and the first short side edge point cloud; and calculating a fourth included angle between the second long side edge point cloud and the second short side edge point cloud; wherein, the generating the first judgment result if one of the edge point cloud included angles is greater than the preset included angle includes: generating the first judgment result if the third included angle or the fourth included angle is greater than the preset included angle.
[0012] In combination with the first aspect, in a possible implementation manner, after fusing the radar point cloud data and the image data to obtain the point cloud fusion data of the target container, the container door abnormality recognition method further includes: obtaining the number of layers or the top surface height of the target container; obtaining a corresponding point cloud density according to the number of layers or the top surface height; and performing point cloud filtering on the point cloud fusion data according to the point cloud density.
[0013] Second aspect, the present application provides a container door abnormality recognition system, which is applied to a container crane. The container crane includes a trolley platform, and a spreader is connected to the trolley platform. A lidar and a camera are arranged at the bottom of the trolley platform. The lidar is used to detect lidar point cloud data including the container, and the camera is used to detect image data including the container. The container door abnormality recognition system includes: a data acquisition module, which is respectively communicatively connected to the lidar and the camera, and the data acquisition module is configured to: respectively acquire the lidar point cloud data and the image data detected by the lidar and the camera facing downward; a fusion module, which is communicatively connected to the data acquisition module, and the fusion module is configured to: fuse the lidar point cloud data and the image data to obtain point cloud fusion data of the target container; and a door abnormality detection module, which is communicatively connected to the fusion module, and the door abnormality detection module is configured to: extract the door point cloud and the side point cloud of the target container from the point cloud fusion data; if the included angle between the door point cloud and the side point cloud is greater than a preset included angle, a first judgment result is generated.
[0014] The second aspect is the system corresponding to the first aspect, and the technical effects of the second aspect will not be elaborated here.
[0015] Third aspect, the present application provides a container crane, including: a container crane body; a trolley platform, which is arranged on the container crane body, and a lidar and a camera are arranged at the bottom of the trolley platform. The lidar is used to detect lidar point cloud data including the container, and the camera is used to detect image data including the container; and the aforementioned container door abnormality recognition system.
[0016] The third aspect includes the system of the second aspect, and the technical effects of the third aspect will not be elaborated here.
[0017] Through the solution of the present invention, the lidar and camera devices installed under the mobile trolley can be used to detect the abnormal opening of the container door in the yard. This invention can remotely identify the abnormal door, without the spreader descending to a certain height above the target container for detection, eliminating the influence of spreader shaking on the recognition accuracy and improving the operation efficiency. Moreover, through the lidar and camera fusion perception algorithm, the limitations of a single sensor are eliminated, especially in the case of changing and complex environments, improving the robustness of the door abnormality detection. Description of the Drawings
[0018] Figure 1 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by an embodiment of the present application.
[0019] Figure 2The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0020] Figure 3 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0021] Figure 4 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0022] Figure 5 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0023] Figure 6 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0024] Figure 7 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0025] Figure 8 The figure shows a schematic diagram of the system structure of a container door abnormality recognition system provided by an embodiment of the present application.
[0026] Figure 9 The figure shows a schematic diagram of the abnormal state of the container door.
[0027] Figure 10 The figure shows a schematic diagram of the system structure of a container crane provided by an embodiment of the present application. Detailed implementation manners
[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.
[0029] An exemplary container door abnormality recognition method is as follows:
[0030] The present application provides a method for identifying abnormal container doors, which is applied to a container crane. The container crane includes a moving trolley, and the moving trolley includes a trolley platform and a spreader. A lidar and a camera are provided at the bottom of the trolley platform. Specifically, the lidar and the camera can be installed at the corners of the bottom of the trolley platform to increase the detection range and reduce the detection blind area. The lidar is used to detect lidar point cloud data including the container, and the camera is used to detect image data including the container.
[0031] Figure 1 The following is a schematic diagram of the method steps of a method for identifying abnormal container doors provided by an embodiment of the present application. As Figure 1 shown, the method for identifying abnormal container doors includes:
[0032] Step 110: Obtain lidar point cloud data and image data detected by the lidar and the camera facing downward respectively.
[0033] In this step, the lidar point cloud data is detected by the lidar, and the image data is captured by the camera. This step obtains the detection data of the lidar and the camera.
[0034] Step 120: Fuse the lidar point cloud data and the image data to obtain point cloud fusion data of the target container.
[0035] In this step, since the lidar point cloud data detected by the lidar is likely to contain interfering point clouds. Through data fusion, the lidar point cloud data can be projected onto the image data, and the filtered lidar point cloud data can be obtained according to the image data as the point cloud fusion data to accurately obtain the lidar point cloud data corresponding to the target container.
[0036] Step 130: Extract the door point cloud and the side point cloud of the target container from the point cloud fusion data.
[0037] Step 140: If the angle between the door point cloud and the side point cloud is greater than a preset angle, generate a first judgment result.
[0038] In this step, the included angle between the door point cloud and the side point cloud of the container is calculated. Since the point cloud data is detected from top to bottom, the door point cloud is actually the top edge of the door. Therefore, the preset included angle can be set to 90°. When the door is normally closed, this included angle should be 90°. If this included angle is greater than 90°, it indicates that the door is in the open state at this time. A first judgment result is generated according to the comparison result of the included angle. When the door is open, the first judgment result is 1; when the door is closed, the first judgment result is 0. Specifically, the door of the target container can be a single-leaf door or a double-leaf door. When the target container is a single-leaf door, the target object for calculation is the included angle between the door point cloud of the single-leaf door and the side point cloud; when the target container is a double-leaf door, the target object for calculation is the included angle between the door point cloud of any one of the doors and the side point cloud.
[0039] Step 150: Input the image data into the pre-trained classification model to obtain a second judgment result through discrimination.
[0040] In this step, a deep learning classification model can be pre-trained. The deep learning classification model includes an image model of the door being open. The deep learning classification algorithm is used to compare the image data collected by the camera with the image model of the door being open to obtain the similarity between the image data and the image of the door being open, and the confidence level is output, that is, the second judgment result is output. The higher the similarity, the higher the confidence level. The range of the confidence level is 0 to 1.
[0041] Step 160: Obtain the door abnormality judgment result according to the first judgment result, the second judgment result, and the preset weight parameters.
[0042] In this step, the preset weight parameters include the first weight of the first judgment result and the second weight of the second judgment result. The first weight and the second weight can be adjusted through experiments and experience. For example, when the first judgment result obtained from the radar point cloud is relatively accurate, the first weight can be set in a higher range: 0.5 to 0.9; for example, when the second judgment result obtained from image recognition is relatively accurate, the second weight can be set in a higher range: 0.7 to 0.9. In this step, the following formula is used to calculate the door abnormality judgment result: Door abnormality judgment result = First judgment result * First weight + Second judgment result * Second weight. Generally, when the door abnormality judgment result is greater than 0.8, it can be determined that the door is open.
[0043] This embodiment can fuse radar point cloud data and image data to obtain a more accurate radar point cloud, wherein the radar point cloud of the target container can be accurately screened based on the box contour through the image coordinates, and the shape contour of the target container can be accurately known through the radar point cloud in the point cloud fusion data. The door point cloud and the side point cloud are extracted from the point cloud fusion data corresponding to the target container, and the angle between the door point cloud and the side point cloud is calculated to obtain the first judgment result calculated by the radar point cloud. Then, image recognition is used to detect whether the door is open to obtain the second judgment result. Finally, the weight parameter and the first judgment result and the second judgment result are combined to finally obtain the door abnormality judgment result to know whether the door of the target container is open. This embodiment can comprehensively judge whether the container door is open by combining radar data and image data, and the comprehensive judgment results in many aspects can improve the judgment accuracy.
[0044] Through the solution of the present invention, the laser radar and camera equipment installed under the mobile vehicle can be used to detect abnormal opening of container doors in the yard. This invention can identify abnormal door openings at a distance without the need for the spreader to descend to a certain height above the target container for detection, eliminating the impact of spreader shaking on recognition accuracy and improving work efficiency. Furthermore, through the fusion perception algorithm of the laser radar and camera, the limitations of a single sensor are eliminated, especially in the case of a changeable and complex environment, the robustness of abnormal door detection is improved.
[0045] Figure 2 Shown is a schematic diagram of the method steps of a method for identifying abnormalities in container doors provided by another embodiment of the present application.
[0046] In one embodiment, if Figure 2 As shown, step 120 includes:
[0047] Step 121: Obtain the external parameter matrices of the laser radar and camera.
[0048] Step 122: Based on an image recognition algorithm, the image data is recognized to obtain the container image coordinates of the target container.
[0049] In this step, a deep learning method can be used to pre-set the characteristics of the container, and the yolov8 model can be used to perform feature recognition on the image data, so as to extract the image of the target container from the image data of the camera. The camera specifically uses a deep sensing camera, so that the spatial coordinates of each pixel point can be obtained. After the target container is extracted, the image coordinates of the target container can be obtained, that is, the spatial coordinates of the pixel points of the target container image can be obtained.
[0050] Step 123: project the radar point cloud data onto the image data according to the external parameter matrix, the internal parameter matrix of the camera, the radar point cloud data and the image data.
[0051] In this step, through the external parameter matrix and the internal parameter matrix, the radar point cloud data and the image data can be correlated with each other, so that the radar point cloud data can be projected onto the image data, that is, the reference systems of the radar point cloud data and the image data are unified. After the reference system is unified, the radar point cloud data can be accurately filtered according to the coordinates of the image data. For example, when it is necessary to filter the point cloud data of the target container, according to the spatial coordinates of the target container in the image data, the shape outline of the target container can be obtained, and the point cloud data can be filtered from all the radar point cloud data detected by the lidar according to the shape outline. Specifically, the calculation method of projecting the radar point cloud data onto the image data is as follows:
[0052]
[0053] In the above formula, x, y, and z are the coordinates of the radar point cloud data, u and v are the coordinates of the image data, K is the internal parameter matrix of the camera, and the external parameter matrix includes: the rotation matrix R and the translation matrix T of the lidar relative to the camera.
[0054] Step 124: Filter the target point cloud data of the target container from the radar point cloud data according to the container image coordinates.
[0055] The aforementioned point cloud fusion data includes target point cloud data. In this step, based on the image coordinates of the target container obtained by the image recognition algorithm in step 122 and combined with the corresponding relationship formed by the data projection in step 123, the radar point cloud data within the spatial position of the target container can be extracted and filtered, that is, the radar point cloud corresponding to the target container is obtained, and the radar point cloud outside the target container is excluded, that is, the target point cloud data corresponding to the target container is obtained. The spatial position of the target container can be clearly obtained through the radar point cloud.
[0056] Based on steps 121-124, step 130 includes:
[0057] Step 131: Extract the door point cloud and the side point cloud from the target point cloud data.
[0058] Figure 3 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0059] In one embodiment, as Figure 3 shown, step 130 includes:
[0060] Step 131: Identify the door edge coordinates of the target container, as well as the first side coordinates and the second side coordinates adjacent to the door edge coordinates, from the image data according to the preset door characteristics.
[0061] In this step, the image features of the top edge of the container door are preset in advance. Based on the container door features, the coordinates of the container door edge at the top of the container door are extracted from the image data. At the same time, according to the coordinates of the container door edge, the two side edges adjacent to the top of the target container and the container door are determined, and the coordinates of the first side edge and the second side edge are extracted.
[0062] Step 132: Extract the container door edge point cloud corresponding to the container door edge coordinates from the point cloud fusion data, and extract the first side edge point cloud and the second side edge point cloud corresponding to the coordinates of the first side edge and the second side edge respectively.
[0063] In this step, based on the aforementioned process of projecting the radar point cloud data onto the image data, since the coordinates of the radar point cloud data and the image data are unified, the container door edge point cloud can be screened out according to the container door edge coordinates. Similarly, the first side edge point cloud and the second side edge point cloud are screened out.
[0064] Step 133: Divide the container door edge point cloud into two halves to obtain the first container door point cloud and the second container door point cloud.
[0065] In this step, the first container door point cloud is adjacent to the first side edge point cloud, and the second container door point cloud is adjacent to the second side edge point cloud. When the container door edge point cloud is divided into two halves, one half corresponds to one container door, that is, the first container door point cloud and the second container door point cloud. Then, linear fitting can be performed on the first container door point cloud, the second container door point cloud, the first side edge point cloud, and the second side edge point cloud.
[0066] Specifically, step 140 includes:
[0067] Step 141: Calculate the first included angle between the first container door point cloud and the first side edge point cloud.
[0068] Step 142: Calculate the second included angle between the second container door point cloud and the second side edge point cloud.
[0069] Step 143: If the first included angle and / or the second included angle is greater than the preset included angle, generate a first judgment result with a value of 1.
[0070] In the above steps, when the first included angle is greater than the preset included angle, it means that the first container door is open; when the second included angle is greater than the preset included angle, it means that the second container door is open. When both container doors are open or one container door is open, the first judgment result is generated. Figure 9 The figure shows a schematic diagram of the abnormal state of the container door. From left to right, they are the closed state of the container door, the open state of the left container door, the open state of the right container door, and the state where both container doors are open. When both the first included angle and the second included angle are equal to the preset included angle, it means that the container door is in the closed state at this time, and a judgment result with a value of 0 is generated.
[0071] Figure 4 The figure shows a schematic diagram of the method steps of a container door abnormal recognition method provided by another embodiment of the present application.
[0072] In one embodiment, as Figure 4 shown, step 130 includes:
[0073] Step 134, extracting two pairs of adjacent length and width edge point clouds of the target container from the point cloud fusion data.
[0074] Step 140 includes:
[0075] Step 144, calculating the included angle between the edge point clouds of the adjacent length and width edge point clouds.
[0076] Step 145, if an included angle between the edge point clouds is greater than a preset included angle, generating a first judgment result with a value of 1.
[0077] In this embodiment, without knowing the position of the container door, by comparing two pairs of adjacent length and width edges, the position of the container door point cloud is determined, that is, by comparing two sets of length and width edges on the top surface of the target container, the edge of the container door will not be missed. Therefore, two pairs of adjacent length and width edge point clouds correspond to two included angles between the edge point clouds. As long as one included angle between the edge point clouds is greater than the preset included angle, it indicates that the container door is in an unclosed state. When an included angle between the edge point clouds is equal to the preset included angle, it indicates that the container door is in a closed state at this time, and a judgment result with a value of 0 is generated.
[0078] Specifically, as Figure 5 shown, step 134 includes:
[0079] Step 1341, determining the first long side coordinate, the second long side coordinate, the first short side coordinate, and the second short side coordinate of the target container according to the image data.
[0080] In this step, the second long side coordinate is located on the opposite side of the first long side coordinate, and the second short side coordinate is located on the opposite side of the first short side coordinate. The top surface of the target container is rectangular. The long side image feature and the short side image feature of the four edges on the top surface of the target container are preset in advance. Based on the long side image feature, two opposite long side coordinates are extracted from the image data as the first long side coordinate and the second long side coordinate. Based on the short side image feature, two opposite short side coordinates are extracted from the image data as the first short side coordinate and the second short side coordinate.
[0081] Step 1342, respectively extracting the first long side point cloud, the second long side point cloud, the first short side point cloud, and the second short side point cloud corresponding to the first long side coordinate, the second long side coordinate, the first short side coordinate, and the second short side coordinate from the point cloud fusion data.
[0082] In this step, based on the aforementioned process of projecting radar point cloud data onto image data, since the coordinates of the radar point cloud data and the image data are unified, the long-edge point cloud can be filtered out according to the long-edge coordinates, and the short-edge point cloud can be filtered out according to the short-edge coordinates. Then, linear fitting is performed on the two long-edge point clouds and the two short-edge point clouds. The first long-edge point cloud and the first short-edge point cloud form a pair of adjacent long and wide edge point clouds, and the second long-edge point cloud and the second short-edge point cloud form a pair of adjacent long and wide edge point clouds. These two pairs of adjacent long and wide edge point clouds correspond to two edge point cloud angles. As long as one of the edge point cloud angles is greater than the preset angle, it indicates that the container door is abnormal.
[0083] Specifically, step 144 includes:
[0084] Step 1441: Calculate the third angle between the first long-edge edge point cloud and the first short-edge edge point cloud.
[0085] Step 1442: Calculate the fourth angle between the second long-edge edge point cloud and the second short-edge edge point cloud.
[0086] In step 1441 and step 1442, calculate the angles between the long-edge point cloud and the short-edge point cloud after linear fitting. When judging the angles, respectively judge the magnitude relationship between the third angle and the preset angle, and the magnitude relationship between the fourth angle and the preset angle.
[0087] Step 145 includes:
[0088] Step 1451: If the third angle or the fourth angle is greater than the preset angle, generate a first judgment result with a value of 1. When both the third angle and the fourth angle are equal to the preset angle, it indicates that the container door is in the closed state at this time, and a judgment result with a value of 0 is generated.
[0089] Figure 6 The figure shows a schematic diagram of the method steps of a container door abnormality recognition method provided by another embodiment of the present application.
[0090] In one embodiment, as Figure 6 shown, after step 120, the container door abnormality recognition method further includes:
[0091] Step 170: Obtain the number of layers or the top surface height of the target container.
[0092] In this step, the number of layers or the top surface height of the target container reflects which layer the current target container is located in the stacked containers, and can be input by the staff or detected by the system.
[0093] Step 180: Obtain the corresponding point cloud density according to the number of layers or the top surface height.
[0094] In this step, due to the detection characteristics of the lidar, the lidar has different point cloud densities for detection results at different height positions. Different point cloud densities corresponding to different layers or top surface heights are preset in advance. After obtaining the number of layers or the top surface height of the target container, in this step, the corresponding point cloud density is determined according to the corresponding relationship.
[0095] Step 190: Perform point cloud filtering on the point cloud fusion data according to the point cloud density.
[0096] In this step, by performing clustering filtering on the point cloud fusion data, the point cloud regions that do not meet the point cloud density can be filtered out, that is, the background noise is filtered out, so as to obtain the point cloud fusion data that meets the point cloud density. The point cloud fusion data at this layer is the point cloud data of the target container, thereby further improving the signal-to-noise ratio of the radar point cloud of the target container. Then, the angle between the door point cloud and the side point cloud of the target container is calculated according to the filtered point cloud fusion data.
[0097] Figure 7 The following shows a schematic flowchart of a method for identifying abnormal container doors provided by an embodiment of the present application. In one embodiment, as Figure 7 shown, first, lidar detection and camera shooting are performed. The image data obtained by camera shooting is subjected to image recognition to obtain the image coordinates of the target container. Then, data fusion of the radar point cloud and the image data is performed, and the angle between the door and the side is calculated according to the fusion data, so as to determine the first judgment result. In addition, through image recognition, the image data is input into the deep learning classification model to obtain the second judgment result. Finally, the abnormal door judgment result is obtained by comprehensively considering the first judgment result and the second judgment result.
[0098] An exemplary container door abnormality recognition system is as follows:
[0099] Figure 8 The following shows a schematic system structure diagram of a container door abnormality recognition system provided by an embodiment of the present application. The present application also provides a container door abnormality recognition system, as Figure 8 shown, this system is applied to a container crane. The container crane includes a trolley platform, and one or more pairs of lidars and cameras are arranged at the bottom of the trolley platform. The lidar is used to detect the radar point cloud data including the container, and the camera is used to detect the image data including the container; the container door abnormality recognition system includes: a data acquisition module 101, a fusion module 102, and a door abnormality detection module 103.
[0100] The data acquisition module 101 is respectively communicatively connected to the lidar and the camera. The data acquisition module 101 is configured to: respectively acquire the radar point cloud data and the image data detected by the lidar and the camera facing downward.
[0101] The fusion module 102 is communicatively connected to the data acquisition module 101, and the fusion module 102 is configured to: fuse the radar point cloud data and the image data to obtain the point cloud fusion data of the target container.
[0102] The container door abnormality detection module 103 is communicatively connected to the fusion module 102, and the container door abnormality detection module 103 is configured to: extract the container door point cloud and the side point cloud of the target container from the point cloud fusion data; if the included angle between the container door point cloud and the side point cloud is greater than a preset included angle, generate a first judgment result.
[0103] In this embodiment, the radar point cloud data and the image data can be fused to obtain a more accurate radar point cloud. Among them, through the image coordinates, the radar point cloud of the target container can be accurately screened based on the container contour. Through the radar point cloud in the point cloud fusion data, the shape contour of the target container can be accurately known. By extracting the container door point cloud and the side point cloud from the point cloud fusion data corresponding to the target container and calculating the included angle between the container door point cloud and the side point cloud, it can be conveniently known whether the container door is in the open state. If the included angle is greater than the preset included angle, it means that the container door is open, and a first judgment result is generated for prompting.
[0104] An exemplary container crane is as follows:
[0105] Figure 10 The following shows a schematic structural diagram of a container crane system provided by an embodiment of the present application. The present application provides a container crane, which includes a container crane body, a moving trolley, and the aforementioned container door abnormality recognition system, as Figure 10 shown, the moving trolley includes a trolley platform 10 and a spreader. The trolley platform 10 is arranged on the container crane body. A lidar 11 and a camera 12 are arranged at the bottom of the trolley platform 10. The lidar 11 is used to detect the radar point cloud data including the container, and the camera 12 is used to detect the image data including the container. Specifically, the lidar 11 and the camera 12 can be adjacently arranged at the same position at the bottom of the trolley platform 10. For example, a set of lidar 11 and camera 12 are arranged at a corner at the bottom of the trolley platform 10, and multiple sets of lidar 11 and camera 12 can be arranged at multiple corners at the bottom of the trolley platform 10. In addition, the lidar 11 and the camera 12 can also be respectively arranged at two corners at the bottom of the trolley platform 10. In some embodiments, multiple pairs of lidar 11 and camera 12 are arranged at the bottom of the trolley platform 10.
[0106] The basic principles of the present application have been described in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are only examples and not limitations. It cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present application. Additionally, the specific details disclosed above are only for illustrative and easy-to-understand purposes, rather than limitations. These details do not limit the present application to necessarily implementing with the above specific details.
[0107] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present application are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or", and can be used interchangeably with it, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with it.
[0108] It should also be noted that in the devices, equipment, and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present application.
[0109] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be very apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features invented herein.
[0110] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present application shall be included within the protection scope of the present application.
Claims
1. A method for identifying abnormal container doors, characterized in that, it is applied to a container crane, the container crane includes a trolley platform, the trolley platform is connected with a spreader, a lidar and a camera are arranged at the bottom of the trolley platform, the lidar is used to detect radar point cloud data including the container, and the camera is used to detect image data including the container; The method for identifying abnormal container doors includes: respectively obtaining the radar point cloud data and the image data detected by the lidar and the camera facing downward; fusing the radar point cloud data and the image data to obtain point cloud fusion data of the target container; extracting the door point cloud and the side point cloud of the target container from the point cloud fusion data; if the included angle between the door point cloud and the side point cloud is greater than a preset included angle, generating a first judgment result; inputting the image data into a pre-trained classification model to obtain a second judgment result by discrimination; and obtaining an abnormal door judgment result according to the first judgment result, the second judgment result and a preset weight parameter.
2. The method for identifying abnormal container doors according to claim 1, characterized in that, the fusing the radar point cloud data and the image data to obtain point cloud fusion data of the target container includes: obtaining the external parameter matrix of the lidar and the camera; identifying the image data based on an image recognition algorithm to obtain the container image coordinates of the target container; projecting the radar point cloud data onto the image data according to the external parameter matrix, the internal parameter matrix of the camera, the radar point cloud data and the image data; and screening the target point cloud data of the target container from the radar point cloud data according to the container image coordinates; the point cloud fusion data includes the target point cloud data; wherein, the extracting the door point cloud and the side point cloud of the target container from the point cloud fusion data includes: extracting the door point cloud and the side point cloud from the target point cloud data.
3. The method for identifying abnormal container doors according to claim 1, characterized in that, the extracting the door point cloud and the side point cloud of the target container from the point cloud fusion data includes: identifying the door edge coordinates of the target container, as well as the first side coordinates and the second side coordinates adjacent to the door edge coordinates from the image data according to a preset door feature; extracting the door edge point cloud corresponding to the door edge coordinates from the point cloud fusion data, and extracting the first side point cloud and the second side point cloud corresponding to the first side coordinates and the second side coordinates respectively; and dividing the door edge point cloud into two halves to obtain a first door point cloud and a second door point cloud; the first door point cloud is adjacent to the first side point cloud, and the second door point cloud is adjacent to the second side point cloud.
4. The method for identifying abnormal container doors according to claim 3, characterized in that, the if the included angle between the door point cloud and the side point cloud is greater than a preset included angle, generating a first judgment result includes: Calculate a first angle between the first container door point cloud and the first side point cloud; Calculate a second angle between the second container door point cloud and the second side point cloud; and If the first angle and / or the second angle is greater than the preset angle, generate the first judgment result.
5. The container door abnormality recognition method according to claim 1, wherein, The extracting the container door point cloud and the side point cloud of the target container from the point cloud fusion data includes: Extracting two pairs of adjacent length and width edge point clouds of the target container from the point cloud fusion data; wherein, the generating the first judgment result if the angle between the container door point cloud and the side point cloud is greater than the preset angle includes: Calculating an edge point cloud angle of the adjacent length and width edge point clouds; and If one of the edge point cloud angles is greater than the preset angle, generate the first judgment result.
6. The container door abnormality recognition method according to claim 5, wherein, The extracting two pairs of adjacent length and width edge point clouds of the target container from the point cloud fusion data includes: Determining a first long side coordinate, a second long side coordinate, a first short side coordinate, and a second short side coordinate of the target container according to the image data; the second long side coordinate is on the opposite side of the first long side coordinate, and the second short side coordinate is on the opposite side of the first short side coordinate; and Respectively extracting a first long side point cloud, a second long side point cloud, a first short side point cloud, and a second short side point cloud corresponding to the first long side coordinate, the second long side coordinate, the first short side coordinate, and the second short side coordinate from the point cloud fusion data.
7. The container door abnormality recognition method according to claim 6, wherein, The calculating the edge point cloud angle of the adjacent length and width edge point clouds includes: Calculating a third angle between the first long side edge point cloud and the first short side edge point cloud; and Calculating a fourth angle between the second long side edge point cloud and the second short side edge point cloud; wherein, the generating the first judgment result if one of the edge point cloud angles is greater than the preset angle includes: If the third angle or the fourth angle is greater than the preset angle, generate the first judgment result.
8. The container door abnormality recognition method according to claim 1, wherein, After fusing the radar point cloud data and the image data to obtain the point cloud fusion data of the target container, the container door abnormality recognition method further includes: Obtaining the number of layers or the top surface height of the target container; Obtaining a corresponding point cloud density according to the number of layers or the top surface height; and Performing point cloud filtering on the point cloud fusion data according to the point cloud density.
9. A container door abnormality recognition system, wherein, Applied to a container crane, the container crane includes a trolley platform, the trolley platform is connected with a spreader, a lidar and a camera are arranged at the bottom of the trolley platform, the lidar is used for detecting radar point cloud data including a container, and the camera is used for detecting image data including a container; the container door abnormality recognition system includes: A data acquisition module, communicatively connected to the lidar and the camera respectively, and the data acquisition module is configured to: acquire the radar point cloud data and the image data detected by the lidar and the camera facing downward respectively; A fusion module, communicatively connected to the data acquisition module, and the fusion module is configured to: fuse the radar point cloud data and the image data to obtain the point cloud fusion data of the target container; and A container door abnormality detection module, communicatively connected to the fusion module, and the container door abnormality detection module is configured to: extract the container door point cloud and the side point cloud of the target container from the point cloud fusion data; if the included angle between the container door point cloud and the side point cloud is greater than a preset included angle, generate a first judgment result.
10. A container crane Characterized in that It includes A container crane body; A trolley platform, arranged on the container crane body, with a lidar and a camera arranged at the bottom of the trolley platform, the lidar is used to detect the radar point cloud data including the container, and the camera is used to detect the image data including the container; and The container door abnormality recognition system according to claim 9.