Container damage detection method, container damage detection system and container damage detection equipment

Abstract

The invention provides a container damage detection method, a container damage detection system and a container damage detection device, and solves or improves the technical problems of low detection speed and high risk due to the fact that the container damage detection mainly adopts manual detection in the prior art. The invention provides a container damage detection method, which comprises the following steps of: calculating first damage data of a container by taking an image as basic data of defect detection and taking theoretical point cloud data corresponding to the image as reference; and meanwhile, calculating second damage data of the container by taking the point cloud data as basic data of defect detection and taking the theoretical image corresponding to the point cloud data as reference, and fusing the first damage data and the second damage data to determine actual damage data of the container. According to the method, multi-modal data is adopted for fusion processing, and the point cloud and image feature complementarity are utilized, so that the limitation of a single sensor on container damage detection is reduced, and the complete detection rate and the accurate detection rate of damage detection are improved.

Description

Technical Field

[0001] The present application relates to the technical field of container detection, and particularly to a method and system for detecting container damage, as well as damage detection equipment. Background Art

[0002] In recent years, with the continuous development of China's economy, the port container business has grown rapidly, and the demand for increasing throughput has become stronger and stronger. And the detection of container body damage is an inevitable inspection work when the container enters the port terminal. The detection of container body damage is usually carried out during the process of loading and unloading containers at the terminal, and is used to detect whether there are obvious damages, defects, depressions and other abnormal conditions on the container surface. The traditional method for detecting container damage at the terminal still stays in the stage of tally clerks' visual judgment. Due to the position of the tally clerks and the environmental restrictions near the crane equipment during operation, it is often only possible to judge whether the container is damaged from one side, resulting in the problems of slow manual processing speed and high danger. Summary of the Invention

[0003] In view of this, the present application provides a method and system for detecting container damage, as well as damage detection equipment, which solve or improve the technical problems of slow detection speed and high danger existing in the prior art that the detection of container damage mainly relies on manual detection.

[0004] According to a first aspect of the present application, a method for detecting container damage is provided, including: obtaining point cloud data and an image of a container, where the point cloud data is the point cloud data of the container detected by a lidar, and the image is an image of the container captured by a camera device; performing coordinate transformation on the coordinate system of the lidar and the pixel coordinate system of the camera device according to the parameters of the camera device and the lidar to determine the corresponding projection relationship between the point cloud data and the image of the container; inputting the image into a trained deep learning model for segmentation into target regions, and performing defect detection on the target regions to output abnormal pixel data of the target regions; determining first damage data of the container according to the corresponding projection relationship and the abnormal pixel data; determining second damage data of the container according to the point cloud data and the corresponding projection relationship; and fusing the first damage data and the second damage data to determine the actual damage data of the container.

[0005] In a possible implementation, according to the parameters of the imaging device and the parameters of the lidar, coordinate transformation is performed between the coordinate system of the lidar and the pixel coordinate system of the imaging device to determine the corresponding projection relationship between the point cloud data of the container and the image, including: jointly calibrating the lidar and the imaging device to determine the external parameter matrix from the lidar to the imaging device, where the external parameter matrix includes the rotation matrix and the translation matrix from the lidar to the imaging device; according to the external parameter matrix from the lidar to the imaging device, the internal parameter matrix of the imaging device, the point cloud data, and the image, projecting the point cloud data onto the image to determine the corresponding projection relationship between the point cloud data and the image.

[0006] In a possible implementation, according to the corresponding projection relationship and the abnormal pixel data, determining the first damage data of the container includes: calculating the theoretical point cloud data in the target area according to the corresponding projection relationship and the abnormal pixel data; determining the number of point clouds in the target area according to the theoretical point cloud data; when the number of point clouds in the target area is less than a preset threshold, determining that the target area is a damaged area and outputting the damage data of the target area, where the damage data of the target area includes: the abnormal pixel data corresponding to the target area; where the first damage data of the container includes the abnormal pixel data of at least one of the target areas.

[0007] In a possible implementation, according to the point cloud data and the corresponding projection relationship, determining the second damage data of the container includes: extracting the point cloud data of the area of interest in the point cloud data of the container based on the parameters of the container; extracting the point cloud data of the top surface of the container in the point cloud data of the container according to the plane normal vector; determining whether the area of interest is a damaged area according to the point cloud data of the area of interest and the point cloud data of the top surface of the container; and when the area of interest is a damaged area, calculating the damage data of the area of interest according to the point cloud data and the corresponding projection relationship; where the second damage data of the container includes the damage data of at least one of the areas of interest.

[0008] In a possible implementation, determining whether the region of interest is a damaged region based on the point cloud data of the region of interest and the point cloud data of the top surface of the container includes: using the RANSAC algorithm to perform plane fitting on the point cloud data of the region of interest and calculating the distance from each point cloud to the fitted plane; when the distance from the point cloud to the fitted plane is greater than the distance threshold, determining that the point cloud is a point cloud outside the plane; clustering the point clouds outside the plane and calculating the maximum value and the minimum value of each cluster; and when the difference between the maximum value and the minimum value of the cluster is greater than a preset value, determining that the cluster is an abnormal cluster; and when the number of the abnormal clusters is greater than a preset number, determining that the region of interest is a damaged region.

[0009] In a possible implementation, when the region of interest is a damaged region, calculating the damaged data of the region of interest according to the point cloud data and the corresponding projection relationship includes: when the region of interest is a damaged region, projecting the point cloud data into an image according to the point cloud data of the region of interest and the corresponding projection relationship between the point cloud data of the container and the image, so as to calculate the theoretical abnormal pixel data of the region of interest; wherein, the damaged data of the region of interest includes the theoretical abnormal pixel data of the region of interest; the second damaged data of the container includes the theoretical abnormal pixel data of at least one region of interest.

[0010] In a possible implementation, fusing the first damaged data and the second damaged data to determine the actual damaged data of the container includes: determining whether there is overlapping pixel data according to the abnormal pixel data of the first damaged data and the theoretical abnormal pixel data in the second damaged data; when there is overlapping pixel data, determining the overlapping pixel data according to the abnormal pixel data and the theoretical abnormal pixel data; and determining the actual damaged data according to the overlapping pixel data.

[0011] In a possible implementation, fusing the first damaged data and the second damaged data to determine the actual damaged data of the container further includes: when there is no overlapping pixel data, determining the actual damaged data of the container based on the point cloud data and the weight assignment rule of the image.

[0012] As a second aspect of the present application, the present application provides a container damage detection system, comprising: a data acquisition module for acquiring point cloud data and images of the container, wherein the point cloud data is the point cloud data of the container detected by a lidar, and the image is an image of the container captured by a camera device; a projection module for calculating a corresponding projection relationship between the point cloud data and the image of the container according to the point cloud data and the image; an image analysis module for inputting the image into a trained deep learning model for segmentation into target regions, detecting defects in the target regions to output abnormal pixel data of the target regions, and determining first damage data of the container according to the corresponding projection relationship and the abnormal pixel data; a point cloud analysis module for determining second damage data of the container according to the point cloud data; and a fusion module for fusing the first damage data and the second damage data to determine actual damage data of the container.

[0013] As a second aspect of the present application, the present application provides a container damage detection device, comprising: a lidar configured to detect point cloud data of the container; a camera device configured to capture an image of the container; and the above-mentioned container damage detection controller; wherein the lidar and the camera device are both communicatively connected to the container damage detection controller.

[0014] A container damage detection method provided by the present application uses an image as basic data for defect detection, takes the theoretical point cloud data corresponding to the image as a reference to calculate the first damage data of the container; at the same time, uses the point cloud data as basic data for defect detection, takes the theoretical image corresponding to the point cloud data as a reference to calculate the second damage data of the container, and fuses the first damage data and the second damage data to determine the actual damage data of the container. That is, multi-modal data is used for fusion processing, and the complementary features of the point cloud and the image are utilized to reduce the limitation of a single sensor for container damage detection, improve the detection completeness rate and the detection accuracy rate of damage detection, and better improve the automation level of the yard. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the drawings, the above and other objects, features and advantages of the present application will become more obvious. The drawings are used to provide a further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation to the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1The figure shows a schematic flowchart of a method for detecting damages of a container provided by an embodiment of the present application.

[0017] Figure 2 The figure shows a schematic flowchart of a method for detecting damages of a container provided by another embodiment of the present application.

[0018] Figure 3 The figure shows a schematic flowchart of a method for detecting damages of a container provided by another embodiment of the present application.

[0019] Figure 4 The figure shows a schematic flowchart of a method for detecting damages of a container provided by another embodiment of the present application.

[0020] Figure 5 The figure shows a schematic flowchart of a method for detecting damages of a container provided by another embodiment of the present application.

[0021] Figure 6 The figure shows a schematic flowchart of a method for detecting damages of a container provided by another embodiment of the present application.

[0022] Figure 7 The figure shows a schematic working principle diagram of a system for detecting damages of a container provided by an embodiment of the present application.

[0023] Figure 8 The figure shows a schematic working principle diagram of a device for detecting damages of a container provided by an embodiment of the present application.

[0024] Figure 9 The figure shows a schematic working principle diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0025] In the description of the present application, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined. In the embodiments of the present application, all directional indications (such as up, down, left, right, front, back, top, bottom...) are only used to explain the relative positional relationship and movement conditions between components in a specific posture (as shown in the drawings). If the specific posture changes, the directional indications will also change accordingly. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0026] In addition, the mention of "embodiment" in this document means that the specific features, structures, or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears at various positions in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0028] Overview of the Application

[0029] The traditional method for detecting container damage in a terminal still remains at the stage of tally clerks' visual judgment. Due to the position of the tally clerks and the environmental restrictions near the crane equipment during operation, it is often only possible to judge whether a container is damaged from a single side, resulting in the problem of slow manual processing speed and relatively high danger. Therefore, in order to improve the detection efficiency and safety, a method of using lidar to detect containers is adopted. The point cloud data of the containers is obtained, and then the point cloud data is analyzed to determine whether there is damage to the container body. However, lidar is limited by resolution. When the distance between the lidar and the container is relatively far, that is, when using lidar for long-distance identification of containers, the accuracy of damage detection is relatively low. In addition to using lidar to detect containers, a camera device can also be used to take pictures of the containers, and based on the images of the containers taken, it is detected whether the containers are damaged. However, since the camera device is greatly affected by the environment when taking pictures of the containers and has weak perception of concave and convex damage, the accuracy of detecting whether the containers are damaged only through images is relatively low.

[0030] Therefore, the present application provides a method for detecting container damage. Taking the image as the basic data for defect detection and using the corresponding theoretical point cloud data as a reference, the first damage data of the container is calculated. At the same time, taking the point cloud data as the basic data for defect detection and using the corresponding theoretical image as a reference, the second damage data of the container is calculated, and the first damage data and the second damage data are fused to determine the actual damage data of the container. That is, multi-modal data is used for fusion processing, and the complementary features of point clouds and images are utilized to reduce the limitations of a single sensor for container damage detection, improve the detection rate and accuracy of damage detection, and better enhance the automation level of the yard.

[0031] Exemplary Method

[0032] As a first aspect of the present application, the present application provides a method for detecting damage to a container. Figure 1 The following is a schematic flowchart of a method for detecting damage to a container provided by an embodiment of the present application. As Figure 1 shown, the method for detecting damage to the container includes the following steps:

[0033] S10: Obtain the point cloud data and image of the container;

[0034] Install a lidar and a camera device around the container. The lidar can detect the point cloud data of the container, and the camera device can capture the image of the container.

[0035] Specifically, the installation positions of the lidar and the camera device can be correspondingly set according to the scene where the container is located. For example, when the container is in the yard and is transferred by a quay crane, the quay crane includes a crossbeam and a trolley. A spreader is fixed under the trolley, and the trolley can move along the crossbeam, so that the container can be lifted to the corresponding position. Both the lidar and the camera device can be installed on the trolley. During the process of the trolley lifting the container, the lidar and the camera device can obtain the point cloud data and image of the container.

[0036] S20: According to the parameters of the camera device and the lidar, perform coordinate transformation on the coordinate system of the lidar and the pixel coordinate system of the camera device to determine the corresponding projection relationship between the point cloud data and the image of the container;

[0037] According to the parameters of the lidar and the camera device, perform coordinate transformation on the coordinate system of the lidar and the pixel coordinate system of the camera device to determine the corresponding projection relationship between the point cloud data and the image. That is, the relationship between the point cloud data and the pixel data in the image. According to this corresponding projection relationship, the theoretical pixel data can be obtained based on the actual point cloud data, and similarly, the theoretical point cloud data can be obtained based on the actual pixel data.

[0038] S30: Input the image into a trained deep learning model for segmentation into target regions, and perform defect detection on the target regions to output abnormal pixel data of the target regions;

[0039] Specifically, the deep learning model is a semi-supervised model Memseg (a memory-based end-to-end segmentation network, a semi-supervised method that uses differences and commonalities to detect surface defects in images). The model Memseg is based on the U-Net network structure and uses a pre-trained ResNet-101 as the encoder. The L1 loss (i.e., the L1 loss function) and the Focal loss (i.e., the Focal loss function) are used as loss functions for training. The semi-supervised model Memseg means that when training the model Memseg, unlabeled samples and labeled samples are used to train the classifier.

[0040] Specifically, the L1 loss function is a commonly used loss function in machine learning. It makes elements in the weight vector less than a certain threshold become 0 to make outliers more robust. The Focal loss function was proposed in the RetinaNet network. Based on the binary cross-entropy CE, it is a dynamically scaled cross-entropy loss. Through a dynamic scaling factor, it can dynamically reduce the weights of easily distinguishable samples during training.

[0041] The model Memseg can determine the abnormal regions in the image in an end-to-end manner to achieve defect detection of the product surface. That is, step S30 can detect the image of the container to obtain abnormal pixel data on the image, so as to obtain the defect data of the container.

[0042] S40: Determine the first damage data of the container according to the corresponding projection relationship and the abnormal pixel data;

[0043] S20 determines the corresponding projection relationship between the image and the point cloud data, that is, determines the corresponding projection relationship between the image pixel data and the point cloud data. Then, according to the corresponding projection relationship and the abnormal pixel data, the theoretical point cloud data corresponding to the abnormal pixel data can be calculated. Then, according to the theoretical point cloud data and the abnormal pixel data, the first damage data of the container is determined. That is, based on the abnormal pixel data as the calculation basis and the theoretical point cloud data as the reference, the first damage data of the container is jointly determined.

[0044] Compared with only using the image to determine the damage data of the container, in this application, while taking advantage of the strong texture of the image, the characteristics of the point cloud with strong 3D perception are also utilized. That is, the characteristics of the point cloud with strong 3D perception are used to make up for the weakness of the image in 3D perception. Therefore, the accuracy of the first damage data is improved.

[0045] The limitations of single-sensor damage detection for the crew route box body, such as weak vision for unevenness and strong perception of texture,

[0046] S50: Determine the second damage data of the container according to the point cloud data and the corresponding projection relationship;

[0047] Similar to the principle of S40, based on the point cloud data and the corresponding projection relationship, the theoretical pixel data corresponding to the point cloud data can be calculated, and then the second damage data of the container can be determined according to the theoretical pixel data and the point cloud data. That is, based on the point cloud data as the calculation basis and with the reference of the theoretical pixel data, the second damage data of the container is jointly determined. Compared with only using the point cloud to determine the damage data of the container, while this application has strong 3D perception using the point cloud, it also utilizes the strong texture feature of the image, that is, uses the strong texture feature of the image to make up for the weak texture of the point cloud, thus improving the accuracy of the second damage data.

[0048] S60: Fuse the first damage data and the second damage data to determine the actual damage data of the container.

[0049] Since both S40 and S50 are calculated based on different actual parameters and different reference factors are added, the first damage data and the second damage data may not be completely the same or may be completely different. Therefore, when finally determining the damage data of the container, the first damage data and the second damage data are fused to determine the actual damage data of the container.

[0050] A container damage detection method provided by this application uses the image as the basic data for defect detection and the corresponding theoretical point cloud data of the image as a reference to calculate the first damage data of the container; at the same time, uses the point cloud data as the basic data for defect detection and the corresponding theoretical image of the point cloud data as a reference to calculate the second damage data of the container, and fuses the first damage data and the second damage data to determine the actual damage data of the container. That is, multi-modal data is used for fusion processing, and the complementary features of the point cloud and the image are utilized to reduce the limitations of a single sensor for container damage detection, improve the detection completeness rate and the detection accuracy rate of damage detection, and better improve the automation level of the yard.

[0051] In a possible implementation manner of this application, as Figure 2 shown, the method for determining the corresponding projection relationship between the point cloud data and the image can adopt the following steps, that is, S20 (coordinate transformation of the coordinate system of the lidar and the pixel coordinate system of the camera device according to the parameters of the camera device and the lidar to determine the corresponding projection relationship between the point cloud data of the container and the image) specifically includes the following steps:

[0052] S201: Jointly calibrate the lidar and the camera device to determine the external parameter matrix from the lidar to the camera device, where the external parameter matrix includes the rotation matrix and the translation matrix from the lidar to the camera device;

[0053] The external parameter matrix [R T] from the lidar to the camera device, where R is the rotation matrix from the lidar to the camera device, and T is the translation matrix from the lidar to the camera device. R and T can represent the relative position relationship between the camera device and the lidar in the common reference coordinate system.

[0054] S202: Project the point cloud data onto the image according to the external parameter matrix from the lidar to the camera device, the internal parameter matrix of the camera device, the point cloud data, and the image, to determine the corresponding projection relationship between the point cloud data and the image.

[0055] That is, convert the coordinates of the 3D coordinate system of the lidar and the pixel coordinate system of the camera device. The specific calculation method is as follows:

[0056]

[0057] where u and v are pixel coordinates, and x, y, and z are point cloud coordinate data. K is the internal parameter matrix of the camera device, and R and T are the rotation matrix and translation matrix from the lidar to the camera device, respectively.

[0058] In a possible implementation manner of the present application, as Figure 3 shown, based on the image and with the point cloud as a reference, the specific method for determining the first damage data of the container may include the following steps, that is, S40 (determining the first damage data of the container according to the corresponding projection relationship and abnormal pixel data) specifically includes the following steps:

[0059] S401: Calculate the theoretical point cloud data in the target area according to the corresponding projection relationship and abnormal pixel data;

[0060] The theoretical point cloud data in the target area can be calculated according to the corresponding projection relationship and abnormal pixel data.

[0061] S402: Determine the number of point clouds in the target area according to the theoretical point cloud data;

[0062] S403: Determine whether the number of point clouds in the target area is less than a preset threshold;

[0063] When the judgment result in S403 is yes, that is, the number of point clouds in the target area is less than the preset threshold, then determine that the target area is a damaged area, and output the damaged data of the target area, that is, the abnormal pixel data of the target area, that is, execute S404.

[0064] When the judgment result in S403 is negative, that is, the number of point clouds in the target area is greater than or equal to the preset threshold, it is determined that the target area is not a damaged area. Then, it is determined that there is no damaged area in the area corresponding to the target area in the container (that is, the target area segmented this time is invalid), which means that there is no damage in the segmented target area. Then, continue to calculate the theoretical point number data for other target areas segmented from the image, that is, execute S401.

[0065] S404: Determine that the target area is a damaged area and output the abnormal pixel data of the target area.

[0066] When S401 - S404 are executed for all target areas segmented from the image, and after all segmented target areas are completed with detection, the first damaged data of the container includes the abnormal pixel data corresponding to at least one damaged area, and the abnormal pixel data in the entire image has been determined, that is, the first damaged data of the container determined according to the entire image has been determined.

[0067] When using an image to determine whether there is damage in the area of the container corresponding to the image, calculate the point cloud data according to the corresponding projection relationship and abnormal pixel data. When the number of point clouds is less than the preset threshold, determine the first damaged data of the container. Utilizing the strong 3D perception feature of point clouds makes up for the weak 3D perception shortcoming of images, thus improving the accuracy of the first damaged data.

[0068] In a possible implementation manner of the present application, as Figure 4 shown, based on the point cloud data and with the pixel data of the image as a reference, the specific method for determining the second damaged data of the container may include the following steps. S50 (determine the second damaged data of the container according to the point cloud data and the corresponding projection relationship) specifically includes the following steps:

[0069] S501: Based on the parameters of the container, extract the point cloud data of the region of interest from the point cloud data of the container;

[0070] Specifically, the parameters of the container include but are not limited to the number of layers of the container.

[0071] S502: According to the plane normal vector, extract the point cloud data of the top surface of the container from the point cloud data of the container;

[0072] S503: According to the point cloud data of the region of interest and the point cloud data of the top surface of the container, determine whether the region of interest is a damaged area; and

[0073] S504: When the region of interest is a damaged area, calculate the damaged data of the region of interest according to the point cloud data and the corresponding projection relationship.

[0074] Optionally, when the region of interest is a damaged area, according to the point cloud data of the region of interest and the corresponding projection relationship between the point cloud data of the container and the image, the point cloud data is projected onto the image to calculate the theoretical abnormal pixel data of the region of interest. That is, when the region of interest is a damaged area, the damaged data of the region of interest is the theoretical abnormal pixel data of the region of interest. Among them, the damaged data of the region of interest includes the theoretical abnormal pixel data of the region of interest;

[0075] The second damaged data of the container includes the theoretical abnormal pixel data of at least one region of interest.

[0076] That is, when the present application determines whether the container is damaged according to the point cloud data, first extracts a region of interest according to the point cloud data, and determines whether the region of interest is a damaged area according to the point cloud data. When it is a damaged area, then calculates the theoretical abnormal pixel data in the region of interest according to the point cloud data of the region of interest and the corresponding projection relationship. While the present application has strong 3D perception using point cloud, it also utilizes the strong texture feature of the image, that is, uses the strong texture feature of the image to make up for the weakness of the point cloud in texture.

[0077] When all the point cloud data has been processed through S501 - S504, the theoretical pixel data of at least one region of interest will be obtained, then the theoretical abnormal pixel data of the entire container can be determined, that is, the second damaged data of the container.

[0078] Optionally, as Figure 5 shown, S503 (determine whether the region of interest is a damaged area according to the point cloud data of the region of interest and the point cloud data of the top surface of the container) specifically includes the following steps:

[0079] S5031: Use the RANSAC algorithm to perform plane fitting on the point cloud data of the region of interest, and calculate the distance from each point cloud to the fitted plane;

[0080] Specifically, RANSAC (Random Sample Consensus) is a commonly used parameter estimation method. Randomly select a part of the data in the data, then perform fitting according to the selected data and statistically calculate the deviation of other data, and finally screen out data with a certain threshold for parameter estimation.

[0081] S5032: When the distance from the point cloud to the fitted plane is greater than the distance threshold, determine that the point cloud is a point cloud outside the plane;

[0082] S5033: Cluster the point clouds outside the plane, and calculate the maximum value and the minimum value of each cluster; and

[0083] S5034: When the difference between the maximum value and the minimum value of the cluster is greater than a preset value, determine that the cluster is an abnormal cluster;

[0084] When the difference between the maximum value and the minimum value of the cluster is greater than a preset value, it is considered that there is a protrusion or a depression in this area.

[0085] S5035: When the number of abnormal clusters is greater than a preset number, determine that the region of interest is a damaged region.

[0086] In a possible implementation manner of the present application, as Figure 6 shown, the specific fusion method for fusing the first damaged data and the second damaged data may include the following steps, that is, S60 (fusing the first damaged data and the second damaged data to determine the actual damaged data of the container) specifically includes the following steps:

[0087] S601: Determine whether there is overlapping pixel data according to the abnormal pixel data of the first damaged data and the theoretical abnormal pixel data in the second damaged data;

[0088] S602: When there is overlapping pixel data, determine the overlapping pixel data according to the abnormal pixel data and the theoretical abnormal pixel data;

[0089] When there are overlapping pixels, it can be explained that there is damage.

[0090] S603: Determine the actual damaged data according to the overlapping pixel data.

[0091] Specifically, according to the overlapping pixel data, the damaged region can be determined and the area of the damaged region can be calculated.

[0092] S604: When there is no overlapping pixel data, determine the actual damaged data of the container based on the point cloud data and the weight assignment rule of the image.

[0093] Specifically, when there is no overlapping pixel data, whether there is a damaged region in the container is determined according to the point cloud data and the weight assignment rule of the image, and the point cloud data and the weight assignment rule of the image are also different in different application scenarios.

[0094] For example, when the weight of the point cloud data is greater than the weight of the image, it means that the damaged data detected according to the point cloud data is more accurate than the damaged data detected based on the image. Therefore, at this time, the damaged data detected according to the point cloud data is the damaged data of the container, that is, the second damaged data is the actual damaged data of the container.

[0095] As the second aspect of the present application, the present application also provides a container damage detection system, as Figure 7 shown, the container damage detection system 100 includes:

[0096] A data acquisition module 101 is configured to acquire point cloud data and images of a container, wherein the point cloud data is the point cloud data of the container detected by a lidar, and the image is the image of the container captured by an imaging device;

[0097] That is, the data acquisition module 101 is used to execute S10 in the above-mentioned container damage detection method.

[0098] A projection module 102 is configured to calculate the corresponding projection relationship between the point cloud data and the image of the container according to the point cloud data and the image;

[0099] That is, the projection module 102 is used to execute S20 in the above-mentioned container damage detection method.

[0100] An image analysis module 103 is configured to input the image into a trained deep learning model to segment it into target regions, perform defect detection on the target regions to output abnormal pixel data of the target regions, and determine the first damage data of the container according to the corresponding projection relationship and the abnormal pixel data;

[0101] That is, the image analysis module 103 is used to execute S30 and S40 in the above-mentioned container damage detection method.

[0102] A point cloud analysis module 104 is configured to determine the second damage data of the container according to the point cloud data;

[0103] That is, the point cloud analysis module 104 is used to execute S50 in the above-mentioned container damage detection method.

[0104] A fusion module 105 is configured to fuse the first damage data and the second damage data to determine the actual damage data of the container.

[0105] That is, the fusion module 105 is used to execute S60 in the above-mentioned container damage detection method.

[0106] A container damage detection system provided by the present application uses an image as the basic data for defect detection, takes the theoretical point cloud data corresponding to the image as a reference to calculate the first damage data of the container; at the same time, uses the point cloud data as the basic data for defect detection, takes the theoretical image corresponding to the point cloud data as a reference to calculate the second damage data of the container, and fuses the first damage data and the second damage data to determine the actual damage data of the container. That is, multi-modal data is used for fusion processing, and the complementary features of the point cloud and the image are utilized to reduce the limitation of a single sensor for container damage detection, improve the detection rate and accuracy of damage detection, and better improve the automation level of the yard.

[0107] As a third aspect of the present application, the present application further provides a container damage detection device, as Figure 8 shown, the container damage detection device includes:

[0108] A lidar 200 configured to detect point cloud data of the container;

[0109] An imaging device 300 configured to capture an image of the container; and

[0110] The above-mentioned container damage detection controller 100;

[0111] Wherein, both the lidar and the imaging device are communicatively connected to the container damage detection controller. The lidar 200 transmits the captured point cloud data to the data acquisition module 101 in the container damage detection controller 100. Similarly, the image of the container captured by the imaging device 300 is transmitted to the data acquisition module 101 in the container damage detection controller 100.

[0112] Optionally, the installation positions of the lidar 200 and the imaging device 300 may be different according to different application scenarios. For example, when the container is in the yard and is transferred by a quay crane, the quay crane includes a crossbeam and a trolley. A spreader is fixed below the trolley, and the trolley can move along the crossbeam, so that the container can be lifted to the corresponding position. Both the lidar and the imaging device can be installed on the trolley. During the process of the trolley lifting the container, the lidar and the imaging device can acquire the point cloud data and the image of the container.

[0113] Next, refer to Figure 9 to describe an electronic device according to an embodiment of the present application.

[0114] Figure 9 The block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0115] As Figure 9 shown, the electronic device 10 includes one or more processors 11 and a memory 12.

[0116] The processor 11 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0117] The memory 12 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage media, and the processor 11 may run the program instructions to implement the container damage detection method of each embodiment of the present application described above and / or other desired functions.

[0118] In one example, the electronic device 10 may further include: an input device 13 and an output device 14, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0119] When the electronic device is a stand-alone device, the input device 13 may be a communication network connector for receiving the collected input signals from the first device and the second device.

[0120] In addition, the input device 13 may further include, for example, a keyboard, a mouse, and so on.

[0121] The output device 14 may output various information to the outside, including the determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, and so on.

[0122] Of course, for simplicity, Figure 9 only some of the components related to the present application in the electronic device 10 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 10 may further include any other appropriate components.

[0123] As the third aspect of the present application, the present application provides a computer-readable storage medium, and the storage medium stores a computer program, and the computer program is used to execute the following steps:

[0124] S10: Obtain the point cloud data and image of the container;

[0125] S20: According to the parameters of the camera device and the parameters of the lidar, perform coordinate transformation on the coordinate system of the lidar and the pixel coordinate system of the camera device to determine the corresponding projection relationship between the point cloud data and the image of the container;

[0126] S30: Input the said image into a trained deep learning model for segmentation into target regions, and perform defect detection on the target regions to output abnormal pixel data of the target regions;

[0127] S40: Determine the first damage data of the container according to the corresponding projection relationship and the abnormal pixel data;

[0128] S50: Determine the second damage data of the container according to the point cloud data and the corresponding projection relationship;

[0129] S60: Fuse the first damage data and the second damage data to determine the actual damage data of the container.

[0130] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program information. When the computer program information is run by a processor, the processor is caused to execute the steps in the container damage detection method according to various embodiments of the present application described in this specification.

[0131] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code may be executed entirely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0132] In addition, an embodiment of the present application may also be a computer-readable storage medium, on which computer program information is stored. When the computer program information is run by a processor, the processor is caused to execute the steps in the container damage detection method according to various embodiments of the present application described in this specification.

[0133] The computer-readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0134] The basic principles of the present application have been described in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are merely examples and not limitations. It cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present application. Additionally, the specific details disclosed above are only for illustrative and easy-to-understand purposes and not limitations. The above details do not limit the present application to necessarily implement using the above specific details.

[0135] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present application are only illustrative examples and do not intend to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms meaning "including but not limited to" and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or" and can be used interchangeably with it, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to" and can be used interchangeably with it.

[0136] It should also be noted that in the devices, equipment, and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of the present application.

Claims

1. A method for detecting damages of a container, characterized in that, it includes: Obtaining the point cloud data and the image of the container, wherein the point cloud data is the point cloud data of the container detected by a lidar, and the image is the image of the container captured by a camera device; According to the parameters of the camera device and the lidar, performing coordinate transformation on the coordinate system of the lidar and the pixel coordinate system of the camera device to determine the corresponding projection relationship between the point cloud data and the image of the container; Inputting the image into a trained deep learning model for segmentation into target regions, and performing defect detection on the target regions to output abnormal pixel data of the target regions; Determining the first damage data of the container according to the corresponding projection relationship and the abnormal pixel data; Determining the second damage data of the container according to the point cloud data and the corresponding projection relationship; and Fusing the first damage data and the second damage data to determine the actual damage data of the container.

2. The method for detecting damages of a container according to claim 1, characterized in that, According to the parameters of the camera device and the lidar, performing coordinate transformation on the coordinate system of the lidar and the pixel coordinate system of the camera device to determine the corresponding projection relationship between the point cloud data and the image of the container, including: Performing joint calibration on the lidar and the camera device to determine the external parameter matrix from the lidar to the camera device, wherein the external parameter matrix includes the rotation matrix and the translation matrix from the lidar to the camera device; According to the external parameter matrix from the lidar to the camera device, the internal parameter matrix of the camera device, the point cloud data and the image, projecting the point cloud data onto the image to determine the corresponding projection relationship between the point cloud data and the image.

3. The method for detecting damages of a container according to claim 1, characterized in that, Determining the first damage data of the container according to the corresponding projection relationship and the abnormal pixel data, including: Calculating the theoretical point cloud data in the target region according to the corresponding projection relationship and the abnormal pixel data; Determining the number of point clouds in the target region according to the theoretical point cloud data; When the number of point clouds in the target region is less than a preset threshold, determining the target region as a damaged region and outputting the damage data of the target region, the damage data of the target region including: the abnormal pixel data corresponding to the target region; Wherein, the first damage data of the container includes the abnormal pixel data of at least one target region.

4. The method for detecting damages of a container according to claim 3, characterized in that, Determining the second damage data of the container according to the point cloud data and the corresponding projection relationship, including: Based on the parameters of the container, extracting the point cloud data of the region of interest in the point cloud data of the container; According to the plane normal vector, extracting the point cloud data of the top surface of the container in the point cloud data of the container; Determine whether the region of interest is a damaged region based on the point cloud data of the region of interest and the point cloud data of the top surface of the container; and When the region of interest is a damaged region, calculate the damage data of the region of interest according to the point cloud data of the region of interest and the corresponding projection relationship; Wherein, the second damage data of the container includes the damage data of at least one region of interest.

5. The container damage detection method according to claim 4, Characterized in that, Determining whether the region of interest is a damaged region based on the point cloud data of the region of interest and the point cloud data of the top surface of the container includes: Using the RANSAC algorithm, perform plane fitting on the point cloud data of the region of interest, and calculate the distance from each point cloud to the fitted plane; When the distance from the point cloud to the fitted plane is greater than the distance threshold, determine that the point cloud is a point cloud outside the plane; Cluster the point clouds outside the plane, and calculate the maximum value and the minimum value of each cluster; and When the difference between the maximum value and the minimum value of the cluster is greater than a preset value, determine that the cluster is an abnormal cluster; and When the number of the abnormal clusters is greater than a preset number, determine that the region of interest is a damaged region.

6. The container damage detection method according to claim 4, Characterized in that, When the region of interest is a damaged region, calculating the damage data of the region of interest according to the point cloud data and the corresponding projection relationship includes: When the region of interest is a damaged region, project the point cloud data into the image according to the point cloud data of the region of interest and the corresponding projection relationship between the point cloud data of the container and the image, so as to calculate the theoretical abnormal pixel data of the region of interest; Wherein, the damage data of the region of interest includes the theoretical abnormal pixel data of the region of interest; The second damage data of the container includes the theoretical abnormal pixel data of at least one region of interest.

7. The container damage detection method according to claim 6, Characterized in that, Fusing the first damage data and the second damage data to determine the actual damage data of the container includes: Determine whether there is overlapping pixel data according to the abnormal pixel data of the first damage data and the theoretical abnormal pixel data in the second damage data; When there is overlapping pixel data, determine the overlapping pixel data according to the abnormal pixel data and the theoretical abnormal pixel data; and Determine the actual damage data according to the overlapping pixel data.

8. The container damage detection method according to claim 7, Characterized in that, Fusing the first damage data and the second damage data to determine the actual damage data of the container further includes: When there is no overlapping pixel data, determine the actual damage data of the container based on the point cloud data and the weight assignment rule of the image.

9. A container damage detection system, Characterized in that, Comprising: A data acquisition module, configured to acquire the point cloud data and images of the container, wherein the point cloud data is the point cloud data of the container detected by a lidar, and the images are the images of the container captured by a camera device; A projection module, configured to calculate the corresponding projection relationship between the point cloud data and the images of the container according to the point cloud data and the images; An image analysis module, configured to input the images into a trained deep learning model for segmentation into target regions, perform defect detection on the target regions to output abnormal pixel data of the target regions, and determine the first damage data of the container according to the corresponding projection relationship and the abnormal pixel data; A point cloud analysis module, configured to determine the second damage data of the container according to the point cloud data; and A fusion module, configured to fuse the first damage data and the second damage data to determine the actual damage data of the container.

10. A container damage detection device, characterized in that it includes: A lidar, configured to detect the point cloud data of the container; A camera device, configured to capture the images of the container; and the container damage detection controller as claimed in claim 9; wherein the lidar and the camera device are both communicatively connected to the container damage detection controller.

Images