Intelligent container checking method, device and system

Abstract

The invention relates to a method, a device and a system for detecting the damage of a container body. The method comprises the following steps: acquiring a 2D image of the container body acquired by a 2D camera; obtaining a plurality of groups of 3D point cloud data of the container body collected by a plurality of 3D cameras, wherein each group of 3D point cloud data corresponds to a local part of the container body; respectively registering each group of 3D point cloud data to a unified coordinate system to obtain complete 3D point cloud data of the container body in the coordinate system; and detecting the container body according to the 2D image and the complete 3D point cloud data. The invention further provides a corresponding container intelligent checking device and system. According to the method, the automation degree, efficiency and inspection quality of container inspection, the registration efficiency and accuracy of the 3D point cloud and the accuracy of 3D point cloud feature extraction can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field related to container inspection, and in particular to a method, device and system for intelligent container inspection using deep learning technology for image recognition. Background Art

[0002] At present, in most cases, container inspections are conducted in the yard. Inspectors mainly inspect the container damage on site through visual recognition and manual measurement. They then use handheld cameras and measuring tools to measure and photograph the damaged parts and upload them to the business system. This may have the following disadvantages:

[0003] On the one hand, manual inspection may be affected by complex environmental factors such as weather conditions and lighting changes. At the same time, there is a high possibility of negligence or errors in human operations. Therefore, the accuracy of traditional manual inspection methods is limited, and the inspection quality fluctuates greatly. Moreover, with the increase in the number of containers, traditional manual inspection methods cannot meet the needs of rapid inspection, resulting in low degree of automation and inspection efficiency.

[0004] On the other hand, with the innovation of inspection equipment and the development of AI technology (Artificial Intelligence) in recent years, it has become possible for automated container inspection to replace manual inspection. Among them, deep learning-based image recognition technology uses convolutional neural networks (CNN) to learn a large amount of 2D and 3D image data of container damage to perform container damage detection. This can not only improve container inspection efficiency and save labor costs, but also improve the quality and stability of container inspection results and reduce communication costs between equipment handover parties. Among them, 3D point cloud data used for AI container inspection often faces the need for registration when used in actual use. However, 3D point cloud registration still faces technical difficulties and challenges such as noise influence, density differences, high computational costs, and real-time registration.

[0005] On the other hand, in the point cloud feature extraction technology applied to AI container inspection, current point cloud processing models such as PointNet have limited ability to capture local structures and mainly focus on global features. This results in the neglect of detailed features when extracting damaged targets from containers, resulting in the loss of important information; when processing complex or unevenly distributed point clouds and complex scenes, the performance degrades significantly; at the same time, it will cause the model to be more sensitive to noise occlusion and unable to accurately restore the true shape of the object.

[0006] Therefore, given the current container inspection methods' low detection efficiency, poor accuracy, low 3D point cloud registration accuracy, high computational cost, difficulty in real-time registration, and limited ability to capture local structures through point cloud feature extraction, how to provide an intelligent container inspection method with a high degree of automation, quick and convenient calculation, high accuracy, and the ability to capture local features of point clouds is a technical problem that needs to be solved. Summary of the Invention

[0007] In view of the above problems in the prior art, this application provides a container intelligent inspection method that can minimize manual intervention, automate the inspection process, and improve the efficiency and accuracy of the inspection. Correspondingly, this application also provides a container intelligent inspection device and system to achieve the above effects.

[0008] To achieve the above objectives, the present application provides a first aspect of a container intelligent inspection method, comprising the following steps:

[0009] Acquire a 2D image of the container body captured by a 2D camera;

[0010] Acquire multiple sets of 3D point cloud data of the container body collected by multiple 3D cameras, wherein each set of 3D point cloud data corresponds to a portion of the container body;

[0011] Registering each set of 3D point cloud data to a unified coordinate system to obtain complete 3D point cloud data of the container body in the coordinate system;

[0012] The container body is inspected based on the 2D image and the complete 3D point cloud data.

[0013] As described above, the intelligent container inspection method described in this application utilizes a camera to extract container body data, facilitating subsequent computer visual recognition and processing. This method can be integrated with automated structures such as robotic arms, minimizing human involvement. The registered point cloud data is more accurate and includes the entire three-dimensional information of the container. Furthermore, by combining 2D images with 3D point cloud data, positioning accuracy is increased, information loss is reduced, and the ability to detect and locate small objects is enhanced. Furthermore, by simplifying the learning problem and utilizing more mature 2D image processing technology, network learning and detection efficiency are increased.

[0014] As a possible implementation of the first aspect, the registering of each set of 3D point cloud data to a unified standard coordinate system in the container intelligent inspection method includes:

[0015] A. Extract the descriptors of each key point included in each set of 3D point cloud data;

[0016] B. For each set of 3D point cloud data, perform the following steps:

[0017] For each source keypoint in the group, a k-nearest neighbor search is performed in the descriptor space to determine a plurality of candidate target keypoints, wherein each source keypoint and the corresponding plurality of candidate target keypoints form a cluster; wherein the source keypoint is located in the group of 3D point cloud data to be registered, and the target keypoint is located in a group of 3D point cloud data in the unified coordinate system;

[0018] Extract the features of each cluster;

[0019] Input the features of each cluster into the first model, and output the correction values ​​and confidence levels of each candidate target keypoint corresponding to the source keypoint in each cluster;

[0020] The optimal rotation and translation matrix used for registration to the unified coordinate system is calculated based on the following formula:

[0021]

[0022] Among them, R * ,t * represents the optimal value in the rotation and translation matrix R,t, represents the source keypoint, represents the correction value of the candidate target key point, represents the confidence, ‖·‖2 represents the L2 norm, and M represents the number of candidate key points;

[0023] The set of 3D point cloud data to be registered is registered to the unified coordinate system using the optimal rotation and translation matrix.

[0024] From the above, the point cloud data registration uses the k-nearest neighbor search method to determine the target key points and form clusters, which has the advantages of being simple and intuitive, easy to implement, highly adaptable, and highly robust. At the same time, since the point cloud data registration method extracts the features of the clusters rather than the features of the point pairs, the amount of computation is reduced from the order of the square of the number of target points to the order of the number of target points, greatly reducing the computational burden. At the same time, the point cloud data registration method can extract and construct a variety of cluster descriptors according to needs; and it is easy to combine with a variety of algorithm models to improve the processing and correction of point cloud registration. Using the L2 norm and confidence, find R,t that minimizes the distance between the source key point and the target key point after the correction after transformation. This method severely penalizes R,t with poor performance, so as to find the optimal transformation R more quickly. * ,t * , optimized the search speed.

[0025] As a possible implementation of the first aspect, the descriptors of the key points are extracted through an Inception model.

[0026] From the above, we can see that the Inception model, as a deep convolutional neural network architecture that performs well in image recognition and classification tasks, can improve the efficiency and accuracy of key point descriptor extraction.

[0027] As a possible implementation of the first aspect, extracting the features of each cluster includes extracting at least one of the following features of each cluster: a geometric feature of the cluster, a descriptor feature of the cluster, and a similarity feature of the cluster.

[0028] From the above, the geometric features, descriptor features, and similarity features of the cluster can remain invariant under the rotation and scale changes of the point cloud, which is crucial for the matching and recognition tasks of point cloud data.

[0029] As a possible implementation of the first aspect, the geometric features of the cluster include at least one of the following: connections between source key points and candidate target key points within the cluster, relative coordinates, and distances between them; or

[0030] The descriptor features of the cluster include at least one of the following: descriptors, saliency, and uncertainty of the source keypoints and each candidate target keypoint within the cluster; or

[0031] The similarity characteristics of the clusters include at least one of the following: bilateral consistency and neighborhood consistency.

[0032] As described above, the connections, coordinates, and distances between source keypoints and candidate target keypoints within a cluster can easily describe the cluster's geometric characteristics. The descriptors, salience, and uncertainty of the source keypoints and candidate target keypoints can accurately describe the point's feature information. The bilateral consistency and neighborhood consistency of the cluster can indicate the similarity between the keypoint and the target point. The construction of these three types of cluster features includes important information about whether the source keypoint and the target keypoint are the same physical keypoint.

[0033] As a possible implementation of the first aspect, the features of each cluster are input into the first model, and the correction values ​​and confidence levels of each candidate target key point corresponding to the source key point in each cluster are output, including:

[0034] Inputting the features of each cluster into the first model including a shared multi-layer perceptron module, a pooling module, and a first activation function module connected in sequence, wherein the first activation function module outputs the attention weight of each candidate key point;

[0035] The output of the shared multilayer perceptron module included in the first model is multiplied by the attention weight and then input into the sequentially connected multilayer perceptron module and the second activation function module, and the second activation function module outputs the initial confidence c i , and calculate the confidence level used according to the following formula

[0036]

[0037] Each candidate key point in the cluster is multiplied by the attention weight to output the correction value of each candidate target key point

[0038] As described above, multiple cluster features are comprehensively considered through a mechanism leading to multi-layer perceptron parameter sharing; the pooling module reduces the dimensionality of the feature map, making the data more robust and generalizing well; the first activation function module outputs attention weights, and multiplies them with each candidate keypoint to obtain a correction value for the target keypoint, which balances accuracy and robustness. The output of the shared multi-layer perceptron module is multiplied by the attention weights to obtain a corrected feature map of the target keypoint. This is then input into the sequentially connected multi-layer perceptron module and the second activation function module to obtain the corrected target keypoint confidence. Since point cloud registration requires consideration of the entirety, the final confidence is determined based on the confidence of all keypoints.

[0039] As a possible implementation of the first aspect, the detecting the container body according to the 2D image and the complete 3D point cloud data includes:

[0040] Detecting 2D target objects in the 2D image to generate a series of 2D bounding boxes, wherein the target objects include a container body;

[0041] generating a corresponding 3D view frustum in the complete 3D point cloud data using the 2D bounding box;

[0042] extracting point cloud data within the 3D viewing cone;

[0043] Extracting global features from the extracted point cloud data, classifying the target object points and background points, and regressing the target object points to predict the 3D boundary of the target object;

[0044] A detection result of the container body is obtained based on the 3D boundary of the target object.

[0045] From the above, using the more mature 2D image object detection and guiding 3D point cloud object detection can reduce the difficulty of learning problems, reduce computational costs, and improve accuracy and generalization capabilities.

[0046] As a possible implementation of the first aspect, extracting global features from the extracted point cloud data includes:

[0047] For each point in the extracted point cloud data, a K-nearest neighbor graph is constructed based on the distance between the points;

[0048] Performing graph convolution on the K nearest neighbor graph through a multi-layer edge convolution module to extract high-level features;

[0049] A pooling operation is performed on the provided high-level features to output the global features.

[0050] From the above, by passing the k-nearest neighbor graph through a multi-layer edge convolution module, global features can be further output without ignoring local structural features, ensuring local recognition performance and detail sensitivity while quickly obtaining robust global features and retrieving damaged targets.

[0051] A second aspect of the present application provides a container damage detection device, comprising:

[0052] A first acquisition unit is configured to acquire a 2D image of the container body captured by a 2D camera;

[0053] a second acquiring unit, configured to acquire multiple sets of 3D point cloud data of the container body acquired by multiple 3D cameras, wherein each set of 3D point cloud data corresponds to a portion of the container body;

[0054] a registration unit, configured to register each set of 3D point cloud data to a unified coordinate system, and obtain complete 3D point cloud data of the container body in the coordinate system;

[0055] A detection unit is used to detect the container body based on the 2D image and the complete 3D point cloud data.

[0056] A third aspect of the present application provides a container damage detection system, comprising:

[0057] A 2D camera, used to capture 2D images of the container body;

[0058] Multiple 3D cameras are provided at different locations, for respectively collecting multiple sets of 3D point cloud data of the container body, wherein each set of 3D point cloud data corresponds to a portion of the container body;

[0059] The back-end service subsystem is used to execute the damage detection method for the container body described in any one of claims 1-8. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1This is a flow chart of the container intelligent inspection method provided by the first embodiment of the present application;

[0061] Figure 2 This is a flow chart of the container intelligent inspection method provided by the second embodiment of the present application;

[0062] Figure 3a This is a system structure diagram of the container intelligent inspection system provided by the second embodiment of the present application;

[0063] Figure 3b This is a structural diagram of the central control subsystem of the container intelligent inspection system provided in the second embodiment of the present application;

[0064] Figure 3c It is the AI ​​visual preprocessing subsystem of the container intelligent inspection system provided in the second embodiment of the present application;

[0065] Figure 3d It is a deep learning training model for point cloud registration in the back-end service subsystem of the container intelligent inspection system provided in the second embodiment of the present application;

[0066] Figure 4 is a schematic diagram of a container damage detection device provided in an embodiment of the present application;

[0067] It should be understood that the structures of the various systems in the above structural diagrams are for reference only and should not constitute an exclusive interpretation of the embodiments of the present invention. DETAILED DESCRIPTION

[0068] The technical solution provided by this application is further described below with reference to the accompanying drawings and examples. It should be understood that the system structure and business scenarios provided in the examples of this application are mainly for illustrating possible implementation methods of the technical solution of this application and should not be interpreted as the sole limitation of the technical solution of this application. It is known to those skilled in the art that with the evolution of the system structure and the emergence of new business scenarios, the technical solution provided by this application is also applicable to similar technical problems.

[0069] It should be understood that the intelligent container inspection solutions provided in the embodiments of this application include intelligent container inspection methods, inspection devices, and inspection systems. Because these technical solutions solve the same or similar problems, some repetitions may not be repeated in the following descriptions of the specific embodiments. However, these specific embodiments should be considered as cross-references and can be combined with each other.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. In the event of any inconsistency, the meanings described in this specification or the meanings derived from the contents recorded in this specification shall prevail. In addition, the terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application. In order to accurately describe the technical content in this application and to accurately understand the present invention, the following explanations or definitions are given for the terms used in this specification before describing the specific embodiments:

[0071] 1) RTSP (Real-Time Streaming Protocol): This protocol is a network control protocol used for real-time transmission of audio and video data. It allows clients to interact with servers to control operations such as playing, pausing, and stopping streaming media. It is widely used in video surveillance and live streaming.

[0072] 2) CNN (Convolutional Neural Network): A convolutional neural network is a deep learning algorithm widely used in image and video recognition, natural language processing, and other fields. It simulates the processing mechanism of the human visual system, using convolutional layers to extract features and recognize complex patterns through a multi-layered structure.

[0073] 3) Optical Character Recognition (OCR): This technology uses image processing to identify printed text in images and convert it into editable and searchable text. This technology is widely used in automated data entry and document digitization.

[0074] 4) SDK (Software Development Kit): This camera software development kit provides APIs and libraries to help developers quickly integrate and control camera functions. It simplifies image capture and processing, supports a variety of camera hardware, and is widely used in mobile app and desktop software development.

[0075] 5) Registration of point clouds, i.e., standard coordinate system: In this application, under the condition of the shape of the object to be measured (the space inside the container) (the space occupied is large), multiple 3D cameras need to be deployed at multiple installation positions to collect local point cloud data respectively. For the multiple 3D cameras deployed, each 3D camera can only obtain local point cloud information, and the point cloud obtained by each 3D camera has an independent coordinate system and cannot be directly spliced. Due to the incompleteness, rotational misalignment, translational misalignment, etc. of the point cloud, it is necessary to align multiple local point clouds to obtain a complete point cloud. In order to obtain the complete point cloud data of the object to be measured, and for subsequent use, it is necessary to determine the appropriate coordinate transformation (i.e., determine the rotation and translation matrix: R, t), and merge the point sets obtained from various perspectives into a unified coordinate system (in this application, the unified coordinate system is referred to as the standard coordinate system) to form a complete data point cloud for subsequent use. This process is the registration of point cloud data.

[0076] The standard coordinate system is the unified coordinate system mentioned above, and may be a coordinate system selected from the plurality of 3D cameras as a reference.

[0077] 6) Feature description, also known as a descriptor, is a set of numerical values ​​that characterize the characteristics of a local area in an image or point cloud by analyzing that area. This set of numerical values ​​is also called a high-dimensional vector, and each element in the vector (that is, each value in the set of numerical values) corresponds to a feature value of the descriptor.

[0078] 7) Descriptor space: refers to the high-dimensional space where all descriptors are located. In the descriptor space, each descriptor is represented as a high-dimensional vector.

[0079] The intelligent container inspection solution provided by the embodiments of this application uses a camera to capture 2D container images and 3D point cloud data, registers the collected 3D container point cloud data, and uses deep learning recognition technology to extract container damage information based on the registered point cloud. This method has a high degree of automation, requires minimal human intervention, is quick and convenient to calculate, has high accuracy, and performs well in extracting local features from the point cloud. The embodiments of this application can be applied to scenarios such as intelligent container inspection and intelligent inspection of three-dimensional objects. The following detailed description of each embodiment of this application is provided with reference to the accompanying figures.

[0080] The first embodiment of the present application provides a container inspection system. Figure 1 , specifically describing the implementation of each step of the method, including steps S10-S30:

[0081] S10: Acquire a 2D image of a container body captured by a 2D camera; acquire multiple sets of 3D point cloud data of the container body captured by multiple 3D cameras, wherein each set of 3D point cloud data corresponds to a portion of the container body.

[0082] In some embodiments, the captured 2D image may be a standard still image, a dynamic range image, or an image captured from a video, captured by a 2D camera. The 2D camera may be a 2D line scan camera or a 2D array camera to meet different usage requirements. Other camera-related parameters and configurations, such as the camera sensor, are not limited herein.

[0083] In some embodiments, 3D point cloud data can be obtained through laser scanning, structured light scanning, multi-angle acquisition, etc., and the selection needs to be made based on the required scanning time, accuracy, and environmental scenario.

[0084] In some embodiments, the process of collecting 2D images and 3D point cloud data can start from the time the container enters the inspection area. Specifically, it may include guiding the vehicle carrying the container to enter and exit the area to be inspected, obtaining vehicle information, guiding the clearance of the inspection area, and deploying cameras. Each link can be performed manually, but in order to reduce human involvement, it can also be completed through automated systems and processes. Specifically, it may include using prompt guidance, text guidance, mechanical lifting bars, automatic ground locks, etc. to guide vehicles to enter and exit the area to be inspected; using OCR text recognition or other image recognition algorithms to obtain license plate and container number information; using tools such as sensors and image recognition algorithms to identify whether there are unrelated people in the inspection area; and using a combination of mechanical arms, mobile devices, and hand and foot trusses to deploy cameras to corresponding collection positions.

[0085] In some embodiments, the objects of collection of 2D images and 3D point cloud data may include the outer surface of the vehicle and the box, the interior of the vehicle and the inner surface of the box.

[0086] S20: aligning each set of 3D point cloud data to a unified coordinate system to obtain complete 3D point cloud data of the container body in the coordinate system.

[0087] In some embodiments, depending on the point cloud registration algorithm used, 3D point cloud data may undergo preprocessing steps such as point cloud filtering, noise removal, normal estimation, and feature extraction. Common filtering methods include bilateral filtering, Gaussian filtering, conditional filtering, pass-through filtering, random sampling consistency filtering, and VoxelGrid filtering. Noise removal can be achieved using clustering algorithms and other processing methods. Feature extraction can include point cloud features such as curvature, edges, and corners. Point cloud feature description and key point extraction can be achieved using algorithms such as the Inception network.

[0088] In some embodiments, if the density of the collected point clouds is different, it may be necessary to normalize the point cloud data, perform point cloud segmentation, density estimation, etc.; if the overlapping area of ​​the collected point clouds is not sufficient to provide sufficient information, it may be necessary to perform overlap calculation and detection, point cloud area growth, and other methods for processing.

[0089] In some embodiments, rigid or non-rigid registration may be employed.

[0090] In some embodiments, 3D point cloud data can be aligned using traditional geometric algorithms such as RANSAC, 4PCS, ICP, NDT, FPFH, or using a deep learning network; wherein, candidate target points for point cloud alignment are obtained using a coarse alignment algorithm, or algorithms such as k-nearest neighbor can be used for search, and the search can be accelerated by combining data structures such as kdtree and octree.

[0091] In some embodiments, when using a deep learning network for registration, the trained network can be used to obtain the optimal target point or its correction value. Specifically, source keypoints and target keypoints can be formed into point pairs or point clusters, and their features can be extracted to obtain the optimal solution, which can then be corrected. Alternatively, a direct end-to-end neural network approach can be used for training. The choice of modules / layers used in the deep learning network is not limited here.

[0092] In some embodiments, when training using point clusters, cluster features may be composed of three components: geometric features, descriptor features, and similarity features. Geometric features may be composed of geometric relationships between points, descriptor features may be derived using the aforementioned point cloud feature description method, and similarity features may be composed of similar relationships between points. It should be understood that the remaining features may be constructed based on the specific network requirements.

[0093] In some embodiments, the rotation matrix and translation matrix obtained by point cloud registration that transform the original key points of the point cloud into the target key points can be represented by R,t, then the optimal R,t is the R,t that makes the distance between the rotated and translated point cloud and the target point cloud closest; wherein, the distance can be represented by Euclidean distance; wherein, in order to make the result more robust and close to the target point cloud, the Euclidean distance between the source target point and the key target point can be multiplied by a weight or confidence.

[0094] S30: Detecting the container body according to the 2D image and the complete 3D point cloud data.

[0095] In some embodiments, container body inspection can utilize 2D or 3D deep learning image feature extraction algorithms to detect damaged areas. 2D image deep learning networks can extract data features using algorithms such as CNN, edge detection, corner detection, and scale-invariant feature transforms; 3D point cloud deep learning networks can extract data features using algorithms such as CNN, PointNet, LIFT, FMP, PV-RCNN, 3D U-Net, and GCNs. 2D image detection and 3D point cloud detection can be combined through a variety of methods to achieve the final detection results.

[0096] In some embodiments, damaged object extraction using a combination of 2D image detection and 3D point cloud analysis may include the following steps: detecting objects in the 2D image and generating a 2D bounding box; using the 2D bounding box to generate a 3D viewing cone and obtain point cloud data within the viewing cone; extracting global features from the obtained point cloud data and performing feature classification and object detection. It should be understood that this is only one possible implementation of the method of this application, and any additions, deletions, or adjustments to this method are still within the scope of the description of this application.

[0097] In some embodiments, the 3D viewing cone may be obtained without detecting objects in a 2D image.

[0098] In some embodiments, when acquiring global features from point cloud data, the detection performance of small targets can be improved, thereby enhancing overall detection accuracy, by enhancing the ability to capture local feature structures. This can be achieved by utilizing edge convolution algorithm modules, optimizing neighborhood searches, and adding local feature descriptors.

[0099] The second embodiment of the present application provides an automated container inspection system that uses a multi-layer perceptron for point cloud registration. It includes four parts: a central control subsystem, an AI visual preprocessing subsystem, an acquisition control subsystem, and a backend service subsystem. The inspection process is as follows: Figure 2 As shown in the system structure diagram Figure 3a The specific steps for verification are as follows:

[0100] S201: The central control subroutine will first guide the vehicle loaded with containers to be inspected to enter the site.

[0101] S202: The AI ​​vision pre-processing subsystem then obtains the vehicle's license plate and container number, instructing the driver to open the container door so the program can inspect the container floor using a camera. After completing the inspection, the program will confirm whether the inspection area is clear.

[0102] S203: If no personnel remain in the inspection area, the acquisition control subsystem will be activated to scan the container exterior to obtain point cloud data. After preliminary processing, the scanned data will be uploaded to the backend service subsystem.

[0103] S204: The acquired point cloud data is registered and damaged by the deep learning algorithm of the backend service subsystem, and the results are finally transmitted to the user for review. The data that passes the review can be directly pushed to the quotation system.

[0104] The following will introduce the composition, function and implementation of each subsystem respectively.

[0105] The central control subsystem consists of hardware such as front and rear gates, guide sentries, radars, photoelectric sensors, central control industrial computers, and software systems such as central control programs, sentry guidance programs, and gate control programs. Figure 3a shown.

[0106] The central control program is responsible for controlling the operation of all subsystems and front-end devices, sending control commands and receiving status information from each subsystem. It coordinates the orderly participation of each subsystem in the overall system workflow based on their status in real time. Furthermore, the central control program receives and processes alarm and fault information from each subsystem, and can output alarm information through audio and visual means.

[0107] The Sentry Guidance Program is designed for drivers of vehicles entering the site. It is responsible for receiving commands from the central control program and guiding the drivers of vehicles entering the site on their routes through voice and text, prompting them to complete a series of operations such as opening and closing the rear door and leaving the detection area after parking the vehicle.

[0108] The barrier control program is the prerequisite for maintaining the orderly progress of the entire container inspection process. It is responsible for receiving commands from the central control program and combining the radar signals and photoelectric sensors of the front and rear barriers to control the orderly entry, inspection and departure of vehicles.

[0109] Specifically:

[0110] 1) When the gate control program receives the "entry" command from the central control program, the gate opens under program control, and the sentry guidance program begins to guide the driver into the inspection;

[0111] 2) When a vehicle enters the detection area and the rear barrier radar does not detect any obstacles, the program controls the rear barrier to close;

[0112] 3) The gate control program sends a "vehicle in position" command to the central control program. After receiving the command, the central control program starts sending a "snap shot" command to the AI ​​vision pre-processing subsystem;

[0113] 4) After receiving the "leave" command from the central control program, the gate control program controls the front gate to open until the vehicle completely leaves the sensing area of ​​the front gate, and then controls the front gate to close and the rear gate to open.

[0114] The AI ​​visual pre-processing subsystem consists of hardware devices such as edge computing devices, high-position cameras (HD capture cameras), surveillance cameras, gate cameras, and software systems such as license plate recognition programs, box number recognition programs, door status and personnel detection programs, and box floor inspection programs. Figure 3b shown.

[0115] The license plate recognition program is connected to the gate camera and is responsible for real-time acquisition of images near the gate, and performing license plate detection and license plate number recognition on the images. After a valid license plate is recognized, the license plate extraction instruction is fed back to the central control program, which is then centrally dispatched.

[0116] The box number recognition program is connected to the high-position camera, and the edge computing device pulls the camera video stream through the RTSP protocol for analysis.

[0117] Specifically:

[0118] 1) When an incoming vehicle enters the camera's field of view, the container number recognition program begins to call the trained CNN model to identify the container door and detect and capture the container number area on the door image;

[0119] 2) Then, using the OCR character recognition algorithm, the container number characters in the area are extracted, verified, and saved;

[0120] 3) After receiving the vehicle "in position" command from the central control program, the program will use the box number that appears most frequently as the final recognition result;

[0121] 4) Finally, the program feeds back the recognition results to the central control program as an auxiliary judgment condition for the central control program.

[0122] The container floor inspection program connects to a high-position camera and processes images captured by fill-light of the container floor. It then uses a trained deep learning-based object detection and segmentation model to detect grease, foreign objects, dirt, cracks, and other contaminants on the floor. It then generates inspection results and uploads them to a data server for storage. Unlike the container number recognition program, the container floor inspection program analyzes single images captured by the high-position camera using the camera SDK.

[0123] Specifically:

[0124] 1) After receiving the "snap shot" command from the central control program, the floor inspection program in the box starts sending the "snap shot" command to the high-definition snapshot camera through the camera's private protocol or physical trigger signal;

[0125] 2) After receiving the "snap" command, the HD camera starts to use the fill light to capture the image and transmits the captured image back to the edge computing device;

[0126] 3) The program parses the returned information and uses a trained deep learning-based object detection and segmentation model to inspect the container's interior floor.

[0127] 4) Finally, the program uploads the test results to the data server of the backend service subsystem through a private protocol for storage.

[0128] The door status and personnel detection program connects to a surveillance camera, and the edge computing device pulls the camera video stream via the RTSP protocol for analysis. After the incoming vehicle stops, this program uses a deep learning-based door and personnel detection algorithm to determine the open and closed status of the container door and the presence of pedestrians in the detection area in real time. The recognition results are fed back to the central control program, which serves as the basis for determining whether the servo motor is on or off in the acquisition control subroutine.

[0129] Specifically:

[0130] 1) After receiving the vehicle "in position" command from the central control program, the door status and personnel detection program starts to call the deep learning-based object detection algorithm to detect the open and closed status of the container door;

[0131] 2) The program will feed back the detection status to the central control program in real time, and the central control program will guide the driver to open the door through the sentry guidance program;

[0132] 3) After the driver opens the door, the door status and personnel detection program detects that the door is open, the high-position camera takes a photo, and the floor inspection program inside the box is started to detect whether there are any defects in the floor and upload the results;

[0133] 4) After the floor inspection program is uploaded, the program switches to the pedestrian detection algorithm to detect whether there are pedestrians in the gantry movement area;

[0134] 5) The program will feed back the detection status to the central control program in real time, and the central control program will be responsible for sending "acquisition" or "interruption" instructions to the acquisition control subsystem.

[0135] The acquisition and control subsystem is used to collect 2D image data and 3D point cloud data from the cabinet. The system consists of a hardware system consisting of an acquisition control unit, servo motors, multiple 3D cameras positioned at different locations, a 2D line scan camera and its light source, and software systems including an electromechanical control program, a 2D image acquisition program, a 3D image acquisition program, and an image preprocessing program.

[0136] Specifically:

[0137] 1) When the acquisition program receives the "acquisition" instruction from the central control program, the electromechanical control program starts and notifies the acquisition program to complete the acquisition position action;

[0138] 2) The servo motor starts to drive the 3D camera and 2D line scan camera to scan and transmit data in real time;

[0139] 3) When the acquisition program receives the "end" command from the electromechanical control program, the acquisition program begins to analyze the 2D line scan camera scan results;

[0140] 4) The image preprocessing program first uses the calibrated rotation matrix to stitch the consecutive scanned images together to form a complete container surface image;

[0141] 5) The image preprocessing program removes excess noise by using statistical filtering and bilateral filtering to reduce the error introduced when the 3D camera collects data, and uses a sliding window for smoothing filtering;

[0142] 6) Afterwards, the image preprocessing program uses clustering to segment the container point cloud data that needs to be detected.

[0143] 7) Based on the extracted point cloud, re-establish a Cartesian coordinate system with a vertex in the container as the coordinate origin, the normal vector of the container surface as the Z axis, and the long edge as the X axis

[0144] 8) Upload the pre-processed point cloud to the back-end service subsystem through a private protocol, which will perform further data fusion, point cloud registration, and damage detection.

[0145] The back-end service subsystem analyzes the two-dimensional and three-dimensional images collected by the acquisition control subsystem. It first fuses the two-dimensional image data with the three-dimensional image data of multiple cameras through an improved three-dimensional fusion algorithm, and aligns them for subsequent damage detection of the container body.

[0146] Specifically, the damage detection method for the container body includes:

[0147] 1) Before the camera is collected, the 2D line scan camera is calibrated by Zhang Zhengyou to determine the internal and external parameters of the camera and calculate the homography matrix for calculating the actual image results

[0148] 2) For 3D cameras, first use a checkerboard grid for geometric calibration, simultaneously determine the internal and external parameters of the 3D camera, and calibrate the laser and structured light source;

[0149] 3) Based on the structured light and internal and external parameter information of the 3D camera, the relative position relationship between the camera, light source and object is guaranteed to be accurate. Through the calibration plate and specific calibration algorithm, the overall error of the system is corrected to improve the measurement accuracy;

[0150] 4) After the multi-camera image acquisition is completed, the feature description of the point cloud data is first extracted. Here, the Inception network is used to implement feature description;

[0151] 5) Based on the key points obtained from the feature description, the source key points and target key points of the point cloud to be registered are recorded as The descriptors are respectively recorded as in, Indicates that the number of key points is M, each key point has three coordinate values ​​(x, y, z), It means that the number of descriptors is M (one descriptor for each key point), and each descriptor has C eigenvalues. S A source key point in We first perform a k-nearest neighbor search in the descriptor space to find X T Among the K candidate key points {x T1 ,…,x TK} and source keypoints form a cluster, such as Figure 3d As shown;

[0152] 6) Construct the cluster features, which consists of three parts: geometric features F G , descriptor feature F D and similarity feature F S . F G It is the connection between the coordinates of the source keypoint and the candidate keypoint; in addition, the relative coordinates and distance between the candidate keypoint and the source keypoint are calculated as additional geometric features. D It consists of the descriptors of the source keypoints and the candidate keypoints; in addition, the saliency uncertainty of the keypoints is also included in F D In. F S It includes bilateral consistency and neighborhood consistency of key points.

[0153] 7) To learn cluster features, first the cluster features will be generated through a three-layer shared multi-layer perceptron (Shared-MLP) to generate feature maps Then, a max pooling layer and a Softmax function are applied to each candidate keypoint x TK Predict the attention weight w TK ; Combine the candidate target key points with the attention weight w TK Multiply and sum to obtain the corrected target key points

[0154] 8) Feature Map It will also go through an MLP and Sigmond function to predict the confidence c, and the final confidence Will be through the formula get;

[0155] 9) Finally, the optimal transformation R for point cloud registration * ,t *It can be obtained by the following calculation formula:

[0156]

[0157] in, and are the corresponding source key points and the corrected target key points, is the corresponding confidence, and ‖·‖2 represents the L2 norm.

[0158] Based on the calculated rotation and translation matrix R * ,t * , aligning the point cloud data collected by the 3D camera to the same coordinate system.

[0159] Referring to the above method, the point cloud data collected by multiple 3D cameras deployed at different locations are aligned to the same coordinate system, thereby completing the splicing of the point cloud data and forming a complete point cloud data. Among them, the process of splicing the point cloud data includes deduplication of the point cloud. For example, in the complete point cloud data, when the distance between two points is lower than a preset threshold (indicating that due to calculation error or camera error, the two points are very close and should be the same point), the extreme value (maximum or minimum value) or the average value can be used to determine the point coordinates to achieve deduplication.

[0160] Using the complete point cloud data from the aforementioned registration, damaged objects are detected using an improved Frustum PointNets network. The final damaged structured data is then presented to the client via a web application for user review. Subsequently, approved data is directly pushed to the higher-level quotation system via the data transmission module.

[0161] Frustum PointNets is a deep learning model for 3D object detection, primarily applied to point cloud data. Proposed by a research team at Stanford University, it aims to improve the accuracy and efficiency of point cloud-based 3D object detection. The key idea behind Frustum PointNets is to leverage 2D object detection results to guide the processing of 3D point cloud data, thereby reducing computational complexity and improving detection accuracy. However, due to its focus on global features, it can lose local detail information, making it ineffective when processing scenes with complex local structures. This patent addresses the aforementioned shortcomings of point clouds, and the improved workflow can be divided into the following steps:

[0162] 1) 2D object detection: First, 2D object detection is performed on the RGB image to generate a series of 2D bounding boxes.

[0163] 2) 3D frustum extraction: A 2D bounding box is used to generate a corresponding 3D frustum in the point cloud data. The frustum is a 3D extension of the spatial region from the camera position to the 2D bounding box.

[0164] 3) Point cloud clipping within the viewing cone: Point cloud data is extracted within the viewing cone, and these point cloud data are considered to be points that may contain target objects.

[0165] 4) For each point in the viewing cone, a K-nearest neighbor (KNN) graph is dynamically constructed based on the distance between points. That is, the neighborhood changes dynamically during each training and inference step to ensure that local geometric changes in the point cloud are captured;

[0166] 5) Use the EdgeConv layer to perform graph convolution on the neighborhood graph. Apply the EdgeConv layer to the neighborhood of each point to extract local neighborhood edge features. By stacking multiple EdgeConv layers, higher-level features are gradually captured.

[0167] EdgeConv is defined as follows:

[0168] h Θ (x i ,x j )=RELU(Θ·(x i -x j )+Φ·x j

[0169] Among them, x i and x j are two points in the point cloud, Θ and Φ are learnable parameters.

[0170] 6) Aggregate global features: Use the global maximum pooling operation (Global Max Pooling) to aggregate the features of all points into a global feature vector, which represents the global features of the entire point cloud.

[0171] 7) The global features of the point cloud will be used as input for subsequent instance segmentation and 3D object detection of Frustum PointNets.

[0172] 8) Instance segmentation: Use PointNet to classify the points in the point cloud and distinguish between target object points and background points.

[0173] 9) 3D bounding box regression: Regress the points of the target object, predict the 3D bounding box, and finally obtain the result of 3D object detection.

[0174] Finally, the target detection results will be sent to the user for review, and the data that passes the review can be directly pushed to the quotation system.

[0175] The third embodiment of the present application provides a container damage detection device, such as Figure 4 Shown, including:

[0176] A first acquisition unit is configured to acquire a 2D image of the container body captured by a 2D camera;

[0177] a second acquiring unit, configured to acquire multiple sets of 3D point cloud data of the container body acquired by multiple 3D cameras, wherein each set of 3D point cloud data corresponds to a portion of the container body;

[0178] a registration unit, configured to register each set of 3D point cloud data to a unified coordinate system, and obtain complete 3D point cloud data of the container body in the coordinate system;

[0179] A detection unit is used to detect the container body based on the 2D image and the complete 3D point cloud data.

Claims

1. A method for detecting damage to a container body, characterized in that: include: Acquire a 2D image of the container body captured by a 2D camera; Acquire multiple groups of 3D point cloud data of the container body respectively collected by multiple 3D cameras, wherein each group of 3D point cloud data corresponds to a part of the container body; Registering each set of 3D point cloud data to a unified coordinate system to obtain complete 3D point cloud data of the container body in the coordinate system; The container body is inspected according to the 2D image and the complete 3D point cloud data.

2. The method according to claim 1, characterized in that The registering each set of 3D point cloud data to a unified standard coordinate system includes: A. Extract the descriptors of each key point included in each set of 3D point cloud data; B. For each set of 3D point cloud data, perform the following steps: For each source key point in the group, a plurality of candidate target key points are determined by performing a k-nearest neighbor search in the descriptor space, wherein each source key point and the corresponding plurality of candidate target key points form each cluster; wherein the source key point is located in the group of 3D point cloud data to be registered, and the target key point is located in a group of 3D point cloud data in the unified coordinate system; Extract the features of each cluster; Input the features of each cluster into the first model, and output the correction values ​​and confidence levels of each candidate target key point corresponding to the source key point in each cluster; The optimal rotation and translation matrix used for registration to the unified coordinate system is calculated based on the following formula: Among them, R * ,t * represents the optimal value in the rotation and translation matrix R,t, represents the source key point, represents the correction value of the candidate target key point, represents the confidence, ‖·‖2 represents the L2 norm, and M represents the number of candidate key points; The group of 3D point cloud data to be registered is registered to the unified coordinate system using the optimal rotation and translation matrix.

3. The method according to claim 2, characterized in that The descriptors of the key points included in each set of 3D point cloud data are extracted through the Inception model.

4. The method according to claim 2, characterized in that: The extracting of the feature of each cluster includes extracting at least one of the following features of each cluster: a geometric feature of the cluster, a descriptor feature of the cluster, and a similarity feature of the cluster.

5. The method according to claim 4, characterized in that: The geometric features of the cluster include at least one of the following: the connection between the source key point in the cluster and each candidate target key point, the relative coordinates and the distance between them; or The descriptor features of the cluster include at least one of the following: descriptors, saliency, and uncertainty of the source key points and each candidate target key point in the cluster; or The similarity characteristics of the clusters include at least one of the following: bilateral consistency and neighborhood consistency.

6. The method according to any one of claims 2 to 5, characterized in that: The features of each cluster are input into the first model, and the correction values ​​and confidence levels of each candidate target key point corresponding to the source key point in each cluster are output, including: Inputting the features of each cluster into a shared multi-layer perceptron module, a pooling module, and a first activation function module connected in sequence, which are included in the first model, and the first activation function module outputs the attention weight of each candidate key point; The output of the shared multilayer perceptron module included in the first model is multiplied by the attention weight and then input into the sequentially connected multilayer perceptron module and the second activation function module, and the second activation function module outputs the initial confidence c i , and the confidence level used is calculated according to the following formula Each candidate key point in the cluster is multiplied by the attention weight to output a correction value of each candidate target key point 7. The method according to claim 1, characterized in that The detecting of the container body according to the 2D image and the complete 3D point cloud data includes: Detecting 2D target objects in the 2D image to generate a series of 2D bounding boxes, wherein the target objects include a container body; generating a corresponding 3D view cone in the complete 3D point cloud data using the 2D bounding box; Extracting point cloud data within the 3D viewing cone; Extracting global features from the extracted point cloud data, classifying the points of the target object and the background points, and regressing the points of the target object to predict the 3D boundary of the target object; A detection result of the container body is obtained based on the 3D boundary of the target object.

8. The method according to claim 7, characterized in that The extracting global features from the extracted point cloud data comprises: For each point in the extracted point cloud data, a K nearest neighbor graph is constructed according to the distance between the points; Performing graph convolution operation on the K nearest neighbor graph through a multi-layer edge convolution module to extract high-level features; A pooling operation is performed on the provided high-level features to output the global features.

9. A damage detection device for a container body, characterized in that: include: A first acquisition unit is used to acquire a 2D image of a container body captured by a 2D camera; A second acquisition unit is used to acquire multiple groups of 3D point cloud data of the container body respectively collected by multiple 3D cameras, wherein each group of 3D point cloud data corresponds to a part of the container body; A registration unit, used for registering each group of 3D point cloud data to a unified coordinate system to obtain complete 3D point cloud data of the container body in the coordinate system; A detection unit is used to detect the container body according to the 2D image and the complete 3D point cloud data.

10. A container damage detection system, characterized in that: include: A 2D camera, used to collect 2D images of the container body; A plurality of 3D cameras disposed at different positions are used to respectively collect a plurality of groups of 3D point cloud data of the container body, wherein each group of 3D point cloud data corresponds to a part of the container body; The back-end service subsystem is used to execute the damage detection method for the container body described in any one of claims 1-8.

Images