A container empty container detection method and system based on multi-modal fusion

Abstract

The application discloses a container empty container detection method and system based on multi-modal fusion. The method comprises the following steps: when the container door is opened, triggering two groups of sensor units to synchronously collect three-dimensional point cloud data and RGB images inside the container; based on the pre-calibrated sensor parameters, the three-dimensional point cloud data collected on both sides is fused into the same coordinate system to form a complete three-dimensional point cloud inside the container; the complete three-dimensional point cloud is texture mapped with the RGB image to generate a fusion point cloud model with color information; based on the fusion point cloud model, by removing the container body structure point cloud and performing clustering analysis on the remaining internal point cloud, it is detected whether there is a residual object in the container; and based on the fusion point cloud model, the inner surface of the container body is fitted, the actual internal size is calculated and compared with the standard size to determine whether there is a sandwich structure; and a detection report containing residual object information and sandwich detection results is output. The application can realize accurate detection of the container empty container.

Description

Technical Field

[0001] This invention relates to the fields of smart customs and artificial intelligence technology, specifically to a method and system for detecting empty containers based on multimodal fusion. Background Technology

[0002] In customs clearance and terminal management processes, the rapid and accurate verification of containers declared as empty is a crucial step in preventing the concealment of prohibited items and maintaining the security of trade operations. When a container is declared empty, customs needs to verify its emptiness at the checkpoint when the external truck (container truck from an external logistics company) leaves the terminal. Currently, the mainstream methods for detecting whether a container is empty all have significant drawbacks:

[0003] Manual inspection: This method relies on customs officers entering containers to conduct sampling inspections. It has inherent drawbacks such as low efficiency, high labor intensity, susceptibility to subjective factors, and inability to guarantee the detection rate. It can easily cause customs congestion during peak hours and may also pose security risks.

[0004] Technical testing:

[0005] 1) Ultrasonic testing: Although it has no radiation, it is not capable of detecting residues that are stationary and close to the box wall, and its accuracy is insufficient.

[0006] 2) Radiation imaging (X-ray): There is a risk of ionizing radiation, the equipment purchase and maintenance costs are high, and the cost-effectiveness for empty box inspection is low.

[0007] 3) Weight analysis: This is an indirect detection method, which cannot locate residues or identify interlayers, and is not sensitive to trace amounts of residue.

[0008] 4) Pure visual inspection: affected by weather, lacks depth information (cannot identify interlayers), and is not effective in identifying trace residues.

[0009] 5) Pure LiDAR Detection: This method typically deploys 3D LiDAR at the gate boom. It has specific requirements for the radar's installation height to ensure a complete scan of the container's internal structure, often involving significant on-site engineering work such as raising the boom, structural reinforcement, and wiring modifications. Furthermore, the long-term operation of the radar mounted on the boom not only affects the overall detection accuracy but also impacts the stress distribution on the boom and the equipment's lifespan due to continuous load and vibration. Simultaneously, this method lacks visual information support, failing to intuitively present the specific shape and appearance of residues inside the container and making it difficult to retain visual evidence of violations, thus limiting its effectiveness in tracing responsibility and verifying data.

[0010] In summary, existing technologies fail to comprehensively utilize the precise three-dimensional geometric information and rich two-dimensional texture and color information inside the container, and cannot achieve a detection solution that combines high-precision three-dimensional measurement, rich visual information, easy deployment, and universality at a low cost. Summary of the Invention

[0011] The purpose of this invention is to provide a method, system, storage medium, and computer program product for detecting empty containers based on multimodal fusion, so as to solve the problems mentioned in the background art.

[0012] The first aspect of this invention provides a method for detecting empty containers based on multimodal fusion, comprising the following steps:

[0013] Step S1: When the container door is opened, two sets of sensor units are triggered to synchronously collect three-dimensional point cloud data and RGB images inside the container; each set of sensor units includes a three-dimensional lidar and a high-definition optical camera.

[0014] Step S2: Based on the pre-calibrated sensor parameters, the three-dimensional point cloud data collected from both sides are fused into the same coordinate system to form a complete three-dimensional point cloud inside the container.

[0015] Step S3: Map the complete 3D point cloud to the RGB image to generate a fused point cloud model with color information;

[0016] Step S4: Based on the fused point cloud model, the container structure point cloud is removed and the remaining internal point cloud is clustered to detect whether there are any remaining objects inside the container; and the internal surface of the container is fitted based on the fused point cloud model to calculate the actual internal dimensions and compare them with the standard dimensions to determine whether there is a sandwich structure.

[0017] Step S5: Output an inspection report containing information on the remaining objects and the results of the interlayer inspection.

[0018] A second aspect of the present invention provides a container empty container inspection system based on multimodal fusion, comprising:

[0019] The data acquisition module is configured to: trigger two sets of sensor units to synchronously acquire three-dimensional point cloud data and RGB images inside the container when the container door is opened; each set of sensor units includes a three-dimensional lidar and a high-definition optical camera;

[0020] The point cloud fusion module is configured to: fuse the three-dimensional point cloud data collected from both sides into the same coordinate system based on pre-calibrated sensor parameters to form a complete three-dimensional point cloud inside the container;

[0021] The texture mapping module is configured to: perform texture mapping between a complete 3D point cloud and an RGB image to generate a fused point cloud model with color information;

[0022] The detection and analysis module is configured to: detect whether there are any remaining objects inside the container by removing the container structure point cloud and performing cluster analysis on the remaining internal point cloud based on the fused point cloud model; and calculate the actual internal dimensions by fitting the inner surface of the container based on the fused point cloud model and comparing them with standard dimensions to determine whether there is a sandwich structure.

[0023] The results output module is configured to output a test report containing information on the remains and the results of the interlayer detection.

[0024] A third aspect of the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method described in any of the preceding claims.

[0025] A fourth aspect of the present invention provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method described in any of the preceding claims.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] 1. High precision and high reliability: By fusing three-dimensional point cloud and two-dimensional image information, and combining point cloud fusion and fine calibration technology, this invention can achieve centimeter-level size measurement and detection of tiny foreign objects, far exceeding other technical solutions.

[0028] 2. Comprehensive Inspection: Simultaneously addresses the two pain points of leftover items and interlayer issues, providing a comprehensive solution for empty container safety inspection.

[0029] 3. High efficiency and automation: The entire testing process is completed in tens of seconds without human intervention, enabling 24 / 7 uninterrupted operation and greatly improving on-site work efficiency.

[0030] 4. The test results are intuitive and reliable: The generated color point cloud model and the two-dimensional image with precise annotations make the test results more intuitive, which facilitates customs officers to conduct rapid verification and decision-making.

[0031] 5. Adaptability and robustness: It combines the reliability of 3D measurement with the intuitiveness of 2D vision, enabling the system to work stably under different lighting and weather conditions (radar is not affected by light). Furthermore, the point cloud fusion and fine calibration process enable the system to have self-learning and self-optimization capabilities, gradually adapting to changes in the field environment and sensor drift, ensuring long-term operational stability.

[0032] 6. Universality and scalability: The proposed installation method (installation on pillars on both sides of the checkpoint road) is a typical layout of customs checkpoints, requiring no modification to existing infrastructure. The solution is easy to replicate and deploy quickly at various customs ports. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a method for detecting empty containers based on multimodal fusion, as disclosed in an embodiment of the present invention.

[0034] Figure 2 This is a schematic diagram of the sensor unit layout disclosed in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the structure of a container empty container detection system based on multimodal fusion disclosed in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] To address the technical problems mentioned in the background section, this invention provides a container empty container detection scheme based on multimodal fusion of dual-sided lidar and camera. Its core idea is to utilize a specific installation layout to acquire complementary point cloud data using dual-sided lidar, and then achieve high-precision fusion and analysis of the point clouds through a sophisticated data processing workflow, thereby enabling accurate detection of empty containers (including abandoned items), interlayers, and other components.

[0038] Please see Figure 1 This invention provides a method for detecting empty containers based on multimodal fusion, comprising the following steps:

[0039] Step S1: When the container door is opened, two sets of sensor units are triggered to synchronously collect three-dimensional point cloud data and RGB images inside the container; each set of sensor units includes a three-dimensional lidar and a high-definition optical camera.

[0040] In this step, when the vehicle arrives at the checkpoint (for example, by analyzing video streams to determine if the truck is parked within a pre-defined detection area), and the container door is determined to be open, the sensor units deployed on both sides of the checkpoint are automatically triggered to work synchronously. The left and right sensor units independently and synchronously collect data on the interior space of the container.

[0041] Among them, the 3D LiDAR is responsible for acquiring dense point cloud data with precise 3D coordinates inside the container. The point cloud data fully reflects the spatial geometry of the container's inner walls and internal objects. The high-definition optical camera works in conjunction with the LiDAR to acquire high-resolution RGB images covering the same field of view at the same time. The RGB images provide rich visual appearance information such as texture and color of the scene inside the container.

[0042] Step S2: Based on the pre-calibrated sensor parameters, the three-dimensional point cloud data collected from both sides are fused into the same coordinate system to form a complete three-dimensional point cloud inside the container.

[0043] In this step, the sensor intrinsic and extrinsic parameters, which were pre-calibrated and acquired during the offline phase, are invoked, primarily the extrinsic parameter matrix between the left and right sensor units. Based on this extrinsic parameter matrix, for example, the 3D point cloud data collected by the right sensor is uniformly transformed to the coordinate system of the left sensor, achieving preliminary alignment and stitching of the point cloud data from both sides, and obtaining a complete 3D point cloud of the container's interior.

[0044] Furthermore, to improve fusion accuracy, this invention uses the geometric features such as parallelism and symmetry that the inner surfaces of the container (e.g., left and right sidewalls) should satisfy in actual physical space as constraints to construct a nonlinear optimization problem. The extrinsic parameter matrices of the initially used left and right sensors are iteratively optimized and reverse-corrected. After optimization, the optimized extrinsic parameters are used for point cloud transformation and stitching. Then, planar segmentation algorithms such as RANSAC are employed to accurately segment the left and right sidewalls, top surface, bottom surface, and innermost end face of the container from the stitched point cloud, thereby forming a high-precision, complete 3D point cloud model of the container's interior.

[0045] Step S3: Map the complete 3D point cloud to the RGB image to generate a fused point cloud model with color information;

[0046] In this step, for each 3D point P(x,y,z) in the fused 3D point cloud model generated in step S20, the pre-calibrated corresponding camera-LiDAR extrinsic parameter matrix and camera intrinsic parameters are queried according to the point's source (left or right sensor unit). Using these parameters, the 3D point P is accurately projected onto the RGB image plane acquired by the corresponding high-definition optical camera, and its two-dimensional pixel coordinates (u,v) in the image are calculated. From these pixel coordinates (u,v), the RGB color value (including the brightness information of the red, green, and blue channels) is extracted, and this color attribute is assigned to the 3D point P.

[0047] By traversing all 3D points, a fused point cloud model with color attributes, combining high-precision 3D geometric information with rich 2D texture colors, is finally generated. In essence, this fused point cloud model possesses both the high-precision 3D geometric information measured by LiDAR and the realistic visual appearance information acquired by the camera.

[0048] Step S4: Based on the fused point cloud model, the container structure point cloud is removed and the remaining internal point cloud is clustered to detect whether there are any remaining objects inside the container; and the internal surface of the container is fitted based on the fused point cloud model to calculate the actual internal dimensions and compare them with the standard dimensions to determine whether there is a sandwich structure.

[0049] In this step, based on the color fusion point cloud model generated in step S3, the following two types of parallel detection analysis are performed:

[0050] Residue Detection: The point cloud of the box structure (each inner surface) segmented in step S2 is removed from the fused point cloud model, retaining only the point cloud of the internal space of the box. Cluster analysis is performed on this internal point cloud to identify object clusters independent of the box walls. For each identified object cluster, its three-dimensional dimensions (length, width, height), spatial coordinates within the box, and color information are calculated and output to comprehensively determine whether there are non-empty relics.

[0051] Layer detection: Based on the fused point cloud model, the planar equations of each inner surface are accurately fitted again, and the intersection lines and intersection points between these planes are calculated to accurately calculate the actual length, width, and height dimensions of the container's internal space. This calculated dimension is then compared with the dynamic standard dimensions and their statistical variance (e.g., 3σ range) of the container type, obtained through self-learning from historical data and stored in the database. If the deviation of a calculated dimension (e.g., length) from the standard value exceeds a certain multiple of its variance (e.g., 3 times the variance), or if the parallelism of the container's relative surfaces exceeds a set threshold, the container is determined to have an illegally constructed layer structure.

[0052] Step S5: Output an inspection report containing information on the remaining objects and the results of the interlayer inspection.

[0053] In this step, the analysis results from step S4 are structured and packaged to generate an inspection report. The inspection report includes at least the overall status of empty / non-empty containers, the determination of the presence of interlayers, and a detailed list of detected remnants (such as quantity, 3D dimensions, and location of each remnant). Simultaneously, the inspection results can be visualized, for example, by drawing a prominent warning box around the detected remnants on the original RGB image. Finally, the structured inspection report and the visualized image are submitted together to the customs business system for subsequent verification and decision-making.

[0054] As an example, the two sets of sensor units are symmetrically deployed on both sides of the checkpoint channel, with their installation lines perpendicular to the lane direction and the installation spacing greater than the maximum width of a standard container truck.

[0055] To achieve comprehensive, blind-spot-free scanning of the interior of a standard shipping container, this invention employs a symmetrical deployment scheme that conforms to on-site engineering specifications. Specifically, as follows... Figure 2 As shown, two sets of sensor units are rigidly mounted on fixed columns on both sides of the customs checkpoint channel in a mirror-symmetrical manner. The line connecting the two mounting points is designed to be substantially perpendicular to the direction of travel of the container truck lane, thus ensuring that the sensors observe from the side of the container. In addition, the horizontal distance between the two mounting points is greater than the maximum outer width of the current standard container truck, thereby ensuring that even when the largest container truck is parked in the detection area, the sensors on both sides still have sufficient unobstructed viewing windows, allowing the emitted laser beams and light to enter from the side of the open container door, completely covering the entire internal cavity space from the near edge of the container door to the far corner (i.e., the inner end face near the front of the truck), laying the physical foundation for subsequent acquisition of complete point cloud data.

[0056] As an example, in step S2, based on the geometric feature constraints of the inner surface of the container, the extrinsic parameter matrices of the point clouds on the left and right sides are iteratively optimized to obtain the complete three-dimensional point cloud.

[0057] During the online inspection phase, relying solely on the initial extrinsic parameter matrix obtained from offline calibration for point cloud stitching may introduce accumulated errors due to factors such as on-site vibration and temperature drift. To address this issue, this invention introduces a self-optimization process based on the scene's geometric characteristics after the initial stitching. The core idea of ​​this self-optimization process is to utilize the container itself as a known calibration reference with regular geometric features.

[0058] Specifically, the first step is to extract a subset of point clouds representing the inner surfaces of the container (especially the left and right sidewalls) from the initially stitched point cloud. Ideally, these inner surfaces should be flat and parallel to each other. Based on this, the present invention constructs a nonlinear optimization problem centered on planar constraints and mirror constraints. The planar constraint requires that the same physical plane (such as the left sidewall) fitted from the left and right point clouds should coincide after transformation; the mirror constraint utilizes the symmetry of the container structure.

[0059] By using iterative optimization algorithms (such as the Levenberg-Marquardt algorithm) to adjust the extrinsic parameter matrices (rotation matrix R and translation vector t) between the left and right sensors, the transformed point clouds on both sides can achieve optimal alignment under these strong geometric constraints.

[0060] This process can significantly correct subtle system errors and improve point cloud stitching accuracy to the millimeter level, ensuring the reliability of subsequent size measurements and the detection of tiny foreign objects.

[0061] As an example, based on the geometric feature constraints of the inner surface of the container, the extrinsic parameter matrices of the point clouds on the left and right sides are iteratively optimized to obtain the complete 3D point cloud, including:

[0062] Geometric features that characterize the rigid structure inside the container are extracted from the left and right point clouds respectively. Based on the extracted geometric features, at least two different types of spatial correspondence constraints with complementary physical meanings are established between the left and right point clouds.

[0063] By fusing at least two of the aforementioned spatial correspondence constraints, a multi-constraint coupled nonlinear optimization model is constructed. The extrinsic parameter matrix is ​​then optimized and updated by iteratively solving the nonlinear optimization model to obtain a high-precision point cloud registration result that satisfies multiple geometric consistency requirements, thereby obtaining the complete 3D point cloud.

[0064] In this implementation, traditional methods typically rely on fitting the inner surface of the container to an ideal plane as the basis for subsequent constraints. However, in non-ideal situations (e.g., slight deformation of the container due to long-term use, dirt or coating peeling on the inner wall, or cargo already attached to the container wall when inspected), the point cloud of the inner surface cannot accurately fit a standard planar model, or the fitted plane can no longer represent the actual position of the container wall. This can lead to the failure of optimization based on planar constraints or introduce significant errors.

[0065] To address this issue, the method of this invention no longer relies on fitting an idealized planar model. Specifically, geometric feature elements characterizing the rigid internal structure of the container are extracted from the point cloud data acquired by the left and right sensor units, respectively. These geometric feature elements differ from the idealized planar model; instead, they are local spatial information directly identified from the point cloud data that is strongly correlated with the physical structure of the container (such as stiffeners and hinge seats). These structural features maintain their basic geometric properties and relative positions even when the container experiences slight deformation or its surface is partially obscured.

[0066] Based on the extracted geometric feature elements, at least two types of spatial correspondence constraints with complementary characteristics in both mathematical expression and physical meaning are constructed and established between the left and right point clouds. One type of constraint focuses on the absolute coordinate alignment accuracy of feature point pairs in three-dimensional space, providing precise anchor points for optimization; the other type of constraint focuses on the relative consistency of the structural line segments formed by feature points in length and direction, exhibiting better tolerance to local deformation and noise, thus providing effective supplementary constraints when planar features are unreliable.

[0067] Subsequently, at least two types of spatial correspondence constraints are fused to form a multi-constraint coupled nonlinear optimization mathematical model. By iteratively solving this mathematical model, the extrinsic parameter matrix parameters between the left and right sensor units can be gradually adjusted and optimized. After this optimization process, a high-precision point cloud registration result that satisfies multiple geometric consistency criteria is finally obtained. This result is the complete 3D point cloud that accurately reflects the true geometric condition inside the container, which is required for subsequent processes.

[0068] This invention, through a strategy based on rigid structural features and multi-constraint fusion, can effectively overcome the technical limitations of traditional methods when the geometry of containers is not ideal.

[0069] As an example, the establishment of at least two different types of spatial correspondence constraints with complementary physical meanings includes establishing rigid constraints based on non-coplanar feature point pairs, including:

[0070] In the left and right point clouds, identify and extract the inherent, non-temporary physical structural feature regions of the container, including but not limited to: the recessed area formed by the door hinge mounting base, the raised strip area formed by the inner wall reinforcing ribs, and the local area where the floor fasteners are located.

[0071] For each identified physical structure feature region, calculate its three-dimensional feature descriptor in its respective point cloud coordinate system and a local reference point representing the spatial location of the region;

[0072] By matching the three-dimensional feature descriptors in the point clouds on both sides, two local reference points that originate from the same physical structure and are located in the left and right point clouds respectively are established as a pair of virtual corresponding points.

[0073] A set of feature points that are not coplanarly distributed in three-dimensional space is formed by multiple pairs of virtual corresponding points, and constraints are established based on this set of feature points to minimize the sum of spatial distances between all corresponding point pairs after the transformation of the extrinsic parameter matrix.

[0074] In this embodiment, the aforementioned spatial correspondence constraint includes a rigid constraint based on non-coplanar feature point pairs, which solves the problem of how to obtain a stable and accurate absolute alignment reference in complex scenarios.

[0075] First, using point cloud data from the left and right sides, local shape analysis and pattern recognition algorithms are employed to automatically identify and extract the characteristic regions corresponding to the inherent, non-temporary physical structure of the container. These physical structure characteristic regions include, but are not limited to, specific recessed areas formed by door hinge mounting brackets on the container structure, regularly raised strips formed by the reinforcing ribs of the container's inner wall, and localized areas containing devices such as latches on the container floor used to secure cargo. These regions, due to their stable structure, distinctive shape features, and resistance to complete obscuring by temporary cargo, serve as reliable sources of baseline features.

[0076] For each successfully identified physical structural feature region, a three-dimensional feature descriptor is calculated in the local coordinate system of the point cloud on that side. This three-dimensional feature descriptor effectively encodes the spatial distribution, normal vector changes, and other local geometric characteristics of the point cloud in that region, and possesses a certain degree of rotation and translation invariance. Simultaneously, a local reference point is calculated and determined to characterize the spatial center position of the feature region, for example, by calculating the centroid or feature peak point of the point cloud in that region.

[0077] Next, by performing similarity matching calculations on the 3D feature descriptors of all extracted feature regions in the left and right point clouds (e.g., using a nearest neighbor search algorithm), feature regions whose descriptor similarity exceeds a preset threshold and are considered to originate from the same physical structure are paired. For each successfully paired region, the local reference points calculated in the left point cloud and the local reference points calculated in the right point cloud are established as a virtual corresponding point pair.

[0078] Ultimately, multiple sets of virtual corresponding point pairs constitute a set of feature points that are non-coplanarly distributed in three-dimensional space. Based on this set of feature points, the mathematical expression of the rigid constraint is constructed: its optimization objective is to find an optimal extrinsic transformation matrix such that, after this matrix transformation, the sum of the squares of the three-dimensional Euclidean distances between any two corresponding points in all virtual corresponding point pairs is minimized. This constraint directly reflects the physical essence that, under rigid transformation, the relative positions between feature points on the same object should remain unchanged.

[0079] As an example, the establishment of at least two different types of spatial correspondence constraints with complementary physical meanings also includes establishing soft constraints based on the topological consistency of the internal structure, including:

[0080] In the left and right point clouds, a set of local reference points are selected as nodes, and a spatial topology connection diagram is formed between the nodes according to the actual physical connection relationship or spatial proximity relationship inside the container.

[0081] In the spatial topology connection diagram, line segment pairs describing the same physical connection component are identified, and constraints are established for each identified line segment pair. The constraints require that after the transformation of the extrinsic parameter matrix, the length of the line segment pair should tend to be consistent, and their three-dimensional spatial directions should tend to be parallel.

[0082] The constraints established based on all identified line segments are summarized to form the soft constraints for the topological consistency of the internal structure.

[0083] In this embodiment, the aforementioned spatial correspondence constraint also includes a soft constraint based on the internal structural topology consistency, which aims to provide a relative geometric relationship correction mechanism that is more robust to local noise and non-rigid deformation.

[0084] Specifically, using the aforementioned extracted and calculated local reference points as basic nodes, connections are established between these nodes in the left and right point clouds, respectively, based on prior knowledge of the container's internal structure (such as the orientation of stiffeners and the relative positions of hinge seats) or the spatial proximity relationships between nodes in the point clouds. This constructs a spatial topology diagram describing the internal skeleton frame of the container. The spatial topology diagram consists of multiple line segments connecting two nodes, with each line segment representing a stable spatial connection relationship between two feature regions.

[0085] Then, in the spatial topology connection diagrams constructed from the left and right point clouds respectively, the line segment pairs describing the same physical connection component are identified by comparing whether the physical structural feature regions corresponding to the nodes at both ends of the line segments match. For example, the line segment connecting two reference points of the reinforcing rib protrusion area in the left point cloud and the line segment connecting the corresponding two reference points of the reinforcing rib protrusion area in the right point cloud constitute a matching line segment pair.

[0086] For each identified pair of line segments, a specific soft constraint is established. This soft constraint includes two levels of requirements: (1) after transformation by the extrinsic parameter matrix, the lengths of the two line segments in the pair should tend to be consistent, i.e., their length difference should be minimized; (2) after transformation, the direction vectors of the two line segments in the pair in three-dimensional space should tend to be parallel, i.e., their direction angle should be minimized. The above two requirements together constitute the constraint on the consistency of the local dimensions and orientation of the structure. It is understandable that this soft constraint has a higher tolerance for non-rigid deformation or point cloud noise of the container.

[0087] Finally, the constraints established based on all identified matching line segments are summarized and integrated to form a soft constraint for the topological consistency of the internal structure. Integrating this soft constraint, along with the aforementioned rigid constraints, into the nonlinear optimization model allows the optimization process to not only focus on the precise alignment of feature points but also enforce the overall correctness of the container's internal skeletal structure, thereby significantly improving the accuracy and stability of point cloud registration in complex real-world scenarios.

[0088] As an example, in step S4, the detection of the remnant is based on its three-dimensional geometric dimensions, color, and texture information; the determination of the sandwich structure is achieved by comparing the actual internal dimensions of the box with the dynamically updated standard dimensions of the box type and their allowable deviation range.

[0089] For object detection, a multimodal information fusion criterion is employed. Specifically, the presence and size of an object are determined not only by the 3D bounding box size of the clustered point cloud (geometric criterion), but more importantly, the RGB color and texture information attached to the object's point cloud are simultaneously fused (visual criterion). For example, a small, dark-colored block, if its color differs significantly from the bottom of the box and its texture is abnormal, will be assigned a higher confidence level and classified as a suspicious object, rather than a stain or shadow on the floor. Understandably, this comprehensive criterion can significantly reduce false alarms for changes in lighting and dust.

[0090] For mezzanine inspection, the core lies in a dynamic and adaptive dimensional comparison standard. Specifically, instead of using fixed theoretical container dimensions as thresholds, a continuously self-learning and updated database is maintained. This database dynamically maintains a set of standard dimensions and corresponding statistical variances for each common container type (e.g., 20GP, 40HQ). The standard dimensions are calculated using a moving average of historical inspection data, while the variance characterizes the dimensional fluctuation range of that container type under normal conditions. During inspection, the real-time calculated inner diameter is compared with the corresponding standard value in the database. Only when the deviation exceeds a certain multiple (e.g., 3σ) of its own variance is it considered an anomaly. Understandably, this method can effectively distinguish between manufacturing tolerances, measurement noise, and genuine illegal mezzanines, improving the system's robustness and adaptability to different container types.

[0091] As an example, before step S1, an offline calibration step is also included to obtain the relationship between the internal parameters and external pose of each group of sensor units and to establish the spatial transformation relationship between the left and right sensor groups.

[0092] The offline preparation phase is an essential and precise calibration process upon which online testing methods rely for proper operation. This process consists of two levels:

[0093] The first level involves calibrating the intrinsic and extrinsic parameters of a single sensor unit. For each LiDAR-camera pair, the camera's intrinsic parameters (focal length, principal point, distortion coefficients) and the rigid transformation relationship (extrinsic parameters) from the LiDAR to the camera need to be calibrated. This establishes a precise projection mapping relationship from 3D point clouds to 2D pixels, which is the mathematical foundation for subsequent point cloud shading or texture mapping.

[0094] The second level is joint calibration between sensor units. After the left and right sensor units are installed in place, a specific calibration object (such as a calibration plate with highly reflective markings) needs to be placed in the common field of view area. By simultaneously collecting data, the spatial transformation relationship between the coordinate systems of the two sensor units, i.e., the initial extrinsic parameter matrix, is calculated.

[0095] Understandably, the accuracy of offline calibration directly determines the upper limit of the final detection performance.

[0096] Please see Figure 3 As shown in the figure, this embodiment of the invention also discloses a multimodal fusion-based empty container detection system 200, comprising:

[0097] The data acquisition module 201 is configured to: trigger two sets of sensor units to synchronously acquire three-dimensional point cloud data and RGB images inside the container when the container door is opened; wherein, each set of sensor units includes a three-dimensional lidar and a high-definition optical camera;

[0098] The point cloud fusion module 202 is configured to: fuse the three-dimensional point cloud data collected from both sides into the same coordinate system based on pre-calibrated sensor parameters to form a complete three-dimensional point cloud inside the container;

[0099] Texture mapping module 203 is configured to: perform texture mapping between a complete 3D point cloud and an RGB image to generate a fused point cloud model with color information;

[0100] The detection and analysis module 204 is configured to: detect whether there are any remaining objects inside the container by removing the container structure point cloud and performing cluster analysis on the remaining internal point cloud based on the fused point cloud model; and calculate the actual internal dimensions by fitting the inner surface of the container based on the fused point cloud model and comparing them with standard dimensions to determine whether there is a sandwich structure.

[0101] The result output module 205 is configured to output a detection report containing information on the residue and the results of the interlayer detection. This invention also discloses a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the foregoing embodiments.

[0102] This invention also discloses a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing embodiments.

[0103] This invention also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing embodiments.

[0104] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

Claims

1. A method for detecting empty containers based on multimodal fusion, characterized in that, Includes the following steps: Step S1: When the container door is opened, two sets of sensor units are triggered to synchronously collect three-dimensional point cloud data and RGB images inside the container; each set of sensor units includes a three-dimensional lidar and a high-definition optical camera. Step S2: Based on the pre-calibrated sensor parameters, the three-dimensional point cloud data collected from both sides are fused into the same coordinate system to form a complete three-dimensional point cloud inside the container. Step S3: Map the complete 3D point cloud to the RGB image to generate a fused point cloud model with color information; Step S4: Based on the fused point cloud model, the container structure point cloud is removed and the remaining internal point cloud is clustered to detect whether there are any remaining objects inside the container; and the internal surface of the container is fitted based on the fused point cloud model to calculate the actual internal dimensions and compare them with the standard dimensions to determine whether there is a sandwich structure. Step S5: Output an inspection report containing information on the remaining objects and the results of the interlayer inspection; In step S2, based on the geometric feature constraints of the inner surface of the container, the extrinsic parameter matrices of the point clouds on the left and right sides are iteratively optimized to obtain the complete 3D point cloud: Geometric features that characterize the rigid structure inside the container are extracted from the left and right point clouds respectively. Based on the extracted geometric features, at least two different types of spatial correspondence constraints with complementary physical meanings are established between the left and right point clouds. At least two of the aforementioned spatial correspondence constraints are fused to construct a multi-constraint coupled nonlinear optimization model. The extrinsic parameter matrix is ​​then optimized and updated by iteratively solving the nonlinear optimization model to obtain a high-precision point cloud registration result that meets multiple geometric consistency requirements, thereby obtaining the complete 3D point cloud. The establishment of at least two different types of spatial correspondence constraints with complementary physical meanings includes establishing rigid constraints based on non-coplanar feature point pairs, including: In the left and right point clouds, identify and extract the inherent, non-temporary physical structural feature regions of the container, including but not limited to: the recessed area formed by the door hinge mounting base, the raised strip area formed by the inner wall reinforcing ribs, and the local area where the floor fasteners are located. For each identified physical structure feature region, calculate its three-dimensional feature descriptor in its respective point cloud coordinate system and a local reference point representing the spatial location of the region; By matching the three-dimensional feature descriptors in the point clouds on both sides, two local reference points that originate from the same physical structure and are located in the left and right point clouds respectively are established as a pair of virtual corresponding points. A set of feature points that are not coplanarly distributed in three-dimensional space is formed by multiple pairs of virtual corresponding points, and constraints are established based on this set of feature points to minimize the sum of spatial distances between all corresponding point pairs after the transformation of the extrinsic parameter matrix.

2. The method for detecting empty containers based on multimodal fusion according to claim 1, characterized in that: The two sets of sensor units are symmetrically deployed on both sides of the checkpoint channel, with their installation lines perpendicular to the lane direction and the installation spacing greater than the maximum width of a standard container truck.

3. The method for detecting empty containers based on multimodal fusion according to claim 1, characterized in that: The establishment of at least two different types of spatial correspondence constraints with complementary physical meanings also includes establishing soft constraints based on the topological consistency of the internal structure, including: In the left and right point clouds, a set of local reference points are selected as nodes, and a spatial topology connection diagram is formed between the nodes according to the actual physical connection relationship or spatial proximity relationship inside the container. In the spatial topology connection diagram, line segment pairs describing the same physical connection component are identified, and constraints are established for each identified line segment pair. The constraints require that after the transformation of the extrinsic parameter matrix, the length of the line segment pair should tend to be consistent, and their three-dimensional spatial directions should tend to be parallel. The constraints established based on all identified line segments are summarized to form the soft constraints for the topological consistency of the internal structure.

4. The method for detecting empty containers based on multimodal fusion according to claim 1, characterized in that: In step S4, the detection of the remnant is based on its three-dimensional geometric dimensions, color, and texture information; the determination of the sandwich structure is achieved by comparing the actual internal dimensions of the box with the dynamically updated standard dimensions of the box and their allowable deviation range.

5. The method for detecting empty containers based on multimodal fusion according to claim 1, characterized in that: Before step S1, an offline calibration step is also included, which is used to obtain the internal parameters and external pose relationship of each group of sensor units and to establish the spatial transformation relationship between the left and right sensor groups.

6. A container empty container inspection system based on multimodal fusion, characterized in that, include: The data acquisition module is configured to: trigger two sets of sensor units to synchronously acquire three-dimensional point cloud data and RGB images inside the container when the container door is opened; each set of sensor units includes a three-dimensional lidar and a high-definition optical camera; The point cloud fusion module is configured to: fuse the three-dimensional point cloud data collected from both sides into the same coordinate system based on pre-calibrated sensor parameters to form a complete three-dimensional point cloud inside the container; The texture mapping module is configured to: perform texture mapping between a complete 3D point cloud and an RGB image to generate a fused point cloud model with color information; The detection and analysis module is configured to: detect whether there are any remaining objects inside the container by removing the container structure point cloud and performing cluster analysis on the remaining internal point cloud based on the fused point cloud model; and calculate the actual internal dimensions by fitting the inner surface of the container based on the fused point cloud model and comparing them with standard dimensions to determine whether there is a sandwich structure. The results output module is configured to output a test report containing information on the remains and the results of the interlayer detection. Based on the geometric feature constraints of the inner surface of the container, the extrinsic parameter matrices of the point clouds on the left and right sides are iteratively optimized to obtain the complete 3D point cloud, including: Geometric features that characterize the rigid structure inside the container are extracted from the left and right point clouds respectively. Based on the extracted geometric features, at least two different types of spatial correspondence constraints with complementary physical meanings are established between the left and right point clouds. At least two of the aforementioned spatial correspondence constraints are fused to construct a multi-constraint coupled nonlinear optimization model. The extrinsic parameter matrix is ​​then optimized and updated by iteratively solving the nonlinear optimization model to obtain a high-precision point cloud registration result that meets multiple geometric consistency requirements, thereby obtaining the complete 3D point cloud. The establishment of at least two different types of spatial correspondence constraints with complementary physical meanings includes establishing rigid constraints based on non-coplanar feature point pairs, including: In the left and right point clouds, identify and extract the inherent, non-temporary physical structural feature regions of the container, including but not limited to: the recessed area formed by the door hinge mounting base, the raised strip area formed by the inner wall reinforcing ribs, and the local area where the floor fasteners are located. For each identified physical structure feature region, calculate its three-dimensional feature descriptor in its respective point cloud coordinate system and a local reference point representing the spatial location of the region; By matching the three-dimensional feature descriptors in the point clouds on both sides, two local reference points that originate from the same physical structure and are located in the left and right point clouds respectively are established as a pair of virtual corresponding points. A set of feature points that are not coplanarly distributed in three-dimensional space is formed by multiple pairs of virtual corresponding points, and constraints are established based on this set of feature points to minimize the sum of spatial distances between all corresponding point pairs after the transformation of the extrinsic parameter matrix.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-5.

Images