Method and system for detecting a box interlayer

Abstract

The present application relates to the technical field of image detection, and provides a kind of detection method and system of box interlayer, comprising: S1: acquisition box external data and acquisition box internal data;S2: according to box internal data, calculate six plane equations in box interior;S3: according to box external data, calculate six plane equations in box exterior;S4: according to six plane equations in box exterior and six plane equations in box interior, obtain the detection result of box interlayer.This scheme is based on the box interlayer detection method of multimodal data fusion, based on multimodal fusion technology, quickly segment target box point cloud;Utilize the box structure feature accurately to extract the inside and outside plane of box, realize the rapid detection of box interlayer by comparing the inside and outside plane equation of box.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method and system for detecting the interlayer of a box. Background Technology

[0002] Truck compartments are often used for concealment, making truck compartment inspection crucial. In recent years, truck compartment inspection has evolved from simply checking goods to checking space, risks, and compliance, becoming the first digital line of defense for smart customs and secure borders.

[0003] In existing technologies, large-scale X-ray scanning systems cannot detect interlayers shielded with heavy metals such as lead plates, and rely on manual image interpretation for judgment; X-ray computed tomography (CT) systems have slow scanning speeds and high costs, making them difficult to scale up.

[0004] Therefore, there is a need to provide a method and system for detecting interlayers in box enclosures. The method is based on multimodal data fusion, which rapidly segments the target box point cloud using multimodal fusion technology. It accurately extracts the inner and outer planes of the box using the structural features of the box, and achieves rapid detection of interlayers by comparing the equations of the inner and outer planes of the box.

[0005] The information disclosed in the background section is only intended to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the problems of slow detection speed and limited detection of interlayer materials in existing technologies, and to provide a method and system for detecting interlayers in boxes. The method is based on multimodal data fusion, which rapidly segments the point cloud of the target box using multimodal fusion technology; accurately extracts the inner and outer planes of the box using the structural features of the box; and achieves rapid detection of interlayers in boxes by comparing the equations of the inner and outer planes of the box.

[0007] To achieve the above objectives, the first aspect of the present invention provides a method for detecting the interlayer of a box, comprising the following steps:

[0008] S1: Collect external data and internal data of the enclosure;

[0009] S2: Calculate the equations of the six planes inside the box based on the internal data of the box;

[0010] S3: Calculate the equations of the six planes outside the box based on the external data of the box;

[0011] S4: The detection results of the interlayer of the box are obtained based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0012] According to an exemplary embodiment of the present invention, in step S1, the collection of external data of the enclosure includes:

[0013] Two contour radars are placed on both sides of the housing, and the pose calibration matrix of the two contour radars is calculated.

[0014] The pose calibration matrix of the two contour radars is used to stitch together the data of the two contour radars and output the external data of the housing.

[0015] According to an exemplary embodiment of the present invention, in step S1, the collection of data inside the enclosure includes:

[0016] Two laser scanning radars are placed on both sides of the housing. The first laser scanning radar is the primary one and the second one is the secondary one. Their pose relationship is calibrated to obtain the pose calibration matrix from the second laser scanning radar to the first laser scanning radar.

[0017] The first laser scanning radar and 2D camera are calibrated, and the pose calibration matrix of the laser scanning radar and 2D camera is calculated.

[0018] The data from the laser scanning radars are stitched together using the pose calibration matrices of the two laser scanning radars to obtain the data inside the enclosure.

[0019] According to an exemplary embodiment of the present invention, in step S2, calculating the equations of the six planes inside the box based on the internal data of the box includes:

[0020] S21: Perform multimodal data segmentation on the box;

[0021] S22: Using the sequential RANSAC model extraction algorithm, multiple plane equations and interior point sets are obtained;

[0022] S23: Perform a coarse fit on the plane to obtain six coarse planes;

[0023] S24: Accurately extract the planes to obtain the equations of the six planes inside the box.

[0024] According to an exemplary embodiment of the present invention, in step S22, obtaining multiple plane equations and interior point sets using the sequential RANSAC model extraction algorithm includes:

[0025] S221: Initialize the model parameter set and the point set within the model, set the initial point cloud to the remaining point cloud and enter the loop;

[0026] S222: Select the minimum point set from the remaining point cloud to extract the model, fit a candidate model with the minimum point set, and obtain the corresponding interior point set of the candidate model;

[0027] S223: Check whether the number of interior points in the candidate model meets the preset minimum value, and whether the sum of interior point errors is within the preset error threshold. If not, the loop terminates.

[0028] S224: If the candidate model is valid, store the model parameters and interior points into the model parameter set and model interior point set respectively, and remove the candidate model interior points from the current remaining point cloud;

[0029] S225: Repeat steps S222 to S225. If the current number of remaining point clouds is lower than the preset value, the loop terminates.

[0030] According to an exemplary embodiment of the present invention, in step S23, the coarse fitting of the plane to obtain six coarse planes includes:

[0031] S231: Define six coarse plane normal vectors;

[0032] S232: Traverse the set of plane equations obtained by the RANSAC plane extraction algorithm, calculate the angle between the plane normal vector and the normal vectors of the six coarse planes, and if the angle is less than the preset threshold, put the interior point corresponding to the plane equation into the interior point set corresponding to the six coarse planes.

[0033] S233: Refit the planes based on the point sets in the six coarse planes to obtain new plane equations, thus obtaining six coarse planes.

[0034] According to an exemplary embodiment of the present invention, in step S24, the precise extraction of the planes to obtain the equations of the six planes inside the box includes:

[0035] Based on the six coarse planes, combine them into groups of three, and calculate the intersection points using the intersection of the three planes, which are the corner points of the box.

[0036] Based on the location information of the corner points of the box, the four corner points of each face are refitted to a plane to obtain the equations of the six planes inside the box.

[0037] According to an exemplary embodiment of the present invention, in step S3, calculating the equations of the six planes outside the box based on the external data of the box includes:

[0038] Segment and extract the point cloud outside the box;

[0039] Construct an OBB volumetric coordinate system and rotate the point cloud outside the box to the OBB volumetric coordinate system;

[0040] Obtain points on the three outer faces of the box;

[0041] Calculate the curvature of the point cloud, and extract the edge point cloud of the three faces based on the change in point cloud curvature;

[0042] Merge the edge point clouds on each surface of the outer box and recalculate the new plane equation.

[0043] According to an exemplary embodiment of the present invention, in step S4, obtaining the detection result of the interlayer of the box based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box includes:

[0044] The four corner points of the internal plane of the box are calculated based on the equations of the six planes inside the box. If the sum of the distances from the four corner points of the internal plane of the box to the corresponding external plane of the vehicle body exceeds a predetermined distance threshold, it is considered that there is an abnormal interlayer in the plane where the corner point is located; otherwise, it is considered that there is no abnormal interlayer.

[0045] As a second aspect of the present invention, the present invention provides a detection system for a box interlayer, comprising: a data acquisition module, an external plane calculation module, an internal plane calculation module, and a comparison detection module;

[0046] The data acquisition module is used to collect data from the outside of the enclosure and data from the inside of the enclosure;

[0047] The external plane calculation module is used to calculate the equations of six external planes based on the external data of the box.

[0048] The internal plane calculation module is used to calculate the equations of the six planes inside the box based on the internal data of the box.

[0049] The comparison detection module is used to obtain the detection results of the interlayer of the box based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0050] The advantages of this invention are:

[0051] This solution is based on a multimodal data fusion-based method for detecting interlayers in vehicle bodies. It rapidly segments the target vehicle body point cloud using multimodal fusion technology; it accurately extracts the inner and outer planes of the vehicle body using the structural features of the body; and it achieves rapid detection of interlayers by comparing the equations of the inner and outer planes of the body. It has low environmental requirements, is harmless to humans, and is highly flexible. It can also display the interlayer situation of the body more intuitively on images and point clouds. Attached Figure Description

[0052] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0053] Figure 1 A schematic diagram of the detection system for the interlayer of the enclosure is shown.

[0054] Figure 2The schematic diagram illustrates the data acquisition module's design scheme.

[0055] Figure 3 The diagram illustrates the steps of the method for detecting the interlayer of the enclosure.

[0056] Figure 4 The diagram illustrates the technical roadmap for extrinsic parameter calibration of laser scanning radar and 2D cameras.

[0057] Figure 5 The diagram illustrates the data acquired by the laser scanning radar. Figure 5 (a) is the RGB diagram of the cabinet. Figure 5 (b) is the point cloud of the box.

[0058] Figure 6 The diagram illustrates the multimodal data segmentation of the target container.

[0059] Figure 7 A schematic diagram of reverse projection is shown. Figure 7 (a) is a diagram illustrating the process of back-projecting pixels from a two-dimensional image into three-dimensional space. Figure 7 (b) is the original two-dimensional image data captured by the camera.

[0060] Figure 8 The diagram schematically shows the four rays corresponding to the coordinates of eight points.

[0061] Figure 9 The diagram schematically illustrates the original state of the enclosure in the laser scanning radar coordinate system.

[0062] Figure 10 The OBB box is illustrated schematically.

[0063] Figure 11 The diagram illustrates the effect of transforming the box-shaped point cloud into the OBB volume coordinate system.

[0064] Figure 12 A schematic diagram of the six sides of the box is shown.

[0065] Figure 13 A schematic diagram of the edge is shown.

[0066] Figure 14 The diagram illustrates the test results when the truck's cargo box is empty.

[0067] Figure 15 The diagram illustrates the test results when the truck has a double-layered structure.

[0068] Figure 16 The diagram illustrates the detection results when a foreign object is found in a truck. Detailed Implementation

[0069] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0070] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0071] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0072] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0073] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0074] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0075] According to a first specific embodiment of the present invention, the present invention provides a detection system for the interlayer of a box, such as... Figure 1 As shown, it includes: a data acquisition module, an external plane calculation module, an internal plane calculation module, and a comparison and detection module.

[0076] The data acquisition module is used to collect data from both the outside and inside of the enclosure.

[0077] like Figure 2 As shown, the data acquisition module includes two contour radars 1, a laser velocity radar 2, two laser scanning radars 3, and a 2D camera 4.

[0078] Two gantry cranes, A and B, are set up. The truck carrying the container passes through gantry cranes A and B sequentially. Two contour radars 1 and a laser speed radar 2 are used together to collect external data of the container. The two contour radars 1 are placed on gantry crane A, located on the upper sides of the container, higher than the container itself. These two contour radars 1 are single-line laser radars, scanning downwards. When the truck carrying the container passes through gantry crane A, they can collect the external contour point cloud data of the container. The laser speed radar 2 is placed on gantry crane B, in the middle, to measure the moving speed and position of the container. It is used to stitch together the data collected by the two contour radars 1 to generate a complete scan point cloud. Two laser scanning radars 3 and a 2D camera 4 are used together to collect internal data of the container. The two laser scanning radars 3 are placed in the middle on both sides of gantry crane A, at a height approximately the same as the middle of the container, to collect internal data of the container. The 2D camera 4 is installed in the middle position above gantry crane A, working in conjunction with the laser scanning radars 3 to collect container data.

[0079] The external plane calculation module is used to calculate the equations of the six external planes of the enclosure based on the external data of the enclosure.

[0080] The internal plane calculation module is used to calculate the equations of the six planes inside the box based on the internal data of the box.

[0081] The comparison detection module is used to obtain the detection results of the interlayer of the box based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0082] According to a second embodiment of the present invention, the present invention provides a method for detecting the interlayer of a box, employing the detection system for the interlayer of a box according to the first embodiment, such as... Figure 3 As shown, the detection method includes the following steps:

[0083] S1: Collect external data and internal data of the enclosure.

[0084] For the exterior of the enclosure, this solution employs two contour radars to provide high-precision, high-density contour point cloud data, while a dedicated laser velocimeter radar 2 accurately measures the enclosure's movement speed. By fusing the data from these three sources, high-resolution surface point cloud data is generated, achieving a precise digital representation of the enclosure's external geometry. The two contour radars 1 are located on the same horizontal plane, higher than the enclosure. The laser surfaces of the two contour radars 1 are vertically set, perpendicular to the enclosure's axis. The two contour radars have overlapping fields of view in the area traversed by the enclosure, greatly expanding the effective sensing range and reducing the blind spots of individual radars. The laser velocimeter radar 2 is located on the same horizontal plane as the contour radars 1, in front of the enclosure's direction of movement, emitting radar signals towards the enclosure from behind. The laser surface of the laser velocimeter radar 2 is vertically set, with the enclosure's axis either on this laser surface or parallel to it, ensuring that the laser velocimeter radar 2 can also scan the enclosure while the contour radars are acquiring data.

[0085] Contour radar 1 accurately calculates the distance to each object by measuring the characteristics of the reflected echo signal after the emitted light pulse hits the nearby object, and outputs the three-dimensional contour information of the object (i.e., the XYZ coordinates of a series of points). Laser velocity radar 2 utilizes the Doppler effect to determine the speed of the target by comparing the frequencies of the emitted and received signals.

[0086] The single-frame data of contour radar 1 is not generated instantaneously. Instead, the laser points are emitted at different times, requiring a certain amount of time. During this time, the target may be moving (e.g., a truck driving), resulting in different relative positions between the laser points and the target object. However, when a typical lidar driver encapsulates a frame of data, it defaults to acquiring all laser points in that frame at the same time and in the same pose. This leads to motion errors in the laser frame, potentially causing blurred, streaked, or distorted contours. Therefore, this solution uses lidar velocity radar 2 to acquire the target object's velocity and calculate its position at different times. The lidar coordinates corresponding to each laser point in each frame of contour radar 1 data are then transformed to their positions at different times to minimize the lidar's motion errors.

[0087] The data collected from the outside of the enclosure includes:

[0088] S111: Place the two contour radars 1 on both sides of the housing and calculate the pose calibration matrix of the two contour radars 1.

[0089] Since two contour radars 1 and one laser velocity radar 2 are used in combination, the data calibration of the acquisition system is equivalent to the pose calibration of the two contour radars 1. The pose calibration of the two contour radars 1 is performed to calculate the transformation matrix between them: a calibration object with obvious features is placed statically within the overlapping area of ​​the field of view of the two contour radars 1, and both contour radars 1 are simultaneously triggered to scan the calibration object; the calibration object is moved multiple times and scanned multiple times to acquire calibration object data pairs under different poses; feature 3D coordinates are extracted from each calibration object data; using the extracted feature point pairs, the transformation matrix is ​​calculated, which is the pose calibration matrix of the two contour radars 1.

[0090] S112: Based on the pose calibration matrix of the two contour radars 1, the data of the two contour radars 1 are stitched together to output the external data of the box, which is the "external point cloud of the box".

[0091] For the interior of the enclosure, this solution uses two laser scanning radars 3 and a 2D camera 4 to collect data about the interior of the enclosure, such as... Figure 6 As shown in (a), Figure 6 (a) is a point cloud image collected on-site by a single laser scanning radar 3. The device is located approximately 5 meters behind the rear of the enclosure, at a height within the middle of the enclosure. The rear of the enclosure is open, and the laser scanning radar 3 is aimed at the interior of the enclosure for scanning. The beam emission is similar to that of a flashlight. A single device needs to be directly facing the scanner, resulting in left-right deviations and potential incomplete scanning of the left and right sides of the enclosure. Therefore, using two laser scanning radars 3 in this solution can reduce blind spots. The RGB image generated by the 2D camera 4 has high-resolution texture, color, and semantic information, with significant edge features. The point cloud data output by the laser radar has accurate, illumination-independent 3D geometric information and distance measurement capabilities, with significant point features. By combining the "3D spatial geometry of the radar" with the "2D semantics of the camera," the laser scanning radar 3 and the 2D camera 4 achieve a perception effect of 1+1>2, which is a core technology necessary for many applications, including autonomous vehicle positioning, environmental mapping, and dynamic object recognition.

[0092] The data collected from inside the enclosure includes:

[0093] S121: Place two laser scanning radars on both sides of the housing. The first laser scanning radar 3 is the primary one and the second one is the secondary one. Calibrate their pose relationship to obtain the pose calibration matrix from the second laser scanning radar 3 to the first laser scanning radar 3.

[0094] 2A. Place a calibration object with obvious features in the overlapping area of ​​the field of view of the two laser scanning radars 3, and simultaneously trigger the two laser scanning radars 3 to collect calibration object data (including point cloud data and RGB image data).

[0095] 2B. Following step 1A, move the calibration object multiple times and collect calibration object data pairs under different poses;

[0096] 2C. Extract feature 3D coordinates from the point cloud data of each calibration object; use the extracted 3D feature point pairs to calculate the transformation matrix, which is the pose calibration matrix of the two laser scanning radars 3.

[0097] S122: Calibrate the first laser scanning radar 3 and 2D camera 4, and calculate the pose calibration matrix of the laser scanning radar 3 and 2D camera 4.

[0098] The extrinsic parameter calibration of the laser scanning radar 3 and the 2D camera 4 is based on the premise that both (2D camera 4 and laser scanning radar 3) are rigidly connected. On-site, the 6DoF rigid transformation relationship (R,t) between the coordinate systems of the two sensors is quickly and accurately determined. 6DoF: six degrees of freedom (3 rotation parameters, 3 translation parameters); (R,t): R is the rotation matrix (which can be converted to and from the 3 rotation parameters), and t is the translation vector (including the 3 translation parameters). The technical approach is as follows: Figure 4 As shown.

[0099] 1A. Place a calibration object with obvious features in the overlapping area of ​​the field of view of the laser scanning radar 3 and the 2D camera 4, and simultaneously trigger the laser scanning radar 3 and the 2D camera 4 to collect data of the calibration object;

[0100] 1B. Following step 1A, move the calibration object multiple times and collect calibration object data pairs under different poses;

[0101] 1C. Extract the 3D coordinates of the features of the point cloud generated by the laser scanning radar 3 in the laser scanning radar coordinate system; calculate the 2D coordinates of the RGB image features generated by the 2D camera 4, and convert them into 3D coordinates in the 2D camera coordinate system according to the intrinsic parameters of the 2D camera 4.

[0102] 1D. Using the extracted feature point pairs, calculate the transformation matrix, which is the pose calibration matrix of the laser scanning radar 3 and the 2D camera 4.

[0103] S123: Based on the pose calibration matrix of the two laser scanning radars 3, the laser scanning radar data is stitched together to obtain the data inside the box.

[0104] like Figure 5 As shown, the laser scanning radar data is stitched together and merged according to the pose calibration matrices of the two laser scanning radars 3, and the resulting output is called the "box point cloud"; the 2D camera 4 outputs the "box RGB image". Among these, Figure 5 (a) is the RGB diagram of the cabinet. Figure 5 (b) is the point cloud of the box.

[0105] This solution employs multimodal fusion technology, specifically multimodal fusion of LiDAR and camera data. Point cloud data exhibits a discrete and sparse spatial distribution, possessing precise geometric measurement capabilities and significant point features; while image data exhibits a continuous and dense pixel distribution, possessing rich texture response capabilities and significant edge features. By adopting a multimodal data fusion method, the limitations of relying on a single sensor are eliminated. By comprehensively utilizing complementary information from different sensors to perceive the current environment, the overall system's perception accuracy and environmental understanding capabilities can be significantly improved.

[0106] Multimodal fusion of LiDAR and cameras effectively combines the three-dimensional spatial geometric information (point cloud, echo intensity, etc.) provided by LiDAR with the two-dimensional semantic information (such as RGB images) provided by the camera through techniques such as spatial alignment, feature transformation, or attention mechanisms in a unified three-dimensional space (such as bird's-eye view / BEV space) or feature space. This leverages the complementary advantages of the two modalities—the precise spatial location information of LiDAR and the rich semantic information of the camera—to enhance the performance of tasks such as 3D target detection, tracking, or segmentation.

[0107] LiDAR camera data fusion is a core technology essential for many applications, including autonomous vehicle localization, environmental mapping, and dynamic object recognition. Key technologies for achieving cross-modal fusion of LiDAR camera data include LiDAR camera extrinsic parameter calibration and point cloud image feature transformation.

[0108] Point clouds are a direct product of LiDAR, while the corresponding images are a direct product of cameras. Feature transformation between the two is a prerequisite for cross-modal fusion and a fundamental technology for subsequent target detection and tracking, and environmental mapping applications. By jointly calibrating the extrinsic parameters of a 2D camera and LiDAR, orthographic and inverse projections can be achieved. Orthographic projection projects the 3D point cloud onto a 2D image plane, assigning color information to each point; inverse projection is a reverse mapping from 2D pixels to the 3D world. A pixel in the image represents a direction, a potential 3D location range, which signifies an infinitely extending line originating from the camera's optical center, passing through that pixel, and extending into the real 3D world. Based on the calibration matrix, a ray in the point cloud coordinate system can be determined.

[0109] S2: Calculate the equations of the six planes inside the box based on the internal data of the box.

[0110] The moment the container stops at the double-rod gantry, the laser scanning radar 3 and the 2D camera 4 simultaneously acquire point cloud data with high-precision three-dimensional geometric information and RGB images with surface texture information inside the container, and perform data processing based on this data.

[0111] Based on the internal data of the enclosure, the equations for the six planes inside the enclosure are calculated as follows:

[0112] S21: Perform multimodal data segmentation on the box.

[0113] Extract the outer rectangular outline (4 sides, 4 corners) of the target box from the image, that is, the side of the box tail parallel to the laser surface of contour radar 1 (i.e., Figure 8 In the lower small image, a rectangular frame (the plane containing the rear side of the box) is used. Four rays from its four corner points can be obtained through back projection. Based on depth information, eight point coordinates can be extracted from these rays. The point cloud is then cropped based on these eight coordinates to obtain the box's point cloud. Its OBB (minimum bounding box) is calculated, and a box OBB volume coordinate system is constructed. The box's point cloud is then corrected to a frontal viewpoint, resulting in the following... Figure 6 The image shows a multimodal data segmentation diagram of the box-shaped target, in which... Figure 6 (a) Raw data collected by a single laser scanning radar; Figure 6 (b) To determine the position of the box through the image, the position of the box in the laser scanning radar point cloud is calculated based on the calibration results of 2D camera 4 and laser scanning radar 3, and the box point cloud is obtained after cropping.

[0114] S211: Extraction of the outer contour of the box target in RGB image.

[0115] In the image, the target box is darker than other areas. First, the RGB image is filtered and denoised. Second, adaptive histogram equalization is used to highlight the target box's location, and gradients are calculated to emphasize its edge information. Then, the image is binarized, and edge extraction is performed to determine the target box's edge contour. Finally, the coordinates of the four corner points of the minimum bounding matrix of the target box's edge contour are calculated.

[0116] S212: Reverse projection.

[0117] like Figure 7 As shown, Figure 7 (a) describes the process of backprojecting pixels from a two-dimensional image into three-dimensional space; Figure 7 (b) is the original 2D image data captured by the camera. Figure 7 (b) is marked with a cross. Figure 7 (a) Ray Corresponding Position. The process of backprojecting pixels from a two-dimensional image into three-dimensional space, i.e., recovering the corresponding three-dimensional point coordinates from pixel coordinates. Its function is to reconstruct a three-dimensional point cloud from an RGB-D image or a binocular parallax map, realizing the correspondence between a two-dimensional image and three-dimensional space. The backprojection formula is:

[0118] ;

[0119] Here, λ is a scale factor representing the distance from the camera's optical center along the ray direction. To obtain a specific 3D point on the ray, the value of λ must be provided.

[0120] This scheme, based on the pose calibration parameters of the 2D camera 4 and the laser scanning radar 3 in step S122, transforms the coordinates of the four corner points of the minimum bounding matrix of the target box edge contour on the RGB image into point cloud data through inverse projection. Mapping the two-dimensional points to the three-dimensional coordinate system yields a ray. Therefore, by setting two different depth parameters, the corresponding eight coordinate points in the point cloud data can be obtained. The depth information can be referenced from the depth-of-field setting of the laser scanning radar.

[0121] S213: Construction of the OBB volumetric coordinate system for the box point cloud and calculation of OBB dimensions.

[0122] like Figure 8 As shown, by inverse projection from the image to the point cloud, four rays are obtained, depth information is set, and the coordinates of eight points in the point cloud are obtained; as shown Figure 9 As shown, the original state of the box in the laser scanning radar coordinate system has a tilt angle. The box point cloud is clipped based on the coordinates of these 8 points, and the OBB (minimum bounding box) is obtained based on the box point cloud. Figure 10 The rectangular line surrounding the point cloud is the calculated OBB (minimum bounding box); transforming it to the OBB coordinate system, as shown... Figure 11 As shown, the box point cloud transformed to the OBB volume coordinate system has no tilt angle, enabling cuboid correction. The eight points resemble the eight vertices of a cuboid. Selecting all points within the cuboid from the point cloud yields the box point cloud; this process is called clipping.

[0123] The box-shaped point cloud is clipped based on the 8 coordinate points calculated from the reverse projection; the box-shaped point cloud OBB will be calculated below:

[0124] (1) Calculate the centroid of the point cloud:

[0125] ;

[0126] Among them, P i is the coordinates of the point, n is the total number of points, and µ is the centroid of the point cloud.

[0127] (2) Calculate the covariance matrix: The covariance matrix describes the distribution of points in three dimensions and the correlation between them;

[0128] ;

[0129] in, ,

[0130] ,

[0131] Where, µ x µ y µ zThe centroids are located in the three directions of the point cloud: X, Y, Z, W, and Q, where X, Y, Z, W, and Q represent different dimensions. i ,q i Let W and Q represent the specific values ​​respectively, and let x represent the specific values ​​corresponding to X, Y, and Z respectively. i ,y i ,z i , that is, the value of a point in the point cloud in three directions. All parameters in C can be obtained by using cov(W, Q).

[0132] Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and solve the characteristic equation:

[0133] ;

[0134] Where λ is the eigenvalue, λ={λ1, λ2, λ3}, and its magnitude represents the variance of the data in that direction; ν is the eigenvector, ν={ν1, ν2, ν3}, representing the principal direction.

[0135] (3) Construct the OBB volume coordinate system: Construct the OBB volume coordinate system with the centroid of the point cloud as the origin and the three feature vectors as the basis vectors of the OBB volume coordinate system; and transform all points in the point cloud from the original coordinate system to the OBB volume coordinate system.

[0136] ;

[0137] Among them, P i P is a point in the original coordinate system. ’ i The vector represents the point in the transformed OBB volume coordinate system, T represents the transpose of the vector, and is the operation sign; ν={ν1, ν2, ν3}, and µ is the centroid of the point cloud.

[0138] Determine OBB dimensions: Find the maximum values ​​(x, y, z) of the point cloud in the OBB volume coordinate system. max ,y max ,z max ) and minimum value (x) min ,y min ,z min If ), then the dimensions of the box point cloud on the three axes are ((x max - x min ) / 2, (y max - y min ) / 2, (z max -z min ) / 2).

[0139] S22: Use the sequential RANSAC model extraction algorithm to obtain multiple plane equations and interior point sets.

[0140] The box has a cuboid structure with six faces, eight corners, and twelve edges. Each face is associated with four corners. Therefore, this scheme achieves accurate plane extraction of the box based on its cuboid features. The sequential RANSAC model extraction algorithm is used to obtain multiple plane equations and interior point sets, including:

[0141] S221: Initialize the model parameter set S and the model internal point set I, set the initial point cloud P to the remaining point cloud PR and enter the loop, and set the model to be extracted as a plane.

[0142] S222: Randomly select the minimum point set of the model to be extracted from the remaining point cloud PR, fit a candidate model with the minimum point set, and obtain the corresponding interior point set of the candidate model.

[0143] S223: Check whether the number of interior points in the candidate model meets the preset minimum value, and whether the interior point error is within the preset error threshold. If not, the loop terminates.

[0144] S224: If the candidate model is valid, store the model parameters and interior points into the model parameter set S and the model interior point set I respectively, and remove the candidate model interior points from the current remaining point cloud.

[0145] S225: Repeat steps S222 to S225. If the current number of remaining point clouds is lower than a preset value, the loop terminates.

[0146] The sequential RANSAC model extraction algorithm is used to obtain multiple plane equations and interior point sets.

[0147] S23: Perform a coarse fit on the plane to obtain six coarse planes.

[0148] S221: Define six coarse plane normal vectors.

[0149] By converting the box point cloud to the OBB volume coordinate system, we can obtain the normal vectors of the six coarse planes of the box: the normal vector of the front side (-1,0,0), the normal vector of the rear side (1,0,0), the normal vector of the left side (0,-1,0), the normal vector of the right side (0,1,0), the normal vector of the top side (0,0,-1), and the normal vector of the bottom side (0,0,1). We can then create the corresponding point set in the empty plane.

[0150] S222: Traverse the set of plane equations obtained by the RANSAC plane extraction algorithm, calculate the angle between the plane normal vector and the normal vectors of the six coarse planes. If the angle is less than a preset threshold, put the interior point corresponding to the plane equation into the interior point set corresponding to the six coarse planes.

[0151] S223: Refit the planes based on the point sets in the six coarse planes to obtain new plane equations, thus obtaining six coarse planes.

[0152] S24: Accurately extract the planes to obtain the equations of the six planes inside the box.

[0153] S241: Based on the six coarse planes, combine them into three groups of three, and calculate the intersection point using the intersection of the three planes, which is the corner point of the box.

[0154] S242: Based on the location information of the corner points of the box, refit the planes of the four corner points of each face to obtain the equations of the six planes inside the box.

[0155] The equations of the six planes inside the box are denoted as follows: .

[0156] S3: Calculate the equations of the six planes outside the box based on the external data of the box.

[0157] The outer surface of the box is a cuboid structure; therefore, the outer surface of the box can be defined by six plane equations.

[0158] S31: Segment and extract the point cloud outside the box.

[0159] Euclidean clustering algorithm was used to segment and extract the point cloud outside the box.

[0160] S32: Construct an OBB volumetric coordinate system and rotate the point cloud outside the box to the OBB volumetric coordinate system.

[0161] Refer to step S213 (3) for constructing the OBB volume coordinate system.

[0162] S33: Obtain points on the three outer faces of the box.

[0163] The three planes are the top plane, the left side plane, and the right side plane, as follows: Figure 12 The diagram showing the six sides of the enclosure is because the contour radar 1 is installed on the left and right corners above the gantry A, and from the scanning perspective, only these three sides can be captured.

[0164] S34: Calculate the curvature of the point cloud and extract the edge point cloud of the three faces based on the change in the curvature of the point cloud.

[0165] like Figure 13 As shown, each of the six faces of the box has two or more edge point clouds.

[0166] S35: Merge the edge point clouds on each surface of the outer side of the box and recalculate the new plane equation.

[0167] like Figure 13 As shown, the edges of the cuboid are the edges extracted from the three faces. A new plane is fitted based on the edge point cloud on each face to obtain a new plane equation.

[0168] Each of the six faces has two or more edges, and two non-overlapping edges can form a plane.

[0169] The equations of the six planes outside the box are obtained, denoted as follows: .

[0170] Steps S2 and S3 can be performed simultaneously.

[0171] S4: The detection results of the interlayer of the box are obtained based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0172] The four corner points of the internal plane of the box are calculated based on the equations of the six planes inside the box. If the sum of the distances from the four corner points of the internal plane of the box to the corresponding external plane of the vehicle body exceeds a predetermined distance threshold, it is considered that there is an abnormal interlayer in the plane where the corner point is located; otherwise, it is considered that there is no abnormal interlayer.

[0173] Figure 14 The image shows the test results when the truck's cargo box is empty. Figure 15 The image shows the detection results when the truck has a sandwich structure, with the sandwich point cloud in the middle. Figure 16 The image shows the detection results when there is a foreign object in the truck, with point clouds of foreign objects appearing inside the truck body.

[0174] Therefore, this solution is a box-type interlayer detection method based on multimodal data fusion. It quickly segments the target box point cloud based on multimodal fusion technology; accurately extracts the inner and outer planes of the vehicle body using the structural features of the box; and achieves rapid detection of box interlayers by comparing the equations of the inner and outer planes of the box. It has low environmental requirements, is harmless to the human body, and is highly flexible. It can more intuitively display the box interlayer situation on images and point clouds.

[0175] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method of detecting a core sandwich of a box, characterized by, The method comprises the following steps: S1: collecting box external data and collecting box internal data; S2: calculating six plane equations inside the box according to the box internal data; S3: calculating six plane equations outside the box according to the box external data; S4: obtaining the detection result of the box interlayer according to the six plane equations outside the box and the six plane equations inside the box; In step S2, the calculation of the six plane equations inside the box according to the box internal data comprises: S21: multi-modal data segmentation of the box; S22: obtaining multiple plane equations and an inlier set by using a sequential RANSAC model extraction algorithm; S23: rough fitting of the planes to obtain six rough planes; S24: accurate extraction of the planes to obtain the six plane equations inside the box; In step S24, the accurate extraction of the planes to obtain the six plane equations inside the box comprises: According to the six rough planes, three of them are combined, and the intersection point is calculated by using three-face intersection, which is the corner point of the box; According to the position information of the box corner point distribution, the four corner points of each face are refitted to obtain the six plane equations inside the box; In step S3, the calculation of the six plane equations outside the box according to the box external data comprises: Segmentation and extraction of the box external point cloud; Construction of an OBB body coordinate system, and rotation of the box external point cloud to the OBB body coordinate system; Obtaining points inside three faces of the box; Calculating the curvature of the point cloud, and extracting the edge point cloud of the three faces according to the curvature change of the point cloud; Combining the edge point cloud on each face of the box outside to recalculate the new plane equation; In step S4, the calculation of the detection result of the box interlayer according to the six plane equations outside the box and the six plane equations inside the box comprises: According to the six plane equations inside the box, the four corner points of the internal plane of the box are calculated, and if the sum of the distances from the four corner points of the internal plane of the box to the corresponding external plane of the vehicle body exceeds a predetermined distance threshold, it is considered that the position plane where the corner point is located has an abnormal interlayer, otherwise, it is considered that there is no abnormal interlayer.

2. The method of claim 1, wherein In step S1, the collection of the box external data comprises: Placing two profile radars on both sides of the box, calculating the pose calibration matrix of the two profile radars; Splicing the data of the two profile radars based on the pose calibration matrix of the two profile radars to output the box external data.

3. The method of claim 1, wherein In step S1, the collection of the box internal data comprises: Placing two laser scanning radars on both sides of the box, taking the first laser scanning radar as the main one and the second laser scanning radar as the auxiliary one, calibrating the pose relationship to obtain the pose calibration matrix of the second laser scanning radar to the first laser scanning radar; Calibrating the first laser scanning radar and the 2D camera to calculate the pose calibration matrix of the laser scanning radar and the 2D camera; Splicing the laser scanning radar data according to the pose calibration matrix of the two laser scanning radars to obtain the box internal data.

4. The method of claim 1, wherein In step S22, the obtaining of multiple plane equations and an inlier set by using a sequential RANSAC model extraction algorithm comprises: S221: initializing a model parameter set and a model inlier set, setting the initial point cloud as the remaining point cloud to enter a loop; S222: select a minimum point set from the remaining point cloud to extract a model, fit a candidate model with the minimum point set, and obtain the corresponding inlier set of the candidate model; S223: check whether the number of inliers of the candidate model meets the preset minimum value, and whether the inlier error and the inlier error are within the preset error threshold, if not, the loop is terminated; S224: if the candidate model is valid, the model parameters and the inliers are stored in the model parameter set and the model inlier set respectively, and the inliers of the candidate model are removed from the current remaining point cloud; S225: repeat steps S222 to S225, if the number of current remaining point clouds is lower than the preset value, the loop is terminated.

5. The method of claim 1, wherein In step S23, the rough fitting of the planes includes: S231: set six rough plane normal vectors; S232: traverse the plane equation set obtained by the RANSAC plane extraction algorithm, calculate the included angle between the plane normal vector and the six rough plane normal vectors, and if the included angle is less than a preset threshold, put the inliers corresponding to the plane equation into the inlier set corresponding to the six rough planes; S233: re-fit the planes according to the six rough plane inlier sets, obtain new plane equations, and obtain the six rough planes.

6. A system for detecting a core sandwich of a box, the system comprising: It includes: a data acquisition module, an external plane calculation module, an internal plane calculation module, and a comparison and detection module; The data acquisition module is used for collecting box external data and collecting box internal data; The external plane calculation module is used for calculating six plane equations of the box exterior according to the box external data; The internal plane calculation module is used for calculating six plane equations of the box interior according to the box internal data; Therefore, the comparison and detection module is used for obtaining the detection result of the box interlayer according to the six plane equations of the box exterior and the six plane equations of the box interior; The calculation of the six plane equations of the box interior according to the box internal data includes: S21: multi-modal data segmentation of the box; S22: obtaining multiple plane equations and inlier sets by using a sequential RANSAC model extraction algorithm; S23: rough fitting of the planes to obtain six rough planes; S24: accurate extraction of the planes to obtain six plane equations of the box interior; The accurate extraction of the planes to obtain the six plane equations of the box interior includes: According to the six rough planes, three of them are combined, and the intersection points are calculated by using three-face intersection, which are the corner points of the box; According to the position information of the box corner point distribution, the four corner points of each face are re-fitted to obtain six plane equations of the box interior; The calculation of the six plane equations of the box exterior according to the box external data includes: segmenting and extracting the box external point cloud; constructing an OBB body coordinate system and rotating the box external point cloud to the OBB body coordinate system; obtaining points within three faces of the box exterior; calculating the point cloud curvature and extracting the edge point cloud of the three faces according to the point cloud curvature change; combining the edge point cloud on each face of the box exterior and recalculating the new plane equation; The detection result of the box body interlayer obtained according to the six plane equations of the box exterior and the six plane equations of the box interior includes: According to six plane equations inside the box, four corner points of the plane inside the box are calculated, and if the sum of distances from the four corner points of the plane inside the box to the corresponding plane outside the vehicle body exceeds a predetermined distance threshold, it is considered that the position plane where the corner point is located has an abnormal sandwich, otherwise it is considered that there is no abnormal sandwich.

Images