Box interlayer detection method and system

Abstract

The invention relates to the technical field of image detection, and provides a box interlayer detection method and system, and the method comprises the steps: S1, collecting the external data of a box and the internal data of the box; s2, calculating six plane equations in the box body according to the internal data of the box body; s3, calculating six external plane equations of the box body according to the external data of the box body; and S4, obtaining a detection result of the interlayer of the box body according to the six plane equations outside the box body and the six plane equations inside the box body. According to the box interlayer detection method based on multi-modal data fusion, the point cloud of the target box is quickly segmented based on the multi-modal fusion technology; the inner and outer planes of the box body are accurately extracted by using the structural features of the box body, and the box body interlayer is rapidly detected by comparing the inner and outer plane equations of the box body.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image detection, in particular to a detection method and system for box interlayer. BACKGROUND

[0002] The interlayer of a truck carriage is often used for concealment, and truck interlayer detection is of great significance. In recent years, truck carriage interlayer detection has been upgraded from "checking goods" to "checking space, checking risk, and checking compliance", becoming the first digital defense line of smart customs and safe borders.

[0003] In the prior art, large X-ray scanning systems cannot detect interlayers shielded by heavy metals such as lead plates, and rely on manual reading of images for judgment. X-ray computed tomography (CT) systems have slow scanning speeds and high costs, making it difficult to scale up.

[0004] Therefore, it is necessary to provide a detection method and system for box interlayer, a box interlayer detection method based on multi-modal data fusion, and a multi-modal fusion technology for quickly segmenting target box point clouds. The box structure features are used to accurately extract the inner and outer planes of the box, and the box interlayer is quickly detected by comparing the equations of the inner and outer planes of the box.

[0005] The above information disclosed in the background section is only used to enhance the understanding of the background of the present application, and therefore it can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The main purpose of the present application is to overcome the problems of slow detection speed and limited interlayer material detection in the prior art, and to provide a detection method and system for box interlayer, a box interlayer detection method based on multi-modal data fusion, and a multi-modal fusion technology for quickly segmenting target box point clouds. The box structure features are used to accurately extract the inner and outer planes of the box, and the box interlayer is quickly detected by comparing the equations of the inner and outer planes of the box.

[0007] To achieve the above purpose, the first aspect of the present application provides a detection method for box interlayer, comprising the following steps: S1: collecting box external data and collecting box internal data; S2: calculating six plane equations of the box interior according to the box internal data; S3: calculating six plane equations of the box exterior according to the box external data; S4: obtaining the detection result of the box interlayer according to the six plane equations of the box exterior and the six plane equations of the box interior.

[0008] According to an example embodiment of the present application, in step S1, the collection of box external data comprises: Two profile radars are placed on both sides of the box, and pose calibration matrices of the two profile radars are calculated; Two profile radars are placed on both sides of the box, and pose calibration matrices of the two profile radars are calculated;

[0009] According to an example embodiment of the present application, in step S1, the collection of the box interior data includes: Two laser scanning radars are placed on both sides of the box, the first laser scanning radar is the main one, and the second laser scanning radar is the auxiliary one, the pose relationship of the two laser scanning radars is calibrated, and a pose calibration matrix of the second laser scanning radar to the first laser scanning radar is obtained; The first laser scanning radar and the 2D camera are calibrated, and a pose calibration matrix of the laser scanning radar and the 2D camera is calculated; According to an example embodiment of the present application, in step S1, the collection of the box interior data includes:

[0010] According to an example embodiment of the present application, in step S2, the calculation of the six plane equations in the box interior according to the box interior data includes: S21: multi-modal data segmentation is performed on the box; S22: a plurality of plane equations and an inlier set are obtained by using a sequential RANSAC model extraction algorithm; S23: six rough planes are obtained by rough fitting of the planes; S24: six plane equations in the box interior are obtained by accurate extraction of the planes.

[0011] According to an example embodiment of the present application, in step S22, the obtaining of the plurality of plane equations and the inlier set by using the sequential RANSAC model extraction algorithm includes: S221: a model parameter set and a model inlier set are initialized, and an initial point cloud is set as a remaining point cloud to enter a loop; S222: a minimum point set is selected from the remaining point cloud to extract a model, a candidate model is fitted out by using the minimum point set, and an inlier set corresponding to the candidate model is obtained; S223: whether the number of inliers of the candidate model meets a preset minimum value, and whether the inlier error and the inlier error threshold are within a preset error threshold are checked, and if not, the loop is terminated; S224: if the candidate model is valid, the model parameter and the inlier are respectively stored in the model parameter set and the model inlier set, and the inlier of the candidate model is removed from the current remaining point cloud; S225: steps S222 to S225 are repeated, and if the number of the current remaining point cloud is lower than a preset value, the loop is terminated.

[0012] According to an example embodiment of the present application, in step S23, the rough fitting of the planes includes: S231: setting six rough plane normal vectors; S232: traversing the plane equation set obtained by the RANSAC plane extraction algorithm, calculating the angle between the plane normal vector and the six rough plane normal vectors, and if the angle is less than a preset threshold, then putting the inliers corresponding to the plane equation into the inlier set corresponding to the six rough planes; S233: re-fitting the planes according to the six rough plane inlier sets to obtain new plane equations, and obtaining the six rough planes.

[0013] According to an example embodiment of the present application, in step S24, the accurate extraction of the planes includes: According to the six rough planes, combining three of them to calculate the intersection point by three-plane intersection, which is the corner point of the box; According to the position information of the box corner point distribution, re-fitting the four corner points of each face to obtain the six plane equations inside the box.

[0014] According to an example embodiment of the present application, in step S3, the calculation of the six plane equations outside the box according to the box exterior data includes: segmenting and extracting the box exterior point cloud; constructing an OBB coordinate system and rotating the box exterior point cloud to the OBB coordinate system; obtaining the points inside the three faces of the box exterior; calculating the point cloud curvature and extracting the edge point cloud of the three faces according to the point cloud curvature change; merging the edge point cloud on each face of the box exterior and re-calculating the new plane equation.

[0015] According to an example embodiment of the present application, in step S4, the detection result of the box interlayer according to the six plane equations outside the box and the six plane equations inside the box includes: calculating the four corner points of the plane inside the box according to the six plane equations inside the box, and if the sum of the distances from the four corner points of the plane inside the box to the corresponding plane outside the vehicle body exceeds a predetermined distance threshold, then considering that the position plane where the corner point is located has an abnormal interlayer, otherwise considering that there is no abnormal interlayer.

[0016] As a second aspect of the present application, the present application provides a box interlayer detection system, comprising: a data acquisition module, an external plane calculation module, an internal plane calculation module and a comparison and detection module; The data acquisition module is used to acquire box exterior data and acquire box interior data; The external plane calculation module is used to calculate the six plane equations outside the box according to the box exterior data; The internal plane calculation module is used to calculate the equations of the six planes inside the box based on the internal data of the box. The comparison detection module is used to obtain the detection results of the interlayer of the box based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0017] The advantages of this invention are: This solution is based on a multimodal data fusion-based method for detecting interlayers in vehicle bodies. It rapidly segments the target vehicle body point cloud using multimodal fusion technology; it accurately extracts the inner and outer planes of the vehicle body using the structural features of the body; and it achieves rapid detection of interlayers by comparing the equations of the inner and outer planes of the body. It has low environmental requirements, is harmless to humans, and is highly flexible. It can also display the interlayer situation of the body more intuitively on images and point clouds. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0019] Figure 1 A schematic diagram of the detection system for the interlayer of the enclosure is shown.

[0020] Figure 2 The schematic diagram illustrates the data acquisition module's design scheme.

[0021] Figure 3 The diagram illustrates the steps of the method for detecting the interlayer of the enclosure.

[0022] Figure 4 The diagram illustrates the technical roadmap for extrinsic parameter calibration of laser scanning radar and 2D cameras.

[0023] Figure 5 The diagram illustrates the data acquired by the laser scanning radar. Figure 5 (a) is the RGB diagram of the cabinet. Figure 5 (b) is the point cloud of the box.

[0024] Figure 6 The diagram illustrates the multimodal data segmentation of the target container.

[0025] Figure 7 A schematic diagram of reverse projection is shown. Figure 7 (a) is a diagram illustrating the process of back-projecting pixels from a two-dimensional image into three-dimensional space. Figure 7 (b) is the original two-dimensional image data captured by the camera.

[0026] Figure 8The diagram schematically shows the four rays corresponding to the coordinates of eight points.

[0027] Figure 9 The diagram schematically illustrates the original state of the enclosure in the laser scanning radar coordinate system.

[0028] Figure 10 The OBB box is illustrated schematically.

[0029] Figure 11 The diagram illustrates the effect of transforming the box-shaped point cloud into the OBB volume coordinate system.

[0030] Figure 12 A schematic diagram of the six sides of the box is shown.

[0031] Figure 13 A schematic diagram of the edge is shown.

[0032] Figure 14 The diagram illustrates the test results when the truck's cargo box is empty.

[0033] Figure 15 The diagram illustrates the test results when the truck has a double-layered structure.

[0034] Figure 16 The diagram illustrates the detection results when a foreign object is found in a truck. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0036] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0039] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of this application. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0040] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing this application, and therefore cannot be used to limit the scope of protection of this application.

[0041] According to a first specific embodiment of the present invention, the present invention provides a detection system for the interlayer of a box, such as... Figure 1 As shown, it includes: a data acquisition module, an external plane calculation module, an internal plane calculation module, and a comparison and detection module.

[0042] The data acquisition module is used to collect data from both the outside and inside of the enclosure.

[0043] like Figure 2 As shown, the data acquisition module includes two contour radars 1, a laser velocity radar 2, two laser scanning radars 3, and a 2D camera 4.

[0044] Two gantry cranes, A and B, are set up. The truck carrying the container passes through gantry cranes A and B sequentially. Two contour radars 1 and a laser speed radar 2 are used together to collect external data of the container. The two contour radars 1 are placed on gantry crane A, located on the upper sides of the container, higher than the container itself. These two contour radars 1 are single-line laser radars, scanning downwards. When the truck carrying the container passes through gantry crane A, they can collect the external contour point cloud data of the container. The laser speed radar 2 is placed on gantry crane B, in the middle, to measure the moving speed and position of the container. It is used to stitch together the data collected by the two contour radars 1 to generate a complete scan point cloud. Two laser scanning radars 3 and a 2D camera 4 are used together to collect internal data of the container. The two laser scanning radars 3 are placed in the middle on both sides of gantry crane A, at a height approximately the same as the middle of the container, to collect internal data of the container. The 2D camera 4 is installed in the middle position above gantry crane A, working in conjunction with the laser scanning radars 3 to collect container data.

[0045] The external plane calculation module is used to calculate the equations of the six external planes of the enclosure based on the external data of the enclosure.

[0046] The internal plane calculation module is used to calculate the equations of the six planes inside the box based on the internal data of the box.

[0047] The comparison detection module is used to obtain the detection results of the interlayer of the box based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0048] According to a second embodiment of the present invention, the present invention provides a method for detecting the interlayer of a box, employing the detection system for the interlayer of a box according to the first embodiment, such as... Figure 3 As shown, the detection method includes the following steps: S1: Collect external data and internal data of the enclosure.

[0049] For the exterior of the enclosure, this solution employs two contour radars to provide high-precision, high-density contour point cloud data, while a dedicated laser velocimeter radar 2 accurately measures the enclosure's movement speed. By fusing the data from these three sources, high-resolution surface point cloud data is generated, achieving a precise digital representation of the enclosure's external geometry. The two contour radars 1 are located on the same horizontal plane, higher than the enclosure. The laser surfaces of the two contour radars 1 are vertically set, perpendicular to the enclosure's axis. The two contour radars have overlapping fields of view in the area traversed by the enclosure, greatly expanding the effective sensing range and reducing the blind spots of individual radars. The laser velocimeter radar 2 is located on the same horizontal plane as the contour radars 1, in front of the enclosure's direction of movement, emitting radar signals towards the enclosure from behind. The laser surface of the laser velocimeter radar 2 is vertically set, with the enclosure's axis either on this laser surface or parallel to it, ensuring that the laser velocimeter radar 2 can also scan the enclosure while the contour radars are acquiring data.

[0050] Contour radar 1 accurately calculates the distance to each object by measuring the characteristics of the reflected echo signal after the emitted light pulse hits the nearby object, and outputs the three-dimensional contour information of the object (i.e., the XYZ coordinates of a series of points). Laser velocity radar 2 utilizes the Doppler effect to determine the speed of the target by comparing the frequencies of the emitted and received signals.

[0051] The single-frame data of contour radar 1 is not generated instantaneously. Instead, the laser points are emitted at different times, requiring a certain amount of time. During this time, the target may be moving (e.g., a truck driving), resulting in different relative positions between the laser points and the target object. However, when a typical lidar driver encapsulates a frame of data, it defaults to acquiring all laser points in that frame at the same time and in the same pose. This leads to motion errors in the laser frame, potentially causing blurred, streaked, or distorted contours. Therefore, this solution uses lidar velocity radar 2 to acquire the target object's velocity and calculate its position at different times. The lidar coordinates corresponding to each laser point in each frame of contour radar 1 data are then transformed to their positions at different times to minimize the lidar's motion errors.

[0052] The data collected from the outside of the enclosure includes: S111: Place the two contour radars 1 on both sides of the housing and calculate the pose calibration matrix of the two contour radars 1.

[0053] Since two contour radars 1 and one laser velocity radar 2 are used in combination, the data calibration of the acquisition system is equivalent to the pose calibration of the two contour radars 1. The pose calibration of the two contour radars 1 is performed to calculate the transformation matrix between them: a calibration object with obvious features is placed statically within the overlapping area of ​​the field of view of the two contour radars 1, and both contour radars 1 are simultaneously triggered to scan the calibration object; the calibration object is moved multiple times and scanned multiple times to acquire calibration object data pairs under different poses; feature 3D coordinates are extracted from each calibration object data; using the extracted feature point pairs, the transformation matrix is ​​calculated, which is the pose calibration matrix of the two contour radars 1.

[0054] S112: Based on the pose calibration matrix of the two contour radars 1, the data of the two contour radars 1 are stitched together to output the external data of the box, which is the "external point cloud of the box".

[0055] For the interior of the enclosure, this solution uses two laser scanning radars 3 and a 2D camera 4 to collect data about the interior of the enclosure, such as... Figure 6 As shown in (a), Figure 6(a) is a point cloud image collected on-site by a single laser scanning radar 3. The device is located approximately 5 meters behind the rear of the enclosure, at a height within the middle of the enclosure. The rear of the enclosure is open, and the laser scanning radar 3 is aimed at the interior of the enclosure for scanning. The beam emission is similar to that of a flashlight. A single device needs to be directly facing the scanner, resulting in left-right deviations and potential incomplete scanning of the left and right sides of the enclosure. Therefore, using two laser scanning radars 3 in this solution can reduce blind spots. The RGB image generated by the 2D camera 4 has high-resolution texture, color, and semantic information, with significant edge features. The point cloud data output by the laser radar has accurate, illumination-independent 3D geometric information and distance measurement capabilities, with significant point features. By combining the "3D spatial geometry of the radar" with the "2D semantics of the camera," the laser scanning radar 3 and the 2D camera 4 achieve a perception effect of 1+1>2, which is a core technology necessary for many applications, including autonomous vehicle positioning, environmental mapping, and dynamic object recognition.

[0056] The data collected from inside the enclosure includes: S121: Place two laser scanning radars on both sides of the housing. The first laser scanning radar 3 is the primary one and the second one is the secondary one. Calibrate their pose relationship to obtain the pose calibration matrix from the second laser scanning radar 3 to the first laser scanning radar 3.

[0057] 2A. Place a calibration object with obvious features in the overlapping area of ​​the field of view of the two laser scanning radars 3, and simultaneously trigger the two laser scanning radars 3 to collect calibration object data (including point cloud data and RGB image data). 2B. Following step 1A, move the calibration object multiple times and collect calibration object data pairs under different poses; 2C. Extract feature 3D coordinates from the point cloud data of each calibration object; use the extracted 3D feature point pairs to calculate the transformation matrix, which is the pose calibration matrix of the two laser scanning radars 3.

[0058] S122: Calibrate the first laser scanning radar 3 and 2D camera 4, and calculate the pose calibration matrix of the laser scanning radar 3 and 2D camera 4.

[0059] The extrinsic parameter calibration of the laser scanning radar 3 and the 2D camera 4 is based on the premise that both (2D camera 4 and laser scanning radar 3) are rigidly connected. On-site, the 6DoF rigid transformation relationship (R,t) between the coordinate systems of the two sensors is quickly and accurately determined. 6DoF: six degrees of freedom (3 rotation parameters, 3 translation parameters); (R,t): R is the rotation matrix (which can be converted to and from the 3 rotation parameters), and t is the translation vector (including the 3 translation parameters). The technical approach is as follows: Figure 4 As shown.

[0060] 1A. Place a calibration object with obvious features in the overlapping area of ​​the field of view of the laser scanning radar 3 and the 2D camera 4, and simultaneously trigger the laser scanning radar 3 and the 2D camera 4 to collect data of the calibration object; 1B. Following step 1A, move the calibration object multiple times and collect calibration object data pairs under different poses; 1C. Extract the 3D coordinates of the point cloud features generated by the laser scanning radar 3 in the laser scanning radar coordinate system; calculate the 2D coordinates of the RGB image features generated by the 2D camera 4, and convert them into 3D coordinates in the 2D camera coordinate system according to the intrinsic parameters of the 2D camera 4. 1D. Using the extracted feature point pairs, calculate the transformation matrix, which is the pose calibration matrix of the laser scanning radar 3 and the 2D camera 4.

[0061] S123: Based on the pose calibration matrix of the two laser scanning radars 3, the laser scanning radar data is stitched together to obtain the data inside the box.

[0062] like Figure 5 As shown, the laser scanning radar data is stitched together and merged according to the pose calibration matrices of the two laser scanning radars 3, and the resulting output is called the "box point cloud"; the 2D camera 4 outputs the "box RGB image". Among these, Figure 5 (a) is the RGB diagram of the cabinet. Figure 5 (b) is the point cloud of the box.

[0063] This solution employs multimodal fusion technology, specifically multimodal fusion of LiDAR and camera data. Point cloud data exhibits a discrete and sparse spatial distribution, possessing precise geometric measurement capabilities and significant point features; while image data exhibits a continuous and dense pixel distribution, possessing rich texture response capabilities and significant edge features. By adopting a multimodal data fusion method, the limitations of relying on a single sensor are eliminated. By comprehensively utilizing complementary information from different sensors to perceive the current environment, the overall system's perception accuracy and environmental understanding capabilities can be significantly improved.

[0064] Multimodal fusion of LiDAR and cameras effectively combines the three-dimensional spatial geometric information (point cloud, echo intensity, etc.) provided by LiDAR with the two-dimensional semantic information (such as RGB images) provided by the camera through techniques such as spatial alignment, feature transformation, or attention mechanisms in a unified three-dimensional space (such as bird's-eye view / BEV space) or feature space. This leverages the complementary advantages of the two modalities—the precise spatial location information of LiDAR and the rich semantic information of the camera—to enhance the performance of tasks such as 3D target detection, tracking, or segmentation.

[0065] LiDAR camera data fusion is a core technology essential for many applications, including autonomous vehicle localization, environmental mapping, and dynamic object recognition. Key technologies for achieving cross-modal fusion of LiDAR camera data include LiDAR camera extrinsic parameter calibration and point cloud image feature transformation.

[0066] Point clouds are a direct product of LiDAR, while the corresponding images are a direct product of cameras. Feature transformation between the two is a prerequisite for cross-modal fusion and a fundamental technology for subsequent target detection and tracking, and environmental mapping applications. By jointly calibrating the extrinsic parameters of a 2D camera and LiDAR, orthographic and inverse projections can be achieved. Orthographic projection projects the 3D point cloud onto a 2D image plane, assigning color information to each point; inverse projection is a reverse mapping from 2D pixels to the 3D world. A pixel in the image represents a direction, a potential 3D location range, which signifies an infinitely extending line originating from the camera's optical center, passing through that pixel, and extending into the real 3D world. Based on the calibration matrix, a ray in the point cloud coordinate system can be determined.

[0067] S2: Calculate the equations of the six planes inside the box based on the internal data of the box.

[0068] The moment the container stops at the double-rod gantry, the laser scanning radar 3 and the 2D camera 4 simultaneously acquire point cloud data with high-precision three-dimensional geometric information and RGB images with surface texture information inside the container, and perform data processing based on this data.

[0069] Based on the internal data of the enclosure, the equations for the six planes inside the enclosure are calculated as follows: S21: Perform multimodal data segmentation on the box.

[0070] Extract the outer rectangular outline (4 sides, 4 corners) of the target box from the image, that is, the side of the box tail parallel to the laser surface of contour radar 1 (i.e., Figure 8 In the lower small image, a rectangular frame (the plane containing the rear side of the box) is used. Four rays from its four corner points can be obtained through back projection. Based on depth information, eight point coordinates can be extracted from these rays. The point cloud is then cropped based on these eight coordinates to obtain the box's point cloud. Its OBB (minimum bounding box) is calculated, and a box OBB volume coordinate system is constructed. The box's point cloud is then corrected to a frontal viewpoint, resulting in the following... Figure 6 The image shows a multimodal data segmentation diagram of the box-shaped target, in which... Figure 6 (a) Raw data collected by a single laser scanning radar; Figure 6 (b) To determine the position of the box through the image, the position of the box in the laser scanning radar point cloud is calculated based on the calibration results of 2D camera 4 and laser scanning radar 3, and the box point cloud is obtained after cropping.

[0071] S211: Extraction of the outer contour of the box target in RGB image.

[0072] In the image, the target box is darker than other areas. First, the RGB image is filtered and denoised. Second, adaptive histogram equalization is used to highlight the target box's location, and gradients are calculated to emphasize its edge information. Then, the image is binarized, and edge extraction is performed to determine the target box's edge contour. Finally, the coordinates of the four corner points of the minimum bounding matrix of the target box's edge contour are calculated.

[0073] S212: Reverse projection.

[0074] like Figure 7 As shown, Figure 7 (a) describes the process of backprojecting pixels from a two-dimensional image into three-dimensional space; Figure 7 (b) is the original 2D image data captured by the camera. Figure 7 (b) is marked with a cross. Figure 7 (a) Ray Corresponding Position. The process of backprojecting pixels from a two-dimensional image into three-dimensional space, i.e., recovering the corresponding three-dimensional point coordinates from pixel coordinates. Its function is to reconstruct a three-dimensional point cloud from an RGB-D image or a binocular parallax map, realizing the correspondence between a two-dimensional image and three-dimensional space. The backprojection formula is: ; Here, λ is a scale factor representing the distance from the camera's optical center along the ray direction. To obtain a specific 3D point on the ray, the value of λ must be provided.

[0075] This scheme, based on the pose calibration parameters of the 2D camera 4 and the laser scanning radar 3 in step S122, transforms the coordinates of the four corner points of the minimum bounding matrix of the target box edge contour on the RGB image into point cloud data through inverse projection. Mapping the two-dimensional points to the three-dimensional coordinate system yields a ray. Therefore, by setting two different depth parameters, the corresponding eight coordinate points in the point cloud data can be obtained. The depth information can be referenced from the depth-of-field setting of the laser scanning radar.

[0076] S213: Construction of the OBB volumetric coordinate system for the box point cloud and calculation of OBB dimensions.

[0077] like Figure 8 As shown, by inverse projection from the image to the point cloud, four rays are obtained, depth information is set, and the coordinates of eight points in the point cloud are obtained; as shown Figure 9 As shown, the original state of the box in the laser scanning radar coordinate system has a tilt angle. The box point cloud is clipped based on the coordinates of these 8 points, and the OBB (minimum bounding box) is obtained based on the box point cloud. Figure 10 The rectangular line surrounding the point cloud is the calculated OBB (minimum bounding box); transforming it to the OBB coordinate system, as shown... Figure 11As shown, the box point cloud transformed to the OBB volume coordinate system has no tilt angle, enabling cuboid correction. The eight points resemble the eight vertices of a cuboid. Selecting all points within the cuboid from the point cloud yields the box point cloud; this process is called clipping.

[0078] The box-shaped point cloud is clipped based on the 8 coordinate points calculated from the reverse projection; the box-shaped point cloud OBB will be calculated below: (1) Calculate the centroid of the point cloud: ; Among them, P i is the coordinates of the point, n is the total number of points, and µ is the centroid of the point cloud.

[0079] (2) Calculate the covariance matrix: The covariance matrix describes the distribution of points in three dimensions and the correlation between them; ; in, , , Where, µ x µ y µ z The centroids are located in the three directions of the point cloud: X, Y, Z, W, and Q, where X, Y, Z, W, and Q represent different dimensions. i ,q i Let W and Q represent the specific values ​​respectively, and let x represent the specific values ​​corresponding to X, Y, and Z respectively. i ,y i ,z i , that is, the value of a point in the point cloud in three directions. All parameters in C can be obtained by using cov(W, Q).

[0080] Calculate the eigenvalues ​​and eigenvectors of the covariance matrix, and solve the characteristic equation: ; Where λ is the eigenvalue, λ={λ1, λ2, λ3}, and its magnitude represents the variance of the data in that direction; ν is the eigenvector, ν={ν1, ν2, ν3}, representing the principal direction.

[0081] (3) Construct the OBB volume coordinate system: Construct the OBB volume coordinate system with the centroid of the point cloud as the origin and the three feature vectors as the basis vectors of the OBB volume coordinate system; and transform all points in the point cloud from the original coordinate system to the OBB volume coordinate system.

[0082] ; Among them, P i P is a point in the original coordinate system. ’i The vector represents the point in the transformed OBB volume coordinate system, T represents the transpose of the vector, and is the operation sign; ν={ν1, ν2, ν3}, and µ is the centroid of the point cloud.

[0083] Determine OBB dimensions: Find the maximum values ​​(x, y, z) of the point cloud in the OBB volume coordinate system. max ,y max ,z max ) and minimum value (x) min ,y min ,z min If ), then the dimensions of the box point cloud on the three axes are ((x max - x min ) / 2, (y max - y min ) / 2, (z max -z min ) / 2).

[0084] S22: Use the sequential RANSAC model extraction algorithm to obtain multiple plane equations and interior point sets.

[0085] The box has a cuboid structure with six faces, eight corners, and twelve edges. Each face is associated with four corners. Therefore, this scheme achieves accurate plane extraction of the box based on its cuboid features. The sequential RANSAC model extraction algorithm is used to obtain multiple plane equations and interior point sets, including: S221: Initialize the model parameter set S and the model internal point set I, set the initial point cloud P to the remaining point cloud PR and enter the loop, and set the model to be extracted as a plane.

[0086] S222: Randomly select the minimum point set of the model to be extracted from the remaining point cloud PR, fit a candidate model with the minimum point set, and obtain the corresponding interior point set of the candidate model.

[0087] S223: Check whether the number of interior points in the candidate model meets the preset minimum value, and whether the interior point error is within the preset error threshold. If not, the loop terminates.

[0088] S224: If the candidate model is valid, store the model parameters and interior points into the model parameter set S and the model interior point set I respectively, and remove the candidate model interior points from the current remaining point cloud.

[0089] S225: Repeat steps S222 to S225. If the current number of remaining point clouds is lower than a preset value, the loop terminates.

[0090] The sequential RANSAC model extraction algorithm is used to obtain multiple plane equations and interior point sets.

[0091] S23: Perform a coarse fit on the plane to obtain six coarse planes.

[0092] S221: Define six coarse plane normal vectors.

[0093] By converting the box point cloud to the OBB volume coordinate system, we can obtain the normal vectors of the six coarse planes of the box: the normal vector of the front side (-1,0,0), the normal vector of the rear side (1,0,0), the normal vector of the left side (0,-1,0), the normal vector of the right side (0,1,0), the normal vector of the top side (0,0,-1), and the normal vector of the bottom side (0,0,1). We can then create the corresponding point set in the empty plane.

[0094] S222: Traverse the set of plane equations obtained by the RANSAC plane extraction algorithm, calculate the angle between the plane normal vector and the normal vectors of the six coarse planes. If the angle is less than a preset threshold, put the interior point corresponding to the plane equation into the interior point set corresponding to the six coarse planes.

[0095] S223: Refit the planes based on the point sets in the six coarse planes to obtain new plane equations, thus obtaining six coarse planes.

[0096] S24: Accurately extract the planes to obtain the equations of the six planes inside the box.

[0097] S241: Based on the six coarse planes, combine them into three groups of three, and calculate the intersection point using the intersection of the three planes, which is the corner point of the box.

[0098] S242: Based on the location information of the corner points of the box, refit the planes of the four corner points of each face to obtain the equations of the six planes inside the box.

[0099] The equations of the six planes inside the box are denoted as follows: .

[0100] S3: Calculate the equations of the six planes outside the box based on the external data of the box.

[0101] The outer surface of the box is a cuboid structure; therefore, the outer surface of the box can be defined by six plane equations.

[0102] S31: Segment and extract the point cloud outside the box.

[0103] Euclidean clustering algorithm was used to segment and extract the point cloud outside the box.

[0104] S32: Construct an OBB volumetric coordinate system and rotate the point cloud outside the box to the OBB volumetric coordinate system.

[0105] Refer to step S213 (3) for constructing the OBB volume coordinate system.

[0106] S33: Obtain points on the three outer faces of the box.

[0107] The three planes are the top plane, the left side plane, and the right side plane, as follows: Figure 12 The diagram showing the six sides of the enclosure is because the contour radar 1 is installed on the left and right corners above the gantry A, and from the scanning perspective, only these three sides can be captured.

[0108] S34: Calculate the curvature of the point cloud and extract the edge point cloud of the three faces based on the change in the curvature of the point cloud.

[0109] like Figure 13 As shown, each of the six faces of the box has two or more edge point clouds.

[0110] S35: Merge the edge point clouds on each surface of the outer side of the box and recalculate the new plane equation.

[0111] like Figure 13 As shown, the edges of the cuboid are the edges extracted from the three faces. A new plane is fitted based on the edge point cloud on each face to obtain a new plane equation.

[0112] Each of the six faces has two or more edges, and two non-overlapping edges can form a plane.

[0113] The equations of the six planes outside the box are obtained, denoted as follows: .

[0114] Steps S2 and S3 can be performed simultaneously.

[0115] S4: The detection results of the interlayer of the box are obtained based on the six plane equations of the outer side of the box and the six plane equations of the inner side of the box.

[0116] The four corner points of the internal plane of the box are calculated based on the equations of the six planes inside the box. If the sum of the distances from the four corner points of the internal plane of the box to the corresponding external plane of the vehicle body exceeds a predetermined distance threshold, it is considered that there is an abnormal interlayer in the plane where the corner point is located; otherwise, it is considered that there is no abnormal interlayer.

[0117] Figure 14 The image shows the test results when the truck's cargo box is empty. Figure 15 The image shows the detection results when the truck has a sandwich structure, with the sandwich point cloud in the middle. Figure 16 The image shows the detection results when there is a foreign object in the truck, with point clouds of foreign objects appearing inside the truck body.

[0118] Therefore, this solution is a box-type interlayer detection method based on multimodal data fusion. It quickly segments the target box point cloud based on multimodal fusion technology; accurately extracts the inner and outer planes of the vehicle body using the structural features of the box; and achieves rapid detection of box interlayers by comparing the equations of the inner and outer planes of the box. It has low environmental requirements, is harmless to the human body, and is highly flexible. It can more intuitively display the box interlayer situation on images and point clouds.

[0119] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method of detecting a core sandwich of a box, characterized by, The method comprises the following steps: S1: collecting box external data and collecting box internal data; S2: calculating six plane equations inside the box according to the box internal data; S3: calculating six plane equations outside the box according to the box external data; S4: obtaining the detection result of the box interlayer according to the six plane equations outside the box and the six plane equations inside the box.

2. The method of claim 1, wherein In step S1, the collection of box external data comprises: Placing two profile radars on both sides of the box, calculating the pose calibration matrix of the two profile radars; Splicing the data of the two profile radars based on the pose calibration matrix of the two profile radars, and outputting the box external data.

3. The method of claim 1, wherein In step S1, the collection of box internal data comprises: Placing two laser scanning radars on both sides of the box, taking the first laser scanning radar as the main one and the second laser scanning radar as the auxiliary one, calibrating the pose relationship thereof, and obtaining the pose calibration matrix of the second laser scanning radar to the first laser scanning radar; Calibrating the first laser scanning radar and the 2D camera, and calculating the pose calibration matrix of the laser scanning radar and the 2D camera; Splicing the laser scanning radar data according to the pose calibration matrix of the two laser scanning radars to obtain the box internal data.

4. The method of claim 1, wherein In step S2, the calculation of six plane equations inside the box according to the box internal data comprises: S21: multi-modal data segmentation of the box; S22: obtaining multiple plane equations and an inlier set by using a sequential RANSAC model extraction algorithm; S23: rough fitting of the planes to obtain six rough planes; S24: accurate extraction of the planes to obtain six plane equations inside the box.

5. The method of claim 4, wherein, In step S22, the obtaining of multiple plane equations and an inlier set by using a sequential RANSAC model extraction algorithm comprises: S221: initializing a model parameter set and a model inlier set, and setting the initial point cloud as the remaining point cloud to enter a loop; S222: selecting a minimum point set from the remaining point cloud to extract a model, fitting a candidate model with the minimum point set, and obtaining an inlier set corresponding to the candidate model; S223: checking whether the number of inliers of the candidate model meets a preset minimum value, and whether the inlier error and the inlier error threshold are within a preset error threshold, and if not, the loop is terminated; S224: if the candidate model is valid, the model parameters and the inliers are respectively stored in the model parameter set and the model inlier set, and the inliers of the candidate model are removed from the current remaining point cloud; S225: repeating steps S222 to S225, and if the number of the current remaining point cloud is lower than a preset value, the loop is terminated.

6. The method of claim 4, wherein In step S23, the rough fitting of the planes to obtain six rough planes comprises: S231: setting six rough plane normal vectors; S232: traversing the plane equation set obtained by the RANSAC plane extraction algorithm, calculating the included angle between the plane normal vector and the six rough plane normal vectors, and if the included angle is less than a preset threshold, the inliers corresponding to the plane equation are put into the inlier set corresponding to the six rough planes; S233: re-fitting the planes according to the inlier sets of the six rough planes, obtaining new plane equations, and obtaining the six rough planes.

7. The method of claim 4, wherein In step S24, the accurate extraction of the planes to obtain six plane equations inside the box comprises: According to six rough planes, three of which are combined, the intersection points are calculated by three-face intersection, which are the corner points of the box; According to the position information of the box corner point distribution, the four corner points of each face are refitted to a plane to obtain six plane equations inside the box.

8. The method of claim 1, wherein, In step S3, the calculation of the six plane equations outside the box according to the box exterior data comprises: Segmenting and extracting the box exterior point cloud; Constructing an OBB body coordinate system and rotating the box exterior point cloud to the OBB body coordinate system; Obtaining points inside the three faces of the box; Calculating the curvature of the point cloud and extracting the edge point cloud of the three faces according to the change of the point cloud curvature; Combining the edge point cloud on each face of the box and recalculating the new plane equation.

9. The method of claim 1, wherein, In step S4, the detection result of the box body interlayer obtained according to the six plane equations outside the box and the six plane equations inside the box comprises: According to the six plane equations inside the box, the four corner points of the interior plane are calculated. If the sum of the distances from the four corner points of the interior plane to the corresponding exterior plane of the vehicle body exceeds the predetermined distance threshold, it is considered that the position plane where the corner point is located has an abnormal interlayer, otherwise it is considered that there is no abnormal interlayer.

10. A system for detecting a core sandwich of a box, the system comprising: Comprise: Data acquisition module, exterior plane calculation module, interior plane calculation module and comparison detection module; The data acquisition module is used to collect box exterior data and collect box interior data; The exterior plane calculation module is used to calculate the six plane equations outside the box according to the box exterior data; The interior plane calculation module is used to calculate the six plane equations inside the box according to the box interior data; Therefore, the comparison inspection module is used to obtain the detection result of the box interlayer according to the six plane equations outside the box and the six plane equations inside the box.

Images