Corrugated board warping detection method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of corrugated board detection, in particular to a corrugated board warping detection method and system based on multi-modal data fusion. The method comprises the following steps: acquiring three-dimensional point cloud data of a corrugated board; a paperboard contour main body point set is obtained through a preset spatial clustering algorithm; performing point cloud feature extraction to obtain a point cloud global feature vector; performing cubic polynomial fitting on the paperboard contour main body point set to obtain a standard image; performing image feature extraction on the standard image to obtain an image global feature vector; performing fusion processing on the point cloud global feature vector and the image global feature vector to obtain a multi-modal fusion feature vector; and inputting the multi-modal fusion feature vector into a pre-trained classifier, and outputting the warping category of the corrugated board. The obtained multi-modal fusion feature vector gives full play to the geometric accuracy of the three-dimensional point cloud data and the semantic richness advantage of the standard image, and the detection accuracy of the corrugated board warping is improved.

Description

Technical Field

[0001] This application relates to the field of corrugated board inspection technology, and in particular to a method and system for detecting corrugated board warping based on multimodal data fusion. Background Technology

[0002] As a core material in the modern packaging industry, the quality of corrugated cardboard directly determines the strength, printing quality, and stacking stability of subsequent boxes. Warping is one of the most common and extremely harmful quality defects in corrugated cardboard production lines. Currently, the industry mainly uses manual visual inspection or imported automated equipment for quality control, but these methods have significant limitations.

[0003] In the field of automated inspection, existing technologies for extracting cardboard contours using line structured light face two major technical bottlenecks. First, at the data preprocessing level, traditional methods struggle to effectively handle raw point cloud data collected from industrial sites. Due to interference from factors such as production line vibration and environmental dust, raw point clouds typically contain a large number of noise points and outliers.

[0004] Secondly, at the defect identification algorithm level, existing methods mostly use single-modal data for analysis. While two-dimensional image-based methods are fast, they are sensitive to lighting conditions and cannot accurately capture the three-dimensional deformation features of cardboard. 3D point cloud-based methods, although able to acquire geometric information, lack surface texture features and are easily affected by the quality of the point cloud during feature extraction. Both of these single-modal methods have unavoidable technical limitations, making it difficult to achieve high-precision multi-class warp classification. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a method and system for detecting corrugated cardboard warpage based on multimodal data fusion, which can improve the accuracy of corrugated cardboard warpage detection.

[0006] According to one aspect of the embodiments of this application, a method for detecting the warpage of corrugated cardboard based on multimodal data fusion is proposed, the method comprising: Three-dimensional point cloud data of corrugated cardboard are acquired using a line structured light sensor; The three-dimensional point cloud data is converted into a set of main points representing the outline of the corrugated cardboard using a preset spatial clustering algorithm. Point cloud features are extracted from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard. A standard image of the corrugated cardboard is obtained by performing a cubic polynomial fitting on the main point set of the cardboard outline. Image feature extraction is performed on the standard image to obtain a global feature vector for the corrugated cardboard. The point cloud global feature vector and the image global feature vector are fused to obtain a multimodal fused feature vector; The multimodal fusion feature vector is input into a pre-trained classifier, which outputs the warping category of the corrugated cardboard to determine the warping shape of the corrugated cardboard based on the warping category.

[0007] In the above scheme, the step of converting the three-dimensional point cloud data into a set of main points representing the corrugated cardboard outline using a preset spatial clustering algorithm includes: The initial two-dimensional contour point set of the corrugated cardboard is extracted from the three-dimensional point cloud data using the preset spatial clustering algorithm. Adaptive DBSCAN spatial clustering is performed on the initial two-dimensional contour point set to obtain the largest cluster used to characterize the paperboard contour of the corrugated paperboard; The outlier removal of the largest cluster is performed using the RANSAC algorithm to obtain a clean set of main points for the cardboard outline.

[0008] In the above scheme, the step of performing adaptive DBSCAN spatial clustering on the initial two-dimensional contour point set to obtain the largest cluster used to characterize the cardboard contour of the corrugated cardboard includes: Determine the distribution density of the three-dimensional point cloud data; The neighborhood radius and minimum number of points (minPts) parameter of the 3D point cloud data are dynamically adjusted according to the distribution density. Adaptive DBSCAN spatial clustering is performed based on the adjusted neighborhood radius and the adjusted minimum number of points (minPts) parameter to obtain the maximum cluster.

[0009] In the above scheme, the step of performing a cubic polynomial fitting on the main point set of the cardboard outline to obtain the standard image of the corrugated cardboard includes: The pure cardboard outline main point set is fitted with least squares using a preset first formula to obtain a smooth outline curve for the corrugated cardboard. The smooth contour curve is rendered to generate a grayscale image or a binary image of a preset size, and the grayscale image or the binary image is used as the standard image.

[0010] In the above scheme, the step of extracting point cloud features from the main point set of the cardboard outline to obtain a global point cloud feature vector for the corrugated cardboard includes: The point cloud feature extraction branch is used to extract point cloud features from the main point set of the cardboard outline to obtain the global point cloud feature vector for the corrugated cardboard. The point cloud feature extraction branch is based on a PointNet network structure, which includes a one-dimensional convolutional layer and a max pooling layer.

[0011] In the above scheme, the step of extracting image features from the standard image to obtain a global feature vector for the corrugated cardboard image includes: Image features are extracted from the standard image using a pre-trained ResNet18 backbone network to obtain a global feature vector for the corrugated cardboard image.

[0012] In the above scheme, the step of fusing the point cloud global feature vector and the image global feature vector to obtain a multimodal fused feature vector includes: The point cloud global feature vector and the image global feature vector are concatenated to obtain the multimodal fusion feature vector.

[0013] According to one aspect of the embodiments of this application, a corrugated cardboard warpage detection system based on multimodal data fusion is proposed, the system comprising: The acquisition unit is used to acquire three-dimensional point cloud data of corrugated cardboard through a line structured light sensor; The first extraction unit is used to convert the three-dimensional point cloud data into a set of main points representing the outline of the corrugated cardboard using a preset spatial clustering algorithm. The second extraction unit is used to extract point cloud features from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard. The fitting unit is used to perform cubic polynomial fitting on the main point set of the cardboard outline to obtain a standard image of the corrugated cardboard. The third extraction unit is used to extract image features from the standard image to obtain a global feature vector for the corrugated cardboard. The fusion processing unit is used to fuse the global feature vector of the point cloud and the global feature vector of the image to obtain a multimodal fusion feature vector; The output unit is used to input the multimodal fused feature vector into a pre-trained classifier and output the warping category of the corrugated cardboard to determine the warping shape of the corrugated cardboard based on the warping category.

[0014] The beneficial effects of this application are as follows: This application converts the three-dimensional point cloud data of corrugated cardboard into a two-dimensional cardboard outline main point set that is easier to analyze, and performs point cloud feature extraction to obtain a point cloud global feature vector representing geometric features. On the other hand, by performing cubic polynomial fitting on the cardboard outline main point set, a standard image of the corrugated cardboard is obtained, that is, an image global feature vector representing image features is obtained. Based on this, multimodal feature fusion is used, that is, the point cloud global feature vector and the image global feature vector are fused. The resulting multimodal fused feature vector fully leverages the geometric accuracy of the three-dimensional point cloud data and the semantic richness of the standard image. As a result, the detected warpage category of the corrugated cardboard is more accurate, and the detection accuracy of corrugated cardboard warpage is improved. Attached Figure Description

[0015] Figure 1 This is a system architecture diagram of the corrugated cardboard warpage detection method based on multimodal data fusion provided in the embodiments of this application; Figure 2 A flowchart illustrating the corrugated cardboard warpage detection method based on multimodal data fusion provided in this application embodiment; Figure 3 The image shows the effect of the adaptive DBSCAN algorithm in selecting the outline of the cardboard. Figure 4 The image shows the outlier removal effect of the RANSAC algorithm. Figure 5 Fit a grayscale image to a cubic polynomial; Figure 6 The model proposed in this application f A flowchart of a multimodal network; Figure 7 The model proposed in this application f The multimodal network structure diagram; Figure 8 This is a schematic diagram illustrating the warping shape of corrugated cardboard as an example. Figure 9 A block diagram of a corrugated cardboard warpage detection system based on multimodal data fusion provided in an embodiment of this application. Detailed Implementation

[0016] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] It should be noted that while some processes described in the specification, claims, and accompanying drawings include multiple steps appearing in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not themselves represent any execution order. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. "Multiple" in this document refers to at least two.

[0018] It is worth noting that in the specific embodiments of this application, 3D point cloud data, image data, and other related data of corrugated cardboard are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the target object is required, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, when an embodiment of this application needs to obtain 3D point cloud data, image data, and other related data of corrugated cardboard, separate permission or consent from the target object can be obtained through pop-up windows or redirection to a confirmation page. After obtaining separate permission or consent from the target object, the necessary 3D point cloud data, image data, and other related data of corrugated cardboard for the normal operation of the embodiment of this application can be obtained.

[0019] Please see Figure 1 , Figure 1 This is a system architecture diagram of the corrugated cardboard warpage detection method based on multimodal data fusion provided in this application embodiment. It includes a terminal 140, an Internet connection 130, a gateway 120, a server 110, etc.

[0020] Terminal 140 can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, forming a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0021] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0022] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0023] The following provides a detailed description of the specific implementation methods of the embodiments of this application: Please see Figure 2 , Figure 2 This is a flowchart illustrating the corrugated cardboard warpage detection method based on multimodal data fusion provided in this application embodiment. The corrugated cardboard warpage detection method based on multimodal data fusion can be implemented by server 110 and / or terminal 140. Figure 2 The corrugated cardboard warpage detection method shown includes: Step 210: Collect 3D point cloud data of corrugated cardboard using a line structured light sensor; Step 220: Convert the three-dimensional point cloud data into a set of main points representing the outline of the corrugated cardboard using a preset spatial clustering algorithm; Step 230: Extract point cloud features from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard; Step 240: Perform cubic polynomial fitting on the main point set of the cardboard outline to obtain the standard image of the corrugated cardboard; Step 250: Extract image features from the standard image to obtain a global feature vector for the corrugated cardboard. Step 260: The point cloud global feature vector and the image global feature vector are fused to obtain a multimodal fused feature vector; Step 270: Input the multimodal fusion feature vector into the pre-trained classifier and output the warping category of the corrugated cardboard to determine the warping shape of the corrugated cardboard based on the warping category.

[0024] The complete embodiment of this application will be explained in detail below with reference to steps 210-270: In step 210, by using a line structured light sensor and an industrial camera arranged at the exit end of the corrugated cardboard production line, the curve contour point cloud data of the vertical surface of the corrugated cardboard, i.e., three-dimensional point cloud data, can be extracted.

[0025] In step 220, the three-dimensional point cloud data is converted into the main point set of the corrugated cardboard outline using a preset spatial clustering algorithm. Here, the main point set of the cardboard outline is a pure two-dimensional point set.

[0026] In some embodiments, converting the three-dimensional point cloud data into a set of main points representing the corrugated cardboard outline using a preset spatial clustering algorithm includes: The initial two-dimensional contour point set of the corrugated cardboard is extracted from the three-dimensional point cloud data using the preset spatial clustering algorithm. Adaptive DBSCAN spatial clustering is performed on the initial two-dimensional contour point set to obtain the largest cluster used to characterize the paperboard contour of the corrugated paperboard; The outlier removal of the largest cluster is performed using the RANSAC algorithm to obtain a clean set of main points for the cardboard outline.

[0027] Specifically, the acquired 3D point cloud data is preprocessed. First, the adaptive DBSCAN algorithm is used to accurately extract the cardboard outline from the 3D point cloud data, such as... Figure 3 As shown. Then, based on the distribution density of the 3D point cloud data, the neighborhood radius and minimum point threshold are dynamically adjusted to adapt to different density distributions, accurately extracting the pure cardboard outline main point set. The neighborhood radius is dynamically adjusted based on the distribution density of the point cloud data. and minimum points The parameters are calculated using formulas (1), (2), and (3): ; In equation (1) For the first point cloud i One point; Point The first in i The nearest neighbor; Point The first in i The Euclidean distance between the nearest neighbors; in equations (2) and (3) Indicates Centered on, The set of points in the neighborhood of radius; This represents the threshold for the minimum number of neighboring locations of a core point. The dynamic adjustment is based on 3D point cloud data. This can be achieved using distance maps or local point density statistics (i.e., distribution density). To enhance robustness under varying surface textures and lighting conditions, the neighborhood radius conservation is adaptively determined rather than manually fixed. Using... Calculate the average of the nearest neighbors. k- Distance curve, and select the 95th percentile distance value as the benchmark. Local density gradient and second-order gradient are introduced, and further adaptive refinement is carried out according to the point density distribution using formula (4), which is as follows: ; In equation (4) The baseline radius, For point Local density, dense areas use smaller Values, sparse regions use larger values value; The first-order gradient of the discretized local density approximates the rate at which the density changes with location. The density second gradient is the discretized value.

[0028] In step 230, point cloud features are extracted from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard.

[0029] In the PRF-Net network, the extraction process of the global feature vector of the point cloud aims to transform the disordered and sparse 2D contour point cloud (i.e., the main point set of the cardboard contour) into a fixed-length global feature vector that is invariant to the point order, so as to fuse it with image branch features. Here, the RANSAC algorithm is used to remove outlier point cloud data from the extracted main point set of the cardboard contour, resulting in a clean 2D contour point set. In each iteration, a subset of points from the main point set of the cardboard contour is randomly selected to fit the model. f (The model f includes a pre-trained classifier), and the residuals of all point subsets relative to the model are calculated using formula (5), which is as follows: ; In equation (5) ei Point pi Vertical distance to the model; f Indicates the currently fitted model; dist ( pi f () represents the distance from a point to the curve model. Given a threshold... ,like ei< If , then it is an interior point; when When <5 mm, residual threshold This has a significant impact on data purity, eliminating excessive inner layers, especially at local curvature transitions; when When the surface reflection noise is greater than 10 mm, the abnormal values ​​caused by the surface reflection noise persist. = 7 mm provides the best trade-off between noise suppression and contour preservation. The model with the most interior points is ultimately selected as the final result, such as... Figure 4 As shown.

[0030] In step 240, a cubic polynomial fit is performed on the main point set of the cardboard outline to obtain a standard image of the corrugated cardboard.

[0031] Here, a cubic polynomial fitting is used to transform the main point set of the purified cardboard outline into a smooth outline curve, such as... Figure 5 As shown. The clean cardboard outline main point set is fitted with least squares using the cubic polynomial of formula (6) to obtain a smooth outline curve, and a standardized 224×224 pixel grayscale image is generated by rendering. The formula (6) is as follows: ; in, x The horizontal pixel coordinates representing the cross-sectional contour represent the vertical pixel coordinates of the cross-sectional contour at the corresponding position. a, b, c, d This represents the fitting constant, which can be calibrated. The smooth contour curve is rendered to generate a grayscale image or binary image of a fixed size (preset size), which serves as the standard image.

[0032] In step 250, the image feature extraction branch uses a pre-trained ResNet18 as the backbone network and removes its original fully connected layers to extract high-level semantic features from the standard image. That is, the resulting global feature vector is used to represent the high-level semantic features of the standard image. In the PRF-Net network, the extraction of the global feature vector aims to transform the two-dimensional image representing the cardboard outline into a compact, semantically rich global feature vector. This process leverages the powerful feature learning capabilities of convolutional neural networks pre-trained on large datasets and adapts it for this industrial inspection task through transfer learning and network structure adjustments. In the image feature extraction branch, a ResNet-18 model pre-trained on the ImageNet dataset is used as the feature extraction backbone network. By removing its original fully connected classification layers, the network is transformed into a feature extractor, outputting a 512-dimensional global feature vector. To adapt to the contour image processing task of this application, channel adaptation processing is performed on the input layer to convert the single-channel contour image into a three-channel input. A layered fine-tuning strategy is adopted to optimize the high-level parameters of the network in a progressive unfreezing manner on the corrugated cardboard warping dataset. This allows the model to focus on learning contour geometric features while maintaining general feature extraction capabilities, significantly improving the relevance of feature representation and classification accuracy.

[0033] In step 260, the global feature vector of the point cloud and the global feature vector of the image are fused to obtain a multimodal fused feature vector. The global feature vector of the point cloud and the global feature vector of the image are fused into a feature-level concatenation fusion, that is, the feature vectors output by the two branches are concatenated and fused to form a joint feature representation containing 1536 dimensions.

[0034] In step 270, a neural network based on multimodal data fusion is used to classify corrugated cardboard into nine categories (such as...). Figure 8 As shown, Figure 8 (The diagram illustrates several exemplary warping patterns). The preset classifier is designed as a nine-classifier, mapping the fused multimodal feature vector to nine specific warping categories, including upward warping (type 1, 2, 3), downward warping (type 1, 2, 3), S-shaped warping, L-shaped warping, and normal cardboard. Through end-to-end training, the model can automatically learn the distinguishing features of different warping patterns. Please refer to the following steps (in conjunction with...). Figure 6 (Explanation required) Step 1: Feature concatenation. The 1024-dimensional feature vector output from the point cloud feature extraction branch is concatenated with the 512-dimensional feature vector output from the image branch to obtain a 1536-dimensional fused feature vector. Step 2: A fusion bottleneck module consisting of a fully connected layer, a batch normalization layer, a ReLU activation function, and a Dropout layer was introduced for processing; Step 3: Normalization layers accelerate training convergence and improve stability; Step 4: The 256-dimensional feature vector after fusion and dimensionality reduction is fed into a classifier implemented by a single fully connected layer, as shown in formulas (7) and (8). (7) (8) Finally, the high-level features are mapped to the output space of 9 categories, and then transformed into a probability distribution through the Softmax function to complete the classification. The model architecture is as follows: Figure 7 As shown.

[0035] In summary, this application ensures data quality (i.e., multimodal fusion feature vector) through an innovative preprocessing procedure and achieves high-precision warp classification by utilizing multimodal feature fusion, effectively solving the problems of low detection accuracy, poor robustness, and coarse classification granularity in existing technologies.

[0036] Please see Figure 9 , Figure 9This is a schematic diagram of the structure of a corrugated cardboard warpage detection system based on multimodal data fusion provided in an embodiment of this application. This system is applied to computer equipment and may include: Acquisition unit 401 is used to acquire three-dimensional point cloud data of corrugated cardboard through a line structured light sensor; The first extraction unit 402 is used to convert the three-dimensional point cloud data into a set of main points of the corrugated cardboard outline by a preset spatial clustering algorithm. The second extraction unit 403 is used to extract point cloud features from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard. Fitting unit 404 is used to perform cubic polynomial fitting on the main point set of the cardboard outline to obtain a standard image of the corrugated cardboard. The third extraction unit 405 is used to extract image features from the standard image to obtain a global feature vector for the corrugated cardboard. The fusion processing unit 406 is used to perform fusion processing on the global feature vector of the point cloud and the global feature vector of the image to obtain a multimodal fusion feature vector; The output unit 407 is used to input the multimodal fusion feature vector into a pre-trained classifier and output the warping category of the corrugated cardboard to determine the warping shape of the corrugated cardboard based on the warping category. Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0037] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0038] It should be understood that in the description of the embodiments of this application, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0039] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0040] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0041] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0042] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0043] It should also be understood that the various implementation methods provided in this application can be combined arbitrarily to achieve different technical effects.

[0044] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0045] The above is a detailed description of the embodiments of this application. However, this application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A method for detecting the warpage of corrugated cardboard based on multimodal data fusion, characterized in that, The method includes: Three-dimensional point cloud data of corrugated cardboard are acquired using a line structured light sensor; The three-dimensional point cloud data is converted into a set of main points representing the outline of the corrugated cardboard using a preset spatial clustering algorithm. Point cloud features are extracted from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard. A standard image of the corrugated cardboard is obtained by performing a cubic polynomial fitting on the main point set of the cardboard outline. Image feature extraction is performed on the standard image to obtain a global feature vector for the corrugated cardboard. The point cloud global feature vector and the image global feature vector are fused to obtain a multimodal fused feature vector; The multimodal fusion feature vector is input into a pre-trained classifier, which outputs the warping category of the corrugated cardboard to determine the warping shape of the corrugated cardboard based on the warping category.

2. The method for detecting corrugated cardboard warpage based on multimodal data fusion according to claim 1, characterized in that, The step of converting the 3D point cloud data into a set of main points representing the corrugated cardboard outline using a preset spatial clustering algorithm includes: The initial two-dimensional contour point set of the corrugated cardboard is extracted from the three-dimensional point cloud data using the preset spatial clustering algorithm. Adaptive DBSCAN spatial clustering is performed on the initial two-dimensional contour point set to obtain the largest cluster used to characterize the paperboard contour of the corrugated paperboard; The outlier removal of the largest cluster is performed using the RANSAC algorithm to obtain a clean set of main points for the cardboard outline.

3. The method for detecting corrugated cardboard warpage based on multimodal data fusion according to claim 2, characterized in that, The step of performing adaptive DBSCAN spatial clustering on the initial two-dimensional contour point set to obtain the largest cluster for characterizing the corrugated cardboard contour includes: Determine the distribution density of the three-dimensional point cloud data; The neighborhood radius and minimum number of points (minPts) parameter of the 3D point cloud data are dynamically adjusted according to the distribution density. Adaptive DBSCAN spatial clustering is performed based on the adjusted neighborhood radius and the adjusted minimum number of points (minPts) parameter to obtain the maximum cluster.

4. The method for detecting corrugated cardboard warpage based on multimodal data fusion according to claim 2, characterized in that, The step of performing a cubic polynomial fitting on the main point set of the cardboard outline to obtain a standard image of the corrugated cardboard includes: The pure cardboard outline main point set is fitted with least squares using a preset first formula to obtain a smooth outline curve for the corrugated cardboard. The smooth contour curve is rendered to generate a grayscale image or a binary image of a preset size, and the grayscale image or the binary image is used as the standard image.

5. The method for detecting corrugated cardboard warpage based on multimodal data fusion according to claim 1, characterized in that, The step of extracting point cloud features from the main point set of the cardboard outline to obtain a global point cloud feature vector for the corrugated cardboard includes: The point cloud feature extraction branch is used to extract point cloud features from the main point set of the cardboard outline to obtain the global point cloud feature vector for the corrugated cardboard. The point cloud feature extraction branch is based on a PointNet network structure, which includes a one-dimensional convolutional layer and a max pooling layer.

6. The method for detecting corrugated cardboard warpage based on multimodal data fusion according to claim 1, characterized in that, The step of extracting image features from the standard image to obtain a global feature vector for the corrugated cardboard includes: Image features are extracted from the standard image using a pre-trained ResNet18 backbone network to obtain a global feature vector for the corrugated cardboard image.

7. The method for detecting corrugated cardboard warpage based on multimodal data fusion according to claim 1, characterized in that, The process of fusing the global feature vector of the point cloud and the global feature vector of the image to obtain a multimodal fused feature vector includes: The point cloud global feature vector and the image global feature vector are concatenated to obtain the multimodal fusion feature vector.

8. A corrugated cardboard warpage detection system based on multimodal data fusion, characterized in that, The system includes: The acquisition unit is used to acquire three-dimensional point cloud data of corrugated cardboard through a line structured light sensor; The first extraction unit is used to convert the three-dimensional point cloud data into a set of main points representing the outline of the corrugated cardboard using a preset spatial clustering algorithm. The second extraction unit is used to extract point cloud features from the main point set of the cardboard outline to obtain a global feature vector of the point cloud for the corrugated cardboard. The fitting unit is used to perform cubic polynomial fitting on the main point set of the cardboard outline to obtain a standard image of the corrugated cardboard. The third extraction unit is used to extract image features from the standard image to obtain a global feature vector for the corrugated cardboard. The fusion processing unit is used to fuse the global feature vector of the point cloud and the global feature vector of the image to obtain a multimodal fusion feature vector; The output unit is used to input the multimodal fused feature vector into a pre-trained classifier and output the warping category of the corrugated cardboard to determine the warping shape of the corrugated cardboard based on the warping category.

Images