Online Quality Inspection Method for Toy Plastic Coating Based on Multimodal Visual Perception

Abstract

This invention discloses an online quality inspection method for toy plastic coatings based on multimodal visual perception, relating to the field of visual inspection of toy quality. It constructs a dynamic graph structure from feature maps of each modality, and achieves the co-evolution of multimodal features through iterative message passing via a graph neural network. The method outputs a comprehensive quality index and defect classification results, and performs sorting control based on the comprehensive quality index. This invention improves the accuracy and comprehensiveness of toy plastic coating quality inspection by achieving the co-evolution of multimodal features through the collaborative work of a multimodal visual perception system combined with physical mechanism constraints. Compared with existing technologies, this invention can effectively identify thickness defects, material defects, appearance defects, and abnormal material structure coupling, solving the problems of incomplete single-modal detection and poor fusion effects of simple feature splicing.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology for toy quality, and in particular to an online quality inspection method for the plastic coating of toys based on multimodal visual perception. Background Technology

[0002] The plastic coating layer of toys serves as a crucial protective and decorative structure on the toy surface, and its quality directly impacts the toy's safety, durability, and appearance. Currently, quality inspection of toy plastic coating layers primarily employs single-modal testing methods or simple splicing of multimodal features, resulting in low detection accuracy, incomplete defect identification, and a lack of physical mechanism support. Single-modal testing cannot simultaneously address surface defects, internal thickness anomalies, and material property abnormalities in the plastic coating layer, while simple splicing of multimodal features fails to effectively integrate information from various modalities, making it impossible to identify abnormal defects arising from the coupling of material and structure. This approach falls short of meeting the high-precision, high-efficiency online inspection requirements of toy production lines.

[0003] Meanwhile, existing detection methods lack utilization of the optical transmission characteristics of the plastic coating layer, and the feature extraction process lacks physical constraints, resulting in poor interpretability of the detection results and difficulty in adapting to the detection scenarios of plastic coating layers of different types of toys. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an online quality inspection method for the plastic coating layer of toys based on multimodal visual perception. The technical solution adopted is as follows:

[0005] An online quality inspection method for the plastic coating of toys based on multimodal visual perception includes the following steps:

[0006] Step 1: Construct and calibrate a multimodal visual perception system. The system includes a multispectral imaging unit, a structured light 3D imaging unit, and a hyperspectral imaging unit. The calibration includes system spatial coordinate alignment and optical transmission characteristic parameters of the toy's plastic coating material.

[0007] Step 2: The multimodal visual perception system acquires multimodal image data of the toy under test. The multimodal image data includes ultraviolet fluorescence images, near-infrared transmission images, structured light stripe image sequences, and hyperspectral image cubes.

[0008] Step 3: Extract fluorescence feature maps from ultraviolet fluorescence images, calculate theoretical fluorescence distribution by combining calibrated optical transmission characteristic parameters, and generate the first anomaly guide map by the residual between measured and theoretical fluorescence features.

[0009] Step 4: Using the first anomaly guidance map as the spatial attention weight, collaboratively extract the transmission feature map from the near-infrared transmission image, and update and generate the second anomaly guidance map based on the transmission feature map.

[0010] Step 5: Extract the three-dimensional morphology and thickness distribution features from the structured light stripe image sequence, and establish a thickness spectral correlation constraint model in combination with the calibrated optical transmission characteristic parameters.

[0011] Step 6: Using thickness distribution features as spatial constraints, collaboratively extract spectral feature maps from the hyperspectral image cube, and perform consistency verification through the thickness spectral correlation constraint model to identify material structure coupling anomalies.

[0012] Step 7: Construct a dynamic graph structure for each modality feature map, realize the collaborative evolution of multimodal features through iterative message passing of graph neural network, output the comprehensive quality index and defect classification results, and perform sorting control according to the comprehensive quality index.

[0013] Optionally, in step 1, the multispectral imaging unit is an ultraviolet-visible-near-infrared multispectral camera, the structured light three-dimensional imaging unit is a structured light three-dimensional sensor, and the hyperspectral imaging unit is a hyperspectral camera; the spatial coordinate alignment adopts the checkerboard calibration method, and the optical transmission characteristic parameters are calibrated jointly by a spectrophotometer and a thickness measuring instrument.

[0014] Optionally, in step 2, the synchronous acquisition of multimodal image data is achieved by synchronously controlling each imaging unit through trigger signals. The acquisition frequency is matched with the transmission speed of the toy production line to ensure that the complete image data of the plastic coating layer of each toy under test is captured synchronously.

[0015] Optionally, in step 3, the feature extraction method for the ultraviolet fluorescence image is as follows: a multi-scale Gabor filter bank is used to extract the texture feature map, a morphological reconstruction algorithm is used to extract the microcrack line feature map, and the texture feature map and the microcrack feature map are fused to obtain the fluorescence feature map; the normalization process uses the Sigmoid function to make the value range of the first anomaly guide map between [0,1], and the higher the value, the greater the probability that the corresponding area is a defect area.

[0016] Optionally, in step 4, the transmission feature extraction network is a convolutional neural network, and a spatial attention mechanism is introduced into its convolutional layer. Specifically, during the convolution process, the first anomaly guidance map is multiplied element-wise with the output feature map of the current convolutional layer, and multiplied by the guidance coefficient α to achieve enhanced extraction of features in the abnormal region. The weight coefficient β of the weighted fusion takes a value range of 0.3-0.7 to ensure that the initial guidance information of the first anomaly guidance map is effectively combined with the newly added anomaly information of the transmission features.

[0017] Optionally, in step 5, the three-dimensional topographic feature map includes a thickness distribution map, a thickness gradient feature map, and a surface roughness feature map; the thickness spectral correlation constraint model constructed using Kubelka-Munk theory is expressed as follows:

[0018] ;in Theoretical spectral reflectance, For the reflectivity of the toy substrate, The absorption coefficient is... denoted as scattering coefficient, and t as the thickness of the plastic coating layer.

[0019] Optionally, in step 6, weighted spectral sampling is performed in the hyperspectral image cube using thickness distribution features as spatial constraints, and spectral feature maps are extracted collaboratively. The spectral feature maps are substituted into the thickness spectral correlation constraint model to calculate the residual between the measured spectral features and the theoretical spectral reflectance. When the residual exceeds a preset threshold, it is marked as an abnormal coupling region of the material structure.

[0020] Weights of weighted spectral sampling The calculation formula is:

[0021] ,in The thickness values ​​are the pixel values ​​in the thickness distribution feature map. This is the standard thickness value. The allowable thickness deviation is set; the preset threshold is determined through sample training and is used to distinguish between normal areas and areas with abnormal material structure coupling.

[0022] Optionally, in step 7, the dynamic graph structure uses superpixel regions divided by the spatial domain as graph nodes. The initial features of each node are formed by concatenating the feature vectors of that region in each modal feature map. Spatial adjacent edges, physical associated edges, and abnormal associated edges are constructed based on spatial adjacency, thickness spectral correlation, and anomaly guidance. A heterogeneous graph convolutional network is used to iteratively message-pass the dynamic graph structure. A physical constraint function is introduced during the message-passing process to achieve the co-evolution of multimodal features. The comprehensive quality index of the plastic coating layer and the defect classification results are output through the graph readout function.

[0023] Superpixel regions are divided using the SLIC algorithm, with the number of divisions adaptively adjusted according to image resolution. The condition for constructing the physically associated edges is that the difference in thickness features between two superpixel regions is less than a thickness threshold. Furthermore, the spectral angular distance between the two regions is less than the spectral threshold. The construction condition for abnormal associated edges is that both superpixel regions are high-response regions in either the first or second abnormal guidance graph.

[0024] Optionally, in step 7, the message passing process of the heterogeneous graph convolutional network is as follows: the next iteration feature of each node is obtained by processing the node's current feature with the weight matrix, adding the sum of the current features of all spatially adjacent nodes of the node after processing with the corresponding weight matrix, adding the sum of the current features of all physically associated nodes of the node after element-wise multiplication with the output of the physical constraint function and processing with the corresponding weight matrix, and finally obtaining the result through a nonlinear transformation by the activation function; the physical constraint function is constructed based on the thickness spectral mapping relationship and is used to constrain the feature passing between physically associated nodes.

[0025] Optionally, in step 7, the defect classification results include thickness defects, material defects, appearance defects, and material structure coupling defects. The sorting control of the toy under test is performed based on the comprehensive quality index. The comprehensive quality index is obtained by reading the graph-level features through global average pooling and mapping them through a fully connected layer. The value range is [0,1]. The defect classification results are output through the Softmax function. Among them, the material structure coupling defect specifically refers to the region where the residual between the thickness distribution feature and the spectral feature under the thickness-spectral correlation constraint model exceeds the preset threshold, which corresponds to the defect caused by the doping or uneven curing of the plastic coating material.

[0026] In summary, the present invention has at least one of the following beneficial technical effects:

[0027] This invention provides an online quality inspection method for toy plastic coatings based on multimodal visual perception. Through the collaborative operation of a multimodal visual perception system, combined with physical mechanism constraints, the method achieves the co-evolution of multimodal features, improving the accuracy and comprehensiveness of toy plastic coating quality inspection. Compared to existing technologies, this invention can effectively identify thickness defects, material defects, appearance defects, and abnormal material structure coupling, solving the problems of incomplete single-modal detection and poor fusion effects of simple feature splicing. By introducing a thickness spectral correlation constraint model as a physical prior, the interpretability of the detection results is enhanced. Simultaneously, the construction of a dynamic graph structure enables efficient fusion and co-optimization of multimodal features, adapting to the online inspection needs of toy production lines, improving inspection efficiency, reducing missed and false detection rates, and ensuring the quality and safety of toy plastic coatings. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the online quality inspection method for the plastic coating layer of toys based on multimodal visual perception, as per the present invention.

[0029] Figure 2 This is a schematic diagram of ultraviolet fluorescence images collected in a specific embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of a near-infrared transmission image acquired in a specific embodiment of the present invention;

[0031] Figure 4This is a schematic diagram of a structured light stripe image sequence acquired in a specific embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of a hyperspectral image cube acquired in a specific embodiment of the present invention;

[0033] Figure 6 This is a schematic diagram of the detection results of a specific embodiment of the present invention. Detailed Implementation

[0034] The present invention will be further described in detail below with reference to the accompanying drawings. See details below. Figures 1-6 .

[0035] This invention discloses an online quality inspection method for the plastic coating layer of toys based on multimodal visual perception.

[0036] Example 1

[0037] An online quality inspection method for the plastic coating of toys based on multimodal visual perception includes the following steps:

[0038] Step 1: Construct and calibrate a multimodal visual perception system. The system includes a multispectral imaging unit, a structured light 3D imaging unit, and a hyperspectral imaging unit. The calibration includes system spatial coordinate alignment and optical transmission characteristic parameters of the toy's plastic coating material.

[0039] Step 2: The multimodal visual perception system acquires multimodal image data of the toy under test. The multimodal image data includes ultraviolet fluorescence images, near-infrared transmission images, structured light stripe image sequences, and hyperspectral image cubes.

[0040] Step 3: Extract fluorescence feature maps from ultraviolet fluorescence images, calculate theoretical fluorescence distribution by combining calibrated optical transmission characteristic parameters, and generate the first anomaly guide map by the residual between measured and theoretical fluorescence features.

[0041] Step 4: Using the first anomaly guidance map as the spatial attention weight, collaboratively extract the transmission feature map from the near-infrared transmission image, and update and generate the second anomaly guidance map based on the transmission feature map.

[0042] Step 5: Extract the three-dimensional morphology and thickness distribution features from the structured light stripe image sequence, and establish a thickness spectral correlation constraint model in combination with the calibrated optical transmission characteristic parameters.

[0043] Step 6: Using thickness distribution features as spatial constraints, collaboratively extract spectral feature maps from the hyperspectral image cube, and perform consistency verification through the thickness spectral correlation constraint model to identify material structure coupling anomalies.

[0044] Step 7: Construct a dynamic graph structure for each modality feature map, realize the collaborative evolution of multimodal features through iterative message passing of graph neural network, output the comprehensive quality index and defect classification results, and perform sorting control according to the comprehensive quality index.

[0045] By adopting the above technical solution, a sensing system incorporating multispectral, structured light 3D, and hyperspectral imaging is first constructed and calibrated, providing a precise equipment foundation and physical parameter support for subsequent detection. Comprehensive image information related to the surface, interior, and material properties of the plastic coating layer is obtained by simultaneously acquiring multiple modal images. Features are sequentially extracted from each modal image to generate anomaly guidance maps, achieving dynamic guidance between modalities. Feature verification is then performed using a physical constraint model. Finally, multimodal feature co-evolution is achieved through dynamic graph structures and graph neural networks, ultimately outputting a quality index and defect classification results. This completes sorting control, ensuring the integrity, systematicity, and accuracy of the detection process, covering the detection needs of various defects in the plastic coating layer.

[0046] Example 2

[0047] In step 1, the multispectral imaging unit is an ultraviolet-visible-near-infrared multispectral camera, the structured light three-dimensional imaging unit is a structured light three-dimensional sensor, and the hyperspectral imaging unit is a hyperspectral camera; spatial coordinate alignment adopts the checkerboard calibration method, and optical transmission characteristic parameters are calibrated by a spectrophotometer and a thickness measuring instrument.

[0048] By adopting the above technical solutions and determining specific hardware equipment, the stability and acquisition accuracy of the multimodal visual perception system are ensured. The ultraviolet-visible-near-infrared multispectral camera, structured light 3D sensor, and hyperspectral camera respectively address the image acquisition needs of different modalities, comprehensively capturing the fluorescence, transmission, 3D morphology, and spectral information of the plastic coating layer. The checkerboard calibration method enables precise alignment of the spatial coordinates of each imaging unit, avoiding feature extraction errors caused by image misalignment. Joint calibration with a spectrophotometer and thickness gauge accurately acquires the optical transmission characteristic parameters of the plastic coating material, providing reliable physical parameter support for subsequent theoretical fluorescence distribution calculations and the establishment of thickness-spectral correlation constraint models, thus ensuring the accuracy of the detection.

[0049] Example 3

[0050] In step 2, the synchronous acquisition of multimodal image data is achieved by synchronously controlling each imaging unit through trigger signals. The acquisition frequency is matched with the transmission speed of the toy production line to ensure that the complete image data of the plastic coating layer of each toy under test is captured synchronously.

[0051] By adopting the above technical solution and synchronously controlling each imaging unit through trigger signals, image misalignment caused by differences in the acquisition timing of different units can be eliminated, ensuring that various image data such as ultraviolet fluorescence and near-infrared transmission correspond to the same plastic coating layer of the toy under test. Matching the acquisition frequency with the production line transmission speed can avoid image loss or repeated acquisition due to toy movement, ensuring that the image information of each plastic coating layer of the toy under test can be completely captured. This provides a complete and synchronous data source for subsequent feature extraction and anomaly identification, avoiding the impact of data loss or misalignment on the test results.

[0052] Example 4

[0053] In step 3, the feature extraction method for the ultraviolet fluorescence image is as follows: a multi-scale Gabor filter bank is used to extract the texture feature map, a morphological reconstruction algorithm is used to extract the microcrack line feature map, and the texture feature map and the microcrack feature map are fused to obtain the fluorescence feature map; the normalization process uses the Sigmoid function to make the value range of the first anomaly guide map between [0,1], and the higher the value, the greater the probability that the corresponding area is a defect area.

[0054] By employing the above technical solutions, the multi-scale Gabor filter bank can effectively capture texture details in ultraviolet fluorescence images, and the morphological reconstruction algorithm can accurately extract the linear features of microcracks. The fusion of these two methods can comprehensively acquire the fluorescence features of the plastic coating surface, covering information related to appearance defects such as texture anomalies and microcracks. Sigmoid normalization can map the residual between measured and theoretical fluorescence features to a fixed value range, allowing the values ​​of the anomaly guidance map to intuitively reflect the defect probability. This provides accurate guidance for the subsequent collaborative extraction of near-infrared transmission features, reducing the missed detection of anomaly areas.

[0055] Example 5

[0056] In step 4, the transmission feature extraction network is a convolutional neural network, and a spatial attention mechanism is introduced into its convolutional layer. Specifically, during the convolution process, the first anomaly guidance map is multiplied element-wise with the output feature map of the current convolutional layer, and then multiplied by the guidance coefficient α to achieve enhanced extraction of features in the anomaly region. The weight coefficient β of the weighted fusion takes a value range of 0.3-0.7 to ensure that the initial guidance information of the first anomaly guidance map is effectively combined with the newly added anomaly information of the transmission features.

[0057] By adopting the above technical solution, the convolutional neural network possesses powerful feature extraction capabilities. Introducing a spatial attention mechanism into its convolutional layers, and combining the first anomaly guidance map as weights with the convolutional output feature map, enhances feature extraction from anomaly regions, focuses on high-probability defect areas, and improves the targeting of feature extraction. Appropriate weight coefficient values ​​can balance initial guidance information with newly added anomaly information, avoiding guidance bias caused by single information, ensuring that the second anomaly guidance map accurately conveys anomaly information, and providing reliable guidance for subsequent collaborative extraction of thickness and spectral features.

[0058] Example 6

[0059] In step 5, the three-dimensional topographic feature map includes a thickness distribution map, a thickness gradient feature map, and a surface roughness feature map; the thickness spectral correlation constraint model constructed using Kubelka-Munk theory is expressed as follows:

[0060] ;in Theoretical spectral reflectance, For the reflectivity of the toy substrate, The absorption coefficient is... denoted as scattering coefficient, and t as the thickness of the plastic coating layer.

[0061] By adopting the above technical solution, the three-dimensional morphological feature map covers thickness, thickness gradient, and surface roughness, which can comprehensively reflect the structural characteristics of the coating layer and provide a structural basis for subsequent spectral feature constraints. Based on the Kubelka-Munk theory, a thickness-spectral correlation constraint model is constructed, which can establish a clear correlation between thickness and theoretical spectral reflectance. Using calibrated optical transmission characteristic parameters, the material's optical properties are combined with the structural thickness to form a quantifiable constraint model. This provides reliable physical theoretical support for the consistency verification of subsequent spectral features and ensures the accurate identification of material structural coupling anomalies.

[0062] Example 7

[0063] In step 6, using thickness distribution features as spatial constraints, weighted spectral sampling is performed in the hyperspectral image cube to collaboratively extract spectral feature maps; the spectral feature maps are substituted into the thickness spectral correlation constraint model to calculate the residual between the measured spectral features and the theoretical spectral reflectance. When the residual exceeds a preset threshold, it is marked as an abnormal coupling region of the material structure.

[0064] Weights of weighted spectral sampling The calculation formula is:

[0065] ,in The thickness values ​​are the pixel values ​​in the thickness distribution feature map. This is the standard thickness value. The allowable thickness deviation is set; the preset threshold is determined through sample training and is used to distinguish between normal areas and areas with abnormal material structure coupling.

[0066] By employing the above technical solution and using thickness distribution characteristics as spatial constraints, spectral feature extraction can be focused on areas of thickness anomalies, improving the targeting and efficiency of feature extraction and avoiding interference from invalid regions. Weighted spectral sampling quantifies the deviation between the thickness and the standard value and assigns different sampling weights to ensure that the spectral features of areas of thickness anomalies are fully captured. Substituting the spectral features into the thickness-spectral correlation constraint model, the degree of matching between the measured spectrum and the theoretical spectrum can be determined by comparing the residuals. When the residual exceeds a preset threshold, it indicates a mismatch between the thickness and material properties, thus marking it as an abnormal coupling of the material structure, achieving accurate identification of this type of special defect.

[0067] Example 8

[0068] In step 7, the dynamic graph structure uses superpixel regions divided by the spatial domain as graph nodes. The initial features of each node are formed by concatenating the feature vectors of that region in each modal feature map. Spatial adjacent edges, physical associated edges, and abnormal associated edges are constructed based on spatial adjacency, thickness spectral correlation, and anomaly guidance. A heterogeneous graph convolutional network is used to iteratively message-pass the dynamic graph structure. A physical constraint function is introduced during the message passing process to achieve the co-evolution of multimodal features. The comprehensive quality index of the plastic coating layer and the defect classification results are output through the graph readout function.

[0069] Superpixel regions are divided using the SLIC algorithm, with the number of divisions adaptively adjusted according to image resolution. The condition for constructing the physically associated edges is that the difference in thickness features between two superpixel regions is less than a thickness threshold. Furthermore, the spectral angular distance between the two regions is less than the spectral threshold. The construction condition for abnormal associated edges is that both superpixel regions are high-response regions in either the first or second abnormal guidance graph.

[0070] By adopting the above technical solution, superpixel regions serve as graph nodes, balancing spatial resolution and feature integrity. Each node integrates feature vectors from various modalities, ensuring the comprehensiveness of node features. Different types of edges are constructed based on spatial, physical, and anomaly-guided relationships, accurately reflecting the connections between nodes and enabling the graph structure to embody the intrinsic relationships between modalities. Heterogeneous graph convolutional networks, through iterative message passing, achieve interactive optimization of features across modalities. Combined with physical constraint functions, this ensures that feature evolution conforms to the physical characteristics of the coating layer. Finally, the graph readout function outputs accurate comprehensive quality indices and defect classification results, providing a reliable basis for sorting control.

[0071] Example 9

[0072] In step 7, the message passing process of the heterogeneous graph convolutional network is as follows: the next iteration feature of each node is obtained by processing the node's own current feature with the weight matrix, adding the sum of the current features of all spatially adjacent nodes of the node after processing with the corresponding weight matrix, adding the sum of the current features of all physically associated nodes of the node after element-wise multiplication with the output of the physical constraint function and processing with the corresponding weight matrix, and finally obtaining the result through nonlinear transformation by the activation function; the physical constraint function is constructed based on the thickness spectral mapping relationship and is used to constrain the feature passing between physically associated nodes.

[0073] By adopting the above technical solution, the computational logic for updating node iterative features takes into account the features of the node itself, the features of spatially adjacent nodes, and the features of physically related nodes. Through the processing of different weight matrices, the importance of various relationships can be distinguished. The physical constraint function is constructed based on the thickness spectral mapping relationship, which can constrain the feature transfer between physically related nodes, prevent feature evolution from deviating from the physical properties of the plastic coating layer, and ensure the scientific nature of multimodal feature fusion. The nonlinear transformation of the activation function can enhance the expressive power of the features, so that the final output node features can accurately reflect the quality status of the plastic coating layer, providing reliable support for subsequent quality index and defect classification.

[0074] Example 10

[0075] In step 7, the defect classification results include thickness defects, material defects, appearance defects, and material structure coupling defects. The sorting control of the toy under test is performed according to the comprehensive quality index. The comprehensive quality index is obtained by reading the graph-level features through global average pooling and mapping them through a fully connected layer. The value range is [0,1]. The defect classification results are output through the Softmax function. Among them, material structure coupling defects specifically refer to the region where the residual between the thickness distribution features and the spectral features under the thickness-spectral correlation constraint model exceeds the preset threshold, which corresponds to defects caused by doping or uneven curing of the plastic coating material.

[0076] By adopting the above technical solution, defect classification covers four categories of defects: thickness, material, appearance, and material structure coupling, comprehensively covering potential quality problems in the plastic coating layer. The comprehensive quality index is generated through graph-level feature mapping, with a fixed value range that intuitively reflects the overall quality level of the plastic coating layer. Defect classification results are output through a specific function to ensure classification accuracy. The clear definition of material structure coupling defects distinguishes between traditional single defects and coupled defects, avoiding missed or false detections. Combined with the comprehensive quality index for sorting control, it enables graded screening of toy plastic coating layers, adapting to the high-efficiency inspection needs of production lines.

[0077] The following describes the implementation principle of the present invention using specific embodiments:

[0078] It is used for online quality inspection on toy plastic coating production lines. The sample to be tested is a children's plastic toy that has been injection molded and then plastic coated. The plastic coating thickness is 0.8mm and the material is semi-transparent polyvinyl chloride.

[0079] A multimodal visual perception system was constructed. The system employs an ultraviolet-visible-near-infrared multispectral camera, a structured light 3D sensor, and a hyperspectral camera. Each imaging unit is arranged sequentially along the production line direction, maintaining a fixed working distance from the toy under test.

[0080] A checkerboard calibration method was used to align the spatial coordinates of each imaging unit, ensuring a one-to-one correspondence between pixels in different modal images. The absorption and scattering coefficients of the plastic coating material were measured using a spectrophotometer, and standard thickness parameters were obtained using a thickness gauge to complete the calibration of optical transmission characteristic parameters.

[0081] Each imaging unit is synchronously controlled by a trigger signal, and the acquisition frequency is set to 10Hz to match the transmission speed of the production line.

[0082] Simultaneously acquire four modal images of the toy under test:

[0083] Ultraviolet fluorescence imaging: capturing microcracks and areas of abnormal fluorescence attenuation on the surface of the plastic coating layer; such as Figure 2 As shown;

[0084] Near-infrared transmission images: reflect the uniformity of the internal material of the plastic coating layer; such as Figure 3 As shown;

[0085] Structured light stripe image sequences: used to reconstruct 3D topography and thickness distribution; such as Figure 4 As shown;

[0086] Hyperspectral image cube: Acquiring the spectral characteristics of the plastic coating material. For example... Figure 5 As shown;

[0087] After the data acquisition is completed, the four types of images are previewed synchronously in the system interface to ensure that the data is complete and without misalignment.

[0088] Fluorescence feature extraction and anomaly-guided generation:

[0089] Feature extraction was performed on the ultraviolet fluorescence images: a multi-scale Gabor filter bank was used to extract texture feature maps, and a morphological reconstruction algorithm was used to extract microcrack line feature maps. These were then fused to obtain the fluorescence feature map. The theoretical fluorescence distribution at the standard thickness was calculated by combining the calibrated fluorescence material attenuation coefficient and excitation light distribution.

[0090] The residual between the measured fluorescence feature map and the theoretical fluorescence distribution is calculated, and the first anomaly guide map is generated by normalizing it with the Sigmoid function. The value range is from 0 to 1, and the high value region corresponds to surface microcracks and fluorescence attenuation anomalies.

[0091] The first anomaly guidance map is used as a spatial attention weight and multiplied element-wise with the current output feature map in the convolutional layer. The result is multiplied by the guidance coefficient α to enhance the extraction of transmission features in the anomaly region and obtain the transmission feature map.

[0092] The transmissivity anomaly region is calculated based on the transmissivity feature map, and then weighted and fused with the first anomaly guidance map with a weight coefficient β=0.5 to update and generate the second anomaly guidance map, thereby realizing the progressive propagation of anomaly guidance information.

[0093] Phase demodulation and height reconstruction are performed on the structured light stripe image sequence to extract the thickness distribution map, thickness gradient feature map and surface roughness feature map to form a three-dimensional morphology feature map.

[0094] A thickness spectral correlation constraint model is constructed using Kubelka-Munk theory, and its expression is as follows:

[0095] ;in Theoretical spectral reflectance, For the reflectivity of the toy substrate, The absorption coefficient is... Here, t is the scattering coefficient, and t is the thickness of the plastic coating layer.

[0096] Substitute the calibration parameters for this case: =0.32mm⁻¹, =1.25mm⁻¹, =0.18, t=0.8mm, establish a nonlinear mapping relationship between thickness and theoretical spectral reflectance, and form a thickness-thickness spectral correlation constraint model.

[0097] Weighted spectral sampling is performed in the hyperspectral image cube using thickness distribution characteristics as spatial constraints.

[0098] The formula for calculating sampling weight is: ,in =0.8mm, =0.05mm, which tilts the sampling weight towards areas with larger thickness deviations, and collaboratively extracts spectral feature maps.

[0099] Substituting the spectral feature map into the thickness spectral correlation constraint model, the L2 norm residual between the measured spectral features and the theoretical spectral reflectance is calculated. When the residual exceeds a preset threshold of 0.12, the region is marked as having abnormal material structure coupling, corresponding to uneven doping or curing of the coating layer material.

[0100] Multimodal feature co-evolution and detection result output:

[0101] The fluorescence feature map, transmission feature map, three-dimensional morphology feature map, and spectral feature map are constructed into a dynamic graph structure.

[0102] The SLIC algorithm is used to divide the spatial domain into 200 superpixel regions as graph nodes. The initial features of each node are formed by concatenating the feature vectors of the four modalities of that region. Based on spatial adjacency, thickness-spectral correlation, and anomaly guidance, spatial adjacency edges, physical correlation edges, and anomaly correlation edges are constructed.

[0103] Heterogeneous graph convolutional networks are used for iterative message passing. During the passing process, a physical constraint function based on the thickness spectral mapping relationship is introduced to achieve the co-evolution of multimodal features. After 8 iterations, graph-level features are read out through global average pooling and mapped through fully connected layers to obtain a comprehensive quality index of 0.592.

[0104] The Softmax function outputs the defect classification result as a material defect. Combined with the comprehensive quality index, a sorting control instruction is generated to mark the toy under test as awaiting re-inspection and send it to the re-inspection line for secondary testing. The test results are as follows: Figure 6 As shown.

[0105] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An online quality inspection method for the plastic coating layer of toys based on multimodal visual perception, characterized in that, Includes the following steps: Step 1: Construct and calibrate a multimodal visual perception system. The system includes a multispectral imaging unit, a structured light 3D imaging unit, and a hyperspectral imaging unit. The calibration includes system spatial coordinate alignment and optical transmission characteristic parameters of the toy's plastic coating material. Step 2: The multimodal visual perception system acquires multimodal image data of the toy under test. The multimodal image data includes ultraviolet fluorescence images, near-infrared transmission images, structured light stripe image sequences, and hyperspectral image cubes. Step 3: Extract fluorescence feature maps from ultraviolet fluorescence images, calculate theoretical fluorescence distribution by combining calibrated optical transmission characteristic parameters, and generate the first anomaly guide map by the residual between measured and theoretical fluorescence features. Step 4: Using the first anomaly guidance map as the spatial attention weight, a transmission feature extraction network is used to collaboratively extract transmission feature maps from the near-infrared transmission image, and the second anomaly guidance map is generated based on the transmission feature maps. Step 5: Extract the three-dimensional morphological feature map and thickness distribution features from the structured light stripe image sequence, and establish a thickness spectral correlation constraint model in combination with the calibrated optical transmission characteristic parameters. Step 6: Using thickness distribution features as spatial constraints, collaboratively extract spectral feature maps from the hyperspectral image cube, and perform consistency verification through the thickness spectral correlation constraint model to identify material structural coupling defects. Step 7: Construct a dynamic graph structure for each modality feature map, realize the collaborative evolution of multimodal features through iterative message passing of graph neural network, output the comprehensive quality index and defect classification results, and perform sorting control according to the comprehensive quality index; In step 7, the dynamic graph structure uses superpixel regions divided by the spatial domain as graph nodes. The initial features of each node are formed by concatenating the feature vectors of that region in each modal feature map. Spatial adjacent edges, physical associated edges, and abnormal associated edges are constructed based on spatial adjacency, thickness spectral correlation, and anomaly guidance. A heterogeneous graph convolutional network is used to iteratively message-pass the dynamic graph structure. A physical constraint function is introduced during the message passing process to achieve the co-evolution of multimodal features. The comprehensive quality index of the plastic coating layer and the defect classification results are output through the graph readout function. Superpixel regions are divided using the SLIC algorithm, with the number of divisions adaptively adjusted according to image resolution. The condition for constructing the physically associated edges is that the difference in thickness features between two superpixel regions is less than a thickness threshold. Furthermore, the spectral angular distance between the two regions is less than the spectral threshold. The construction condition for abnormal associated edges is that both superpixel regions are high-response regions in either the first or second abnormal guidance graph. The defect classification results include thickness defects, material defects, appearance defects, and material-structure coupling defects.

2. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 1, characterized in that: In step 1, the multispectral imaging unit is an ultraviolet-visible-near-infrared multispectral camera, the structured light three-dimensional imaging unit is a structured light three-dimensional sensor, and the hyperspectral imaging unit is a hyperspectral camera; spatial coordinate alignment adopts the checkerboard calibration method, and optical transmission characteristic parameters are calibrated by a spectrophotometer and a thickness measuring instrument.

3. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 2, characterized in that: In step 2, the synchronous acquisition of multimodal image data is achieved by synchronously controlling each imaging unit through trigger signals. The acquisition frequency is matched with the transmission speed of the toy production line to ensure that the complete image data of the plastic coating layer of each toy under test is captured synchronously.

4. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 3, characterized in that, In step 3, the feature extraction method for the ultraviolet fluorescence image is as follows: a multi-scale Gabor filter bank is used to extract the texture feature map, a morphological reconstruction algorithm is used to extract the microcrack line feature map, and the texture feature map and the microcrack feature map are fused to obtain the fluorescence feature map. The normalization process uses the Sigmoid function to make the value range of the first anomaly guide map between [0,1]. The higher the value, the greater the probability that the corresponding area is a defect area.

5. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 4, characterized in that, In step 4, the transmission feature extraction network is a convolutional neural network, and a spatial attention mechanism is introduced into its convolutional layer. Specifically, during the convolution process, the first abnormal guidance map is multiplied element-wise with the output feature map of the current convolutional layer, and then multiplied by the guidance coefficient α to achieve enhanced extraction of abnormal region features. The weighting coefficient β of the weighted fusion is in the range of 0.3-0.7 to ensure that the initial guidance information of the first anomaly guidance map is effectively combined with the newly added anomaly information of the transmission features.

6. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 5, characterized in that, In step 5, the three-dimensional topographic feature map includes a thickness distribution map, a thickness gradient feature map, and a surface roughness feature map; the thickness spectral correlation constraint model constructed using Kubelka-Munk theory is expressed as follows: ;in Theoretical spectral reflectance, For the reflectivity of the toy substrate, The absorption coefficient is... denoted as scattering coefficient, and t as the thickness of the plastic coating layer.

7. The online quality inspection method for the plastic coating layer of toys based on multimodal visual perception according to claim 6, characterized in that, In step 6, using thickness distribution features as spatial constraints, weighted spectral sampling is performed in the hyperspectral image cube to collaboratively extract spectral feature maps; the spectral feature maps are substituted into the thickness spectral correlation constraint model to calculate the residual between the measured spectral features and the theoretical spectral reflectance. When the residual exceeds a preset threshold, it is marked as a material structure coupling defect region. Weights of weighted spectral sampling The calculation formula is: ,in The thickness values ​​are the pixel values ​​in the thickness distribution feature map. This is the standard thickness value. For thickness tolerance; The preset threshold is determined through sample training and is used to distinguish between normal areas and material structure coupling defect areas.

8. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 7, characterized in that, In step 7, the message passing process of the heterogeneous graph convolutional network is as follows: the next iteration feature of each node is obtained by processing the node's own current feature with the weight matrix, adding the sum of the current features of all spatially adjacent nodes of the node after processing with the corresponding weight matrix, adding the sum of the current features of all physically associated nodes of the node after element-wise multiplication with the output of the physical constraint function and processing with the corresponding weight matrix, and finally obtaining the result through nonlinear transformation by the activation function; the physical constraint function is constructed based on the thickness spectral mapping relationship and is used to constrain the feature passing between physically associated nodes.

9. The online quality inspection method for toy plastic coating based on multimodal visual perception according to claim 8, characterized in that, In step 7, the sorting control of the toys to be tested is performed according to the comprehensive quality index; the comprehensive quality index is obtained by reading the graph-level features through global average pooling and mapping through a fully connected layer, with a value range of [0,1]; the defect classification results are output through the Softmax function, where the material structure coupling defect specifically refers to the region where the residual between the thickness distribution feature and the spectral feature under the thickness-spectral correlation constraint model exceeds the preset threshold, corresponding to defects caused by doping or uneven curing of the plastic coating material.

Images