Injection molding intelligent defect early warning regulation system based on multi-modal sensor fusion

Abstract

The application discloses a multi-modal sensor fusion-based intelligent defect early warning regulation system for injection molding, and relates to the technical field of industrial automation.The system is composed of a plurality of functional modules and comprises a data acquisition module, which acquires multi-modal sensor data, including visual images, pressure parameters and temperature parameters, and outputs original image sequences and parameter sequences; wherein the visual images include visual sub-path one images and visual sub-path two images; a data analysis module, which pre-processes the visual images, performs image recognition and constructs a light ray tracing model, introduces a lightweight graph neural network into the light ray tracing model to obtain an improved light ray tracing model, inputs plastic refraction deviation and visual images into the improved light ray tracing model, and obtains defect geometric features; and a data fusion module, which performs multi-layer perception processing on the defect geometric features, adaptively weights the processed defect geometric features by using an attention mechanism, and fuses pressure parameters and temperature parameters to learn general representations.

Description

Technical Field

[0001] This invention relates to the field of industrial automation technology, specifically to an intelligent defect early warning and control system for injection molding based on multimodal sensor fusion. Background Technology

[0002] Injection molding, as one of the most widely used plastic processing technologies in the industrial field, serves a wide range of industries. Its high production efficiency and the high complexity of molded products make it a core technology for mass production of plastic products. As downstream industries continue to raise their requirements for product precision, appearance, and performance, such as the micron-level surface flatness requirements for plastic shells in the electronics field and the stringent requirements for bubble-free internal plastic parts in medical devices, defect detection and process control of injection molded products have become key links in determining production quality and economic benefits. However, in the actual production process, various defects may occur due to the influence of factors such as materials, molds, equipment, or process parameters.

[0003] In the current technology, most small and medium-sized enterprises still rely on manual visual inspection. Inspectors need to check each product one by one at the end of the production line to determine whether there are defects such as warping, short shot, burning or surface scratches. This is not only inefficient but also prone to false detection, causing defective products to flow into downstream links, increasing rework costs and quality risks. Although some injection molding production lines are equipped with a single type of sensor detection module, it is difficult to fully correlate the causal relationship between defects and process parameters due to the single detection dimension. Summary of the Invention

[0004] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an intelligent defect early warning and control system for injection molding based on multimodal sensor fusion, which solves the problems mentioned in the background technology.

[0005] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: An intelligent defect early warning and control system for injection molding based on multimodal sensor fusion, the system includes: The data acquisition module acquires multimodal sensor data, including visual images, pressure parameters, and temperature parameters, and outputs raw image sequences and parameter sequences. The visual images include visual sub-path one image and visual sub-path two image; The data analysis module preprocesses and recognizes visual images to build a ray tracing model. A lightweight graph neural network is introduced into the ray tracing model to obtain an improved ray tracing model. The pre-calibrated plastic refractive deviation and visual image are input into the improved ray tracing model to obtain the geometric features of the defects. The data fusion module performs multi-layer perception processing on the geometric features of defects, uses an attention mechanism to adaptively weight the processed geometric features of defects, and fuses pressure and temperature parameters to learn a general representation. The early warning and control module performs defect classification, prediction, and closed-loop adjustment based on general characterization, and uses long short-term memory networks to predict defects and automatically optimize control parameters.

[0006] Furthermore, acquiring multimodal sensor data includes: Visual sub-path one captures images of bubble refraction through illumination from multiple angles; visual sub-path two scans images of the mold surface using a camera. The injection pressure is monitored in real time using a pressure sensor array, and the injection pressure sequence is output. The temperature of the molten plastic is acquired in real time through a distributed temperature sensor network, and the temperature sequence of the molten plastic is output.

[0007] Furthermore, the process of constructing the ray tracing model is as follows: By tracing the path of light propagation, the system captures the refraction disturbance of light caused by the bubble, and calculates the position, size, and three-dimensional shape of the bubble; at the same time, it obtains the wavelength of the LED light source, the refractive index of the transparent plastic, the initial refractive index of the bubble, as well as the camera's installation position and imaging parameters. By simulating the emission angle of each LED light source, the light beam is set as a parallel beam. When the light enters the transparent plastic from the air, the first deflection of the light direction is simulated according to the optical refraction law. The propagation path of the deflected light inside the plastic is calculated. When the light encounters the bubble boundary, a second deflection occurs due to the difference in refractive index between the plastic and the bubble. The propagation path of each light beam inside the bubble is tracked, and the change in direction of the light after passing through the bubble is recorded. When the light passes through the plastic again into the air, a third deflection occurs, and the light propagates to the camera sensor. The pixel position of each light beam reaching the camera is recorded and correlated with the actual captured image pixels to establish a mapping relationship between the light trajectory and the image pixels, i.e., the ray tracing model.

[0008] Furthermore, the process of obtaining the improved ray tracing model is as follows: The 3D physical space of ray tracing is abstracted into a graph structure that can be processed by a graph neural network, and ray nodes and medium nodes are set separately. Based on the standard GAT, the number of attention heads is pruned and the feature dimension is reduced, and local neighborhood sampling is performed. The interaction rules between rays and medium are learned through the message passing mechanism of the graph neural network, replacing the existing ray-by-ray calculation. The final deflection direction and deflection intensity of each ray node are output through multiple rounds of message passing.

[0009] Furthermore, the process of obtaining the unified defect geometric feature set is as follows: An improved ray tracing model is invoked, and multi-angle preprocessed difference images are input. The improved ray tracing model uses the ray deflection data output by the graph neural network to locate the coordinates of the optical nodes in the deflection set and determine the two-dimensional position of the bubble. Then, based on the coverage of ray deflection at different angles, the equivalent radius of the bubble is calculated. Combining the deflection differences in multiple directions, the three-dimensional ellipsoid parameters of the bubble are inferred to obtain the geometric features of the bubble defect. The U-Net_s segmentation network outputs a defect mask, excluding noisy connected components smaller than 5 pixels; for warped defects, the mask boundary curve is fitted and the average curvature is calculated; for short-shot defects, the ratio of the mask area to the expected mold filling area is calculated; at the same time, the pixel distribution of the defect mask is statistically analyzed to determine the two-dimensional position and shape contour of the surface defect, and the geometric features of the surface defect are obtained; the combined features form a unified defect geometric feature set containing bubbles and surface defects.

[0010] Furthermore, the process of performing multi-layer sensing processing on the unified defect geometric feature set is as follows: By standardizing and unifying the geometric features of bubbles and surface defects and mapping them with depth features using a multilayer perceptron, the geometric features of the two types of defects are combined to form the input vector of the multilayer perceptron. The 12-dimensional standardized features are mapped to 256 dimensions through a fully connected layer. The basic fusion features are extracted using the ReLU activation function to enhance feature interaction, capture the potential correlation between bubble and surface defect features, compress the feature dimension, and output a unified feature vector of 128 dimensions.

[0011] Furthermore, the process of learning the general representation is as follows: A unified 128-dimensional feature vector, pressure parameter, and temperature parameter are obtained respectively. The importance weights of the three types of features are dynamically calculated through an attention layer. The three types of features are used as value vectors V, and corresponding query vectors Q and key vectors K are generated. Q and K are mapped from the input features by a fully connected layer. The similarity of each modality is calculated through similarity calculation, and softmax processing is applied to the similarity results to obtain the weights of the three types of features. Each type of feature vector is multiplied by the corresponding attention weight to obtain the weighted geometric feature, pressure feature, and temperature feature. The weighted three types of features are concatenated and input into a fully connected layer. The ReLU activation function is used to perform interaction between features, and a 64-dimensional fused feature is output. The fused feature is normalized by a BatchNorm layer to generate the final 64-dimensional general representation.

[0012] Furthermore, the defect classification prediction and closed-loop adjustment process is as follows: The 64-dimensional general representation is input into the defect classifier to identify the current defect type. A softmax multi-class classification structure is adopted. The input is a general representation that integrates defect geometric features with pressure and temperature parameters. Deep correlation features are extracted through two fully connected layers. The output is the probability of bubble, warpage and short shot defect types, and the production line is adjusted in a closed loop according to the defects.

[0013] Furthermore, the process of automatically optimizing the control parameters is as follows: Based on historical general characterization sequences, the defect risk of future cycles is predicted. The input is the general characterization sequence of the next 10 injection molding cycles. After being converted into 128-dimensional time-series features by the embedding layer, it is input into the LSTM network. The LSTM outputs the probability of each defect in the next 3 cycles by memorizing the correlation between historical defects and process parameters. When the predicted probability of future defects exceeds the threshold, the linkage optimization of process parameters and sensing parameters is triggered.

[0014] (III) Beneficial Effects This invention provides an intelligent defect early warning and control system for injection molding based on multimodal sensor fusion, which has the following beneficial effects: (1) This invention acquires multimodal data of vision, pressure and temperature through the data acquisition module, preprocesses and extracts features through the data analysis module, and performs weighted fusion through the attention mechanism of the data fusion module, thus avoiding the limitations of detection by a single sensor. By improving the ray tracing model, the three-dimensional ellipsoidal parameters of the air bubbles inside the transparent plastic are accurately inferred. At the same time, the U-Net_s segmentation network is used to identify the curvature, area ratio and other features of defects such as warping and short shot on the mold surface. The potential causal relationship between pressure fluctuation, temperature change and defects is also associated, which greatly reduces the rate of missed detection and false detection, and achieves comprehensive and accurate detection of injection molding defects.

[0015] (2) In view of the problem of high computational load and high latency of traditional ray tracing models, this invention introduces a lightweight graph neural network into the ray tracing model. Through the abstraction of the interaction between light and medium by graph structure, the reduction of attention head number, local neighborhood sampling and other designs, the defect detection latency is reduced to less than 10ms as required by industry. It is adapted to edge devices and the computing power limitation of edge devices. At the same time, the improved ray tracing model can accurately output the geometric features of defects. It can meet the rapid detection needs of injection molding production lines without relying on complex hardware upgrades, and avoid the generation of batch non-conforming products due to detection lag. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 This embodiment provides an intelligent defect early warning and control system for injection molding based on multimodal sensor fusion. The detection system includes: The data acquisition module acquires multimodal sensor data, including visual images, pressure parameters, and temperature parameters, and outputs raw image sequences and parameter sequences. The visual images include visual sub-path one image and visual sub-path two image; The process of obtaining the image of visual sub-path one is as follows: By constructing a multi-angle light source matrix and a high-speed imaging linkage system, the optical characteristics of air bubbles inside transparent plastic products were captured, resulting in a visual sub-path image.

[0019] By deploying four high-performance green LED light sources, with the experimental wavelength strictly controlled at 550nm (green), the light transmittance of transparent plastic is significantly optimized. These LEDs are positioned directly above the product (0° illumination angle, vertical incidence), directly below (180° illumination angle, vertical reverse incidence), horizontally to the left (90° illumination angle), and horizontally to the right (270° illumination angle). This omnidirectional layout ensures that light penetrates the product from multiple directions, enabling the optically non-uniform elements such as air bubbles in the transparent plastic product to produce identifiable refraction and scattering effects.

[0020] An industrial-grade high-speed CMOS camera is mounted on a precision bracket 20cm directly in front of the product. The camera is designed for high-speed scenarios, with a frame rate of 1000fps and a resolution of 1920×1080 pixels, which meets the requirements for capturing details of micron-level defects. It acquires sequential images in a single-light-source, single-image mode, meaning that when each light source is lit individually, the camera simultaneously captures an image, ultimately forming an image set containing information from four light sources.

[0021] The process of obtaining the image of visual subpath two is as follows: A high-resolution dynamic scanning scheme is used to acquire information on defects such as warping, short shots, and scorching on the surface of plastic molds.

[0022] An industrial-grade CCD camera with a resolution of 2560×1440 pixels and a frame rate of 500fps is mounted on a custom slide 15cm directly above the mold. It features high sensitivity in low light and can dynamically adjust the shooting position according to the injection cycle to cover the entire surface of the mold. With the injection cycle as the time axis, it continuously captures N_s frames of images in each cycle, which not only obtains the static texture of the mold surface, but also the dynamic changes of defects as the injection process progresses.

[0023] Obtaining pressure parameters: By integrating a pressure sensor into a key node of the injection unit of the injection molding machine, it directly contacts the molten plastic flow channel, ensuring the authenticity and timeliness of the pressure data. The sensor outputs an injection pressure sequence at a high frequency synchronized with the injection cycle, with a sampling rate of ≥1000Hz, in MPa, and completely records the pressure fluctuation curve from the start of injection to the holding and cooling stages.

[0024] Obtaining temperature parameters: Temperature sensors, such as thermocouples or infrared temperature sensors, are deployed in key areas of the molten plastic, such as the barrel, nozzle, and mold flow channel, to form a multi-node temperature monitoring network. The sensors output the temperature sequence of the molten plastic in real time, in °C, accurately reflecting the temperature change of the plastic from the molten state to the cooling and solidification process.

[0025] The data analysis module preprocesses visual images and performs image recognition to construct a ray tracing model. A lightweight graph neural network is introduced into the ray tracing model to obtain an improved ray tracing model. Plastic refractive deviation and visual images are input into the improved ray tracing model to obtain a unified set of defect geometric features. The preprocessing and feature extraction process for visual images is as follows: Based on the optical properties of bubbles inside transparent plastic, the presence, location, size, and three-dimensional shape features of the bubbles are extracted through a multi-step process.

[0026] The visual sub-path image preprocessing involves calculating the difference image between adjacent images from the four acquired light source image sequences, then using Gaussian filtering and smoothing the difference image with a kernel size of 3×3 to suppress noise. By calculating the mean and standard deviation of the difference image, an adaptive threshold is used, and the mean plus twice the standard deviation is used to extract potential bubble regions.

[0027] Four difference images are stitched together and input into a convolutional neural network binary classification model. The output is the probability of bubble existence. If the probability is greater than 0.5, a bubble is determined to exist. The centroid coordinates of the bubble region are calculated using the pixel values ​​of the difference images as weights. The pixel area of ​​the bubble region is calculated. The equivalent radius representing the bubble size is obtained through equivalent radius or PCA principal component analysis. The bubble is modeled as an ellipsoid. A multi-angle refraction model is fitted by least squares optimization. Finally, the three axis length parameters of the ellipsoid are output.

[0028] The second visual sub-path image preprocessing involves converting the RGB image to a grayscale image, enhancing contrast through contrast-limited CLAHE adaptive histogram equalization, and calculating edge intensity using the Sobel operator to obtain the edge image.

[0029] The edge image is input into the U-Net_s segmentation network, which outputs a defect mask to identify defective pixels. Spline curves are fitted to the boundary point set of the defect mask to calculate the curvature. If the average curvature is greater than 0.1, it is considered warped. The ratio of the defect mask area to the expected mold filling area is calculated. If the ratio is greater than 0.05, an alarm is triggered. At the same time, noisy connected components are eliminated through morphological closing operations to ensure accurate area calculation. Features such as the total number of pixels in the defect mask are counted to form a feature vector.

[0030] Building a ray tracing model: By tracing the path of light propagation, the refraction disturbance of light by the bubble is captured, and then the position, size and three-dimensional shape of the bubble are calculated. The basic parameters need to match the actual hardware and material properties, including the wavelength of the four green LED light sources, the refractive index of the transparent plastic is fixed at 1.58, the initial refractive index of the bubble is set to 1.0 according to the refractive index of air, the camera is installed 20cm away from the product, and the imaging parameters are frame rate and resolution. A physical model of the light propagation path is constructed to simulate the light emission angle of each LED light source, corresponding to four fixed directions: 0°, 90°, 180°, and 270°. The light is set as a parallel beam to match the focusing characteristics of industrial LEDs, ensuring that the initial direction of each beam is perfectly aligned with the actual illumination direction of the light source. When the light enters the transparent plastic (1.58 refractive index from air, 1.0), the first deflection of the light direction is simulated according to the optical refraction law. The propagation path of the deflected light inside the plastic is calculated, eliminating interface reflection interference. By setting a low reflection coefficient, it matches the surface treatment process of plastic in actual industrial scenarios. Assuming the bubble is an ellipsoid, it adapts to the characteristic that most bubbles in reality are not spherical. When the light encounters the bubble boundary, a second deflection occurs due to the difference in refractive index between the plastic and the bubble. The model needs to track the propagation path of each beam of light inside the bubble, including possible single or multiple refractions, and record the change in the direction of the light after passing through the bubble, corresponding to the brightness difference in the differential image captured by the camera. When light passes through the plastic and out into the air again, it is deflected for the third time and finally reaches the camera sensor. The model needs to record the pixel position of each ray of light reaching the camera and correspond it one-to-one with the actual image pixels to establish a mapping relationship between the ray trajectory and the image pixels.

[0031] The improved ray tracing model acquisition process is as follows: By abstracting the physical relationships of light propagation through a graph structure and combining it with a lightweight graph neural network to replace the iterative calculation of each ray in traditional ray tracing, real-time performance is improved while maintaining accuracy. Traditional models require calculating the refraction and reflection trajectories of light rays in air, plastic, and bubbles one ray at a time, and iteratively solving physical formulas for each ray. When dealing with multi-angle light sources and high-resolution images, such as tens of millions of light rays corresponding to a resolution of 1920×1080 pixels, the computational load increases exponentially, and the latency often exceeds 100ms, which cannot meet the real-time requirements of injection molding inspection. Therefore, we decided to use a graph neural network to transform the interaction between light and medium into a graph structure that can be computed in parallel. By leveraging the node aggregation capability of the graph neural network, we can learn the interaction rules between light and medium, replacing the iterative calculation of physical formulas one ray at a time. At the same time, the lightweight design is adapted to the computing power limitations of industrial edge devices.

[0032] The traditional 3D physical space of ray tracing is abstracted into a graph structure that can be processed by GNN. By setting two types of core nodes, it covers the entire process of light propagation, including ray nodes and medium nodes. Each ray node represents an actual ray beam. Its features include the initial direction of the ray, corresponding to the light source angles of 0°, 90°, 180° and 270°, 550nm wavelength, current propagation position and initial intensity. Each ray beam corresponds to 1 R-Node. They are grouped according to the light source angle into 4 groups. Each group corresponds to all the rays of one light source. The media nodes are divided into plastic nodes (Mp-Node) and bubble candidate nodes (Mg-Node). The features of Mp-Node include a plastic refractive index of 1.58 and a spatial distribution range corresponding to the three-dimensional dimensions of transparent plastic products. Mg-Node is generated from potential bubble regions obtained from previous image preprocessing, and its features include the region centroid coordinates and the initial area extracted from the difference image. Edges are used to represent the physical interaction relationships between nodes to avoid redundant connections and reduce weight. These include associated edges and interactive edges. Associated edges connect light ray nodes and plastic nodes, representing that the light ray is passing through the plastic in that area. The edge weight is set to a fixed relationship of 1. Interactive edges connect light ray nodes and bubble candidate nodes only when the light ray propagation path passes through the bubble area. The edge weight is initially set to the overlap between the light ray and the bubble area, which is 0 to 1. The higher the overlap, the greater the weight.

[0033] GAT's attention mechanism can accurately learn the interaction weights between light rays and bubbles, and its structure is easily pruned. It uses the simplified graph attention network standard GAT as its foundation, avoiding full-graph matrix operations by abandoning complex graph convolutional networks (GCN) or message-passing neural networks (MPNN). The standard GAT's eight attention heads are reduced to two, retaining only local interaction attention to learn the direct interaction between light rays and bubbles and global trend attention, learning the overall deflection pattern of light rays at the same angle, reducing the total number of parameters by 75%. The initial feature dimensions of light ray nodes and medium nodes, such as the 12-dimensional features of light rays and the 8-dimensional features of the medium, are compressed to 4 dimensions using 1×1 convolutions, reducing the computational cost of message passing in each round. Message passing is only performed between the k nearest neighbors of each node (k=5), meaning each light ray node interacts only with its five most relevant medium nodes, avoiding full graph traversal and improving inference speed.

[0034] By employing the message passing mechanism of the Generative Neural Network (GNN), the interaction between light and medium is learned, replacing the existing ray-by-ray calculation. The ray-medium graph constructed in step two, along with the initial physical parameters of traditional ray tracing, such as refractive index and ray direction, are used as input features of the GNN to ensure that the GNN's learning is based on physical laws. Through multiple rounds of message passing: In the first round of local interaction, the ray node R-Node transmits the ray direction and intensity features to the bubble candidate node Mg-Node via the interaction edge R-Mg. After Mg-Node aggregates these features, it outputs the potential deflection effect features of the bubble on the light and feeds them back to the corresponding R-Node. In the second round of global correction, R-Nodes with light sources at the same angle are connected only to rays at the same angle via trend edges and newly added lightweight edges, sharing deflection features. The global attention head of the GNN averages these features to eliminate noise interference from individual rays, such as abnormal deflection of individual rays due to sensor errors. In the output layer transformation, the GNN outputs the final deflection direction and deflection intensity of each R-Node, directly replacing the deflection results calculated by the law of refraction in traditional ray tracing, eliminating the need to solve physical formulas for each ray and significantly reducing the computational load.

[0035] By training the GNN with labeled data, we ensured that its output conformed to the laws of optical physics and avoided the bias of pure data-driven approaches. Based on a standard sample library with known bubble parameters, we obtained 1,000 sets of transparent plastic test blocks containing bubbles of fixed size, shape and position. We collected multi-angle light source images and accurate deflection results of traditional ray tracing for each set of test blocks as labels. A dual loss function, combining physical error and prediction error, is used to calculate the difference between the light deflection direction output by the GNN and the theoretical deflection direction based on the law of refraction. The simulated image corresponding to the deflection result output by the GNN is calculated. The pixel difference between the virtual image generated by the deflection direction and the actual acquired image is analyzed. After training, a channel pruning algorithm is used to remove redundant convolutional kernels in the GNN, further reducing the model parameters. At the same time, the model weights are converted from 32-bit floating-point numbers to 16-bit numbers through quantization, improving the inference speed of edge devices.

[0036] The process of obtaining the unified defect geometric feature set is as follows: An improved ray tracing model is invoked, and multi-angle preprocessed differential images are input. The model uses the ray deflection data output by the graph neural network to locate the coordinates of optical nodes in the deflection set and determine the two-dimensional position of the bubble. Then, based on the coverage of ray deflection at different angles, the equivalent radius and area conversion of the bubble are calculated. Combining the differences in deflection in multiple directions, the major, middle and minor axis lengths of the bubble's three-dimensional ellipsoid parameters are inferred to complete the extraction of the bubble's geometric features. The U-Net_s segmentation network outputs a defect mask, excluding noisy connected components smaller than 5 pixels. For warped defects, the mask boundary curve is fitted, and the average curvature is calculated; those exceeding 0.1 are considered warped. For short-shot defects, the ratio of the mask area to the expected mold filling area is calculated. Simultaneously, the pixel distribution of the defect mask is statistically analyzed to determine the two-dimensional position and shape contour of the surface defect, forming a bubble-containing feature set. The bubble geometric features and the pixel distribution of the defect mask are integrated to form a unified defect geometric feature set containing the bubble position and ellipsoidal parameters, the position, curvature, area ratio, and contour of the surface defect.

[0037] The data fusion module performs multi-layer perception processing on a unified set of defect geometric features, uses an attention mechanism to adaptively weight the processed defect geometric features, and fuses pressure and temperature parameters to learn a general representation. The process of multi-layer sensing processing is as follows: The geometric features of bubbles and surface defects are standardized and mapped to depth features using a multilayer perceptron. Based on the geometric features of the two types of defects, the input vector of the MLP is formed. The bubble features include the two-dimensional position of the bubble, the three-dimensional ellipsoidal parameters of the major axis, median axis and minor axis length, and the existence probability, for a total of 7 feature dimensions. The surface defect features include the average curvature of the warp, the ratio of the short-shot area, the two-dimensional position of the defect, and the total number of defect mask pixels, for a total of 5 feature dimensions. The two types of features are concatenated into a 12-dimensional initial input vector. To address the differences in the dimensions of different features, such as position being a pixel value and curvature being a dimensionless value, standardization is performed through the MLP input layer. The mean and standard deviation of each feature in historical samples are calculated. By using (feature value - mean) / standard deviation, all features are normalized to the [-1,1] interval, eliminating the interference of dimensions on subsequent calculations.

[0038] A deep processing MLP structure with 3 hidden layers and 1 output layer is adopted: Hidden layer 1 has 256 dimensions, and the 12-dimensional normalized features are mapped to 256 dimensions through a fully connected layer, and the basic fusion features are extracted using the ReLU activation function; Hidden layer 2 has 256 dimensions, which further enhances feature interaction and captures the potential correlation between bubble and surface defect features, such as the correlation between bubble position and warped area; Hidden layer 3 has 128 dimensions, which compresses the feature dimension and reduces the amount of subsequent computation; The output layer outputs a unified 128-dimensional feature vector, which integrates the geometric information of the two types of defects, and directly inputs it into the subsequent attention fusion module.

[0039] The process of learning a general representation is as follows: Multimodal feature weights are dynamically allocated through an attention mechanism, and then pressure and temperature parameters are fused to generate a general representation that reflects the correlation between injection molding defects and the process. Defect geometric features, pressure parameters, and temperature parameters are used as input features. The importance weights of the three types of features are dynamically calculated through an attention layer. The weights are initialized, and the three types of features are respectively used as value vectors V, generating corresponding query vectors Q and key vectors K, each with a dimension of 64. Q and K are obtained from the input features by a fully connected layer. The similarity between modes is calculated through similarity to reflect the correlation between features, such as the correlation between pressure fluctuations and warpage defects. The similarity results are processed by softmax to obtain the weights of the three types of features. For example, when the probability of bubbles is greater than 0.5, the weight of geometric features is automatically increased by 20%; when the pressure fluctuation exceeds the threshold, the weight of pressure parameters is increased by 15%, realizing adaptive adjustment of defect priority and anomaly priority.

[0040] The three types of features are fused based on attention weights. Each feature vector is multiplied by its corresponding attention weight to obtain weighted geometric features, pressure features, and temperature features. The weighted features are then concatenated and input into a fully connected layer. The ReLU activation function is used to interact between the features, and a 64-dimensional fused feature is output.

[0041] The early warning and control module performs defect classification, prediction, and closed-loop adjustment based on general characterization, and uses long short-term memory networks to predict future defects and automatically optimize control parameters. The 64-dimensional general representation is input into the defect classifier to accurately identify the current defect type. The classifier adopts a softmax multi-class classification structure. The input is a general representation that integrates defect geometric features with pressure and temperature parameters. Deep correlation features are extracted through two fully connected layers, and the probability of defect types such as bubbles, warping, and short shots is output, such as bubble probability P1, warping probability P2, etc. The type with the highest probability is the current main defect. Based on historical general characterization sequences, the defect risk of future cycles is predicted. The input is a general characterization sequence of the last 10 injection molding cycles, forming 10×64-dimensional time series data. After being converted into 128-dimensional time series features by the embedding layer, it is input into the LSTM network. The LSTM outputs the probability of each defect in the next 3 cycles, such as P1_pred and P2_pred, by memorizing the correlation between historical defects and process parameters, such as the tendency of high temperature and low pressure to cause short shots. The loss function uses mean squared error to ensure prediction accuracy.

[0042] When the predicted probability of future defects exceeds a threshold, such as P_pred being greater than 0.6, a closed-loop adjustment and optimization of process parameters and sensing parameters is triggered. The speed is dynamically reduced by the current speed × (1-0.2×P_pred) and constrained within the range of 10-100 cm / s to alleviate bubbles or short shots caused by excessive speed. The speed is extended by the current time and 5×P_pred to reduce warping caused by insufficient cooling. The gradient of the general characterization to the light source angle is calculated by automatic differentiation, and the LED light source angle is finely adjusted to ensure clearer defect features in subsequent image acquisition.

[0043] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0044] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0045] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An intelligent defect early warning and control system for injection molding based on multimodal sensor fusion, characterized in that: The system includes: The data acquisition module acquires multimodal sensor data, including visual images, pressure parameters, and temperature parameters, and outputs raw image sequences and parameter sequences. The visual images include visual sub-path one image and visual sub-path two image. By constructing a multi-angle light source matrix and a high-speed imaging linkage system, the optical characteristics of air bubbles inside the transparent plastic product are captured to obtain visual sub-path one image. A high-resolution dynamic scanning scheme is used to obtain visual sub-path two image for the surface of the plastic mold. The data analysis module constructs a ray tracing model, introduces a lightweight graph neural network into the ray tracing model to obtain an improved ray tracing model, and inputs the pre-calibrated plastic refractive deviation and visual sub-path image into the improved ray tracing model to obtain the geometric features of the bubble defect. Specifically, the process involves: subtracting two images from adjacent angles to obtain a difference image; calling an improved ray tracing model; inputting the multi-angle preprocessed difference image; using the ray deflection data output by the improved ray tracing model through a graph neural network to locate the coordinates of optical nodes in the deflection set and determine the two-dimensional position of the bubble; then calculating the equivalent radius of the bubble based on the coverage of ray deflection at different angles; and combining the differences in deflection in multiple directions to infer the three-dimensional ellipsoidal parameters of the bubble and obtain the geometric characteristics of the bubble defect. The U-Net_s segmentation network processes the image of visual sub-path 2 and outputs a defect mask, eliminating noisy connected components smaller than 5 pixels; for warped defects, the mask boundary curve is fitted and the average curvature is calculated; for short-shot defects, the ratio of the mask area to the expected mold filling area is calculated; at the same time, the pixel distribution of the defect mask is statistically analyzed to determine the two-dimensional position and shape contour of the surface defect, and the geometric features of the surface defect are obtained. By combining the geometric features of bubble defects and surface defects, a unified set of defect geometric features containing both bubbles and surface defects is formed. The data fusion module performs multi-layer perception processing on a unified set of defect geometric features, uses an attention mechanism to adaptively weight the processed defect geometric features, and fuses pressure and temperature parameters to learn a general representation. The early warning and control module performs defect classification, prediction, and closed-loop adjustment based on general characterization, and uses long short-term memory networks to predict defects and automatically optimize control parameters.

2. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 1, characterized in that: The acquisition of multimodal sensor data includes: Visual sub-path one captures images of bubble refraction through illumination from multiple angles; visual sub-path two scans images of the mold surface using a camera. The injection pressure is monitored in real time using a pressure sensor array, and the injection pressure sequence is output. The temperature of the molten plastic is acquired in real time through a distributed temperature sensor network, and the temperature sequence of the molten plastic is output.

3. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 2, characterized in that: The process of constructing the ray tracing model is as follows: By tracing the path of light propagation, the system captures the refraction disturbance of light caused by the bubble, and calculates the position, size, and three-dimensional shape of the bubble; at the same time, it obtains the wavelength of the LED light source, the refractive index of the transparent plastic, the initial refractive index of the bubble, as well as the camera's installation position and imaging parameters. By simulating the light emission angle of each LED light source, the light is set as a parallel beam. When the light enters the transparent plastic from the air, the first deflection of the light direction is simulated according to the optical refraction law. The propagation path of the deflected light inside the plastic is calculated. When the light encounters the bubble boundary, a second deflection occurs due to the difference in refractive index between the plastic and the bubble. The propagation path of each beam of light inside the bubble is tracked, and the change in direction of the light after passing through the bubble is recorded. When light passes through the plastic and out into the air again, it is deflected for the third time and propagates to the camera sensor. The pixel position of each ray of light reaching the camera is recorded and matched with the actual image pixels to establish a mapping relationship between the ray trajectory and the image pixels, which is the ray tracing model.

4. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 3, characterized in that: The process of obtaining the improved ray tracing model is as follows: The 3D physical space of ray tracing is abstracted into a graph structure that can be processed by a graph neural network, and ray nodes and medium nodes are set separately. Based on the standard GAT, the number of attention heads is pruned and the feature dimension is reduced, and local neighborhood sampling is performed. The interaction rules between rays and medium are learned through the message passing mechanism of the graph neural network, replacing the existing ray-by-ray calculation. The final deflection direction and deflection intensity of each ray node are output through multiple rounds of message passing.

5. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 1, characterized in that: The process of performing multi-layer sensing processing on the unified defect geometric feature set is as follows: By standardizing and unifying the geometric features of bubbles and surface defects and mapping them with depth features using a multilayer perceptron, the geometric features of the two types of defects are combined to form the input vector of the multilayer perceptron. The 12-dimensional standardized features are mapped to 256 dimensions through a fully connected layer. The basic fusion features are extracted using the ReLU activation function to enhance feature interaction, capture the potential correlation between bubble and surface defect features, compress the feature dimension, and output a unified feature vector of 128 dimensions.

6. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 5, characterized in that: The process of learning the general representation is as follows: A unified 128-dimensional feature vector, pressure parameter, and temperature parameter are obtained respectively. The importance weights of the three types of features are dynamically calculated through an attention layer. The three types of features are used as value vectors V, and corresponding query vectors Q and key vectors K are generated. Q and K are mapped from the input features by a fully connected layer. The similarity of each modality is calculated through similarity calculation, and softmax processing is applied to the similarity results to obtain the weights of the three types of features. Each type of feature vector is multiplied by the corresponding attention weight to obtain the weighted geometric feature, pressure feature, and temperature feature. The weighted three types of features are concatenated and input into a fully connected layer. The ReLU activation function is used to perform interaction between features, and a 64-dimensional fused feature is output. The fused feature is normalized by a BatchNorm layer to generate the final 64-dimensional general representation.

7. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 1, characterized in that: The defect classification prediction and closed-loop adjustment process is as follows: The 64-dimensional general representation is input into the defect classifier to identify the current defect type. A softmax multi-class classification structure is adopted. The input is a general representation that integrates defect geometric features with pressure and temperature parameters. Deep correlation features are extracted through two fully connected layers. The output is the probability of bubble, warpage and short shot defect types, and the production line is adjusted in a closed loop according to the defects.

8. The intelligent defect early warning and control system for injection molding based on multimodal sensor fusion according to claim 1, characterized in that: The process of automatically optimizing control parameters is as follows: Based on historical general characterization sequences, the defect risk of future cycles is predicted. The input is the general characterization sequence of the next 10 injection molding cycles. After being converted into 128-dimensional time-series features by the embedding layer, it is input into the LSTM network. The LSTM outputs the probability of each defect in the next 3 cycles by memorizing the correlation between historical defects and process parameters. When the predicted probability of future defects exceeds the threshold, the linkage optimization of process parameters and sensing parameters is triggered.

Images