LPBF multi-spectrum composite multi-mode online monitoring method and LPBF multi-spectrum composite multi-mode online monitoring device

Abstract

The invention is suitable for the field of additive manufacturing, and provides an LPBF multispectral composite multi-mode online monitoring method and device, and the method comprises the steps: S1, constructing a multispectral composite monitoring sensor device; the method comprises the following steps: S1, acquiring multi-spectral monitoring information of each layer of a machined part by using a multi-spectral monitoring sensor device according to a spectral information acquisition step, S2, performing multi-modal feature extraction and cross-modal attention interaction by using a multi-branch Transform architecture, S4, performing multi-modal feature extraction and cross-modal attention interaction by using a multi-branch Transform architecture, and S5, performing multi-modal feature extraction and cross-modal attention interaction. And carrying out defect classification and defect size prediction by adopting a classification task or a regression task based on the multi-modal features. According to the invention, multispectral composite on-line monitoring is adopted, the comprehensiveness of information acquisition is improved, and the acquisition of image information of different spectrums is realized. According to the method, the multi-modal feature extraction and the cross-modal attention interaction are carried out by adopting the multi-branch Transform architecture, so that the accuracy and the real-time performance of defect prediction in the LPBF processing process are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of additive manufacturing, and in particular relates to a multi-spectral composite multi-modal online monitoring method and device for laser powder bed melting. Background Art

[0002] Although laser powder bed fusion (LPBF) has been widely used in the manufacture of complex components for various types of equipment due to its significant advantages in the precision manufacturing of complex structures, the process is characterized by complex thermophysical and non-equilibrium metallurgical processes, which can easily lead to internal quality defects such as porosity, lack of fusion, and microcracks. Due to the potential for these internal quality defects, non-destructive testing (NDT) such as X-rays or industrial CT is often required before delivery of aviation parts. However, NDT is extremely expensive and can only be performed offline after the fact. Furthermore, due to the complex control systems and their heavy reliance on process plans, occasional manufacturing anomalies in LPBF manufacturing systems can cause significant losses due to equipment stability and lack of process personnel experience. These include powder supply shortages caused by powder supply system anomalies and part warpage caused by inappropriate process plans. Therefore, the development of online monitoring and identification technologies for fusion defects is urgently needed. This would reduce the frequency of costly NDT testing and allow for timely containment or intervention when manufacturing anomalies occur.

[0003] Visual monitoring technology has a certain degree of feasibility. Currently proposed visual monitoring methods mainly include two approaches: visible light monitoring using CCD or CMOS cameras, and infrared monitoring using thermal imagers. However, each approach has its own shortcomings. Using CCD or CMOS for visible light image monitoring presents numerous drawbacks. For example, surface monitoring limitations: visible light can only monitor information on the melt surface and cannot directly detect defects within the structure. For example, sensitivity to melt pool brightness: During the LPBF process, the light intensity in the melt pool is very high, which can easily cause camera overexposure and loss of image detail. Furthermore, thermal radiation and spatter surrounding the melt pool can interfere with image quality. For example, resolution limitations: visible light cameras may have insufficient resolution for accurate identification of small defects (such as microcracks). Inability to reflect thermal processes: Visible light monitoring cannot directly reflect the melt pool's heat distribution and cooling process, information that is crucial for determining the formation of metallurgical defects (such as thermal cracks). Furthermore, ambient light interference: In industrial environments, changes in ambient light can interfere with visible light monitoring, requiring additional shielding and calibration measures.

[0004] However, the use of infrared thermal imagers for infrared temperature measurement technology also has limitations. For example, the influence of surface emissivity: the measurement accuracy of infrared thermal imagers is greatly affected by the surface emissivity of the material. During the LPBF process, the surface state of the molten pool (such as oxidation and roughness) will continue to change, resulting in unstable emissivity, which in turn affects the accuracy of temperature measurement. For example, the spatial resolution is insufficient: the spatial resolution of infrared thermal imagers is usually low. A typical thermal imager has only 1280×1024 pixels, which makes it difficult to accurately capture the temperature distribution of small areas of the molten pool (such as microcracks or unfused areas). For example, thermal radiation interference: thermal radiation, spatter, and environmental heat sources around the molten pool may interfere with the measurement results of the infrared thermal imager, leading to misjudgment. For example, data processing is complex: the amount of thermal image data generated by the infrared thermal imager is large, and complex algorithms are required to analyze changes in the temperature field, which may not be real-time.

[0005] Clearly, monitoring methods using a single spectral band have significant limitations. Furthermore, simply integrating a visible light camera (CCD / CMOS) and an infrared thermal imager into the equipment, using the CMOS / CCD to capture cross-sections of the powdering and forming layers, respectively, and using a thermal imager to capture the temperature field during forming, also present significant drawbacks. First, visible light and infrared images are data-isolated. Conventional CMOS / CCD resolutions typically range from millions to tens of millions of pixels, while infrared thermal imagers typically only have a resolution of 1280 × 1024 pixels. This significant resolution difference makes it difficult to establish a precise physical spatial correlation. Furthermore, if only a simple spatial registration criterion is used, only physical spatial registration is achieved, rather than information fusion. This makes it impossible to perform cross-modal feature coupling analysis of defects, thus failing to overcome the limitations of characterizing internal defects and thermal process characteristics. Summary of the Invention

[0006] In response to the above technical problems, the present invention proposes a multi-spectral composite multi-modal online monitoring method and device. This method can break through the bottleneck of a single information dimension, and combine with a deep learning algorithm to realize cross-modal feature analysis of the powder bed melting process, thereby improving the information comprehensiveness, sensitivity and reliability of online monitoring.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-spectral composite multi-modal online monitoring method for LPBF includes the following steps:

[0009] Step S1: construct a multi-spectral composite monitoring sensor device, use a single image sensor to achieve imaging in multiple bands, and perform image data analysis and processing to ensure the consistency of image pixel accuracy in different spectrums;

[0010] Step S2: The multispectral monitoring sensor device is installed above or to the side of the LPBF equipment forming chamber in a paraxial integration manner to ensure that the sensor's field of view can achieve panoramic coverage of the forming cross section;

[0011] Based on the processing sequence of the parts, the multispectral monitoring sensor device is used to obtain the multispectral monitoring information of each layer of the processed parts according to the spectral information collection steps;

[0012] Step S3: using a multi-branch Transformer architecture to perform multimodal feature extraction and cross-modal attention interaction; the multi-branch Transformer architecture includes an input layer, a branch extraction layer, a concatenate layer, and an output layer; the branch extraction layer includes an image branch module, a time series branch module, and a process vector branch module;

[0013] After obtaining the feature vectors of the branch extraction layer, a cross-modal feature fusion module is used to perform a cross-modal feature fusion operation to combine the feature vectors of different modalities and use the information of each modality to capture the potential relationship therein;

[0014] The cross-modal attention mechanism is used to learn the importance of each modality in the defect prediction task, enabling the model to dynamically adjust the influence of different modal features based on the contribution of each modality to defect prediction;

[0015] Step S4: Based on the multimodal features, a classification task or a regression task is used to classify defects of printed parts and predict defect sizes.

[0016] Preferably, the present invention provides a multi-spectral composite monitoring sensor device, comprising a housing 101, a protective window 102, a filter holder 103, a filter 104, a data transmission interface 105, a CMOS or CCD sensor 106, a lens 107, a servo 108, a control arm 109, a data cable 110 and an acquisition computing card 111; the housing 101 is internally installed with a CMOS or CCD sensor 106; the optical protective window 102 can transmit all wavelengths from 400nm to 1000nm; the housing 101 is equipped with a data transmission cable interface 105 for exposing the data interface on the internal acquisition computing card 111 and connecting it to a printer during subsequent device integration. The filter holder 103 is connected to the IPC of the machine; the filter 104 with a near-infrared band having a fixed wavelength range of 900nm-1000nm is installed; the spectral response range of the CMOS or CCD image sensor 106 covers both the visible light band of 400nm-700nm and the near-infrared band of 900nm-1000nm, and the quantum efficiency of the near-infrared band of 900nm-1000nm is ≥10% and the resolution is ≥20M pixels; the optical lens 107 adopts an array high-resolution fixed-focus lens with a resolution of ≥20M pixels; the electric servo 108 controls the filter holder 103 and the near-infrared band filter 104 through the control arm 109.

[0017] Furthermore, the spectral information collection step described in S2 specifically includes the following steps:

[0018] Step S21: In the initial printing state, the filter holder in the multispectral sensor device is in a control posture for visible light imaging;

[0019] Step S22: After the part surface is powdered, the printer IPC sends a command to the multi-spectral monitoring sensor device, and the multi-spectral monitoring sensor device captures a visible light image;

[0020] Step S23: After completing visible light image capture, the servo 108 releases the control arm 109, placing the filter holder 103 and the filter 104 thereon in front of the lens 107, thereby achieving near-infrared filtered imaging. At this time, the laser beam 8 is controlled by the laser deflection and focusing system to scan the molten metal powder according to the scanning path. Simultaneously, the multispectral monitoring sensor device captures the near-infrared radiation spectrum energy of the laser melt in real time, and utilizes the acquisition computing card 111 in combination with signal processing and image fusion technology to fuse the captured energy changes in real time into a gradient image.

[0021] At the end of the scan, step S24 forms a plurality of near-infrared tomographic fusion images, i.e., obtains a plurality of melting thermal history images during the laser processing process. Simultaneously, the multispectral monitoring sensor device controls the filter holder 103 to be in a visible light imaging posture to obtain a visible light image after melting, thereby completing the processing of a layer of cross-section of the printed part and the collection of multispectral information.

[0022] Step S25: Each layer of the printed part then executes the process of steps S21-S24, thereby completing the processing of each layer of the printed part and obtaining the multi-spectral monitoring information.

[0023] Furthermore, the input layer inputs visible light images after powder spreading, visible light images after melting, near-infrared thermal history images, scanning vector parameters and time dimension data.

[0024] Furthermore, the input data of the image branch module are the visible light image of the input layer, the thermal history image data, and the two-dimensional image drawn by the current layer scanning path. The image branch processing flow includes the following steps:

[0025] S51 spatial alignment, which registers the visible light image and thermal history image to the two-dimensional image of the scanning path through radiometric transformation, so that the three are in a unified coordinate system;

[0026] S52 constructs a multi-stream CNN model, which includes a global CNN model and two lightweight CNN models. The global CNN model is used to identify changes in the spatial physical environment; one lightweight CNN model extracts local thermal history abnormal areas, and the other lightweight CNN model extracts abnormal processing conditions areas;

[0027] S53 finally merges the multi-stream features through the Concatenate layer and outputs the image feature vector.

[0028] Preferably, the timing branch module is used to capture the timing changes and their evolution during the processing process, especially the impact of the thermal cycle effect accumulated layer by layer on the fused microstructure, to identify the long-range dependence across layers. The processing flow includes the following steps:

[0029] S61, constructing spatiotemporal features, combining the image feature vector of the current nth layer and the image feature vector of the previous ni layers into an N-layer × D-dimensional time series tensor data and inputting it into the time series branch module;

[0030] After S62 inputs the time series data into the Transformer, it performs parallel computations through multiple independent attention heads. Each head can learn different attention patterns, and ultimately their outputs are concatenated or weighted together to form a richer time series feature representation.

[0031] S63. Input the temporal features calculated by multiple independent attention heads into a feedforward neural network to further extract nonlinear features. In this process, residual connections are used to avoid the vanishing gradient problem and maintain information flow.

[0032] S64. Output the feature vectors that integrate the temporal dependencies. These feature vectors can reflect the temporal evolution characteristics of each layer during the processing.

[0033] Preferably, the input data of the process vector branch module mainly comes from the structured process parameters of the LPBF equipment itself. The input data includes geometric parameters, energy parameters, and environmental parameters. The processing flow of the process vector branch module includes the following steps:

[0034] S71. Perform data preprocessing to standardize or normalize process parameters of different dimensions and numerical ranges;

[0035] S72 uses an embedding layer to convert discrete process parameters into dense, high-dimensional vector representations, allowing discrete data to be effectively processed together with continuous data. Continuous parameters are directly processed through fully connected layers.

[0036] S73 uses a multi-layer perceptron model to learn the high-dimensional feature representation of process parameters, represents these process parameters into high-dimensional feature vectors, and then passes them to the cross-modal fusion module together with the feature vectors of the image branch module and the timing branch module.

[0037] Furthermore, after obtaining the feature vectors of the three branch modules, the cross-modal fusion module combines the feature vectors of different modalities, analyzes the information of each modality, and captures the potential relationship therein. Specifically, the feature vectors f obtained by the image branch module, the time series branch module, and the process branch module are combined. image 、f time and f peocess , splicing along the feature dimension to obtain a higher dimensional fusion feature vector f fused , the fused feature vector ffused contains information of all modalities, and its length is the sum of the three branch-specific dimensions. The expression is as follows:

[0038] f fused =[f image ,f time ,f peocess ].

[0039] Furthermore, defect classification and defect size prediction using classification or regression tasks based on multimodal features include the following steps:

[0040] S91 inputs the cross-modal fused feature vector into the fully connected layer of defect prediction, performs nonlinear transformation and combination on the feature vector, and further extracts high-level abstract features;

[0041] S92 uses nonlinear activation functions to enhance the nonlinear expression ability of the model;

[0042] S93 passes the feature vectors processed by multiple fully connected layers into the output layer. When the task objective is defined as predicting specific numerical information about the defect, such as the area, length, width or position of the defect, a regression task is adopted, and the output layer uses a linear activation function to output a specific numerical value. When the task objective is defined as the category or probability distribution of the defect, the output layer uses a Softmax activation function to output the predicted defect category and the corresponding probability value.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Compared with images obtained by single-band information, the present invention adopts multi-spectral composite online monitoring, which not only monitors the nodes after powder laying and melting, but also collects thermal history data during the printing process, covering the entire processing and printing process, improving the comprehensiveness of the collected information, and filling the information gap of current online monitoring technology for melting process monitoring.

[0045] 2. The integrated design of a single sensor enables the acquisition of image information of different spectra without increasing the space occupied by the equipment; at the same time, it overcomes the problem of mismatched pixel accuracy of different spectral images obtained by conventional means.

[0046] 3. This paper adopts a multi-branch Transformer architecture for multimodal feature extraction and cross-modal attention interaction, which significantly improves the accuracy and real-time performance of defect prediction in the LPBF (laser powder bed fusion) processing process.

[0047] 4. The present invention introduces a timing branch into the multi-branch Transformer architecture, which can realize cross-layer capture of the spatiotemporal evolution of the processing process, and models the dependency relationship between each layer in the processing process through a multi-head self-attention mechanism in the timing branch, effectively capturing the spatiotemporal evolution of the processing process and improving the ability to predict defect evolution.

[0048] 5. The present invention supports multi-task learning and can realize the prediction of different types of defects. Through the flexible setting of regression and classification tasks, it can realize both quantitative prediction of defects (defect size information) and qualitative prediction of defects (classification and probability distribution).

[0049] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Schematic diagram of a LPBF multi-spectral composite multi-modal online monitoring method of the present invention;

[0051] Figure 2 This is a structural disassembly diagram of the multi-spectral composite online monitoring sensor device;

[0052] Figure 3 To observe the interior of the sensor device after removing the optical protection window;

[0053] Figure 4 This is the filter control posture diagram for visible light imaging;

[0054] Figure 5 A diagram showing the structure of a multi-spectral composite monitoring sensor integrated into a laser powder bed fusion printing device;

[0055] Figure 6 This is a schematic diagram of the multi-branch Transformer architecture of the present invention;

[0056] Figure 7 Schematic diagram of the attention mechanism of the multi-branch Transformer architecture of the present invention;

[0057] Figure 8 A flowchart for defect classification and defect size prediction using classification tasks or regression tasks based on multimodal features;

[0058] 1 forming chamber; 2 powder; 3 powder spreading mechanism; 4 feeding mechanism; 5 forming platform; 6 formed part; 7 laser; 8 laser beam; 9 laser deflection and focusing system; 10 multispectral composite monitoring sensor; 101 multispectral sensor housing; 102 protective window; 103 filter holder; 104 filter; 105 data transmission interface; 106 CMOS or CCD sensor; 107 lens; 108 servo; 109 control arm; 110 data cable; 111 acquisition computing card. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] The specific implementation of the present invention is described in detail below with reference to specific embodiments. Specific embodiment 1

[0062] See also Figure 1 The present invention provides a LPBF multi-spectral composite multi-modal online monitoring method, comprising the following steps:

[0063] Step S1: construct a multi-spectral composite monitoring sensor device, use a single image sensor to achieve imaging in multiple bands, and perform image data analysis and processing to ensure the consistency of image pixel accuracy in different spectrums;

[0064] Step S2: The multispectral monitoring sensor device is installed above or to the side of the LPBF equipment forming chamber in a paraxial integration manner to ensure that the sensor's field of view can achieve panoramic coverage of the forming cross section;

[0065] Based on the processing sequence of the parts, the multispectral monitoring sensor device is used to obtain the multispectral monitoring information of each layer of the processed parts according to the spectral information collection steps;

[0066] Step S3: using a multi-branch Transformer architecture to perform multimodal feature extraction and cross-modal attention interaction; the multi-branch Transformer architecture includes an input layer, a branch extraction layer, a concatenate layer, and an output layer; the branch extraction layer includes an image branch module, a time series branch module, and a process vector branch module;

[0067] After obtaining the feature vectors of the branch extraction layer, a cross-modal feature fusion module is used to perform a cross-modal feature fusion operation to combine the feature vectors of different modalities and use the information of each modality to capture the potential relationship therein;

[0068] The cross-modal attention mechanism is used to learn the importance of each modality in the defect prediction task, enabling the model to dynamically adjust the influence of different modal features based on the contribution of each modality to defect prediction;

[0069] Step S4: Based on the multimodal features, a classification task or a regression task is used to classify defects of printed parts and predict defect sizes. Specific embodiment 2

[0071] As attached Figure 2-5As shown, the present invention provides a multi-spectral composite monitoring sensor device for realizing a multi-spectral composite multi-modal online monitoring method for LPBF, the multi-spectral composite monitoring sensor device includes a housing 101, a protective window 102, a filter holder 103, a filter 104, a data transmission interface 105, a CMOS or CCD sensor 106, a lens 107, a servo 108, a control arm 109, a data cable 110 and an acquisition computing card 111; the housing 101 is internally installed with a CMOS or CCD sensor 106; the optical protective window 102 can transmit all wavelengths from 400nm to 1000nm; the housing 101 is installed with a data transmission cable interface 105 for transmitting data on the internal acquisition computing card 111 to the optical protection window 102; the optical protection ... The interface is exposed and connected to the IPC of the printer during subsequent device integration; the filter fixture 103 is installed with a filter 104 with a fixed wavelength range of 900nm-1000nm in the near-infrared band; the spectral response range of the CMOS or CCD image sensor 106 covers both the visible light band of 400nm-700nm and the near-infrared band of 900nm-1000nm, and the quantum efficiency of the near-infrared band of 900nm-1000nm is ≥10% and the resolution is ≥20M pixels; the optical lens 107 adopts a planar array high-resolution fixed-focus lens with a resolution of ≥20M pixels; the electric servo 108 controls the filter fixture 103 and the near-infrared band filter 104 through the control arm 109. When visible light imaging is required, the servo 108 controls the control arm 109 to pull the filter holder 103 and the near-infrared band filter 104 so that they are not placed in front of the optical lens 107; and when near-infrared imaging is required, the servo 108 releases the control arm 109 to place the filter holder 103 and the near-infrared band filter 104 in front of the optical lens 107, thereby achieving near-infrared filtered imaging. Figure 1 As shown. A USB 3.0 or GigE cable 110 is used for digital image transmission, transmitting the captured visible light or near-infrared image to an image acquisition and computing card 111 in real time. Image acquisition and computing card 111 is used to fuse and analyze the captured visible light and near-infrared images and feed the processed results back to the host computer (IPC). Furthermore, to meet increased environmental adaptability requirements, the sensor device can also be designed with a nitrogen positive pressure dustproof structure and water or air cooling.

[0072] The multispectral monitoring sensor device is installed above or on the side of the LPBF equipment forming chamber 1 in a paraxial integrated manner. It is necessary to ensure that the sensor's field of view can achieve a panoramic coverage of the forming cross section. In the initial printing state, the filter holder in the multispectral sensor device is in the attached Figure 3The control posture for visible light imaging. After processing begins, the powder spreading mechanism applies powder 2 to the surface of the formed part 6. Once the powder spreading is complete, the printer IPC sends a command to the multispectral monitoring sensor device, which then captures a visible light image. After capturing the visible light image, the servo 108 in the multispectral sensor device releases the control arm 109, positioning the filter holder 103 and its filter 104 in front of the lens 107, thereby achieving near-infrared filtered imaging. At this point, the laser beam 8, controlled by the laser deflection and focusing system, scans the molten metal powder along the scanning path. Simultaneously, the sensor captures the near-infrared radiation spectral energy of the laser melt in real time. Using the acquisition computing card 111, signal processing, and image fusion technology, the captured energy variations are fused in real time into a gradient image. At the end of the scan, multiple near-infrared tomographic fusion images are generated, such as a weighted integral fusion image and a weighted extreme value fusion image. This means that multiple images of the melting thermal history are captured during the laser processing process. Simultaneously, the filter holder is controlled to a visible light imaging posture to capture a visible light image after melting. The above process completes the processing of one layer of the part cross-section and the collection of multispectral information. The same process is performed on each subsequent layer to complete the processing of each layer and the acquisition of multispectral monitoring information.

[0073] Through the above two steps, it has been achieved to obtain multispectral image information of the LPBF processing process layer by layer. However, since LPBF is a cyclic thermal processing process, there is the influence of non-equilibrium thermal cycle process during the layer-by-layer processing, so the molten microstructure and defects will evolve. The existing monitoring methods only focus on the acquisition of current layer data and cannot capture the spatiotemporal evolution of microstructural features. To solve this problem, the present invention uses a multi-branch Transformer architecture to extract modal-specific features and perform cross-modal attention interaction fusion on the heterogeneous data (visible light and thermal history images, process time series, process vectors) generated during the LPBF processing.

[0074] The input layer of the multi-branch Transformer architecture contains two visible light images (one after powder application and one after melting), a near-infrared thermal history image (a thermal history gradient fusion map captured by a multispectral sensor, a weighted integral map, and a weighted extreme value map, etc.), scanning vector parameters (structured data such as laser power, scanning rate, and path coordinates), and a time dimension (sequence data with the number of layers as the time dimension).

[0075] The branch extraction layer of the multi-branch Transformer architecture mainly includes three branches, namely image branch, timing branch and process vector branch.

[0076] The input to the image branch is the visible light image of the current layer, the thermal history image data, and the two-dimensional image drawn from the scanning path of the current layer. The image branch's processing flow is as follows: First, spatial alignment is performed. Using a radiometric transformation, the visible light image and thermal history image are aligned to the two-dimensional image of the scanning path, aligning the three to a unified coordinate system. A multi-stream CNN model is then employed: a global CNN model is used to identify changes in the spatial physical environment, a lightweight CNN model (such as a Unet model or MobileNet-V3 model) extracts local thermal history anomalies (such as thermal history distribution anomalies), and another lightweight CNN model (such as a Unet model or MobileNet-V3 model) extracts processing condition anomalies (such as powder bed anomalies). Finally, a concatenate layer combines the multi-stream features to output an image feature vector.

[0077] The purpose of the timing branch is to capture the timing changes and their evolution during the processing, especially the impact of the layer-by-layer thermal cycling effects on the melt microstructure. Since LPBF is a dynamically changing thermal processing process, extracting timing information is crucial for predicting defect evolution.

[0078] The processing flow of the time series branch: First, the spatiotemporal features are constructed. To reduce redundant calculations, the image branch has completed image data processing and extracted rich features. Therefore, the image feature vector of the current nth layer and the image feature vector of the previous ni layers can be combined into an N-layer (depending on the number of previous layers) × D-dimensional time series tensor data and input into the time series branch.

[0079] Because the Transformer model can effectively capture the dependencies between data of long time spans based on the self-attention mechanism (Self-Attention), and automatically learn the weights of each moment, it focuses on those moments that have the greatest impact on the evolution of defects. The core of the self-attention mechanism is to calculate the relationship between the feature vector of each moment (for example, the nth layer) and the feature vectors of other moments (such as the n-1th layer, n-2th layer, etc.), and assign different attention levels through attention weights, automatically determining which moments have the greatest impact on the prediction of abnormal defects at the current moment. Therefore, in the present invention, after the time series data is input into the Transformer, it is calculated in parallel by multiple independent attention heads. Each head can learn different attention patterns, and finally their outputs are spliced ​​or weighted together to form a richer time series feature representation. The time series features calculated by multiple independent attention heads are then input into a feedforward neural network to further extract nonlinear features. In this process, residual connections are used to avoid the gradient disappearance problem and maintain the flow of information. The final output is a feature vector that integrates the time series dependencies. These feature vectors can reflect the time evolution characteristics of each layer during the processing process. This process can identify long-range dependencies across layers, such as the impact of abnormal heat distribution in layer n-5 on the current layer).

[0080] The designed multi-head self-attention mechanism refers to a "local-global" hierarchical attention mechanism. Local attention is achieved by setting the sliding window size (e.g., 3 layers), which focuses more on the thermal residual effect between adjacent layers. Global sparse attention, on the other hand, performs sparse sampling on distant layers (e.g., layers 5 or more apart) to reduce computational complexity.

[0081] The input data of the process vector branch mainly comes from the LPBF equipment itself, mainly structured process parameters, including geometric parameters (path coordinates, layer thickness, spot diameter, etc.), energy parameters (power, speed, energy density, etc.) and environmental parameters (substrate temperature, oxygen content, airflow velocity, etc.). First, data preprocessing is performed to standardize or normalize process parameters of different dimensions and numerical ranges. Then, the discrete process parameters are converted into dense, high-dimensional vector representations using an embedding layer, so that discrete data can be effectively processed together with continuous data. For continuous parameters (such as laser power, scanning speed, ambient temperature, etc.), they are directly processed through the fully connected layer. A multi-layer perceptron model is used to learn the high-dimensional feature representation of the process parameters, and these process parameters are represented as high-dimensional feature vectors, which are subsequently passed to the cross-modal fusion module together with the feature vectors of the image branch and the timing branch.

[0082] After obtaining the feature vectors of the three branches, cross-modal feature fusion is performed. This primarily combines the feature vectors of different modalities (image, time series, and process), leveraging the information from each modality and capturing potential relationships. The first step is feature vector concatenation. Feature vectors fimage, ftime, and fprocess are obtained from the image branch, time series branch, and process branch, respectively. These vectors are concatenated along the feature dimensions to produce a higher-dimensional fused feature vector, ffused. This fused feature vector contains information from all modalities and has a length equal to the sum of the feature dimensions of the three branches.

[0083] f fused =[f image ,f time ,f peocess ]

[0084] In addition to concatenating the feature vectors of the three branches, this paper uses a cross-modal attention mechanism to better capture the interaction between different modalities. This mechanism learns the importance of each modality in the defect prediction task, enabling the model to dynamically adjust the impact of different modal features based on the contribution of each modality to defect prediction.

[0085] Furthermore, classification tasks or regression tasks are used based on multimodal features to predict defect classification and defect size. Specifically, the feature vector after cross-modal fusion is input into the fully connected layer (MLP) of defect prediction, and the feature vector is nonlinearly transformed and combined to further extract high-level abstract features, and a nonlinear activation function (such as ReLU) is used to enhance the nonlinear expression ability of the model. The feature vector processed by multiple fully connected layers enters the output layer. When the task goal is defined as predicting specific numerical information about the defect, such as the area, length, width or position of the defect, a regression task is used, and the output layer uses a linear activation function to output a specific value. When the task goal is defined as the category or probability distribution of the defect, the output layer uses the Softmax activation function to output the predicted defect category and the corresponding probability value.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A LPBF multi-spectral composite multi-modal online monitoring method, characterized in that: The following steps are involved: Step S1: construct a multi-spectral composite monitoring sensor device, use a single image sensor to achieve imaging in multiple bands, and perform image data analysis and processing to ensure the consistency of image pixel accuracy in different spectrums; Step S2: The multispectral monitoring sensor device is installed above or to the side of the LPBF equipment forming chamber in a paraxial integration manner to ensure that the sensor's field of view can achieve panoramic coverage of the forming cross section; Based on the processing sequence of the parts, the multispectral monitoring sensor device is used to obtain the multispectral monitoring information of each layer of the processed parts according to the spectral information collection steps; Step S3: using a multi-branch Transformer architecture to perform multimodal feature extraction and cross-modal attention interaction; the multi-branch Transformer architecture includes an input layer, a branch extraction layer, a concatenate layer, and an output layer; the branch extraction layer includes an image branch module, a time series branch module, and a process vector branch module; After obtaining the feature vectors of the branch extraction layer, a cross-modal feature fusion module is used to perform a cross-modal feature fusion operation to combine the feature vectors of different modalities and use the information of each modality to capture the potential relationship therein; The cross-modal attention mechanism is used to learn the importance of each modality in the defect prediction task, enabling the model to dynamically adjust the influence of different modal features based on the contribution of each modality to defect prediction; Step S4: Based on the multimodal features, a classification task or a regression task is used to classify defects of printed parts and predict defect sizes.

2. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 1, characterized in that: The multi-spectral composite monitoring sensor device comprises a housing (101), a protective window (102), a filter holder (103), a filter (104), a data transmission interface (105), a CMOS or CCD sensor (106), a lens (107), a steering gear (108), a control arm (109), a data cable (110) and an acquisition and calculation card (111); the housing (101) is internally installed with a CMOS or CCD sensor (106); the optical protective window (102) can transmit all wavelengths from 400 nm to 1000 nm; the housing 101 is installed with a data transmission cable interface 105 for exposing a data interface on the internal acquisition and calculation card 111 and When the device is subsequently integrated, it is connected to the printer's IPC; the filter fixture 103 is installed with a filter 104 with a fixed wavelength range of 900nm-1000nm in the near-infrared band; the spectral response range of the CMOS or CCD image sensor 106 covers both the visible light band of 400nm-700nm and the near-infrared band of 900nm-1000nm, and the quantum efficiency of the near-infrared band of 900nm-1000nm is ≥10% and the resolution is ≥20M pixels; the optical lens 107 adopts an array high-resolution fixed-focus lens with a resolution of ≥20M pixels; the electric servo 108 controls the filter fixture 103 and the near-infrared band filter 104 through the control arm 109.

3. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 2, characterized in that: The spectral information acquisition step described in step S2 specifically includes the following steps: Step S21: In the initial printing state, the filter holder in the multispectral sensor device is in a control posture for visible light imaging; Step S22: After the part surface is powdered, the printer IPC sends a command to the multi-spectral monitoring sensor device, and the multi-spectral monitoring sensor device captures a visible light image; Step S23: After completing the visible light image capture, the servo (108) releases the control arm (109), so that the filter holder (103) and the filter (104) thereon are placed in front of the lens (107), thereby realizing near-infrared filter imaging; at this time, the laser beam (8) is controlled by the laser deflection and focusing system to scan the molten metal powder according to the scanning path; at the same time, the multi-spectral monitoring sensor device captures the near-infrared radiation spectrum energy of the laser melt in real time, and uses the acquisition calculation card (111) in combination with signal processing and image fusion technology to fuse the captured energy changes in real time into a gradient image; At the end of the scanning, step S24 forms a plurality of near-infrared tomographic fusion images, i.e., obtains a plurality of melting thermal history images during the laser processing process; at the same time, the multispectral monitoring sensor device controls the filter holder (103) to be in a visible light imaging posture, obtains a visible light image after melting, and completes the processing of a layer of cross-section of the printed part and the collection of multispectral information; Step S25: Each layer of the printed part then executes the process of steps S21-S24, thereby completing the processing of each layer of the printed part and obtaining the multi-spectral monitoring information.

4. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 2, characterized in that: The input layer inputs visible light images after powder spreading, visible light images after melting, near-infrared thermal history images, scanning vector parameters and time dimension data.

5. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 4, characterized in that: The input data of the image branch module are the visible light image of the input layer, the thermal history image data and the two-dimensional image drawn by the current layer scanning path. The processing flow of the image branch includes the following steps: S51 spatial alignment, which registers the visible light image and thermal history image to the two-dimensional image of the scanning path through radiometric transformation, so that the three are in a unified coordinate system; S52 constructs a multi-stream CNN model, which includes a global CNN model and two lightweight CNN models. The global CNN model is used to identify changes in the spatial physical environment; one lightweight CNN model extracts local thermal history abnormal areas, and the other lightweight CNN model extracts abnormal processing conditions areas; S53 finally merges the multi-stream features through the Concatenate layer and outputs the image feature vector.

6. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 4, characterized in that: The timing branch module is used to capture the timing changes and their evolution during the processing process, especially the impact of the thermal cycling effect accumulated layer by layer on the fused microstructure, and to identify the long-range dependence across layers. The processing flow includes the following steps: S61, constructing spatiotemporal features, combining the image feature vector of the current nth layer and the image feature vector of the previous ni layers into an N-layer × D-dimensional time series tensor data and inputting it into the time series branch module; After S62 inputs the time series data into the Transformer, it performs parallel computations through multiple independent attention heads. Each head can learn different attention patterns, and ultimately their outputs are concatenated or weighted together to form a richer time series feature representation. S63. Input the temporal features calculated by multiple independent attention heads into a feedforward neural network to further extract nonlinear features. In this process, residual connections are used to avoid the vanishing gradient problem and maintain information flow. S64. Output the feature vectors that integrate the temporal dependencies. These feature vectors can reflect the temporal evolution characteristics of each layer during the processing.

7. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 4, characterized in that: The input data of the process vector branch module mainly comes from the structured process parameters of the LPBF equipment itself. The input data includes geometric parameters, energy parameters, and environmental parameters. The processing flow of the process vector branch module includes the following steps: S71. Perform data preprocessing to standardize or normalize process parameters of different dimensions and numerical ranges; S72 uses an embedding layer to convert discrete process parameters into dense, high-dimensional vector representations, allowing discrete data to be effectively processed together with continuous data. Continuous parameters are directly processed through fully connected layers. S73 uses a multi-layer perceptron model to learn the high-dimensional feature representation of process parameters, represents these process parameters into high-dimensional feature vectors, and then passes them to the cross-modal fusion module together with the feature vectors of the image branch module and the timing branch module.

8. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 7, characterized in that: After obtaining the feature vectors of the three branch modules, the cross-modal fusion module combines the feature vectors of different modalities, analyzes the information of each modality, and captures the potential relationship between them. Specifically, the feature vectors f obtained by the image branch module, the time series branch module, and the process branch module are combined. image 、f time and f peocess , splicing along the feature dimension to obtain a higher dimensional fusion feature vector f fused , the fused feature vector fused contains information of all modalities, and its length is the sum of the three branch-specific dimensions. The expression is as follows: f fused =[f image ,f time ,f peocess ]。 9. The LPBF multi-spectral composite multi-modal online monitoring method according to claim 8, characterized in that: Defect classification and defect size prediction using classification or regression tasks based on multimodal features include the following steps: S91 inputs the cross-modal fused feature vector into the fully connected layer of defect prediction, performs nonlinear transformation and combination on the feature vector, and further extracts high-level abstract features; S92 uses nonlinear activation functions to enhance the nonlinear expression ability of the model; S93 passes the feature vectors processed by multiple fully connected layers into the output layer. When the task objective is defined as predicting specific numerical information about the defect, such as the area, length, width or position of the defect, a regression task is adopted, and the output layer uses a linear activation function to output a specific numerical value. When the task objective is defined as the category or probability distribution of the defect, the output layer uses a Softmax activation function to output the predicted defect category and the corresponding probability value.

Images