A multi-modal fusion laser directed energy deposition real-time monitoring method and system

Abstract

The application provides a multi-modal fusion laser directed energy deposition real-time monitoring method and system, and belongs to the technical field of additive manufacturing; the method comprises the following steps: synchronously triggering through a multi-source sensor to collect real-time data of a molten pool area in a laser directed energy deposition process; a distributed clock network is constructed, environment influence parameters are collected in real time in combination with a working condition sensing unit, dynamic modeling and correction are performed on asymmetric link delays in a clock synchronization process, and spatio-temporal alignment of multi-source sensor data is realized; after the multi-source sensor data subjected to spatio-temporal alignment is preprocessed, data level fusion, feature level fusion and decision level fusion operations are sequentially performed. The application can monitor the molten pool state in real time and accurately, and improves the quality and stability of the laser directed energy deposition process.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of additive manufacturing, and particularly relates to a multi-modal fusion laser directed energy deposition real-time monitoring method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Laser directed energy deposition (L-DED) as a very promising additive manufacturing technology, has unique advantages in manufacturing complex structures and high-performance parts. In the process of laser directed energy deposition, the multi-physical field synchronous monitoring of the molten pool state (temperature, morphology, kinetic characteristics) is the core to ensure the quality of the parts, but the existing technology has three key defects, and these defects are particularly prominent under dynamic working conditions:

[0004] (1) Poor spatiotemporal synchronization adaptability and large precision fluctuation: the asymmetric delay correction of the traditional IEEE 1588v2 protocol relies on a fixed mathematical model, without considering the influence of temperature dynamic changes (generally 300-800℃) and electromagnetic interference intensity fluctuations in the L-DED process on the link transmission characteristics, resulting in a synchronization accuracy fluctuation of more than 50% with the change of working conditions; and the spatial registration only relies on the static calibration of a single calibration plate, and cannot utilize the complementarity of multi-sensor data, thus failing to correct the thermal expansion and position drift of the sensor optical system under actual working conditions, and the registration error is often more than 50μm, which is difficult to meet the fine fusion requirements.

[0005] (2) Lack of intelligent adaptation in data fusion and insufficient physical rationality: the existing weighted fusion only combines the inherent accuracy of the sensor and the synchronization accuracy, without considering the influence of dynamic changes in working conditions on the effectiveness of the features; and the physical constraints are mostly injected by a single thermodynamic law, which cannot reflect the internal coupling relationship of the molten pool temperature, morphology and kinetic characteristics (such as temperature rise leading to expansion of the molten pool area and change of vibration frequency), resulting in that the fusion result is easy to deviate from the actual physical process, and the process control decision deviation occurs.

[0006] (3) Synchronization-fusion-feature extraction is disconnected and does not form a closed loop: there is a lack of linkage mechanism between synchronization accuracy, fusion quality and feature extraction depth, such as high-precision synchronization cannot fully utilize fine-grained features, and low-precision synchronization still uses complex feature extraction strategies, resulting in a mismatch between data processing delay and feature effectiveness, further affecting the real-time and accuracy of process control. SUMMARY

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a multi-modal fusion laser directed energy deposition real-time monitoring method and system, which can monitor the molten pool state in real time and accurately, and improve the quality and stability of the laser directed energy deposition process.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] The first aspect of this invention provides a method for real-time monitoring of laser-directed energy deposition using multimodal fusion.

[0010] A multimodal fusion method for real-time monitoring of laser-directed energy deposition includes:

[0011] Real-time data acquisition of the molten pool region during the laser directional energy deposition process is achieved through synchronous triggering of multiple source sensors.

[0012] A distributed clock network is constructed, and environmental impact parameters are collected in real time by the working condition sensing unit. The asymmetric link delay in the clock synchronization process is dynamically modeled and corrected to achieve spatiotemporal alignment of multi-source sensor data.

[0013] After preprocessing the spatiotemporally aligned multi-source sensor data, data-level fusion, feature-level fusion, and decision-level fusion operations are performed sequentially. Specifically, the data-level fusion employs a three-dimensional dynamic weighting strategy to weight the multi-source sensor data. The feature-level fusion uses a lightweight neural network model deployed on edge devices to extract the geometric, thermodynamic, and dynamic features of the molten pool from the fused multi-source sensor data and construct a feature correlation matrix. Simultaneously, constraint loss based on multi-physics coupling laws is injected into the model training. The decision-level fusion combines prior knowledge of physical processes, probabilistic reasoning models, and evidence theory to comprehensively evaluate and decide on the state of the molten pool.

[0014] Furthermore, the multi-source sensor includes: an infrared camera for acquiring temperature field data, an X-ray detector for acquiring topographic field data including keyhole topography, an acoustic emission sensor for acquiring dynamic characteristics including oscillation signals, and a spectrometer for acquiring spectral information including plasma spectra.

[0015] Furthermore, the distributed clock network is constructed using the IEEE 1588v2 protocol, which achieves synchronization based on a delay response mechanism, specifically by using timestamps carried in IEEE 1588v2 messages; wherein, the IEEE 1588v2 messages include synchronization messages, follow messages, delay request messages, and delay request response messages.

[0016] Furthermore, the clock synchronization process of the distributed clock network includes: First, a synchronization message is periodically broadcast from the master clock, and the master clock records the transmission time t1 at the moment the transmission ends; then, the master clock encapsulates the timestamp corresponding to the transmission time t1 into a follow-up message and sends it to the slave clock; next, the slave clock records the time t2 when the synchronization message arrives at the slave clock; then, the slave clock sends a delay request message and records the transmission time t3 at the moment the transmission ends; next, the master clock records the arrival time t4 when the delay request message arrives at the master clock; finally, the master clock sends a delay request message carrying the timestamp information corresponding to the arrival time t4 to the slave clock.

[0017] Furthermore, after a complete clock synchronization process, the clock register stores four timestamps: t1, t2, t3, and t4. These four timestamps are used to calculate the link delay and clock skew.

[0018] Furthermore, the dynamic modeling and correction includes: defining a dynamic asymmetric delay correction amount, using the asymmetric delay correction amount to compensate for the average link delay and master-slave clock deviation calculated based on the symmetric delay assumption, and using a filtering algorithm to estimate the fluctuation of link delay and suppress noise in real time.

[0019] Furthermore, the decision-level fusion includes: injecting prior knowledge of the physical process of the molten pool as a physical constraint into the decision model; constructing a Bayesian network model to describe and infer the probabilistic dependencies between features; and applying DS evidence theory to fuse the molten pool state data provided by multiple sensors to obtain the evaluation result of the molten pool state.

[0020] A second aspect of the present invention provides a real-time monitoring system for laser-directed energy deposition based on multimodal fusion.

[0021] A multimodal fusion-based real-time monitoring system for laser-directed energy deposition includes:

[0022] The data acquisition module is configured to: acquire real-time data of the molten pool region during the laser directional energy deposition process by synchronously triggering multiple source sensors;

[0023] The spatiotemporal alignment module is configured to: construct a distributed clock network, combine the working condition sensing unit to collect environmental impact parameters in real time, and dynamically model and correct the asymmetric link delay in the clock synchronization process to achieve spatiotemporal alignment of multi-source sensor data;

[0024] The three-level fusion operation module is configured to: preprocess multi-source sensor data that has undergone spatiotemporal alignment, and then sequentially perform data-level fusion, feature-level fusion, and decision-level fusion operations; wherein, the data-level fusion adopts a three-dimensional dynamic weight strategy to weight the multi-source sensor data; the feature-level fusion extracts the geometric, thermodynamic, and dynamic features of the molten pool from the fused multi-source sensor data through a lightweight neural network model deployed on edge devices, and constructs a feature correlation matrix, while injecting constraint loss based on the coupling law of multi-physics fields into the model training; the decision-level fusion combines prior knowledge of physical processes, probabilistic reasoning models, and evidence theory to comprehensively evaluate and decide on the state of the molten pool.

[0025] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in the first aspect of the present invention.

[0026] The fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a multimodal fusion laser directional energy deposition real-time monitoring method as described in the first aspect of the present invention.

[0027] The above one or more technical solutions have the following beneficial effects:

[0028] (1) This invention constructs a distributed clock network, combines it with a working condition sensing unit to collect environmental impact parameters in real time, and dynamically models and corrects the asymmetric link delay during clock synchronization to achieve spatiotemporal alignment of multi-source sensor data. This effectively compensates for the impact of working condition fluctuations on the transmission link, breaks the limitations of static calibration in the prior art, and enables the time synchronization accuracy to remain stable at the nanosecond level under dynamic working conditions. The spatial registration error is greatly reduced, laying a precise spatiotemporal reference for the reliable fusion of subsequent multi-source data.

[0029] (2) This invention performs data-level fusion, feature-level fusion, and decision-level fusion operations sequentially on spatiotemporally aligned multi-source sensor data. The data-level fusion employs a three-dimensional dynamic weighting strategy to weight the multi-source sensor data. This weighting integrates the inherent accuracy of the sensors, real-time synchronization accuracy, and operational reliability based on the current feature signal-to-noise ratio, achieving intelligent dynamic allocation of data weights. Simultaneously, in both data-level and feature-level fusion, multi-physics coupling constraints are actively injected, embedding the inherent correlation between multiple physical quantities in the molten pool into the algorithm core in the form of a loss function or residual term. This design ensures that the fusion process and results always closely align with the actual physical process, significantly improving the physical rationality and decision credibility of the molten pool state estimation.

[0030] (3) This invention clearly divides the data processing flow into a three-level fusion process: data level, feature level, and decision level, and ensures that they are closely connected and work together. On the one hand, the high-precision synchronization and fusion results output from the previous level can provide high-quality input for feature extraction and decision-making in the subsequent level; on the other hand, the lightweight neural network is trained by introducing constraints containing physical coupling laws, and the deep features it extracts feed back into and verify the quality of the previous level fusion. This closed-loop linkage mechanism can adaptively adjust the processing strategy according to the real-time data quality, ensuring that computing resources are optimized under the premise of high data fidelity, and ultimately achieving real-time, accurate, and efficient monitoring and evaluation of the melt pool state.

[0031] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0033] Figure 1 This is a flowchart of a multimodal fusion method for real-time monitoring of laser directional energy deposition according to Embodiment 1 of the present invention.

[0034] Figure 2 This is a flowchart of spatiotemporal alignment of multi-source sensor data in Embodiment 1 of the present invention.

[0035] Figure 3 This is a modular flowchart of a multimodal fusion laser directional energy deposition real-time monitoring system according to Embodiment 2 of the present invention. Detailed Implementation

[0036] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0038] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0039] Example 1

[0040] This embodiment discloses a multimodal fusion-based real-time monitoring method for laser-directed energy deposition.

[0041] like Figure 1 As shown, a multimodal fusion method for real-time monitoring of laser directional energy deposition includes:

[0042] Step S1: Real-time data acquisition of the molten pool region during the laser directional energy deposition process is performed by synchronously triggering multiple source sensors.

[0043] Step S2: Construct a distributed clock network, combine it with the working condition sensing unit to collect environmental impact parameters in real time, and dynamically model and correct the asymmetric link delay in the clock synchronization process to achieve spatiotemporal alignment of multi-source sensor data.

[0044] Step S3: After preprocessing the spatiotemporally aligned multi-source sensor data, data-level fusion, feature-level fusion, and decision-level fusion operations are performed sequentially. Specifically, the data-level fusion employs a three-dimensional dynamic weighting strategy to weight the multi-source sensor data. The feature-level fusion uses a lightweight neural network model deployed on edge devices to extract the geometric, thermodynamic, and dynamic features of the molten pool from the fused multi-source sensor data and construct a feature correlation matrix. Simultaneously, constraint loss based on multi-physics coupling laws is injected into the model training. The decision-level fusion combines prior knowledge of physical processes, probabilistic reasoning models, and evidence theory to comprehensively evaluate and decide on the state of the molten pool.

[0045] Based on the above process, this invention can monitor the molten pool state in real time and accurately, improving the quality and stability of laser-directed energy deposition (LDED) processes. To facilitate understanding of the technical solution of this invention, the specific implementation methods are further explained and described below.

[0046] In step S1, real-time data acquisition is performed on the molten pool region during the laser directional energy deposition process by synchronously triggering multiple source sensors.

[0047] The multi-source sensors used include: an infrared camera for acquiring temperature field data, an X-ray detector for acquiring topographic field data including keyhole topography, an acoustic emission sensor for acquiring dynamic characteristics including oscillation signals, and a spectrometer for acquiring spectral information including plasma spectra.

[0048] In step S2, a distributed clock network is constructed, and environmental impact parameters are collected in real time by the working condition sensing unit. The asymmetric link delay in the clock synchronization process is dynamically modeled and corrected to achieve spatiotemporal alignment of multi-source sensor data.

[0049] A distributed clock network was constructed using the IEEE 1588v2 Precision Time Protocol (PTP) to achieve nanosecond-level time synchronization (deviation <100ns) for an infrared camera (100kHz), a high-speed X-ray detector (1MHz), an acoustic emission sensor (10MHz), and a spectrometer.

[0050] During the synchronization process, the master and slave clocks exchange precise timestamps and estimated timestamps in the form of messages. After receiving these timestamps, each slave clock calculates the link transmission delay and clock deviation with the master clock, and then adjusts the slave clock to achieve synchronization with the master clock.

[0051] The synchronization principle of the IEEE 1588v2 protocol is a delayed response mechanism, which is accomplished by carrying timestamps in IEEE 1588v2 messages. There are four types of IEEE 1588v2 messages: Sync message, Follow_up message, Delay_req message, and Delay_resp message.

[0052] like Figure 2 As shown, a complete clock synchronization process for spatiotemporal alignment of multi-source sensor data includes: 1) Sync synchronization messages are periodically broadcast from the master clock, and the master clock records the precise transmission time t1 when the Sync synchronization message leaves the master clock at the moment the transmission ends; 2) The master clock encapsulates the precise timestamp t1 into a Follow_up message and sends it to the slave clock; 3) The slave clock records the precise time point t2 when the Sync synchronization message arrives at the slave clock; 4) The slave clock sends a Delay_req delay request message and records the precise transmission time t3 at the moment the transmission ends; 5) The master clock records the precise arrival time t4 when the Delay_req delay request message arrives at the master clock; 6) The master clock sends a Delay_resp delay request message carrying the precise timestamp information t4 to the slave clock.

[0053] After clock synchronization, four precise timestamps are stored from the clock register. These four timestamps can be used to calculate link delay and clock skew. Specifically:

[0054] Let the link delay from the master clock to the slave clock, Master_(Slave_delay), be:

[0055] Master_(Slave_delay) = t1 - t2;

[0056] At this point, the time deviation offset1 between the master clock and the slave clock is:

[0057] Offset1 = t2-t1-Master_(Slave_delay);

[0058] The message is generated by the link delay from the slave clock to the master clock, Slave_(Master_delay).

[0059] Slave_(Master_delay) = t3 - t4;

[0060] At this point, the time offset2 between the slave clock and the master clock is:

[0061] Offset2 = t4-t3-Slave_(Master_delay);

[0062] Since the IEEE 1588v2 protocol assumes that the link delay is symmetrical, then:

[0063] Master_(Slave_delay) = Slave_(Master_delay);

[0064] Therefore, the average link delay, mean_delay, can be expressed as:

[0065] mean_delay = [(t4-t1)-(t3-t2)] / 2;

[0066] At this point, the master-slave clock offset is:

[0067] Offset = (t2-t1)-mean_delay;

[0068] Further calculations reveal that the master-slave clock offset is:

[0069] Offset = [(t2-t1)-(t4-t3)] / 2.

[0070] Furthermore, compensation is applied to the slave clock, i.e., to achieve synchronization with the master clock through dynamic modeling and correction. Dynamic modeling and correction include: defining a dynamic asymmetric delay correction amount; using this asymmetric delay correction amount to compensate for the average link delay and master-slave clock deviation calculated based on the symmetric delay assumption; and using a filtering algorithm to estimate link delay fluctuations and suppress noise in real time. Specifically, this can be achieved through the following methods:

[0071] 1) Asymmetric delay dynamic modeling.

[0072] A "condition sensing unit" (integrating a PT1000 temperature sensor and an electromagnetic interference detector) is connected in series between the master and slave clock nodes to collect ambient temperature T and electromagnetic interference intensity E in real time, and to define a dynamic asymmetric delay correction amount to replace the traditional single signal strength correction:

[0073] ;

[0074] Where k1 represents the link signal strength correction coefficient, k2 represents the temperature correction coefficient, and k3 represents the electromagnetic interference correction coefficient; Indicates real-time link signal strength. To calibrate signal strength; This represents the dynamic asymmetric delay correction amount, which is the real-time calculated asymmetric delay compensation value of the master-slave link. It is used to correct the calculation errors of link delay and clock offset under the traditional symmetric assumption.

[0075] Therefore, the average link delay, mean_delay, can be re-expressed as:

[0076] mean_delay = [(t4-t1)-(t3-t2)+ ] / 2;

[0077] At this point, the master-slave clock offset can be re-expressed as:

[0078] Offset = (t2-t1)-mean_delay;

[0079] Further calculations reveal that the master-slave clock offset is:

[0080] Offset = [(t2-t1)-(t4-t3)- ] / 2.

[0081] 2) Real-time estimation using second-order Kalman filtering models the mapping relationship between link delay fluctuations and temperature and electromagnetic interference, thus suppressing asymmetric noise.

[0082] Data collected by each sensor is transmitted to the edge computing device via a gigabit Ethernet high-speed communication interface, ensuring high efficiency and stability of data transmission and reducing signal attenuation and interference.

[0083] By aligning the multi-sensor coordinate system using a spatial calibration matrix (4×4 Homography matrix), the spatial registration problem between the molten pool morphology and the temperature field is solved (error <50μm). This ensures that all sensors begin data acquisition under the same time reference, enabling precise control of the start-up time of each sensor and guaranteeing strict temporal correspondence of the acquired multiphysics data, providing an accurate time reference for subsequent data fusion and analysis.

[0084] The standardized communication protocol OPC UA is adopted to realize data interaction between sensors and edge computing devices, as well as between edge computing devices and host computers, to facilitate system integration and expansion. To achieve real-time feature extraction and physical model fusion at the edge, four sensors (infrared camera, X-ray detector, acoustic emission sensor, and spectrometer) collect data during the data acquisition and preprocessing stage. The data undergoes spatiotemporal alignment, data cleaning to remove invalid or abnormal sampling points, noise suppression, and normalization.

[0085] In step S3, after preprocessing the spatiotemporally aligned multi-source sensor data, data-level fusion, feature-level fusion, and decision-level fusion operations are performed sequentially.

[0086] Step S3-1: Data-level fusion.

[0087] Data-level fusion includes weighted Kalman filtering, spatiotemporal consistency verification, data stitching, and outlier removal, culminating in physical field reconstruction. Specifically, data-level fusion employs a three-dimensional dynamic weighting strategy to weight and fuse multi-source sensor data. This involves constructing a fusion architecture of "three-dimensional dynamic weights + multi-physics coupling constraints" to replace traditional fixed weights and single physical constraints, thereby improving the intelligent adaptability and physical rationality of data fusion. This can be achieved through the following methods:

[0088] 1) Three-dimensional dynamic weight design.

[0089] Through sensor weights Integrate multi-source data to update state estimates; among which, weights Weighted by sensor accuracy Synchronization precision weight and operating condition characteristics reliability weight Weighted composition, specifically:

[0090] Sensor accuracy weight Based on the sensor's factory-set accuracy;

[0091] Synchronization Precision Weight :Synchronization deviation with sensor Negative correlation, meaning the more precise the synchronization accuracy, the higher the data weight:

[0092] ;

[0093] in, Indicates sensor synchronization deviation The corresponding time length is 50 nanoseconds in this embodiment.

[0094] Operating condition characteristic reliability weight : Based on dynamic calculation of the feature signal-to-noise ratio (SNR), that is:

[0095] ;

[0096] in, Indicates the first The signal-to-noise ratio of a sensor characteristic (e.g., if the signal-to-noise ratio of a spectrometer increases at high temperatures, then...) Increase; the signal-to-noise ratio of the acoustic emission sensor under electromagnetic interference Decrease, then (Decrease).

[0097] 2) Multiphysics coupling constraint injection.

[0098] Fusion equation optimization: Injecting "thermodynamic-kinetic-geometric feature coupling constraints" into the weighted Kalman filter to replace the single Fourier heat conduction residual. Specifically:

[0099] Weighted Kalman filter update equation incorporating physical features:

[0100] ;

[0101] Represents the state vector at the current moment; Indicates the first Kalman gain of each sensor; Indicates the first The observations from each sensor. Indicates the first The observation matrix of each sensor; This represents the multiphysics coupling residuals, including Fourier thermal conduction residuals. Dynamic residuals and geometric residuals .

[0102] Hybrid model loss function optimization: A coupling constraint loss term is added to the lightweight U-Net model, forming a three-dimensional loss function of "segmentation loss + physics loss + coupling loss", ensuring that the model prediction conforms to the multiphysics coupling law.

[0103] ;

[0104] in, Indicates mixed loss; This represents the cross-entropy loss for semantic segmentation. Represents the loss due to a single physical constraint. It represents the geometric-thermodynamic-kinetic coupling loss of the molten pool (the residual of multiple linear regression of molten pool area with average temperature and vibration frequency).

[0105] Step S3-2: Feature-level fusion.

[0106] Building upon the aforementioned multimodal data streams that have undergone high-precision spatiotemporal alignment and data-level fusion, the core task of this stage is to perform deep feature extraction and multi-dimensional correlation analysis to construct a comprehensive, robust, and physically meaningful molten pool state feature system. Specifically, the system deploys a specially optimized lightweight U-Net convolutional neural network on an edge computing device (using NVIDIA Jetson AGX Xavier as an example). This network takes fused multi-channel image data (mainly including infrared thermal images and X-ray topography images) as input and utilizes its encoder-decoder architecture for real-time semantic segmentation, accurately outputting pixel-level masks of the molten pool region and keyhole. Based on this mask, subsequent algorithms automatically calculate a series of quantitative geometric features, including but not limited to: the projected area, perimeter, aspect ratio, and contour compactness of the molten pool, as well as the molten pool volume, maximum melt depth, and keyhole aspect ratio obtained through 3D point cloud reconstruction technology.

[0107] Simultaneously, the system extracts dynamic evolution features in parallel from the synchronously acquired time-series data. Spatiotemporal analysis is performed on the infrared thermographic sequences to extract key thermodynamic features, such as the average temperature, peak temperature, two-dimensional temperature gradient field, and local cooling rate of the molten pool core region, and further calculates the heat flux density distribution. Short-time Fourier transform or wavelet transform analysis is performed on the acoustic emission sensor signals to extract dynamic features characterizing process stability, including the root mean square value, master oscillation frequency, amplitude, and burst characteristic parameters reflecting abrupt events in key frequency bands (e.g., 50-200 kHz). After preprocessing (background subtraction, spectral line calibration), the data acquired by the spectrometer extracts spectral features reflecting the state of the molten pool plasma, such as the absolute intensity of characteristic spectral lines of specific elements, the intensity ratio between different spectral lines (often used as an indicator of plasma electron temperature), and spectral line broadening information.

[0108] Based on the above feature-level fusion, a lightweight U-Net model is deployed on a Jetson AGX edge device to perform semantic segmentation on the molten pool image, extracting key parameters such as aspect ratio and area fluctuation rate. A physical constraint loss function is constructed using the Fourier heat conduction equation, and a hybrid model (Hybrid Physics-Guided CNN) is trained, reducing the prediction error of the molten pool cooling rate from 15% to below 5%. Geometric, thermodynamic, and kinetic features are extracted to form a correlation matrix.

[0109] Step S3-3: Decision-level fusion.

[0110] This stage is the core of information extraction and final interpretation. Its goal is to transform the multi-level, multi-source feature information obtained from the previous fusion into a deterministic understanding of the melt pool state and executable control decisions. To achieve this goal, the system adopts a hierarchical, progressive, hybrid reasoning decision architecture, sequentially performing rapid screening based on physical rules, deep state reasoning based on probabilistic graphical models, and uncertainty fusion based on evidence theory.

[0111] First, a rapid screening and safety alarm based on physical rules is implemented. Prior knowledge of the molten pool's physical processes is injected into the decision model as physical constraints. These constraints include: the material's boiling point (3287 for titanium alloys). Stainless steel is 2860. The system uses a series of validated process knowledge bases and physical limit thresholds as hard constraint rules, with a temperature gradient of 5℃ / μm corresponding to the critical Marangoni number and a keyhole aspect ratio of 3. Threshold judgment is performed immediately after each frame of data fusion. Once any feature value exceeds these safety boundaries (e.g., a local temperature instantaneously exceeds the material's boiling point), the system will immediately trigger a high-level alarm and generate clear intervention suggestions (e.g., "immediately reduce laser power by 10%"). This process does not rely on complex calculations, ensuring millisecond-level response and providing the first line of defense for core process safety.

[0112] Secondly, a dynamic Bayesian network is constructed for state-level deep reasoning and trend prediction. A dynamic Bayesian network model is built, with network nodes comprising six core input feature variables (melt pool area fluctuation rate, average temperature, temperature gradient, keyhole aspect ratio, acoustic emission dominant frequency, and spectral intensity ratio), two hidden state nodes (melt pool stability level and defect occurrence risk), and two output nodes (state category and trend prediction). Based on 5000 sets of historical process data, the conditional probability distribution between nodes is learned through maximum likelihood estimation. According to the multiphysics coupling mechanism of the molten pool and historical process data, the conditional probability distribution relationship between nodes is learned and determined. During real-time operation, the feature vectors output by the feature-level fusion module are used as observation evidence input to the network, and efficient Bayesian inference algorithms (such as connection tree algorithms) are used to update the posterior probability distribution of all nodes. For example, when two pieces of evidence—an abnormal increase in the keyhole aspect ratio and a sudden increase in acoustic emission energy at a specific frequency—occur simultaneously, the network can calculate a significantly increased posterior probability of the event "imminent keyhole collapse." Meanwhile, the temporal connections in the network enable the system to make short-term predictions on the evolution trend of the molten pool state, achieving forward-looking early warning.

[0113] Finally, the DS evidence theory is introduced to address the uncertainty and conflict of multi-source information. To address the potential ambiguity or conflict in feature information from different sensor sources (e.g., infrared thermometry indicating overheating while morphological features show normality), this layer defines a basic probability allocation function for each independent information source (e.g., the "infrared feature extraction module," "X-ray morphology module," and "acoustic emission spectrum module"). This function transforms the feature vectors provided by each module at the current moment into a probability distribution for the recognition framework. The confidence levels of various propositions within the range {“Normal,” “Overheated,” “Splatter,” “Unfused,” …} are assigned. Subsequently, Dempster's combination rule is applied to fuse the confidence levels from all independent evidence sources, resulting in a comprehensive confidence level assignment with significantly reduced conflict. The final decision selects the proposition with the highest fused confidence level as the evaluation conclusion for the current melt pool state, along with its confidence level.

[0114] Based on the above decision-level fusion, prior knowledge of the molten pool physical process is injected into the decision model as a physical constraint; a Bayesian network model is constructed to describe and infer the probabilistic dependencies between various features; DS evidence theory is applied to fuse molten pool state data provided by multiple sensors to obtain the evaluation result of the molten pool state. A lightweight rendering engine is developed based on WebGL, using an octree structure to compress point cloud data (compression ratio >90%) to achieve real-time display of the 3D topography of the molten pool (refresh rate >30Hz). An HSV-α multidimensional shading model is designed to overlay and display temperature (H channel), stress (S channel), and defect density (V channel), and to display them through transparency. Mapping confidence.

[0115] Example 2

[0116] This embodiment discloses a multimodal fusion-based real-time monitoring system for laser-directed energy deposition.

[0117] like Figure 3 As shown, a multimodal fusion laser directional energy deposition real-time monitoring system includes:

[0118] The data acquisition module is configured to: acquire real-time data of the molten pool region during the laser directional energy deposition process by synchronously triggering multiple source sensors;

[0119] The spatiotemporal alignment module is configured to: construct a distributed clock network, combine the working condition sensing unit to collect environmental impact parameters in real time, and dynamically model and correct the asymmetric link delay in the clock synchronization process to achieve spatiotemporal alignment of multi-source sensor data;

[0120] The three-level fusion operation module is configured to: preprocess multi-source sensor data that has undergone spatiotemporal alignment, and then sequentially perform data-level fusion, feature-level fusion, and decision-level fusion operations; wherein, the data-level fusion adopts a three-dimensional dynamic weight strategy to weight the multi-source sensor data; the feature-level fusion extracts the geometric, thermodynamic, and dynamic features of the molten pool from the fused multi-source sensor data through a lightweight neural network model deployed on edge devices, and constructs a feature correlation matrix, while injecting constraint loss based on the coupling law of multi-physics fields into the model training; the decision-level fusion combines prior knowledge of physical processes, probabilistic reasoning models, and evidence theory to comprehensively evaluate and decide on the state of the molten pool.

[0121] Based on the above systematic design, in the specific implementation process:

[0122] The first step involves simultaneously acquiring multiphysics data of the molten pool using an infrared camera, X-ray detector, acoustic emission sensor, and spectrometer. The acquired data is then spatiotemporally aligned to ensure consistency across all sensors. Next, data cleaning is performed to remove noise and outliers, and to fill in or correct missing data. Following this, filtering algorithms are applied to suppress noise and improve data quality. Finally, the data is normalized to scale the data from different sensors to the same scale.

[0123] The second step involves dynamically weighting the data from each sensor through data-level fusion, assigning weights based on sensor accuracy and reliability. Then, a weighted Kalman filter algorithm is applied to filter the data and estimate the optimal value of the molten pool state. Next, a spatiotemporal consistency check is performed to identify and correct any inconsistencies in the data. Finally, the data from each sensor are concatenated to form a complete dataset, and outliers that fail the consistency check are removed. Based on the fused data, a multiphysics model of the molten pool, including its temperature field, morphology field, and spectral field, is reconstructed.

[0124] The third step involves extracting geometric features, such as aspect ratio and area, from the molten pool morphology data through feature-level fusion. Thermodynamic features, including average temperature, temperature gradient, and heat flux density, are extracted from the temperature field data. Dynamic features, such as vibration frequencies, are extracted from the acoustic emission and spectral data. These extracted features are then subjected to correlation analysis to form a feature correlation matrix.

[0125] The fourth step involves injecting prior knowledge and physical constraints of the molten pool's physical processes into the decision model through decision-level fusion. A Bayesian network model is constructed to describe the probabilistic dependencies between features and to perform probabilistic inference. DS evidence theory is applied to fuse evidence from different sensors to evaluate the molten pool state. Finally, by combining the results of physical constraint injection, Bayesian network, and DS evidence theory, a final evaluation of the molten pool state is performed, providing real-time control decisions relative to the laser power controller, scanning path planning, and powder feed flow valve.

[0126] Example 3

[0127] The purpose of this embodiment is to provide a computer-readable storage medium.

[0128] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in Embodiment 1 of this disclosure.

[0129] Example 4

[0130] The purpose of this embodiment is to provide an electronic device.

[0131] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in Embodiment 1 of this disclosure.

[0132] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0133] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0134] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal fusion method for real-time monitoring of laser-directed energy deposition, characterized in that, include: Real-time data acquisition of the molten pool region during the laser directional energy deposition process is achieved through synchronous triggering of multiple source sensors. A distributed clock network is constructed, and environmental impact parameters are collected in real time by the working condition sensing unit. The asymmetric link delay in the clock synchronization process is dynamically modeled and corrected to achieve spatiotemporal alignment of multi-source sensor data. The distributed clock network is built using the IEEE 1588v2 protocol. The IEEE 1588v2 protocol achieves synchronization based on a delay response mechanism, specifically by using timestamps carried in IEEE 1588v2 messages. The IEEE 1588v2 message includes synchronization message, follow message, delay request message and delay request response message; wherein, the dynamic modeling and correction includes: defining a dynamic asymmetric delay correction amount, using the asymmetric delay correction amount to compensate for the average link delay and master-slave clock deviation calculated based on the symmetric delay assumption, and using a filtering algorithm to estimate the fluctuation of link delay and suppress noise in real time. After preprocessing the spatiotemporally aligned multi-source sensor data, data-level fusion, feature-level fusion, and decision-level fusion operations are performed sequentially. The data-level fusion employs a three-dimensional dynamic weighting strategy to weight the multi-source sensor data, integrating the inherent accuracy of the sensors, real-time synchronization accuracy, and operational reliability based on the current feature signal-to-noise ratio. The feature-level fusion uses a lightweight neural network model deployed on edge devices to extract the geometric, thermodynamic, and dynamic features of the molten pool from the fused multi-source sensor data and construct a feature correlation matrix. Simultaneously, constraint loss based on multi-physics coupling laws is injected into the model training. The decision-level fusion combines prior knowledge of physical processes, probabilistic reasoning models, and evidence theory to comprehensively evaluate and decide on the state of the molten pool.

2. The multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in claim 1, characterized in that, The multi-source sensor includes: an infrared camera for acquiring temperature field data, an X-ray detector for acquiring topographic field data including keyhole topography, an acoustic emission sensor for acquiring dynamic characteristics including oscillation signals, and a spectrometer for acquiring spectral information including plasma spectra.

3. The multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in claim 1, characterized in that, The clock synchronization process of the distributed clock network includes: First, a synchronization message is periodically broadcast from the master clock, and the master clock records the transmission time t1 at the moment the transmission ends; then, the master clock encapsulates the timestamp corresponding to the transmission time t1 into a follow-up message and sends it to the slave clock; next, the slave clock records the time t2 when the synchronization message arrives at the slave clock; then, the slave clock sends a delay request message and records the transmission time t3 at the moment the transmission ends; next, the master clock records the arrival time t4 when the delay request message arrives at the master clock; finally, the master clock sends a delay request message carrying the timestamp information corresponding to the arrival time t4 to the slave clock.

4. The multimodal fusion method for real-time monitoring of laser directional energy deposition as described in claim 3, characterized in that, After a complete clock synchronization process, the clock register stores four timestamps: t1, t2, t3, and t4. These four timestamps are used to calculate the link delay and clock skew.

5. The multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in claim 1, characterized in that, The decision-level fusion includes: injecting prior knowledge of the physical process of the molten pool as a physical constraint into the decision model; constructing a Bayesian network model to describe and infer the probabilistic dependencies between features; and applying DS evidence theory to fuse the molten pool state data provided by multiple sensors to obtain the evaluation result of the molten pool state.

6. A multimodal fusion-based real-time monitoring system for laser-directed energy deposition, characterized in that, include: The data acquisition module is configured to: acquire real-time data of the molten pool region during the laser directional energy deposition process by synchronously triggering multiple source sensors; The spatiotemporal alignment module is configured to: construct a distributed clock network, combine the working condition sensing unit to collect environmental impact parameters in real time, and dynamically model and correct the asymmetric link delay in the clock synchronization process to achieve spatiotemporal alignment of multi-source sensor data; The distributed clock network is built using the IEEE 1588v2 protocol. The IEEE 1588v2 protocol achieves synchronization based on a delay response mechanism, specifically by using timestamps carried in IEEE 1588v2 messages. The IEEE 1588v2 message includes synchronization message, follow message, delay request message and delay request response message; wherein, the dynamic modeling and correction includes: defining a dynamic asymmetric delay correction amount, using the asymmetric delay correction amount to compensate for the average link delay and master-slave clock deviation calculated based on the symmetric delay assumption, and using a filtering algorithm to estimate the fluctuation of link delay and suppress noise in real time. The three-level fusion operation module is configured to: preprocess multi-source sensor data after spatiotemporal alignment, and then sequentially perform data-level fusion, feature-level fusion, and decision-level fusion operations; wherein, the data-level fusion adopts a three-dimensional dynamic weight strategy to weight the multi-source sensor data, and the weight integrates the inherent accuracy of the sensors, the real-time synchronization accuracy, and the operational reliability based on the current feature signal-to-noise ratio; the feature-level fusion extracts the geometric, thermodynamic, and dynamic features of the molten pool from the fused multi-source sensor data through a lightweight neural network model deployed on edge devices, and constructs a feature correlation matrix, while injecting constraint loss based on the coupling law of multi-physics fields into the model training; the decision-level fusion combines prior knowledge of physical processes, probabilistic reasoning models, and evidence theory to comprehensively evaluate and decide on the state of the molten pool.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the multimodal fusion-based real-time monitoring method for laser directional energy deposition as described in any one of claims 1-5.

8. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal fusion laser directional energy deposition real-time monitoring method as described in any one of claims 1-5.

Images