Method and device for defect detection in additive manufacturing based on spatiotemporal feature fusion of molten pool images

Abstract

This application relates to the field of additive manufacturing technology, and provides a method and apparatus for defect detection in additive manufacturing based on spatiotemporal feature fusion of molten pool images. The method establishes a precise correspondence between a 3D defect model and the molten pool point cloud, and extracts local molten pool data using cubic spatial sampling based on the defect formation mechanism. Upsampling and center sampling strategies are designed to improve the targeting of data extraction. By introducing a spatial information fusion module, including a spatial-feature interaction attention submodule and a position-aware feature transformation submodule, dynamic fusion of molten pool image features and spatial coordinates is achieved. This addresses the shortcomings of existing deep learning models that ignore spatial position information, enhancing the robustness and accuracy of feature representation, and is particularly suitable for defect identification in complex additive manufacturing environments. A temporal information fusion module is used to perform temporal modeling of the spatially enhanced features, capturing the dynamic changes during the molten pool evolution process, thus improving the real-time performance and prediction accuracy of defect detection.

Description

Technical Field

[0001] This application relates to the field of additive manufacturing technology, and in particular to an additive manufacturing defect detection method and apparatus based on spatiotemporal feature fusion of molten pool images. Background Technology

[0002] Near-net-shape additive manufacturing technologies, represented by selective laser bed fusion (LPBF), offer a new approach for processing complex hot-end components such as hollow air-cooled turbine blades for advanced aero-engines due to their advantages of high forming accuracy and flexibility.

[0003] However, selective laser melting (SLM) involves numerous and highly variable process parameters, resulting in unavoidable and difficult-to-eliminate defects in the formed components. These defects lead to localized mechanical property degradation and premature microcrack initiation, limiting its application in load-bearing critical components. Load-bearing critical components typically possess characteristics such as large size, high density, and complex structure, rendering existing offline inspection methods inadequate in terms of accessibility and required accuracy.

[0004] Related technologies offer methods for real-time monitoring of additive manufacturing processes to ensure high-quality molded parts supporting critical components. However, most of these methods rely on manually designed feature extraction rules, which struggle to adapt to the complex changes in the molten pool state. Furthermore, machine learning algorithms are insufficient in temporal information modeling, failing to capture the dynamic evolution of defect formation. Some deep learning-based solutions primarily focus on the instantaneous spatial characteristics of the molten pool, neglecting to fully consider the temporal characteristics related to defect formation inherent in the dynamic evolution of the molten pool, and also failing to consider the modulating effect of the molten pool's three-dimensional spatial position within the molded body on the defect formation mechanism. Summary of the Invention

[0005] In view of this, embodiments of this application provide an additive manufacturing defect detection method and apparatus based on spatiotemporal feature fusion of molten pool images, in order to solve the problem of insufficient accuracy in real-time defect detection of additive manufacturing processes in the prior art.

[0006] A first aspect of this application provides an additive manufacturing defect detection method based on spatiotemporal feature fusion of molten pool images, comprising:

[0007] Acquire the sequence of molten pool images, the three-dimensional spatial coordinates of the molten pool, and the tomographic scan images of the molded body during the additive manufacturing process;

[0008] During the offline training phase, a 3D reconstruction technique is used to construct a 3D defect model based on the scanned images of the molded body; the 3D defect model includes at least porosity and incomplete fusion defects.

[0009] A 3D point cloud of the molten pool is constructed based on the 3D spatial coordinates of the molten pool, and the correspondence between the 3D defect model and the 3D point cloud of the molten pool is established.

[0010] Based on the defect formation mechanism, local molten pool point clouds are extracted around the defect, and the local molten pool point clouds are matched with the molten pool image sequence to obtain the defect-related molten pool image sequence.

[0011] Spatial and temporal information of molten pool images are extracted using deep learning networks, and defect detection is achieved based on the spatial and temporal information.

[0012] A second aspect of this application provides an additive manufacturing defect detection device based on spatiotemporal feature fusion of molten pool images, comprising:

[0013] The acquisition module is configured to acquire a sequence of molten pool images, three-dimensional spatial coordinates of the molten pool, and tomographic scan images of the molded body during the additive manufacturing process.

[0014] The pre-training module is configured to construct a 3D defect model based on scanned images of the molded body using 3D reconstruction technology during the offline training phase; the 3D defect model includes at least porosity and incomplete fusion defects.

[0015] The registration module is configured to construct a 3D point cloud of the molten pool based on the 3D spatial coordinates of the molten pool, and establish the correspondence between the 3D defect model and the 3D point cloud of the molten pool.

[0016] The extraction module is configured to extract local molten pool point clouds around the defect based on the defect formation mechanism, and match the local molten pool point clouds with the molten pool image sequence to obtain a defect-related molten pool image sequence.

[0017] The detection module is configured to extract spatial location information and temporal information of the molten pool image through a deep learning network, and to perform defect detection based on the spatial location information and temporal information.

[0018] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0019] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0020] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment establishes a precise correspondence between the three-dimensional defect model and the molten pool point cloud, and extracts local molten pool data by cubic spatial sampling based on the defect formation mechanism. It designs upsampling and center sampling strategies to improve the targeting of data extraction. By introducing a spatial information fusion module, including a spatial-feature interaction attention submodule and a position-aware feature transformation submodule, it realizes the dynamic fusion of molten pool image features and spatial coordinates. Compared with the shortcomings of existing deep learning models that ignore spatial position information, it enhances the robustness and accuracy of feature representation, and is especially suitable for defect identification in complex additive manufacturing environments. The temporal information fusion module is used to perform temporal modeling of spatial enhancement features, capturing the dynamic changes in the molten pool evolution process, and improving the real-time performance and prediction accuracy of defect detection. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating an additive manufacturing defect detection method based on spatiotemporal feature fusion of molten pool images, provided in an embodiment of this application.

[0023] Figure 2 This is a flowchart illustrating the method for establishing the correspondence between a three-dimensional defect model and a three-dimensional point cloud of a molten pool, as provided in an embodiment of this application.

[0024] Figure 3 This is a flowchart illustrating the method for extracting local melt pool point clouds around defects, as provided in an embodiment of this application.

[0025] Figure 4 This is a schematic diagram of a cubic space constructed with the defect centroid as the center of the cube.

[0026] Figure 5 This is a schematic diagram of a cubic space constructed with the lowest point of the defect as the center of the bottom face of the cube.

[0027] Figure 6 This is a flowchart illustrating the method provided in this application for extracting features from a molten pool image, encoding spatial coordinate information, and fusing spatial location information and temporal information of the molten pool image using a deep learning network.

[0028] Figure 7 This is a flowchart illustrating another additive manufacturing defect detection method based on spatiotemporal feature fusion of molten pool images provided in this application embodiment.

[0029] Figure 8 This is a schematic diagram of the structure of the deep learning model based on the spatiotemporal features of fused melt pool proposed in the embodiments of this application.

[0030] Figure 9 This is a schematic diagram of the structure of an additive manufacturing defect detection device based on spatiotemporal feature fusion of molten pool images provided in an embodiment of this application.

[0031] Figure 10 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] The following will describe in detail, with reference to the accompanying drawings, an additive manufacturing defect detection method and apparatus based on spatiotemporal feature fusion of molten pool images according to embodiments of this application.

[0034] As mentioned above, critical load-bearing components typically possess characteristics such as large size, high density, and complex structure, rendering existing offline inspection methods inadequate in terms of accessibility and accuracy. Therefore, real-time monitoring of the additive manufacturing process is crucial for ensuring high-quality molded parts that support critical load-bearing components. In the SLM process, the molten pool, as the core region for energy transfer and phase transition, is closely related to defect formation mechanisms.

[0035] Early research on real-time monitoring of additive manufacturing processes primarily employed traditional image processing techniques to extract features such as shape, temperature, and light intensity from molten pool images, combining these with machine learning algorithms for defect detection. For example, some studies used infrared thermal imagers to acquire infrared images of the molten pool, extracting grayscale gradient features, size features, and shape features, and then feeding the fused feature vector into a classification algorithm for defect prediction. However, the molten pool state is influenced by multiple coupled parameters, including laser power, scanning speed, and powder characteristics, exhibiting a high dynamic range and complex texture variations. This method relies on manually designed feature extraction rules, which struggle to adapt to the complex changes in the molten pool state. Furthermore, machine learning algorithms have limitations in temporal information modeling, making it difficult to capture the dynamic evolution of defect formation.

[0036] With the development of deep learning technology, researchers have begun to apply deep learning networks to the field of defect detection in additive manufacturing, using them to process molten pool images to achieve automatic feature extraction and defect classification. However, current deep learning-based defect detection methods mainly focus on the instantaneous spatial features of the molten pool, failing to fully consider the temporal features related to defect formation inherent in the dynamic evolution of the molten pool, nor the modulation effect of the three-dimensional spatial position of the molten pool in the formed body on the defect formation mechanism.

[0037] In view of this, this application provides an additive manufacturing defect detection method based on spatiotemporal feature fusion of molten pool images. It establishes a precise correspondence between the 3D defect model and the molten pool point cloud, and extracts local molten pool data using cubic spatial sampling based on the defect formation mechanism. Upsampling and center sampling strategies are designed to improve the targeting of data extraction. By introducing a spatial information fusion module, including a spatial-feature interaction attention submodule and a position-aware feature transformation submodule, dynamic fusion of molten pool image features and spatial coordinates is achieved. This addresses the shortcomings of existing deep learning models that ignore spatial position information, enhancing the robustness and accuracy of feature representation, and is particularly suitable for defect identification in complex additive manufacturing environments. A temporal information fusion module is used to perform temporal modeling of the spatially enhanced features, capturing the dynamic changes during the molten pool evolution process, thus improving the real-time performance and prediction accuracy of defect detection.

[0038] Figure 1 This is a schematic flowchart of an additive manufacturing defect detection method based on spatiotemporal feature fusion of molten pool images, provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0039] In step S101, the sequence of molten pool images, the three-dimensional spatial coordinates of the molten pool, and the tomographic scan image of the molded body are acquired during the additive manufacturing process.

[0040] In step S102, during the offline training phase, a 3D model of the defect is constructed based on the scanned image of the molded body using 3D reconstruction technology.

[0041] The three-dimensional defect model includes at least porosity and non-fusion defects.

[0042] In step S103, a three-dimensional point cloud of the molten pool is constructed based on the three-dimensional spatial coordinates of the molten pool, and the correspondence between the three-dimensional defect model and the three-dimensional point cloud of the molten pool is established.

[0043] In step S104, based on the defect formation mechanism, local molten pool point clouds are extracted around the defect, and the local molten pool point clouds are matched with the molten pool image sequence to obtain the defect-related molten pool image sequence.

[0044] In step S105, spatial location information and temporal information of the molten pool image are extracted by a deep learning network, and defect detection is achieved based on the spatial location information and temporal information.

[0045] In some embodiments of this application, the method can be executed by a server or by a terminal device with certain processing capabilities, and is used to predict whether defects will occur after the part is formed based on the melt pool image and position during the additive manufacturing printing process.

[0046] In some embodiments of this application, the sequence of molten pool images, the three-dimensional spatial coordinates of the molten pool, and the tomographic scan images of the molded body during the additive manufacturing process can be obtained first.

[0047] In some embodiments of this application, during the offline training phase, a three-dimensional model of defects can be constructed based on the scanned images of the molded body using three-dimensional reconstruction technology. This three-dimensional model of defects includes at least pores and unfused defects.

[0048] In other words, a 3D defect model can be pre-trained, and the training dataset can be a pair of molten pool point clouds and defect label data. The defect label indicates whether a defect exists in the corresponding formed part and the type of defect. The presence and type of defect can be determined using computed tomography (CT).

[0049] In some embodiments of this application, a three-dimensional point cloud of the molten pool can be constructed based on the three-dimensional spatial coordinates of the molten pool, and a correspondence between the three-dimensional defect model and the three-dimensional point cloud of the molten pool can be established.

[0050] Then, based on the defect formation mechanism, local molten pool point clouds are extracted around the defect, and the local molten pool point clouds are matched with the molten pool image sequence to obtain the defect-related molten pool image sequence.

[0051] Based on the defect formation mechanism, it is known that defect formation is related to the characteristics of the molten pool in the layer containing the defect. Furthermore, during printing, subsequent layers remelt the preceding layers, and the already formed defect is also affected by the state of the molten pool in subsequent layers. Therefore, it can be determined that the final state of the defect depends on a multi-layer thermal cycling process. Based on this, a defect-related molten pool image sequence can be obtained by matching the local molten pool point cloud around the defect with the molten pool image sequence.

[0052] In some embodiments of this application, spatial location information and temporal information of the molten pool image are extracted by a deep learning network, and defect detection is achieved based on the spatial location information and temporal information.

[0053] According to the technical solution provided in the embodiments of this application, by establishing a precise correspondence between the three-dimensional defect model and the point cloud of the molten pool, and by extracting local molten pool data using cubic spatial sampling based on the defect formation mechanism, upsampling and center sampling strategies are designed to improve the targeting of data extraction; by introducing a spatial information fusion module, including a spatial-feature interaction attention submodule and a position-aware feature transformation submodule, dynamic fusion of molten pool image features and spatial coordinates is achieved. Compared with the shortcomings of existing deep learning models that ignore spatial position information, the robustness and accuracy of feature representation are enhanced, which is especially suitable for defect identification in complex additive manufacturing environments; a temporal information fusion module is used to perform temporal modeling of spatial enhancement features, capturing the dynamic changes in the molten pool evolution process, and improving the real-time performance and prediction accuracy of defect detection.

[0054] Figure 2 This is a flowchart illustrating the method for establishing the correspondence between a 3D defect model and a 3D point cloud of a molten pool, as provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0055] In step S201, a three-dimensional model of the molded body is determined based on the scanned image of the molded body using three-dimensional reconstruction technology.

[0056] In this model, the 3D model of the formed object and the 3D model of the defect share the same coordinate system.

[0057] In step S202, the surface point cloud of the three-dimensional model of the molded body is extracted, and the surface point cloud of the three-dimensional model of the molded body is matched with the three-dimensional point cloud of the molten pool through a registration algorithm to obtain the registered three-dimensional point cloud of the molten pool.

[0058] In step S203, the three-dimensional defect model is registered with the registered three-dimensional point cloud of the molten pool to obtain the spatial positional correspondence between the three-dimensional defect model and the three-dimensional point cloud of the molten pool.

[0059] In some embodiments of this application, when establishing the correspondence between the 3D defect model and the 3D point cloud of the molten pool, 3D reconstruction technology can first be used to determine the 3D model of the molded body that shares a coordinate system with the 3D defect model based on the scanned image of the molded body. Then, the surface point cloud of the 3D model of the molded body is extracted, and the surface point cloud of the 3D model of the molded body is matched with the 3D point cloud of the molten pool using a registration algorithm to obtain the registered 3D point cloud of the molten pool. Finally, the 3D defect model and the registered 3D point cloud of the molten pool are registered to obtain the spatial positional correspondence between the 3D defect model and the 3D point cloud of the molten pool.

[0060] In other words, based on the tomographic scan image of the additively manufactured body, a three-dimensional reconstruction technique can be used to obtain the body model and the defect three-dimensional model; point cloud extraction technique can be used to extract the surface point cloud of the body three-dimensional model, and the point cloud can be registered with the molten pool point cloud through a registration algorithm; the defect three-dimensional model and the body model share a coordinate system to obtain the spatial correspondence between the defect three-dimensional model and the molten pool point cloud; the molten pool point cloud can be matched with the molten pool image to obtain the molten pool image sequence related to the defect.

[0061] Figure 3 This is a schematic flowchart illustrating the method for extracting local melt pool point clouds around defects, as provided in an embodiment of this application. Figure 3 As shown, the method includes the following steps:

[0062] In step S301, a cube space of a preset size is constructed based on the corresponding position of the target defect location in the three-dimensional point cloud of the molten pool.

[0063] The target defect is any defect in the three-dimensional defect model.

[0064] In step S302, the point cloud data of the molten pool in the constructed cubic space is extracted to obtain the local point cloud of the molten pool around the target defect.

[0065] Existing research indicates that defect formation is related to the characteristics of the molten pool in the layer containing the defect. Furthermore, since subsequent layers remelt previous layers during printing, the formed defect is affected by the molten pool state of subsequent layers. Therefore, the final state of the defect depends on the multi-layer thermal cycling process. In view of this, this application proposes a solution to obtain the local molten pool point cloud around the defect by constructing a cubic space of a preset size based on the defect location and then extracting molten pool point cloud data within this cubic space.

[0066] In other words, in some embodiments of this application, extracting local melt pool point clouds around a defect can be done by first constructing a cube space of a preset size based on the corresponding position of the target defect in the three-dimensional point cloud of the melt pool, and then extracting the melt pool point cloud data in the constructed cube space to obtain the local melt pool point cloud around the target defect.

[0067] The cube space of the preset size is constructed in the following ways: the cube space is constructed with the centroid of the target defect as the center of the cube and the preset size as the side length of the cube; or, the cube space is constructed with the lowest point of the target defect as the center of the bottom surface of the cube and the preset size as the side length of the cube.

[0068] In other words, a method for extracting local melt pool point clouds around a defect can be to construct a cube space of a preset size based on the defect location, and then extract the melt pool point cloud data contained within that cube space. The method for constructing the cube space based on the defect location can be either to construct the cube space with the defect centroid as the cube center, or to construct the cube space with the lowest point of the defect as the center of the cube's bottom surface.

[0069] Among them, the cubic space constructed with the centroid of the defect as the center of the cube is as follows: Figure 4 As shown, the cubic space constructed with the lowest point of the defect as the center of the cube's base is as follows: Figure 5 As shown in the diagram, the large sphere located inside the cube represents the defect.

[0070] Figure 6 This is a flowchart illustrating the method provided in this application for extracting features from a molten pool image using a deep learning network, encoding spatial coordinate information, and fusing spatial location information and temporal information of the molten pool image. Figure 6 As shown, the method includes the following steps:

[0071] In step S601, image features of each frame of the molten pool image in the defect-related molten pool image sequence are extracted using a deep learning network.

[0072] In step S602, the three-dimensional spatial coordinates of the molten pool are encoded into spatial coordinates by an encoder.

[0073] In step S603, the spatial coordinates and image features are fused by the spatial information fusion module to obtain spatially enhanced molten pool image features.

[0074] In step S604, the spatially enhanced molten pool image features are modeled temporally using the time information fusion module to obtain molten pool image features that include spatial location information and temporal information of the molten pool image.

[0075] In step S605, the classifier outputs the defect detection result based on the features of the molten pool image, which includes spatial location information and temporal information of the molten pool image.

[0076] In some embodiments of this application, image features of each frame of the molten pool image in a defect-related molten pool image sequence can be extracted using a deep learning network, and the three-dimensional spatial coordinates of the molten pool can be encoded into spatial coordinates using an encoder. The deep learning network can be, for example, ResNet34.

[0077] Then, the spatial coordinates and image features are fused using the spatial information fusion module to obtain spatially enhanced molten pool image features. The temporal information fusion module is then used to perform temporal modeling on the spatially enhanced molten pool image features to obtain molten pool image features that include both spatial location information and temporal information of the molten pool image.

[0078] Finally, the defect detection results are output by the classifier based on the features of the molten pool image, which include spatial location information and temporal information of the molten pool image.

[0079] The process of fusing spatial coordinates and image features through the spatial information fusion module can be further divided into several steps. First, a spatial-feature interaction attention submodule receives spatial coordinates and image features. Then, a neural network dynamically generates attention weights based on the spatial coordinates of the molten pool image in a three-dimensional coordinate system. Next, a position-aware feature transformation submodule converts the spatial coordinates into modulation factors using a multilayer perceptron. These modulation factors are then used to perform element-wise modulation of the image features, resulting in position-modulated features. Finally, a feature weighting and fusion submodule weights and fuses the position-modulated features and image features based on the attention weights, and then fuses them with the spatial coordinates to obtain fused high-dimensional features. Finally, a feature dimensionality reduction submodule reduces the fused high-dimensional features to the same dimension as the image features, resulting in spatially enhanced molten pool image features.

[0080] Furthermore, temporal modeling of spatially enhanced molten pool image features through a temporal information fusion module can be achieved by using a gated recurrent unit network architecture to perform temporal modeling of spatially enhanced molten pool image features, thereby obtaining molten pool image features that include both spatial location information and temporal information of the molten pool image.

[0081] In other words, defect detection can be achieved by extracting features from the molten pool image, encoding spatial coordinate information, and fusing the spatial location and temporal information of the molten pool image through a deep learning network. First, features of each frame of the molten pool image are extracted using a convolutional neural network. Then, the three-dimensional spatial coordinates of the molten pool points are encoded into spatial coordinates using an encoder, and the spatial coordinates are fused with the image features using a spatial information fusion module to obtain a spatially enhanced molten pool image feature representation. Temporal modeling of the spatially enhanced molten pool image features is then performed using a temporal information fusion module. Finally, the defect detection results are output using a classifier.

[0082] The spatial information fusion module may include a spatial-feature interaction attention submodule, a location-aware feature transformation submodule, a feature weighting and fusion submodule, and a feature dimensionality reduction submodule.

[0083] The spatial-feature interactive attention submodule is used to generate attention weights by combining melt pool image features and spatial location information. Specifically, the spatial-feature interactive attention submodule receives melt pool image features extracted by the convolutional neural network and encoded high-dimensional spatial coordinate information, and dynamically generates feature importance weights based on the spatial location of the melt pool.

[0084] The position-aware feature transformation submodule is used to modulate image features based on spatial location information. Specifically, the position-aware feature transformation submodule converts high-dimensional spatial coordinates into modulation factors through a multilayer perceptron, and uses the modulation factors to perform element-level modulation on the original image features to achieve feature scaling based on spatial location.

[0085] The feature weighting and fusion submodule is used to perform weighted fusion of the modulation features output by the position-aware feature transformation submodule and the original image features based on attention weights, and then fuse them with high-dimensional space features; the feature dimensionality reduction submodule is used to reduce the fused high-dimensional features to the same dimension as the original image features.

[0086] Furthermore, the temporal information fusion module employs a gated recurrent unit network architecture to perform temporal modeling of image features containing spatial location information. The modeling process can involve sequentially inputting enhanced molten pool image features (in the order in which the molten pool images are captured during the printing process) into the gated recurrent unit network. The network then learns these features sequentially and outputs features with temporal information.

[0087] Figure 7 This is a schematic flowchart of another additive manufacturing defect detection method based on spatiotemporal feature fusion of molten pool images provided in an embodiment of this application. Figure 7 As shown, the method includes the following steps: First, construct a three-dimensional model of the molded body and the defect, and construct a molten pool point cloud based on the three-dimensional coordinates of the molten pool; the point cloud of the molded body model can be extracted using the three-dimensional model, and the point cloud of the model can be registered with the point cloud of the molten pool, and then the three-dimensional model of the defect can be registered with the point cloud of the molten pool.

[0088] Then, defect-related melt pool point clouds can be extracted based on the cube sampling method, the melt pool point clouds can be mapped to the melt pool image, and then deep learning networks can be used to extract the features of the melt pool image. At the same time, the three-dimensional coordinates of the melt pool can be mapped to a high dimension through a fully connected layer.

[0089] Next, the spatial information fusion module is used to spatially enhance the image features, the temporal information fusion module is used to perform temporal modeling of the image features, and a fully connected layer is used to output the binary classification probability for defect detection, thereby obtaining the defect detection result.

[0090] Figure 8 This is a schematic diagram of the structure of the deep learning model based on the spatiotemporal features of molten pool fusion proposed in an embodiment of this application. Figure 8 As shown, the model includes a convolutional neural network used to extract image features from each frame of the molten pool image sequence to obtain the original image features. The molten pool image sequence has a certain sequence length, and the original image features contain this sequence length information, as well as the image feature dimensions.

[0091] The model also includes a spatial information fusion module and a temporal information fusion module. The spatial information fusion module fuses the original image features with the spatial location of the molten pool to obtain spatially enhanced molten pool image features. The temporal information fusion module performs temporal modeling on the spatially enhanced molten pool image features to obtain temporal features that include spatial location information of the molten pool image.

[0092] The model also includes a fully connected layer. This fully connected layer classifies temporal features containing spatial location information of the molten pool image to obtain classification probabilities, and then obtains the defect detection results.

[0093] The following example uses the "Overhang Part X16" benchmark dataset for defect detection. This dataset contains high-frequency molten pool image sequences captured by a coaxial camera system, the three-dimensional spatial coordinates of the molten pool, and high-resolution X-ray CT data of the molded body. The detailed steps are as follows:

[0094] The defect 3D model is registered with the molten pool point cloud. Based on the CT image of the molded body, 3D reconstruction software is used to reconstruct the 3D model of the molded body and the defect. Point cloud extraction technology is used to extract the point cloud of the 3D model of the molded body.

[0095] Based on the 3D spatial coordinate data of the molten pool, a 3D point cloud of the molten pool is constructed. Sub-millimeter accuracy is achieved between the 3D point cloud of the molded body model and the point cloud of the molten pool using the Normal Distribution Transform (NDT) algorithm, and the transformation matrix is ​​obtained. Since the 3D defect model shares a coordinate system with the molded body model, registration between the 3D defect model and the point cloud of the molten pool is achieved through the transformation matrix.

[0096] Extract the defect-related melt pool point cloud. Based on the defect location, consider the printing layer thickness, laser scanning path spacing, and camera capture frequency to ensure coverage of the complete scanning trajectory and approximately 10 layers of vertical heat transfer information. Preferably, construct a cubic space with a side length of 200 micrometers (μm). Extract the coordinates of the melt pool point cloud contained in this cubic space and correlate them with the melt pool image. Further methods for constructing the cubic space include: constructing the cubic space with the defect centroid as the center of the cube (center sampling strategy); and constructing the cubic space with the lowest point of the defect as the center of the bottom surface of the cube (upsampling strategy).

[0097] Convolutional neural networks extract image features. For example, ResNet34 backbone network is used to process each frame of molten pool image to extract 512-dimensional high-level spatial features, capturing molten pool morphology, temperature distribution and local texture.

[0098] The multilayer perceptron (MLP) encodes the three-dimensional spatial coordinates of the melt pool point into 256-dimensional spatial coordinate features.

[0099] The spatial information fusion module fuses 256-dimensional spatial coordinate features with 512-dimensional image features to obtain a spatially enhanced molten pool image feature representation. The spatial information fusion module includes a spatial-feature interaction attention submodule, which generates attention weights by combining molten pool image features and spatial location information; a location-aware feature transformation submodule, which modulates image features based on spatial location information; a feature weighting and fusion submodule, which weights the modulated features and the original features based on the attention weights and fuses them with spatial coordinates; and a feature dimensionality reduction submodule, which reduces the dimensionality of the fused high-dimensional features to the same dimension as the original image features.

[0100] The spatial-feature interaction attention submodule in the spatial information fusion module combines melt pool image features and spatial location information to generate attention weights. For example, it receives 512-dimensional melt pool image features extracted by ResNet and encoded 256-dimensional high-dimensional spatial coordinate information, maps the image features to 256 dimensions through MLP, concatenates them and fuses them to 256 dimensions through MLP, and then generates normalized attention weights through MLP and Sigmoid activation function. It dynamically generates feature importance weights based on the spatial location of the melt pool.

[0101] The position-aware feature transformation submodule in the spatial information fusion module converts high-dimensional spatial coordinates into modulation factors (position modulator and position bias generator) through a multilayer perceptron. The modulation factors are then used to perform element-level modulation on the original image features (original features × modulator + bias) to achieve feature scaling based on spatial position.

[0102] The feature weighting and fusion submodule in the spatial information fusion module splices the attention-weighted original features (512 dimensions), the weighted spatially modulated melt pool image features (512 dimensions), and the spatial coding features (256 dimensions) into a 1280-dimensional high-dimensional feature.

[0103] The feature dimensionality reduction submodule in the spatial information fusion module reduces the dimensionality from 1280 to 512 through MLP, resulting in a spatially enhanced image feature representation.

[0104] Subsequently, temporal modeling of the spatially enhanced melt pool image features is performed through a temporal information fusion module. This module employs a gated recurrent unit (GRU) network architecture to perform temporal modeling of image features containing spatial location information. The GRU network receives spatially enhanced feature sequences of [batch, sequence length, 512] with a hidden dimension of 256 and a dropout rate of 0.3. Temporal information is gradually accumulated through update gates, reset gates, candidate hidden states, and the final hidden state, ultimately outputting the 256-dimensional hidden state at the last time step.

[0105] The classifier uses a fully connected layer (256→128→2) to generate binary classification probabilities to achieve defect detection.

[0106] The training process of the network for additive manufacturing defect detection based on spatiotemporal feature fusion of molten pool images includes: constructing a dataset by building 1404 cubic samples from 702 defect samples and an equal number of non-defect samples, and splitting them into a training set (50%), a validation set (30%), and a test set (20%) through hierarchical random sampling, and applying image quality enhancement, geometric transformation, and edge enhancement for data augmentation; using the Adam optimizer with hierarchical learning rate settings (feature extractor 1e-5, position encoder 1e-4, etc.) and dropout hierarchical settings (position encoder 0.1, fusion module 0.3, classifier 0.5), and using the binary cross-entropy loss function; initialization using the Kaiming normal distribution; batch size of 8, training for 100 epochs, and using the ReduceLROnPlateau scheduler (patience 7, factor 0.8) to monitor and validate accuracy.

[0107] The performance evaluation of this method includes: constructing a cube with a side length of 200 μm and adopting an upsampling strategy, the model accuracy is 94.7% and the F1 score is 94.9% under this configuration; ablation studies show that the spatial information fusion module improves the accuracy by 7.1% and the F1 score by 6.1%, while the temporal information fusion module improves both by 8.2%; adopting a center sampling strategy, the model accuracy is 88.0% and the F1 score is 88.6%; ablation studies show that the spatial information fusion module improves the accuracy by 4.6% and the F1 score by 3.9%, while the temporal information fusion module improves both by 3.5%; Gradient-weighted class activation heatmap (Grad-CAM) visualization analysis reveals the hierarchical feature learning mechanism of the model, with shallow layers capturing texture and edges, middle layers extracting morphological features, and deep layers integrating complex textures and gradient features to form a high-level representation related to defects.

[0108] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0109] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0110] Figure 9 This is a schematic diagram of an additive manufacturing defect detection device based on spatiotemporal feature fusion of molten pool images, provided in an embodiment of this application. Figure 9 As shown, the device includes:

[0111] The acquisition module 901 is configured to acquire the sequence of molten pool images, the three-dimensional spatial coordinates of the molten pool, and the tomographic scan images of the molded body during the additive manufacturing process.

[0112] The pre-training module 902 is configured to construct a 3D defect model based on the scanned image of the molded body using 3D reconstruction technology during the offline training phase; the 3D defect model includes at least pores and unfused defects.

[0113] The registration module 903 is configured to construct a 3D point cloud of the molten pool based on the 3D spatial coordinates of the molten pool, and establish the correspondence between the 3D defect model and the 3D point cloud of the molten pool.

[0114] The extraction module 904 is configured to extract local molten pool point clouds around the defect based on the defect formation mechanism, and match the local molten pool point clouds with the molten pool image sequence to obtain a defect-related molten pool image sequence.

[0115] The detection module 905 is configured to extract spatial location information and temporal information of the molten pool image through a deep learning network, and to perform defect detection based on the spatial location information and temporal information.

[0116] According to the technical solution provided in the embodiments of this application, by establishing a precise correspondence between the three-dimensional defect model and the point cloud of the molten pool, and by extracting local molten pool data using cubic spatial sampling based on the defect formation mechanism, upsampling and center sampling strategies are designed to improve the targeting of data extraction; by introducing a spatial information fusion module, including a spatial-feature interaction attention submodule and a position-aware feature transformation submodule, dynamic fusion of molten pool image features and spatial coordinates is achieved. Compared with the shortcomings of existing deep learning models that ignore spatial position information, the robustness and accuracy of feature representation are enhanced, which is especially suitable for defect identification in complex additive manufacturing environments; a temporal information fusion module is used to perform temporal modeling of spatial enhancement features, capturing the dynamic changes in the molten pool evolution process, and improving the real-time performance and prediction accuracy of defect detection.

[0117] In some implementations, establishing the correspondence between the 3D defect model and the 3D point cloud of the molten pool includes: using 3D reconstruction technology to determine the 3D model of the molded body based on the scanned image of the molded body; the 3D model of the molded body and the 3D defect model sharing a coordinate system; extracting the surface point cloud of the 3D model of the molded body, and using a registration algorithm to correspond the surface point cloud of the 3D model of the molded body to the 3D point cloud of the molten pool to obtain the registered 3D point cloud of the molten pool; and registering the 3D defect model with the registered 3D point cloud of the molten pool to obtain the spatial positional correspondence between the 3D defect model and the 3D point cloud of the molten pool.

[0118] In some implementations, extracting local melt pool point clouds around the defect includes: constructing a cube space of a preset size based on the corresponding position of the target defect in the three-dimensional point cloud of the melt pool; the target defect is any defect in the three-dimensional defect model; extracting the melt pool point cloud data in the constructed cube space to obtain the local melt pool point cloud around the target defect.

[0119] In some implementations, the cube space of the preset size is constructed in the following ways: the cube space is constructed with the centroid of the target defect as the center of the cube and the preset size as the side length of the cube; or, the cube space is constructed with the lowest point of the target defect as the center of the bottom surface of the cube and the preset size as the side length of the cube.

[0120] In some implementations, deep learning networks are used to extract features from the molten pool image, encode spatial coordinate information, and fuse the spatial location and temporal information of the molten pool image. This includes: extracting image features from each frame of the molten pool image in a defect-related molten pool image sequence using a deep learning network; encoding the molten pool's three-dimensional spatial coordinates into spatial coordinates using an encoder; fusing the spatial coordinates with the image features using a spatial information fusion module to obtain spatially enhanced molten pool image features; performing temporal modeling on the spatially enhanced molten pool image features using a temporal information fusion module to obtain molten pool image features containing both spatial location and temporal information; and outputting defect detection results using a classifier based on the molten pool image features containing both spatial location and temporal information.

[0121] In some implementations, spatial coordinates and image features are fused using a spatial information fusion module, including: receiving spatial coordinates and image features using a spatial-feature interaction attention submodule, and dynamically generating attention weights based on the spatial coordinates of the molten pool using a neural network; converting spatial coordinates into modulation factors using a position-aware feature transformation submodule via a multilayer perceptron, and using the modulation factors to perform element-level modulation on the image features to obtain position-modulated features; weighting and fusing the position-modulated features and image features based on the attention weights using a feature weighting and fusion submodule, and fusing them with the spatial coordinates to obtain fused high-dimensional features; and reducing the dimensionality of the fused high-dimensional features to the same dimension as the image features using a feature dimensionality reduction submodule to obtain spatially enhanced molten pool image features.

[0122] In some implementations, temporal modeling of spatially enhanced molten pool image features is performed using a temporal information fusion module, including: using a gated recurrent unit network architecture to perform temporal modeling of spatially enhanced molten pool image features to obtain molten pool image features containing spatial location information and temporal information of the molten pool image.

[0123] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0124] Figure 10 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 10As shown, the electronic device 10 of this embodiment includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the memory 1002 and executable on the processor 1001. When the processor 1001 executes the computer program 1003, it implements the steps in the various method embodiments described above. Alternatively, when the processor 1001 executes the computer program 1003, it implements the functions of each module / unit in the various device embodiments described above.

[0125] Electronic device 10 may be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 10 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device 10 and does not constitute a limitation on electronic device 10. It may include more or fewer components than shown, or different components.

[0126] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0127] The memory 1002 can be an internal storage unit of the electronic device 10, such as a hard disk or RAM of the electronic device 10. The memory 1002 can also be an external storage device of the electronic device 10, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, FlashCard, etc., equipped on the electronic device 10. The memory 1002 can also include both internal and external storage units of the electronic device 10. The memory 1002 is used to store computer programs and other programs and data required by the electronic device.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0130] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for detecting defects in additive manufacturing based on spatiotemporal feature fusion of molten pool images, characterized in that, include: Acquire the sequence of molten pool images, the three-dimensional spatial coordinates of the molten pool, and the tomographic scan images of the molded body during the additive manufacturing process; During the offline training phase, a 3D reconstruction technique is used to construct a 3D model of the defects based on the scanned images of the molded body; The three-dimensional model of the defect includes at least porosity and incomplete fusion defects; Based on the three-dimensional spatial coordinates of the molten pool, a three-dimensional point cloud of the molten pool is constructed, and the correspondence between the three-dimensional model of the defect and the three-dimensional point cloud of the molten pool is established. Based on the defect formation mechanism, local molten pool point clouds are extracted around the defect, and the local molten pool point clouds are matched with the molten pool image sequence to obtain a defect-related molten pool image sequence. Spatial location information and temporal information of the molten pool image are extracted by a deep learning network, and defect detection is achieved based on the spatial location information and temporal information.

2. The method according to claim 1, characterized in that, Establishing the correspondence between the 3D defect model and the 3D point cloud of the molten pool includes: A 3D reconstruction technique is used to determine a 3D model of the molded object based on the scanned image of the molded object; the 3D model of the molded object and the 3D model of the defect share a coordinate system; Extract the surface point cloud of the three-dimensional model of the molded body, and use a registration algorithm to match the surface point cloud of the three-dimensional model of the molded body with the three-dimensional point cloud of the molten pool to obtain the registered three-dimensional point cloud of the molten pool. The defect 3D model is registered with the registered molten pool 3D point cloud to obtain the spatial positional correspondence between the defect 3D model and the molten pool 3D point cloud.

3. The method according to claim 1, characterized in that, Extracting local melt pool point clouds around the defect, including: A cubic space of a preset size is constructed based on the corresponding position of the target defect in the three-dimensional point cloud of the molten pool; the target defect is any defect in the three-dimensional defect model. Extract the point cloud data of the molten pool in the constructed cubic space to obtain the local point cloud of the molten pool around the target defect.

4. The method according to claim 3, characterized in that, A cube space of a preset size is constructed in the following manner: Construct a cubic space with the centroid of the target defect as the center of the cube and the preset dimensions as the side length of the cube; Alternatively, construct a cubic space with the lowest point of the target defect as the center of the cube's bottom surface and the cube's side length as the preset dimensions.

5. The method according to claim 1, characterized in that, Deep learning networks are used to extract features from the molten pool image, encode spatial coordinate information, and fuse the spatial location and temporal information of the molten pool image, including: Image features of each frame of the molten pool image in the defect-related molten pool image sequence are extracted using a deep learning network. The three-dimensional spatial coordinates of the molten pool are encoded into spatial coordinates using an encoder; The spatial coordinates are fused with the image features by the spatial information fusion module to obtain spatially enhanced molten pool image features; The spatially enhanced molten pool image features are modeled temporally using a time information fusion module to obtain molten pool image features that include both spatial location information and temporal information of the molten pool image. The classifier outputs defect detection results based on the features of the molten pool image, which includes spatial location information and temporal information of the molten pool image.

6. The method according to claim 5, characterized in that, The spatial coordinates are fused with the image features through a spatial information fusion module, including: The spatial-feature interaction attention submodule receives the spatial coordinates and the image features, and uses a neural network to dynamically generate attention weights based on the spatial coordinates of the melt pool. The spatial coordinates are converted into modulation factors by a multilayer perceptron using a position-aware feature transformation submodule. The image features are then element-wise modulated using the modulation factors to obtain position-modulated features. The feature weighting and fusion submodule uses the attention weight to weight and fuse the position modulation features and the image features, and then fuses them with spatial coordinates to obtain the fused high-dimensional features; The dimensionality reduction submodule is used to reduce the dimensionality of the fused high-dimensional features to the same dimension as the image features, thereby obtaining the spatially enhanced molten pool image features.

7. The method according to claim 5, characterized in that, Temporal modeling of the spatially enhanced molten pool image features is performed using a temporal information fusion module, including: A gated recurrent unit network architecture is used to perform temporal modeling on the spatially enhanced molten pool image features to obtain molten pool image features that include both spatial location information and temporal information of the molten pool image.

8. An additive manufacturing defect detection device based on spatiotemporal feature fusion of molten pool images, characterized in that, include: The acquisition module is configured to acquire a sequence of molten pool images, three-dimensional spatial coordinates of the molten pool, and tomographic scan images of the molded body during the additive manufacturing process. The pre-training module is configured to construct a 3D model of defects based on scanned images of the molded body using 3D reconstruction technology during the offline training phase. The three-dimensional model of the defect includes at least porosity and incomplete fusion defects; The registration module is configured to construct a three-dimensional point cloud of the molten pool based on the three-dimensional spatial coordinates of the molten pool, and establish the correspondence between the three-dimensional model of the defect and the three-dimensional point cloud of the molten pool; The extraction module is configured to extract local molten pool point clouds around the defect based on the defect formation mechanism, and match the local molten pool point clouds with the molten pool image sequence to obtain a defect-related molten pool image sequence. The detection module is configured to extract spatial location information and temporal information of the molten pool image through a deep learning network, and to perform defect detection based on the spatial location information and temporal information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images