A metal additive manufacturing monitoring method based on tracking and adaptive image segmentation

Abstract

The present invention discloses a metal additive manufacturing monitoring method based on tracking and adaptive image segmentation. The method utilizes an image particle filter and a mean-shift algorithm to track a tracking object in an image. The weights in the image particle filter tracking algorithm are then compared with the PHOG features of each particle after tracking using the mean-shift algorithm to determine whether the mean-shift algorithm needs to be used for re-tracking. The method then resamples each particle in the particle filter coordinate set. The coordinate set of the tracking object within the target area at the positioning moment after resampling is calculated. The additive process image is segmented according to the tracking object coordinate set, and useless background is deleted. This optimizes the segmented image and improves product quality. The segmented image can be input into a neural network model for artificial intelligence recognition calculation.

Description

Technical Field

[0001] The present invention belongs to the field of additive manufacturing image recognition and tracking, and in particular relates to a metal additive manufacturing monitoring method based on tracking and adaptive image segmentation. Background Art

[0002] Currently, there are many methods for metal additive manufacturing, which can be divided into two categories: indirect forming and direct forming. Plasma directed energy deposition laser melting is a method of metal powder rapid prototyping technology for direct forming in additive manufacturing. It has great advantages in the overall forming of high-quality, complex, precise and personalized customized metal components. However, the parts formed by plasma directed energy deposition laser melting are prone to defects such as balling, porosity and unfusion. This is caused by the characteristics of diversified process parameters and complex manufacturing process. The complex heat conduction process accompanied by spattering will seriously affect the quality of the formed parts and thus have an adverse effect on subsequent processes. Therefore, it is particularly important to monitor metal additive manufacturing indicators at each moment, such as melt pool morphology, sputtering morphology, and arc morphology.

[0003] Currently, neural networks built using depthwise separable convolutions extract both deep and shallow features to process and monitor metal additive manufacturing metrics at every moment, such as melt pool morphology, sputtering morphology, and arc morphology. However, when high-speed cameras capture images, many images contain overlapping information, and the image characteristics and detection metrics themselves are relatively small, resulting in a large invalid range for each image, which significantly interferes with the extracted metrics. To meet the requirements of faster processing speeds, a metal additive manufacturing monitoring method based on tracking and adaptive image segmentation can better meet these requirements and is of great significance to the development of metal additive manufacturing. Summary of the Invention

[0004] In response to the above problems, the present invention provides a metal additive manufacturing monitoring method based on tracking and adaptive image segmentation. The method uses image particle filtering and Mean-shift algorithms to track the tracking object in the image. Then, the weight in the image particle filter tracking algorithm is compared with the PHOG feature of each particle after tracking by the Mean-shift algorithm to determine whether the Mean-shift algorithm needs to be re-tracked. After that, each particle in the particle filter coordinate set is resampled; the coordinate set of the tracking object in the target area at the positioning moment after resampling is calculated; the additive process image is segmented according to the tracking object coordinate set, and useless background is deleted, so as to optimize the segmented image and improve product quality. The segmented image can be input into a neural network model for artificial intelligence recognition calculation.

[0005] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] A metal additive manufacturing monitoring method based on tracking and adaptive image segmentation, characterized by comprising the following steps:

[0007] Step 1, image acquisition: using an image acquisition device to acquire an image of the additive process, wherein the image of the additive process includes a number of objects to be tracked;

[0008] Step 2: Image preprocessing: performing contrast transformation on all additive process images, and removing salt and pepper and Gaussian noise by random perturbation of each pixel RGB of the additive process images;

[0009] Step 3: perform interval sampling on the pre-processed images, and form an image sample set from the sampled additive process images;

[0010] Step 4: Read the additive process images in the image sample set obtained in step 3 in chronological order. The image time sequence is recorded as k. Select a tracking object, create a target area and mark it. The target area is a rectangular box centered on the tracking object.

[0011] Step 5: Initialize the image particle filter and randomly generate N initial particles for particle filtering in the target area;

[0012] Step 6: Perform particle filter tracking on the target area and calculate the coordinates of each particle in the particle filter coordinate set at the kth positioning moment;

[0013] Step 7: Calculate the weight of each particle in the coordinate set of the particle filter according to the particle coordinates at the kth moment obtained in step 6;

[0014] Step 8: While executing step 7, perform mean-shift algorithm tracking on the target area initialized in step 5;

[0015] Step 9: After tracking with the Mean-shift algorithm, extract the PHOG features of each particle in the target area;

[0016] Step 10: Compare the weight of each particle in the coordinate set of the particle filter at each positioning moment obtained in step 7 with the PHOG features of the corresponding particles in step 9 to obtain the judgment index w m , the formula is as follows;

[0017]

[0018] In the above formula, represents the weight of the mth particle in the particle filter coordinate set at the kth positioning moment, P((x k ,y k ) m ) is the PHOG feature of the mth particle at the kth positioning moment, m ranges from 1 to N, and k is the ordinal number of the positioning moment;

[0019] Step 11: Give the judgment indicator w m Set the threshold w0; if at this moment the target area in the additive process image is within the target area, the judgment index w for all filter particles m If both are greater than the set threshold w0, then execute step 12;

[0020] Otherwise, return to step 8, re-perform the Mean-shift algorithm tracking and recalculate the PHOG features and judgment;

[0021] Step 12: resampling of each particle in the particle filter coordinate set at the k-th positioning moment;

[0022] Step 13: Calculate the coordinate set of the tracked object in the target area at the kth positioning moment after resampling.

[0023] Step 14: According to the coordinate set of the tracked object Segment the additive process image and remove unnecessary background.

[0024] Furthermore, in the metal additive manufacturing monitoring process, the objects tracked include the molten pool, plasma arc and sputtering.

[0025] Furthermore, in step 3, the sampling interval is 50-200 sheets.

[0026] Furthermore, in step 5, the number N of initial particles is set to be 100-500.

[0027] Furthermore, in step 6, the Kalman filter algorithm is used in the particle filter tracking process to determine the coordinates of each particle at a certain positioning moment; and the coordinates of each particle at the positioning moment are composed of the state coordinate set of the particle filter at the positioning moment. The coordinates of each particle at a certain positioning moment are calculated as follows:

[0028]

[0029]

[0030] Among them, k represents the time sequence number, and are the horizontal and vertical coordinates of the kth generation moment of the mth particle; and are the horizontal and vertical coordinates of the k-1th generation time of the mth particle. When k=1, and The horizontal and vertical coordinates represent the initial time of the mth particle; w k is the quasi-perturbed salt and pepper and Gaussian noise of the additive process image at the kth moment, and f is the characteristic function of the Kalman filter algorithm.

[0031] Furthermore, in step 7, the weight calculation formula of each particle in the state coordinate set of the particle filter at the k-th positioning moment is as follows:

[0032]

[0033] in, Represents the weight of the mth particle in the particle filter coordinate set at the kth positioning moment.

[0034] Furthermore, in step 8, during the tracking process of the mean-shift algorithm, the filter particle update formula in the target area of ​​the additive process image is as follows:

[0035]

[0036]

[0037] and are the horizontal and vertical coordinates of the mth particle at the kth generation moment during the tracking process of the Mean-shift algorithm, and are the horizontal and vertical coordinates of the mth particle at the k-1th generation time during the tracking process of the Mean-shift algorithm, and is the center coordinate of the rectangle representing the target area in step 4, written in vector form (x c ,y c ) m , g is the kernel function of the Mean-shift algorithm.

[0038] Furthermore, in step 14, the specific method of segmenting the additive process image is as follows:

[0039] Take the tracking object coordinate set respectively The minimum and maximum values ​​of the horizontal and vertical coordinates are recorded as Subtract the cutting allowance v0 from the minimum value of the horizontal and vertical coordinates and record it as The maximum values ​​of the horizontal and vertical coordinates are added with the cutting allowance v0 and recorded as The new coordinates are organized into four extreme points, namely The image of the additive process is cut by constructing a rectangular frame with four extreme points, retaining the target area containing the tracked object.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] 1) Image preprocessing operations include contrast transformation, removing salt and pepper and Gaussian noise, taking image samples at regular intervals to reduce image memory usage, avoid images with a lot of information overlap, and improve the usability of the image.

[0042] 2) Within the framework of the Mean-shift tracking algorithm, the particle filter method and PHOG feature detection and extraction technology are combined to adaptively select a new round of Mean-shift tracking algorithm based on the threshold. The algorithm has the characteristics of rapidity, uniqueness, high stability and high precision, and can achieve good results in the image positioning of the molten pool, plasma arc and sputtering in the later tracking of metal additive manufacturing process.

[0043] The image size is adaptively processed based on slight differences in the image center and range, increasing the speed of extracting deep and shallow features of the neural network built with deep separable convolution in the metal additive manufacturing process, making the overall accuracy and efficiency higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The present invention provides a flow chart for the metal additive manufacturing monitoring method based on tracking and adaptive image segmentation.

[0045] Figure 2 This is a working principle diagram of the metal additive manufacturing monitoring method based on tracking and adaptive image segmentation implemented by the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] In order to more clearly express the technology of the present invention, the additive manufacturing technology monitored by the present invention is first indirectly described. The plasma processing head of the metal additive manufacturing process monitoring image acquisition device is set at the edge of the experimental platform, emitting a plasma arc as a heat source, and forming metal droplets and subsequent solidification through continuous local melting of powder particles to form the component layer to be manufactured. Subsequently, another powder layer is applied to the processed powder layer, and the process starts again. Therefore, the component is further constructed with each new powder layer, and the construction direction is arranged to be perpendicular to the corresponding plane of the powder layer to achieve the forming of the part; a high-speed camera is set directly above the metal additive manufacturing process experimental platform and connected to a computer; an 808-nanometer laser is set at the bottom of the metal additive manufacturing process experimental platform to emit a light source to illuminate the background, and finally the morphological characteristics of the metal additive manufacturing process that captures clear plasma arcs are uploaded to the image processing system.

[0048] The plasma arc, molten pool and sputtering images taken by the high-speed camera in the metal additive manufacturing process are pre-processed and sampled.

[0049] The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation designed by the present invention includes the following steps:

[0050] Step 1, Image Acquisition: An image acquisition device is used to capture an image of the additive process. The image contains several objects to be tracked. Different images can be obtained by varying the scanning speed and current of the plasma processing head. In this embodiment, the scanning speed of the plasma processing head is 5 mm / s and the current is 25 A. The original additive process image has an 8-bit pixel format and a size of 450 × 512 pixels.

[0051] Step 2: Image preprocessing: performing contrast transformation on all additive process images, and removing salt and pepper and Gaussian noise by random perturbation of each pixel RGB of the additive process images;

[0052] Specifically, the collected images are subjected to contrast transformations, and the value of each pixel on the 8-bit grayscale image is linearly transformed. Gaussian and salt and pepper filters are used to move the central element of the relevant kernel so that it is located directly above the pixels to be processed in the input images of the plasma arc, molten pool, and sputtering. The pixel values ​​of the input image are then used as weights, and the mean of the pixels within the filter window is taken as the output to remove most of the Gaussian and salt and pepper noise.

[0053] Step 3: perform interval sampling on the preprocessed images, and form an image sample set with the sampled additive process images; the sampling interval is 50-200 images. In this embodiment, image samples are taken every 100 images to form an image sample set for subsequent tracking calculation.

[0054] Step 4: Read the additive process images obtained in the image sample set in step 3 in chronological order. The image time sequence is recorded as k. Select a tracking object (such as a melt pool), create a target area and mark it. The target area is a rectangular box centered on the tracking object (melt pool);

[0055] Step 5: Initialize the image particle filter. Randomly generate N initial particles for particle filtering in the target area. N is greater than or equal to 100. In this embodiment, N is set to 200.

[0056] Step 6: Perform particle filter tracking on the target area and calculate the coordinates of each particle in the particle filter coordinate set at the kth positioning moment;

[0057] The present invention uses the Kalman filter algorithm to determine the coordinates of each particle at a certain positioning moment; and the coordinates of each particle at the positioning moment constitute the state coordinate set of the particle filter at the positioning moment. The coordinates of each particle at a certain positioning moment are calculated as follows:

[0058]

[0059]

[0060] Among them, k represents the time sequence number, and is the horizontal and vertical coordinates of the kth generation moment of the mth particle, written in vector form (x k ,y k ) m ; and are the horizontal and vertical coordinates of the k-1th generation time of the mth particle. When k=1, and The horizontal and vertical coordinates represent the initial time of the mth particle; w k is the pseudo-perturbation salt and pepper, Gaussian noise of the additive process image at the kth moment, and f is the characteristic function of the Kalman filter algorithm;

[0061] Step 7: Based on the particle coordinates at the kth moment obtained in step 6, calculate the weight of each particle in the coordinate set of the particle filter. The calculation formula is as follows:

[0062]

[0063] in, It represents the weight of the mth particle in the particle filter coordinate set at the kth positioning moment. When k=0, it represents the weight of each particle at the initial moment.

[0064] Step 8: While executing step 6, perform mean-shift tracking on the target area initialized in step 5. The formula for updating the filter particles in the target area of ​​the additive process image is as follows:

[0065]

[0066]

[0067] and is the horizontal and vertical coordinates of the mth particle at the kth generation time in the tracking process of the Mean-shift algorithm, written in vector form as (x′ k ,y′ k ) m ; and are the horizontal and vertical coordinates of the mth particle at the k-1th generation time during the tracking process of the Mean-shift algorithm, and is the center coordinate of the rectangle representing the target area in step 4, written in vector form (x c ,y c ) m, g is the kernel function of the Mean-shift algorithm.

[0068]

[0069] In the above formula, N() is a function.

[0070] x represents the independent variable; x s , x r The coordinate points (x′ k ,y′ k ) m Pixel position features, color features; m s , m r The coordinate points (x′ k ,y′ k ) m The position and color bandwidth of the particle, e is the natural base, and C is the normalization constant;

[0071] Step 9: After tracking with the Mean-shift algorithm, extract the PHOG features of each particle in the target area;

[0072] Step 10: Compare the weight of each particle in the coordinate set of the particle filter at each positioning moment obtained in step 6 with the PHOG features of the corresponding particles in step 9 to obtain the judgment index w m , the formula is as follows;

[0073]

[0074] In the above formula, represents the weight of the mth particle in the particle filter coordinate set at the kth positioning moment, P((x k ,y k ) m ) is the PHOG feature of the mth particle at the kth positioning moment, m ranges from 1 to N, and k is the ordinal number of the positioning moment;

[0075] Step 11: Give the judgment indicator w m Set the threshold w0; if at this moment the target area in the additive process image is within the target area, the judgment index w for all filter particles m If both are greater than the set threshold w0, then execute step 12;

[0076] Otherwise, return to step 8, re-perform the Mean-shift algorithm tracking and recalculate the PHOG features and judgment;

[0077] The threshold w0 is generally set to 0.5-0.85. In this embodiment, the threshold w0 is set to 0.7.

[0078] Step 12: resampling of each particle in the particle filter coordinate set at the k-th positioning moment;

[0079] Step 13: Calculate the coordinate set of the tracked object in the target area at the kth positioning moment after resampling.

[0080] Step 14: According to the coordinate set of the tracked object Segment the additive process image and remove unnecessary background.

[0081] The specific method for segmenting the additive process image is as follows:

[0082] Take the tracking object coordinate set respectively The minimum and maximum values ​​of the horizontal and vertical coordinates are recorded as Subtract the cutting allowance v0 from the minimum value of the horizontal and vertical coordinates and record it as The maximum values ​​of the horizontal and vertical coordinates are added with the cutting allowance v0 and recorded as The new coordinates are organized into four extreme points, namely The image of the additive process is cut by constructing a rectangular frame with four extreme points, retaining the target area containing the tracked object.

[0083] The segmented image is imported into the deep separable convolution neural network. The deep convolutional network (DCNN) is used as the backbone network and supplemented by the convolutional neural network (CNN). The design adopts multi-scale atrous convolution cascade or parallel to capture features, and uses a pre-trained residual network to extract the image features of the molten pool, plasma arc and sputtering in the metal additive manufacturing process. The obtained deep feature map and shallow feature map are upsampled to the original image size. Finally, the detection indicators of metal additive manufacturing such as the molten pool, sputtering and arc morphology are obtained more completely.

[0084] It should be noted that the resampling in step 12 of the present invention uses the nearest neighbor interpolation method in the prior art. Based on the width (height) ratio of the target image to the source image, the pixel at the corresponding position in the source image is used as the value of the target pixel. The new pixel value must be a pixel value of the original image.

[0085] It should be noted that the tracking objects include molten pools, plasma arcs, sputtering, etc., and one object can be tracked at a time, or multiple objects can be tracked at a time. However, after the final segmentation, a separate cutting image of each tracked object is obtained and input into the neural network for calculation.

[0086] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A metal additive manufacturing monitoring method based on tracking and adaptive image segmentation, characterized in that: The following steps are involved: Step 1, image acquisition: using an image acquisition device to acquire an image of the additive process, wherein the image of the additive process includes a number of objects to be tracked; Step 2: Image preprocessing: performing contrast transformation on all additive process images, and removing salt and pepper and Gaussian noise by random perturbation of each pixel RGB of the additive process images; Step 3: perform interval sampling on the pre-processed images, and form an image sample set from the sampled additive process images; Step 4: Read the additive process images in the image sample set obtained in step 3 in chronological order. The image time sequence is recorded as k. Select a tracking object, create a target area and mark it. The target area is a rectangular box centered on the tracking object. Step 5: Initialize the image particle filter and randomly generate N initial particles for particle filtering in the target area; Step 6: Perform particle filter tracking on the target area and calculate the coordinates of each particle in the particle filter coordinate set at the kth positioning moment; Step 7: Calculate the weight of each particle in the coordinate set of the particle filter according to the particle coordinates at the kth moment obtained in step 6; Step 8: While executing step 7, perform mean-shift algorithm tracking on the target area initialized in step 5; Step 9: After tracking with the Mean-shift algorithm, extract the PHOG features of each particle in the target area; Step 10: Compare the weight of each particle in the coordinate set of the particle filter at each positioning moment obtained in step 7 with the PHOG features of the corresponding particles in step 9 to obtain the judgment index w m , the formula is as follows; In the above formula, represents the weight of the mth particle in the particle filter coordinate set at the kth positioning moment, P((x k ,y k ) m ) is the PHOG feature of the mth particle at the kth positioning moment, m ranges from 1 to N, and k is the ordinal number of the positioning moment; Step 11: Give the judgment indicator w m Set the threshold w0; if at this moment the target area in the additive process image is within the target area, the judgment index w for all filter particles m If both are greater than the set threshold w0, then execute step 12; Otherwise, return to step 8, re-perform the Mean-shift algorithm tracking and recalculate the PHOG features and judgment; Step 12: resampling of each particle in the particle filter coordinate set at the k-th positioning moment; Step 13: Calculate the coordinate set of the tracked object in the target area at the kth positioning moment after resampling. Step 14: According to the coordinate set of the tracked object Segment the additive process image and remove unnecessary background.

2. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 1, characterized in that: During the monitoring process of metal additive manufacturing, the objects being tracked include the molten pool, plasma arc, and sputtering.

3. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 1, characterized in that: In step 3, the sampling interval is 50-200 sheets.

4. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 1, characterized in that: In step 5, the number N of initial particles is set to 100-500.

5. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 1, characterized in that: In step 6, the Kalman filter algorithm is used in the particle filter tracking process to determine the coordinates of each particle at a certain positioning moment; and the coordinates of each particle at that positioning moment constitute the state coordinate set of the particle filter at that positioning moment. The coordinates of each particle at a certain positioning moment are calculated as follows: Among them, k represents the time sequence number, and are the horizontal and vertical coordinates of the kth generation moment of the mth particle; and are the horizontal and vertical coordinates of the k-1th generation time of the mth particle. When k=1, and The horizontal and vertical coordinates represent the initial time of the mth particle; w k is the quasi-perturbed salt and pepper and Gaussian noise of the additive process image at the kth moment, and f is the characteristic function of the Kalman filter algorithm.

6. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 5, characterized in that: In step 7, the weight calculation formula of each particle in the state coordinate set of the particle filter at the kth positioning moment is as follows: in, Represents the weight of the mth particle in the particle filter coordinate set at the kth positioning moment.

7. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 6, characterized in that: In step 8, during the tracking process of the mean-shift algorithm, the filter particle update formula in the target area of ​​the additive process image is as follows: and are the horizontal and vertical coordinates of the mth particle at the kth generation moment during the tracking process of the Mean-shift algorithm, and are the horizontal and vertical coordinates of the mth particle at the k-1th generation time during the tracking process of the Mean-shift algorithm, and is the center coordinate of the rectangle representing the target area in step 4, written in vector form (x c ,y c ) m , g is the kernel function of the Mean-shift algorithm.

8. The metal additive manufacturing monitoring method based on tracking and adaptive image segmentation according to claim 7, characterized in that: In step 14, the specific method for segmenting the additive process image is as follows: Take the tracking object coordinate set respectively The minimum and maximum values ​​of the horizontal and vertical coordinates are recorded as Subtract the cutting allowance v0 from the minimum value of the horizontal and vertical coordinates and record it as The maximum values ​​of the horizontal and vertical coordinates are added with the cutting allowance v0 and recorded as The new coordinates are organized into four extreme points, namely The image of the additive process is cut by constructing a rectangular frame with four extreme points, retaining the target area containing the tracked object.

Images