Visual defect detection and environmental perception integrated method and system for online monitoring metal laser processing process

Abstract

The invention provides a visual defect detection and environmental perception integrated method and system for online monitoring of a metal laser processing process. The method comprises the following steps: S1, carrying out time-synchronous multi-modal data acquisition on the online monitoring of the metal laser processing process; s2, preprocessing the image data; and S3, image defect identification is carried out. And S4, carrying out data fusion and information superposition on the preprocessed image data, defect data and environment data. And S5, carrying out edge coding and cloud pushing on the total image sequence after data fusion and information superposition.

Description

TECHNICAL FIELD

[0001] The application provides a method and system for online monitoring of a laser processing metal process by integrating visual defect detection and environment perception, and relates to the technical field of manufacturing and process online monitoring. BACKGROUND

[0002] Currently, selective laser melting (SLM) technology is an important part of additive manufacturing technology, and has irreplaceable advantages over traditional production and processing technology in manufacturing complex metal parts. It melts metal powder in a specific area through laser scanning, and then accumulates and forms layer by layer. Therefore, it plays an extremely important role in the field of manufacturing complex metal parts.

[0003] During the powder laying process of SLM, powder supply defects have a great influence on the quality and performance of the formed parts. These defects will directly affect the interaction between laser and material, leading to the generation of other defects such as spatter, spheroidization, porosity, poor surface quality, cracks, geometric deformation, etc.

[0004] The powder laying defects in the SLM production process are mostly detected by manual intermittent offline detection method. This method has the problems of easy missed detection, false detection, low efficiency and long response time, which greatly affects the production efficiency and product quality. At the same time, SLM process is also very sensitive to environmental factors such as temperature and humidity in the printing bin. The fluctuation of the environment may be one of the reasons for the powder laying defects. The traditional manual detection method cannot establish the correlation between the defects and the environmental parameters. Although there are researches on automatic visual detection scheme based on PC, these schemes generally have high hardware cost, low system integration, high power consumption, and cannot integrate environmental sensing information, nor can they realize convenient remote monitoring, which does not meet the development requirements of intelligent manufacturing. SUMMARY

[0005] Therefore, in order to make up for the gaps and deficiencies of the prior art, the application provides a method and system for online monitoring of a laser processing metal process by integrating visual defect detection and environment perception. The core of the application is to optimize and transplant the image processing flow that depends on PC and Visual Studio platform to the embedded SOC platform, and innovatively integrate environment perception and cloud streaming media functions, aiming at the difficulties in defect detection during the powder laying process of SLM powder bed, and a series of problems such as missed detection, false detection, low efficiency, inability to correlate environmental factors, and inability to remotely monitor existing detection means.

[0006] The application provides a method and system for online monitoring of a laser processing metal process by integrating visual defect detection and environment perception, comprising the following contents:

[0007] The application provides a method for online monitoring of a laser processing metal process by integrating visual defect detection and environmental perception, characterized in that it comprises the following steps:

[0008] Step S1: Time-synchronized multi-modal data acquisition for online monitoring of a laser processing metal process, comprising

[0009] On an embedded SOC platform, image data and environmental data in the powder laying process of selective laser melting are synchronously collected through a unified timestamp mechanism, wherein the image data is acquired by a CCD / CMOS camera, the environmental data at least includes temperature T, humidity H and air pressure P, and all data are labeled with microsecond-level timestamps;

[0010] Step S2: Preprocessing of the image data, comprising the following steps:

[0011] The following preprocessing operations are sequentially performed on the SOC platform: a two-dimensional gamma function-based uneven illumination image adaptive correction algorithm is used to correct the brightness of the image; a histogram equalization algorithm is used to improve the contrast of the image; a Gaussian filter algorithm is used to remove Gaussian noise in the image; a median filter algorithm is used to remove salt and pepper noise in the image, to obtain the preprocessed image;

[0012] Step S3: Image defect recognition, comprising the following steps:

[0013] An iterative threshold segmentation algorithm is used to segment the image, to identify a powder supply deficiency defect area, and to generate defect feature data D, wherein the defect feature data D includes defect position, area and gray scale features, and wherein the convergence threshold T0 of the iterative threshold segmentation algorithm is dynamically adjusted according to the current environmental parameters;

[0014] Step S4: Data fusion and information superposition of the preprocessed image data, defect data and environmental data, comprising the following steps:

[0015] A data record R containing timestamps, environmental parameters and defect features is established, the defect feature data D and environmental data {T, H, P} are correlated and analyzed, the correlation coefficient of defect occurrence probability P and environmental deviation degree is calculated, and the analysis results are superimposed on the corresponding image frames in text and graphical forms;

[0016] Step S5: Edge encoding and cloud pushing of the total image sequence after data fusion and information superposition, comprising the following steps:

[0017] The image sequence with superimposed visual information is real-time encoded into a video stream by using the hardware video encoder of the SOC, and is pushed to a cloud server through the end-to-end delay of the protocol, wherein the cloud server stores historical data and provides defect-environment correlation trend analysis.

[0018] Further, step S1 includes the following:

[0019] In step S1, the environmental data is collected by sensors, including at least temperature and humidity, the sensors are connected to the SOC through I2C or SPI bus, the sampling frequency is not less than 1Hz, and the image collection is frame-level synchronized.

[0020] Further, step S2 includes the following:

[0021] In step S2, the illumination non-uniform image adaptive correction algorithm based on two-dimensional gamma function includes: extracting multi-level illumination components L of the image by using multi-scale Gaussian function; calculating the global mean μ_L and standard deviation σ_L of the illumination component L; constructing an adaptive two-dimensional gamma function; applying a correction formula to each pixel (x, y) of the original image: realizing pixel-by-pixel adaptive brightness adjustment.

[0022] Further, step S2 also includes the following:

[0023] In step S2, the multi-scale Gaussian function includes:

[0024] {G_1,G_2,...,G_n}

[0025] Where G_n represents Gaussian functions of n different scales, used for noise removal of the image;

[0026] In step S2, the adaptive two-dimensional gamma function includes:

[0027] γ(x,y)=γ_base×[1+k×(L(x,y)-μ_L) / σ_L]

[0028] Where γ_base is the base gamma value, and k is the adaptive adjustment coefficient;

[0029] Applying the correction formula to each pixel (x, y) of the original image includes:

[0030] I_corrected(x,y)=255×[I(x,y) / 255]^(1 / γ(x,y))

[0031] Where I(x, y) represents brightness, and I_corrected(x, y) is used to realize pixel-by-pixel adaptive brightness adjustment.

[0032] Further, step S3 includes the following:

[0033] In step S3, the convergence threshold T0 of the iterative threshold segmentation algorithm is dynamically adjusted according to the current environmental parameters, and the adjustment formula includes:

[0034] T0 = T0_base x [1 + a x (T - T_ref) / T_ref + b x (H - H_ref) / H_ref]

[0035] wherein T0_base is a base convergence threshold, T_ref and H_ref are reference environmental parameters, and a and b are weighting coefficients.

[0036] Further, the step S4 comprises the following contents:

[0037] Step S41: establishing a time series database to store at least 100 consecutive frames of multi-modal data records;

[0038] Step S42: using a sliding window statistical analysis to calculate the Pearson correlation coefficient of the defect occurrence frequency f and the environmental parameters {T, H, P};

[0039] Step S43: when |correlation coefficient|>0.6, displaying a high correlation warning on the image and recording the key environmental threshold;

[0040] Step S44: generating a visual curve to superimposedly display the defect trend and the environmental fluctuation trend in the last 10 minutes.

[0041] Further, the step S5 comprises the following contents:

[0042] The image sequence of the visual information is real-time encoded into an H.264 / H.265 video stream;

[0043] The pushing to the cloud server comprises pushing to the cloud server through a protocol with an end-to-end delay of not more than 200 ms;

[0044] The edge coding and cloud pushing adopt the following optimization strategies: using a SOC hardware encoder acceleration interface, the coding rate is not less than 30 fps; according to the network bandwidth, the code rate is adaptively adjusted, the code rate range is 1-5 Mbps; through the transmission protocol pushing, reliable transmission and line reconnection are realized; a streaming media server is deployed on the cloud to support multi-client concurrent access, and the single-server concurrent number is ≥20.

[0045] According to the second aspect of the present application, the present application provides a system for integrating visual defect detection and environmental perception for online monitoring of laser processing metal process, which is used for executing the method for integrating visual defect detection and environmental perception for online monitoring of laser processing metal process, and is characterized in that the system comprises an image acquisition unit, an environmental perception unit, a core processing unit, an edge intelligent processing module, a cloud storage and analysis unit and a client access unit.

[0046] The image acquisition unit includes an industrial-grade camera, which is connected to the core processing unit via a USB interface, with a resolution of no less than 1920×1080.

[0047] The environmental perception unit includes a multi-functional sensor module, which is connected to the core processing unit via an I2C / SPI interface;

[0048] The core processing unit includes an embedded SOC development board, which contains at least a quad-core processor, RAM, a hardware video encoder, and GPIO interfaces, and runs an embedded operating system.

[0049] The edge intelligence processing module includes an application deployed on the core processing unit, which realizes image preprocessing, defect recognition, data fusion and video encoding functions;

[0050] The cloud storage and analysis unit includes cloud servers that deploy RTSP streaming media services and databases, providing video stream distribution, historical data storage and trend analysis functions;

[0051] The client access unit includes PC or mobile applications that access the cloud server via the network to view monitoring screens and analysis reports in real time.

[0052] According to a third aspect of the present invention, the present invention provides a system for online monitoring of laser-processed metals that integrates visual defect detection and environmental perception, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a method for online monitoring of laser-processed metals that integrates visual defect detection and environmental perception as described in any one of the present invention.

[0053] According to a fourth aspect of the present invention, the present invention provides a system for online monitoring of laser metal processing processes that integrates visual defect detection and environmental perception, comprising a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the method for online monitoring of laser metal processing processes that integrates visual defect detection and environmental perception as described in any one of the present invention.

[0054] The present invention has the following advantages:

[0055] The present application constructs a powerful, low-cost and easy-to-deploy SLM online monitoring system by deeply integrating a set of mature image processing algorithms with modern embedded systems, Internet of Things sensor technology and streaming media technology. It not only realizes the automation of defect detection, but also provides an unprecedented powerful tool for quality control and intelligent management of SLM process through data fusion and remote presentation. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 The step flowchart of the present application is shown.

[0057] Figure 2 The monitoring system schematic diagram of the present application is shown.

[0058] Figure 3 The data flow diagram of the present application is shown. DETAILED DESCRIPTION

[0059] The technical solutions of the present application will be specifically described below in combination with the drawings.

[0060] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0061] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0062] As shown in Figures 1 to 3 The present application proposes a method and system for online monitoring of laser processing metal process integrating visual defect detection and environmental perception, characterized in that it comprises the following contents:

[0063] The present application proposes a method for online monitoring of laser processing metal process integrating visual defect detection and environmental perception, characterized in that it comprises the following contents:

[0064] Step S1: Time-synchronized multi-modal data acquisition for online monitoring of laser processing metal process, including

[0065] On an embedded SOC platform, through a unified timestamp mechanism, image data and environmental data in a selected area laser melting powder laying process are synchronously collected, wherein the image data is acquired by a CCD / CMOS camera, and the environmental data at least includes temperature T, humidity H and air pressure P, and all data are labeled with microsecond-level timestamps.

[0066] Step S2: pre-processing the image data, including the following contents:

[0067] The image data is sequentially executed on the SOC chip The following pre-processing operations are performed: using a two-dimensional gamma function-based adaptive correction algorithm for non-uniform illumination images, the image is corrected for brightness; using a histogram equalization algorithm, the contrast of the image is improved; using a Gaussian filter algorithm, Gaussian noise in the image is removed; using a median filter algorithm, salt and pepper noise in the image is removed, and a pre-processed image is obtained.

[0068] Step S3: image defect recognition, including the following contents:

[0069] An iterative threshold segmentation algorithm is used to segment the image I4, to identify a powder supply deficiency defect area, and to generate defect feature data D, wherein the defect feature data D includes defect location, area, and gray scale features, and wherein the convergence threshold T0 of the iterative threshold segmentation algorithm is dynamically adjusted according to the current environmental parameters.

[0070] Step S4: data fusion and information superposition on the pre-processed image data, defect data and environmental data, including the following contents:

[0071] A data record R containing timestamps, environmental parameters and defect features is established, the defect feature data D and environmental data {T, H, P} are analyzed, the correlation coefficient of defect occurrence probability P and environmental deviation degree is calculated, and the analysis results are superimposed on the corresponding image frames in text and graphical form.

[0072] Step S5: edge encoding and cloud pushing of the total image sequence after data fusion and information superposition, including the following contents:

[0073] The image sequence with superimposed visual information is real-time encoded into a video stream by using the hardware video encoder of the SOC, and is pushed to a cloud server through the end-to-end delay of the protocol, wherein the cloud server stores historical data and provides defect-environment correlation trend analysis.

[0074] Further, step S1 includes the following contents:

[0075] In step S1, the environmental data is collected by sensors connected to the SOC through I2C or SPI bus, including temperature and humidity, with a sampling frequency not less than 1 Hz, and frame-level synchronization with image collection.

[0076] In an embodiment of the present application, first, the present application proposes a SOC development board as a core platform, replacing the traditional PC. A camera (for example, a side-shaft CCD camera, with a specific height difference of 40 cm from the powder laying plane, and assisted by LED lights for illumination) connected to the SOC collects real-time images of the powder bed. Because the side-shaft camera is used for collection, and the influence of the on-site lighting conditions, the obtained pictures have problems such as local brightness unevenness, low contrast, and a large amount of noise. At the same time, through the I2C / SPI interface connected environmental sensors, the temperature, humidity, air pressure and other environmental data in the printing chamber are synchronously collected.

[0077] In an embodiment of the present application, in order to realize synchronous collection of data, a unified timestamp management mechanism is designed to ensure microsecond-level synchronization of image frames and environmental data. A multi-modal data fusion algorithm is proposed. Input: image sequence {I_t}, environmental sequence {E_t} (t is the timestamp) Output: fusion data record set {R_t}.

[0078] Wherein for each image I_t: perform defect detection, extract defect features D_t; query the environmental data E_t' closest to I_t timestamp (time difference < 50 ms); construct fusion record R_t = {t, I_t, D_t, E_t'}.

[0079] Wherein the sliding window analysis (window size N = 100): calculate the defect occurrence rate f = Σ(D_t exists defect) / N; calculate the environment-defect correlation coefficient ρ(D,T), ρ(D,H), ρ(D,P); when |ρ|>0.6, trigger environmental warning

[0080] Wherein the visualization superposition: draw the current environmental parameters on the image; label the defect area (red outline + area); display the environmental trend curve (recent 10 min).

[0081] Further, step S2 includes the following contents:

[0082] In step S2, the light non-uniformity image adaptive correction algorithm based on two-dimensional gamma function includes: extracting the multi-level light components L of the image using a multi-scale Gaussian function; calculating the global mean μ_L and standard deviation σ_L of the light component L; constructing an adaptive two-dimensional gamma function; applying the correction formula to each pixel (x, y) of the original image: realizing pixel-by-pixel adaptive brightness adjustment.

[0083] Further, step S2 further comprises the following:

[0084] In step S2, the multi-scale Gaussian function comprises:

[0085] {G_1,G_2,...,G_n}

[0086] Wherein, G_n represents Gaussian functions of n different scales, which are used for noise removal of the image.

[0087] In step S2, the adaptive two-dimensional gamma function comprises:

[0088] γ(x,y)=γ_base×[1+k×(L(x,y)-μ_L) / σ_L]

[0089] Wherein, γ_base is a base gamma value, and k is an adaptive adjustment coefficient.

[0090] Applying the correction formula to each pixel (x, y) of the original image comprises:

[0091] I_corrected(x,y)=255×[I(x,y) / 255]^(1 / γ(x,y))

[0092] Wherein, I(x, y) represents luminance, and I_corrected(x, y) is used to realize adaptive luminance adjustment of each pixel.

[0093] In an embodiment of the present application, the image preprocessing is automatically performed on the SOC chip. Firstly, for the problem of uneven luminance, one of the key improvements of the present application is to adopt a two-dimensional gamma function-based adaptive correction algorithm for uneven illumination image to correct the luminance. The implementation steps of the algorithm are as follows: the illumination component of the scene is effectively extracted by using a multi-scale Gaussian function, a two-dimensional gamma function is constructed according to the statistical characteristics of the illumination component, the function can adaptively adjust its parameters to match different illumination conditions, and finally the function is applied to adjust the luminance of each pixel, so that the image with uniform luminance is obtained, which is convenient for subsequent operation.

[0094] Secondly, for the problem of low contrast, a histogram equalization algorithm is adopted. The operation is as follows: firstly, the gray level histogram of the original image is calculated; secondly, the cumulative histogram (i.e. cumulative distribution function) is calculated; then, the cumulative histogram is normalized to the gray level range of 0-255, and is used as a gray level mapping function; finally, each pixel of the original image is converted into a new gray level value through the mapping function. By adjusting the histogram distribution of the image, the feature defects are made more clear.

[0095] Then, for the problem of noise existing in the image, a double filtering strategy is adopted. Gaussian filtering is adopted to remove the widely existing Gaussian noise in the image.

[0096] Finally, for randomly appearing black points (salt noise) and white points (pepper noise), median filtering is used to remove. Finally, output a pre-processed high-quality image.

[0097] Further, step S3 includes the following content:

[0098] In step S3, the convergence threshold T0 of the iterative threshold segmentation algorithm is dynamically adjusted according to the current environmental parameters, and the adjustment formula includes:

[0099] T0=T0_base x [1+alpha x (T-T_ref) / T_ref+beta x (H-H_ref) / H_ref]

[0100] Where T0_base is the reference convergence threshold, T_ref and H_ref are reference environmental parameters, and alpha and beta are weighting coefficients.

[0101] Further, step S4 includes the following content:

[0102] Step S41: Establish a time series database to store at least 100 consecutive frames of multi-modal data records;

[0103] Step S42: Use sliding window statistical analysis to calculate the Pearson correlation coefficient of defect occurrence frequency f and environmental parameters {T, H, P};

[0104] Step S43: When |correlation coefficient|>0.6, display high correlation warning on the image and record the key environmental threshold;

[0105] Step S44: Generate a visual curve to superimpose the defect trend and environmental fluctuation trend in the last 10 minutes.

[0106] In an embodiment of the present application, in step S4 of the present application, the data record R containing time stamp, environmental parameters and defect features is established, the defect feature data D and environmental data {T, H, P} are analyzed, the correlation coefficient of defect occurrence probability P and environmental deviation degree is calculated, and the analysis result is superimposed on the corresponding image frame in text and graphical manner using opencv.

[0107] Further, step S5 includes the following content:

[0108] The image sequence of the visual information is encoded into H.264 / H.265 video stream in real time;

[0109] The pushing to the cloud server includes pushing to the cloud server through a protocol with an end-to-end delay of not more than 200ms;

[0110] The edge coding and cloud pushing adopts the following optimization strategies: using the SOC hardware encoder acceleration interface, the coding rate is not less than 30 fps; the code rate is adaptively adjusted according to the network bandwidth, and the code rate range is 1-5 Mbps; reliable transmission and line reconnection are realized through the transmission protocol pushing; the stream media server is deployed in the cloud, and multiple client concurrent access is supported, and the single server concurrent number is greater than or equal to 20.

[0111] In an embodiment of the present application, another key improvement of the present application is that a three-layer architecture is designed, which is edge layer, cloud layer and client layer, wherein the edge layer realizes real-time image processing + defect recognition + video coding, and the delay is less than 50 ms. The cloud layer realizes stream media distribution + historical data storage + big data analysis, and supports 20 concurrent accesses. The client layer realizes multi-terminal access + interactive search, and the access delay is less than 200 ms.

[0112] Specifically, the image sequence superimposed with visual information is real-time encoded into H.264 / H.265 video stream by using the hardware video encoder of the SOC, and is pushed to the cloud server through the protocol with an end-to-end delay of not more than 200 ms. The cloud server stores historical data and provides defect-environment correlation trend analysis. This solves the problem that the existing technology cannot use the automatic degree and intelligent level to identify the powder shortage defect and perform remote and comprehensive monitoring. The present application has the characteristics of high automation degree, high recognition rate, low cost, rich information dimension and remote access, greatly avoids the use of excessive manpower and physical resources for identification and management, and is especially suitable for online monitoring of the powder shortage defect of the powder bed in the SLM manufacturing process.

[0113] According to the second aspect of the present application, a system for integrating visual defect detection and environment perception for online monitoring of laser processing metal process is provided, which is used to execute the method for integrating visual defect detection and environment perception for online monitoring of laser processing metal process, and is characterized by comprising an image acquisition unit, an environment perception unit, a core processing unit, an edge intelligent processing module, a cloud storage and analysis unit and a client access unit.

[0114] The image acquisition unit comprises an industrial-grade camera connected with the core processing unit through a USB interface, and the resolution is not less than 1920*1080.

[0115] The environment perception unit comprises a multifunctional sensor module connected with the core processing unit through an I2C / SPI interface.

[0116] The core processing unit comprises an embedded SOC development board, which at least contains a four-core processor, RAM, a hardware video encoder and a GPIO interface, and runs an embedded operating system.

[0117] The edge intelligence processing module includes an application deployed on the core processing unit, and realizes image preprocessing, defect identification, data fusion and video coding functions.

[0118] The cloud storage and analysis unit includes a cloud server deploying an RTSP streaming media service and a database, and provides video stream distribution, historical data storage and trend analysis functions.

[0119] The client access unit includes a PC end or mobile end application, and accesses the cloud server through a network to realize real-time monitoring of monitoring pictures and analysis reports.

[0120] According to a third aspect of the present application, the present application provides a system for online monitoring of laser processing metal process integrating visual defect detection and environment perception, comprising an electronic device, wherein the electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements a method for online monitoring of laser processing metal process integrating visual defect detection and environment perception according to any one of the present application when executing the computer program.

[0121] According to a fourth aspect of the present application, the present application provides a system for online monitoring of laser processing metal process integrating visual defect detection and environment perception, comprising a computer readable storage medium, wherein the computer readable storage medium stores a computer program, characterized in that the computer program is executed by a processor to implement a method for online monitoring of laser processing metal process integrating visual defect detection and environment perception according to any one of the present application.

[0122] In addition to the above, the present application also has related embodiments, including the following:

[0123] In an embodiment of the present application, the soc platform adopted by the present application is a edge computing platform, which has a performance and architecture much larger than that of a traditional embedded platform, and the soc platform adopted by the present application can realize the function of running an algorithm model on the platform.

[0124] Reference Figure 2 The present application provides a complete monitoring system, and the hardware architecture mainly includes an image acquisition unit 110, an environment perception unit 120, a core processing unit 130 (i.e. a SOC development board) as a core of the system, and a remote cloud server 140 and a user client 150.

[0125] The core processing unit 130 adopts a low-power and high-performance embedded SOC development board, such as a Rockchip RK series development board, which runs an embedded Linux operating system. The unit is the center of all data processing, algorithm running and communication management.

[0126] The image acquisition unit 110 is an industrial CCD or CMOS camera, connected to the core processing unit 130 through a USB interface. The camera is installed in the print chamber of the SLM device and can clearly capture the entire powder bed surface. To ensure image quality, an LED ring light source can be equipped as auxiliary lighting.

[0127] The environmental perception unit 120 is a BME680 multifunctional environmental sensor module, connected to the GPIO pin of the core processing unit 130 through an I2C bus. It is placed in a position that can accurately reflect the internal environment of the print chamber, and is used to monitor parameters such as temperature, humidity, and air pressure in real time.

[0128] Referring to Figure 1 and Figure 2 , the specific implementation process of the method of the present application is as follows:

[0129] Step S1: Multimodal data acquisition:

[0130] When the SLM device completes a powder laying action, a trigger signal (which can be a software signal or a hardware IO signal) is generated. After the main control application program on the core processing unit 130 receives this signal, it immediately starts the data acquisition task in parallel:

[0131] Through the camera driver, the image acquisition unit 110 is controlled to take a high-resolution powder bed image.

[0132] Through the I2C communication protocol, the current temperature, humidity, and other accurate environmental data from the environmental perception unit 120 are read.

[0133] Through synchronous acquisition, it is ensured that each frame of image corresponds to a set of accurate environmental parameters.

[0134] Step S2: Image preprocessing:

[0135] The raw image collected is directly sent to the SOC chip for processing. Due to the influence of factors such as light, camera position, and electromagnetic interference during image acquisition, the image quality cannot be directly used for analysis, so preprocessing must be performed. The preprocessing process is strictly executed on the SOC in the following order:

[0136] a) Brightness correction:

[0137] Firstly, compared with the prior art, there is a problem of uneven brightness in the obtained picture, and one of the key improvements of the present application, an adaptive correction algorithm for uneven illumination image based on two-dimensional gamma function, is adopted. Specifically, the algorithm can effectively estimate the background illumination component of the image by applying a multi-scale different Gaussian kernel to the original image for convolution. Based on the statistical distribution characteristics (such as mean and variance) of the illumination component, the algorithm can adaptively construct a non-linear two-dimensional gamma correction function. This function acts on each pixel of the original image, and the low-brightness area is enhanced by a large amplitude, and the high-brightness area is enhanced by a small amplitude, so as to realize adaptive correction of the illumination of the whole image, so that the subsequent detection picture is uniform in brightness, and the subsequent operation is facilitated.

[0138] b) Contrast enhancement:

[0139] Secondly, in view of the problem of too low contrast in the picture, a histogram equalization algorithm is adopted. The detailed operation steps are as follows: first, the number of pixels of each gray level of 0-255 of the brightness-corrected image is calculated by traversing the image, and a gray level histogram is obtained. Secondly, the cumulative histogram of the histogram is calculated, that is, the sum of the pixel numbers of each gray level and all the previous gray levels. Then, the cumulative histogram is normalized (divided by the total number of pixels and multiplied by 255), thereby creating an equalization mapping from the original gray level to the new gray level. Finally, the image is traversed again, and the gray value of each pixel is replaced by the new gray value obtained by the mapping. This process stretches the gray level, significantly improves the contrast of the image, and makes the boundary between the dark area with insufficient powder and the gray area with normal powder bed more clear.

[0140] c) and d) filter denoising:

[0141] Then, in view of the problem of a large amount of Gaussian noise and salt and pepper noise in the picture, Gaussian filtering and median filtering are adopted in turn. Gaussian filtering can smoothly filter out noise conforming to normal distribution by weighting and averaging the pixel neighborhood through a two-dimensional Gaussian kernel. Subsequent median filtering replaces the current pixel value with the gray value of the median in the pixel neighborhood, which is very effective in removing isolated black points (salt noise) and white points (pepper noise) that appear randomly, while protecting the edge details of the image well. Finally, a clear, uniform and noise-free preprocessed image is output.

[0142] Step S3: image defect recognition:

[0143] The preprocessed image is input into a defect recognition module, which runs another key improvement algorithm of the present application, an iterative threshold segmentation algorithm, on the SOC to identify the defects of insufficient powder. The detailed operation process of the algorithm is as follows:

[0144] Initialization: Choose an initial threshold T. Usually this threshold can be a small positive number, or more commonly, choose the average of the maximum and minimum of the image's gray scale.

[0145] Segmentation: Using the current threshold T, divide all pixels of the image into two sets: foreground set R1 (all pixels with gray scale greater than T) and background set R2 (all pixels with gray scale less than or equal to T).

[0146] Calculate the mean: Calculate the average gray scale of all pixels in sets R1 and R2, respectively, and denote them as u1 and u2.

[0147] Update threshold: According to the calculated mean, update the threshold to T_new = (u1 + u2) / 2.

[0148] Iteration: Determine whether |T_new - T| is less than a preset small positive number T0. T0 is usually a small positive number, such as 0.1. If it is less than, it is considered that the threshold has converged, and the iteration ends, and T_new at this time is the optimal threshold. Otherwise, let T = T_new, and return to step 2 to continue iteration.

[0149] Finally, using the obtained optimal threshold, the image is binarized, and the insufficient powder defect area is accurately segmented. The algorithm outputs the contour and area of the defect area, which serves as the basis for defect marking.

[0150] Step S4: Data fusion and information overlay:

[0151] This is the key step to realize comprehensive monitoring. The main control application program on the SOC formats the defect recognition results obtained in step (3) (such as whether there is a defect, the size of the defect area) and the environmental data collected in step (1) (such as temperature 35.2℃, humidity 15.8%). Then, call the on-board graphics processing library (such as OpenCV) to draw (overlay) these information in text form on the image frame after defect recognition. For example, display the environmental data in the upper left corner of the image, and display the defect status in the upper right corner. If a defect is detected, a red rectangular frame can also be used to enclose the defect area on the image.

[0152] Step S5: Edge encoding and cloud pushing:

[0153] Finally, these image frames overlaid with rich comprehensive information are fed in real time to an FFMPEG sub-process started and managed by the main control program through the pipe. The FFMPEG process encodes the input image sequence into H.264 or H.265 format video stream according to the preset parameters. A typical FFMPEG streaming instruction is as follows:

[0154] ffmpeg-f image2pipe-framerate 15-i--c:v h264_omx-b:v 2M-preset ultrafast-frtsp rtsp: / / [cloud server IP]:[port] / live / slm01.

[0155] Wherein -c:v h264_omx indicates the use of the hardware encoder of the SOC for acceleration. The encoded video stream is actively pushed to the cloud server 140 through the Real-Time Streaming Protocol (RTSP). The RTSP streaming service is deployed on the cloud server, responsible for receiving, managing and distributing video streams. Authorized user clients 150 (such as engineers' computers or managers' mobile phones) can only access the corresponding RTSP address to watch the SLM production site high-definition picture with defect marking and environmental data in real time.

[0156] In summary, the present application combines a set of mature image processing algorithms with modern embedded systems, Internet of Things sensor technology and streaming media technology to build a powerful, low-cost and easy-to-deploy SLM online monitoring system. It not only realizes the automation of defect detection, but also provides an unprecedented powerful tool for quality control and intelligent management of SLM process through data fusion and remote presentation.

[0157] Those skilled in the art will readily understand that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0158] The above is a preferred embodiment of the present application. Any change made according to the technical solution of the present application, which does not exceed the scope of the technical solution of the present application, belongs to the protection scope of the present application.

Claims

1. A method for online monitoring of a laser processing metal process with integrated visual defect detection and environmental perception, characterized in that, The method comprises the following steps: Step S1: time-synchronized multi-modal data acquisition of online monitoring of laser processing metal process, comprising On the embedded SOC platform, through the unified timestamp mechanism, the image data and the environmental data in the powder laying process of the selected laser melting are synchronously collected, wherein the image data is obtained by an industrial camera, and the environmental data at least comprises temperature T, humidity H and air pressure P, and all the data are labeled with microsecond-level timestamps; Step S2: pre-processing of the image data, comprising the following contents: The image data is sequentially subjected to the following pre-processing operations on the SOC platform: adopting a two-dimensional gamma function-based non-uniform illumination image adaptive correction algorithm to correct the brightness of the image; adopting a histogram equalization algorithm to improve the contrast of the image; adopting a Gaussian filtering algorithm to remove Gaussian noise in the image; and adopting a median filtering algorithm to remove salt and pepper noise in the image to obtain the pre-processed image; Step S3: image defect identification, comprising the following contents: An iterative threshold segmentation algorithm is adopted to segment the image, identify the powder supply insufficient defect area, and generate defect feature data D, wherein the defect feature data D comprises defect position, area and gray scale feature, and the convergence threshold T0 of the iterative threshold segmentation algorithm is dynamically adjusted according to the current environmental parameters; Step S4: data fusion and information superposition of the pre-processed image data, defect data and environmental data, comprising the following contents: A data record R containing timestamps, environmental parameters and defect features is established, the defect feature data D and the environmental data {T, H, P} are subjected to correlation analysis, the correlation coefficient of the defect occurrence probability P and the environmental deviation degree is calculated, and the analysis result is superimposed on the corresponding image frame in the form of text and graphics; Step S5: edge coding and cloud pushing of the total image sequence after data fusion and information superposition, Comprising the following contents: The image sequence superimposed with visual information is real-time coded into a video stream by using the hardware video encoder of the SOC, and is pushed to a cloud server through the end-to-end delay of the protocol, wherein the cloud server stores historical data and provides defect-environment correlation trend analysis.

2. A method for online monitoring of laser processing of metal processes with integrated visual defect detection and environmental perception as claimed in claim 1, wherein, Step S1 comprises the following contents: In step S1, the environmental data is collected by a sensor, and at least comprises temperature and humidity, the sensor is connected with the SOC through an I2C or SPI bus, and the sampling frequency is not less than 1 Hz, which is frame-level synchronized with the image collection.

3. A method for online monitoring of laser processing of metal processes with integrated visual defect detection and environmental perception as claimed in claim 1, wherein, Step S2 comprises the following contents: In step S2, the two-dimensional gamma function-based non-uniform illumination image adaptive correction algorithm comprises: extracting a multi-level illumination component L of the image by using a multi-scale Gaussian function; calculating the global mean μ_L and the standard deviation σ_L of the illumination component L; constructing an adaptive two-dimensional gamma function; applying a correction formula to each pixel (x, y) of the original image to realize pixel-by-pixel adaptive brightness adjustment.

4. A method for online monitoring of laser processing of metal processes with integrated visual defect detection and environmental perception as claimed in claim 3, wherein, Step S2 further comprises the following contents: In step S2, the multi-scale Gaussian function comprises: {G_1, G_2, …, G_n} Wherein, G_n represents n different scale Gaussian functions for removing noise from the image; In step S2, the adaptive two-dimensional gamma function includes: γ(x, y) = γ_base × [1 + k × (L(x, y) - μ_L) / σ_L] where γ_base is a base gamma value, and k is an adaptive adjustment coefficient. Applying the correction formula to each pixel (x, y) of the original image includes: I_corrected(x, y) = 255 × [I(x, y) / 255]^(1 / γ(x, y)) where I(x, y) represents the brightness, and I_corrected(x, y) is used to achieve pixel-by-pixel adaptive brightness adjustment.

5. A method for online monitoring of laser processing of metal processes with integrated visual defect detection and environmental perception as claimed in claim 1, wherein, Step S3 includes the following: In step S3, the convergence threshold T0 of the iterative threshold segmentation algorithm is dynamically adjusted according to the current environmental parameters, and the adjustment formula includes: T0 = T0_base × [1 + α × (T - T_ref) / T_ref + β × (H - H_ref) / H_ref] where T0_base is a base convergence threshold, T_ref and H_ref are reference environmental parameters, and α and β are weighting coefficients.

6. A method for online monitoring of laser processing of metal processes with integrated visual defect detection and environmental perception as claimed in claim 1, wherein, Step S4 includes the following: Step S41: Establish a time series database to store at least 100 consecutive frames of multi-modal data records; Step S42: Use sliding window statistical analysis to calculate the Pearson correlation coefficient of the defect occurrence frequency f and the environmental parameters {T, H, P}; Step S43: When |correlation coefficient|>0.6, display a high correlation warning on the image and record the key environmental threshold; Step S44: Generate a visualization curve to superimpose the defect trend and environmental fluctuation trend in the last 10 minutes.

7. A method for online monitoring of laser processing of metal processes with integrated visual defect detection and environmental perception as claimed in claim 1, wherein, Step S5 includes the following: The image sequence of the visualization information is encoded into an H.264 / H.265 video stream in real time; The pushing to the cloud server includes pushing to the cloud server through a protocol with an end-to-end delay of not more than 200 ms; The edge encoding and cloud pushing use the following optimization strategy: using a SOC hardware encoder acceleration interface, the encoding rate is not less than 30 fps; according to the network bandwidth, the code rate is adaptively adjusted, and the code rate range is 1-5 Mbps; through the transmission protocol, reliable transmission and line reconnection are realized; a streaming media server is deployed on the cloud to support multiple client concurrent access, and the single server concurrent number is ≥20.

8. A system for online monitoring of laser processing of metal processes integrating visual defect detection and environmental perception for performing a method of online monitoring of laser processing of metal processes integrating visual defect detection and environmental perception according to any one of claims 1 to 7, characterized in that, It includes an image acquisition unit, an environment perception unit, a core processing unit, an edge intelligent processing module, a cloud storage and analysis unit, and a client access unit; The image acquisition unit includes an industrial-grade camera connected to the core processing unit through a USB interface, with a resolution of not less than 1920×1080; The environment perception unit includes a multifunctional sensor module connected to the core processing unit through an I2C / SPI interface; The core processing unit includes an embedded SOC development board containing at least a quad-core processor, RAM, a hardware video encoder, and a GPIO interface, running an embedded operating system; The edge intelligent processing module includes an application program deployed on the core processing unit, realizing image preprocessing, defect identification, data fusion, and video encoding functions; The cloud storage and analysis unit includes a cloud server deploying an RTSP streaming service and a database, and provides video stream distribution, historical data storage and trend analysis functions. The client access unit includes a PC or mobile terminal application, which accesses the cloud server through a network to view monitoring pictures and analysis reports in real time.

9. A system for online monitoring of a laser processing metal process with integrated visual defect detection and environmental perception, comprising an electronic device, wherein the electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method for online monitoring of a laser processing metal process with integrated visual defect detection and environmental perception according to any one of claims 1 to 7.

10. A system for online monitoring of a laser processing metal process integrating visual defect detection and environmental perception, comprising a computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method for online monitoring of a laser processing metal process with integrated visual defect detection and environmental perception according to any one of claims 1 to 7.

Images