Deep learning-based online monitoring system and method for laser cleaning, and related media

Abstract

This invention discloses a deep learning-based online monitoring system, method, and medium for laser cleaning, relating to the field of laser cleaning technology. It includes an information acquisition mechanism, a laser assembly, a display assembly, and a controller. The controller acquires the current position of the robotic arm and an image of the workpiece surface, initializes the image to obtain initial coordinates, and converts them into three-dimensional coordinates. Subsequently, the controller uses a deep learning model to preprocess and extract features from the image, predicts the cleaning surface based on the processed feature image, obtains the coordinate values ​​of the cleaning surface, and converts them into three-dimensional coordinates to obtain the calculated position of the cleaning surface. Then, the calculated position is subtracted from the coordinates of the current position of the robotic arm to obtain the deviation value, which is transmitted to the controller in real time. This enables the controller to control the laser emitter to perform cleaning movements along the workpiece surface, and displays the cleaning results on the display assembly. This invention solves the problems of existing laser cleaning technologies lacking real-time monitoring and exhibiting low cleaning quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of laser cleaning technology, and more specifically, to a deep learning-based online monitoring system, method, and medium for laser cleaning. Background Technology

[0002] Laser cleaning online monitoring technology is a technique used to monitor and control the laser cleaning process. In laser cleaning, a laser beam is used to remove paint, dirt, and other contaminants from a surface to achieve the purpose of cleaning. Online monitoring technology is used to monitor the effectiveness of the cleaning process in real time, so as to adjust cleaning parameters in a timely manner and ensure the cleaning effect.

[0003] Currently, laser cleaning visual inspection methods are divided into traditional machine vision inspection and deep learning-based visual inspection. Traditional machine vision utilizes mechanical equipment to perform the measurement and evaluation functions of the human eye. It converts the observed target into image signals using optical systems and non-touch sensors. These image signals are then converted into digital signals by an image processing system, which calculates and analyzes information such as pixel arrangement, brightness, and color to identify the required features. Based on these analysis results, it automatically controls the corresponding actions of some mechanical equipment on-site. In contrast, deep learning-based visual inspection, compared to traditional machine vision methods, utilizes complex algorithmic models to process and analyze image data. This method primarily trains deep neural networks, such as CNNs, to automatically learn and recognize features in images without manually encoding feature extraction rules. Deep learning can more effectively identify and distinguish subtle differences between the painted surface and the untreated surface, maintaining high accuracy even under complex or changing environmental conditions.

[0004] Laser cleaning monitoring technology employs both offline and online monitoring methods. Offline monitoring cannot provide immediate feedback, meaning any cleaning defects or errors cannot be detected and corrected instantly, leading to decreased production efficiency and increased costs. In contrast, online monitoring provides real-time data collection and analysis, enabling immediate problem identification and adjustments, significantly improving production efficiency and quality control. Implementing online monitoring allows for faster feedback loops, reduces costs and material waste, and provides richer data to support process optimization.

[0005] In view of this, the applicant hereby submits this application after studying the existing technology. Summary of the Invention

[0006] The present invention aims to provide a laser cleaning online monitoring system, method and medium based on deep learning, so as to monitor the state of the laser-cleaned workpiece surface in real time, and solve the problems of existing laser cleaning technology being unable to monitor in real time, resulting in low cleaning quality and efficiency.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A deep learning-based online monitoring system for laser cleaning includes an information acquisition mechanism, a controller, a laser assembly, and a display assembly, wherein:

[0009] The information acquisition mechanism includes a data acquisition card and a laser vision sensor, used to acquire images of the workpiece surface and send them to the controller.

[0010] The laser vision sensor is connected via a vertically movable mechanism;

[0011] Laser assembly, including workpiece clamping table, robotic arm, and laser emitter;

[0012] The workpiece clamping stage is used to withstand laser energy irradiation outside the sample range and to assist in focusing;

[0013] The robotic arm controls the lifting and focusing of the laser emitter to clean the sample. By adjusting the position of the robotic arm, the angle between the lens and the light source relative to the sample can be changed.

[0014] The robotic arm is mounted at the end of the laser emitter;

[0015] The display component is connected to the controller and transmits display data through the controller.

[0016] The controller includes a processor and a memory, wherein the processor is used to run a computer program stored in the memory to achieve:

[0017] A1. The current position of the robotic arm and the workpiece surface image collected by the information acquisition mechanism are acquired in real time, and the workpiece surface image is preprocessed to obtain the initial cleaning image and the initial coordinates of the image.

[0018] A2, Based on the current position of the robotic arm and the initial coordinates of the image, convert the initial coordinates of the image into three-dimensional coordinates in the base coordinate system of the robotic arm;

[0019] A3, initialize the initial image for cleaning, and after target detection and localization, obtain the predicted cleaned image;

[0020] A4. Convert the coordinate values ​​of the predicted cleaning image into three-dimensional coordinates in the robot arm base coordinate system to obtain the predicted cleaning position;

[0021] A5. Based on the predicted cleaning position, calculate the coordinate difference between the predicted position and the current position of the robotic arm to obtain the deviation value;

[0022] A6. The deviation value is transmitted to the robotic arm, the position of the robotic arm is adjusted according to the deviation value, the laser emitter is controlled to perform cleaning, and the actual result image after cleaning is acquired.

[0023] A7. Based on the actual result image after cleaning, determine whether the cleaning is complete. If it is complete, transmit the actual result image to the display component for display; otherwise, use the actual result image as the surface image and continue the cleaning operation.

[0024] Preferably, the target detection employs an improved YOLOv5 network model, which includes an input layer, a backbone network, intermediate layers, and a prediction layer. The input layer performs data augmentation, image size processing, and adaptive anchor box calculation on the input image. The backbone network performs image segmentation, convolution, and feature extraction. The intermediate layers achieve image feature fusion. The prediction layer generates predicted bounding boxes, predicted categories, and labels for targets in the original image.

[0025] Preferably, the improved Yolov5 network model is trained by the following steps:

[0026] The surface image of the substrate after laser cleaning is acquired, and the surface image is preprocessed to obtain a preprocessed image;

[0027] A classification and annotation tool was used to classify and annotate the detected targets in the preprocessed images to obtain a dataset, which was then divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0028] The model is trained and iteratively optimized using the training set to obtain the optimal model;

[0029] Based on the optimal model, target detection is performed on the validation set to obtain the detection results.

[0030] Preferably, target localization is performed using a U-Net network model, which includes a shrinking path and an expanding path. The shrinking path performs a downsampling operation on the input feature map, and the expanding path performs an upsampling operation on the feature image.

[0031] Preferably, the method further includes training the U-Net network model through the following steps:

[0032] The surface image of the substrate after laser cleaning is acquired, and the surface image is preprocessed to obtain a preprocessed image;

[0033] A classification and annotation tool was used to classify and annotate the detected targets in the preprocessed images to obtain a dataset, which was then divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0034] The model is trained and iteratively optimized using the training set to obtain the optimal model;

[0035] Based on the optimal model, the validation set is segmented to obtain the segmentation results.

[0036] Preferably, when training the improved Yolov5 network model, the Focal EIoU Loss function is used for loss calculation and accuracy optimization. The Focal EIoU Loss function combines the EIoU Loss function with Focal Loss to increase the accuracy of regression. The formula for the Focal EIoU Loss function is as follows:

[0037] Loss F =IOU γ Loss E ;

[0038] Where IoU represents the intersection-over-union ratio of the predicted box and the ground truth box, γ represents the modulation coefficient, and Loss... E The EIoU Loss function includes overlap loss, center distance loss, and width / height loss. The formula for the EIoU Loss function is as follows:

[0039]

[0040] in, ρ represents the center distance loss. 2 (b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes. Indicates width loss. This indicates a significant loss.

[0041] Preferably, the U-Net network model is trained using Caffe's stochastic gradient descent method, and the loss function is calculated using an energy function. The energy function is calculated from the maximum pixel value on the final feature map and the cross-entropy loss function. The formula for calculating the maximum value is:

[0042]

[0043] Where, p k (x) represents the approximate maximum function, a k (x) represents the activation level of feature channel k at pixel location, where K is the number of categories.

[0044] This invention also provides a deep learning-based online monitoring method for laser cleaning, applied to the laser cleaning online monitoring system described above, comprising:

[0045] S1, acquire the image of the cleaned workpiece surface and the current position of the robotic arm, and preprocess the workpiece surface image to obtain the initial cleaned image and the initial coordinates of the image;

[0046] S2, Based on the current position of the robotic arm and the initial coordinates of the image, convert the initial coordinates of the image into three-dimensional coordinates in the base coordinate system of the robotic arm;

[0047] S3, input the initial cleaned image into the improved Yolov5 network model for target detection to obtain the target feature map, input the target feature map into the U-Net network model for target localization, and obtain the predicted cleaned image;

[0048] S4. Clean the image according to the prediction result, obtain the coordinate values ​​of the cleaned image according to the prediction result, and convert them into three-dimensional coordinates in the robot arm base coordinate system to obtain the predicted cleaning position.

[0049] S5. Based on the predicted cleaning position, calculate the coordinate difference between the predicted position and the current position of the robotic arm to obtain the deviation value;

[0050] S6. Based on the deviation value, adjust the position of the robotic arm and perform cleaning, and collect an image of the actual result after cleaning;

[0051] S7. Based on the actual result image after cleaning, determine whether it is clean. If it is clean, the actual result image after cleaning is the final cleaning result image; otherwise, use the actual result image as the surface image and continue the cleaning operation.

[0052] The present invention also provides a computer-readable storage medium, including computer-readable instructions stored on the computer-readable storage medium, wherein when the computer-readable instructions are executed by the processor of the device on which the computer-readable storage medium is located, the computer-readable instructions implement the laser cleaning online monitoring method based on deep learning as described above.

[0053] In summary, compared with existing technologies, this invention, by porting a deep learning model to the controller, allows the controller to acquire the current position of the robotic arm and the workpiece surface image. The acquired workpiece surface image is initialized, its initial coordinates are obtained, and converted into three-dimensional coordinates in the robotic arm's base coordinate system, thus determining the robotic arm's current position. Subsequently, the controller uses the deep learning model for preprocessing and feature extraction. Based on the processed feature image, it predicts the cleaning surface, obtains the coordinate values ​​of the cleaning surface, and converts them into three-dimensional coordinates in the robotic arm's base coordinate system, obtaining the calculated position of the cleaning surface. Next, the calculated position is subtracted from the coordinates of the robotic arm's current position to obtain the deviation value, which is transmitted to the controller in real time. Finally, the controller transmits control signals to the robotic arm, causing it to control the laser emitter to perform cleaning motion along the workpiece surface, and displays the cleaning results on the display component. This invention has the following beneficial effects:

[0054] (1) The present invention transmits the real-time acquired image information to the controller for target detection, analysis and positioning, dynamically adjusts and controls the laser cleaning process, and displays the cleaning results in real time, which solves the problem that the existing laser cleaning technology does not have real-time monitoring and has low cleaning quality and efficiency.

[0055] (2) This invention adds Focal Loss to the EIoU Loss loss function, which solves the sample imbalance problem in bounding box regression task and reduces the optimization contribution of many anchor boxes to BBox regression. Because these anchor boxes have less overlap with the target box, the regression process focuses more on high-quality anchor boxes, solves the problem of penalty term degradation and difficulty in minimizing simultaneously when the shapes are similar, and improves the modeling accuracy of the frame model.

[0056] (3) The present invention uses a target detection network and an image segmentation network to identify and locate one or more objects in an image, which greatly improves the accuracy and stability of sample classification during laser paint removal, and improves the performance and robustness of the system.

[0057] (4) Using the U-Net network, more accurate segmentation results can be generated with less training data. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0059] Figure 1This is a schematic diagram of the overall structure of the laser cleaning online monitoring system provided in the first embodiment of the present invention.

[0060] Figure 2 This is a system flowchart of the laser cleaning online monitoring system provided in the first embodiment of the present invention.

[0061] Figure 3 This is a schematic diagram of the YOLOv5 network structure provided in the first embodiment of the present invention.

[0062] Figure 4 This is a conceptual diagram of the true bounding box and the prediction bounding box of the CIoU loss function provided in the first embodiment of the present invention.

[0063] Figure 5 This is a diagram of the target detection results of the YOLOv5 network provided in the first embodiment of the present invention.

[0064] Figure 6 This is a schematic diagram of the network structure of the U-Net model provided in the first embodiment of the present invention.

[0065] Figure 7 This is a diagram showing the target image segmentation result of the U-Net model provided in the first embodiment of the present invention.

[0066] Figure 8 This is a schematic diagram of the visual acquisition device provided in the first embodiment of the present invention.

[0067] Figure 9 This is a flowchart of the online monitoring method for laser cleaning provided in the second embodiment of the present invention.

[0068] The markings in the diagram are: 1-Information acquisition mechanism; 2-Controller; 3-Display component; 4-Vertical moving mechanism; 5-Workpiece clamping table; 6-Robotic arm; 7-Laser emitter; 8-Data acquisition card; 9-Laser vision sensor.

[0069] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0071] Example 1

[0072] Laser cleaning online monitoring technology is a technique used to monitor and control the laser cleaning process. In laser cleaning, a laser beam is used to remove paint, dirt, and other contaminants from a surface to achieve the purpose of cleaning. Online monitoring technology is used to monitor the effectiveness of the cleaning process in real time, so as to adjust cleaning parameters in a timely manner and ensure the cleaning effect.

[0073] Existing online monitoring technologies for laser cleaning include the use of optical sensors, cameras, or laser scattering technology to monitor and analyze surface features and residual contaminants during the cleaning process. Optical sensors can measure the surface roughness and reflectivity after cleaning to determine the cleaning effect; cameras can capture the cleaning process in real time and perform image processing and analysis to evaluate the cleaning effect; laser scattering technology can obtain information about residual contaminants on the surface by analyzing the scattering signal when the laser interacts with the surface.

[0074] Laser cleaning monitoring is primarily based on the photoacoustic effect theory. When the laser power density exceeds the surface damage threshold of a solid material, laser irradiation raises the surface temperature, causing localized melting and vaporization. The sample surface undergoes ablation, generating laser plasma shock waves, which attenuate to form acoustic waves. When the incident laser power density is less than the surface damage threshold of the solid material, the shallow surface absorbs some laser energy, causing the surface temperature to rise and resulting in volume expansion. With each incident laser pulse, the shallow surface undergoes periodic deformation, radiating periodic acoustic waves into the surrounding medium.

[0075] Currently, laser cleaning visual inspection methods are divided into traditional machine vision inspection and deep learning-based visual inspection. Deep learning-based visual inspection, compared to traditional machine vision methods, utilizes complex algorithmic models to process and analyze image data. This method primarily trains deep neural networks, such as CNNs, to automatically learn and recognize features in images, eliminating the need for manually encoding feature extraction rules. In laser paint removal applications, deep learning can more effectively identify and distinguish subtle differences between the painted surface and the untreated surface, maintaining high accuracy even under complex or changing environmental conditions.

[0076] Please see Figures 1 to 2 The first embodiment of the present invention provides a laser cleaning online monitoring system based on deep learning, including an information acquisition mechanism 1, a controller 2, a laser component, and a display component 3, wherein:

[0077] The information acquisition mechanism 1 includes a data acquisition card 8 and a laser vision sensor 9, which are used to acquire images of the workpiece surface and send them to the controller 2.

[0078] The laser vision sensor 9 is connected via a vertical moving mechanism 4 and its movement is controlled by the vertical moving mechanism 4.

[0079] The laser assembly includes a workpiece clamping table 5, a robotic arm 6, and a laser emitter 7.

[0080] The workpiece clamping table 5 is used to withstand laser energy irradiation outside the sample range and to assist in focusing;

[0081] The robotic arm 6 is used to control the lifting and focusing of the laser emitter to clean the sample. By adjusting the position of the robotic arm 6, the angle between the lens and the light source relative to the sample can be changed.

[0082] The robotic arm 6 is mounted at the end of the laser emitter 7;

[0083] The display component 3 is connected to the controller 2, and the controller controls the transmission of display data.

[0084] The controller 2 includes a processor and a memory, wherein the processor is used to run a computer program stored in the memory to achieve:

[0085] A1. The current position (three-dimensional coordinates) of the robotic arm 6 and the workpiece surface image collected by the information acquisition mechanism 1 are acquired in real time, and the workpiece surface image is preprocessed to obtain the initial cleaning image and the initial coordinates of the image.

[0086] A2, based on the current position of the robotic arm 6 and the initial coordinates of the image, convert the initial coordinates of the image into three-dimensional coordinates in the base coordinate system of the robotic arm 6;

[0087] A3, initialize the initial image for cleaning, and after target detection and localization, obtain the predicted cleaned image;

[0088] A4. Convert the coordinate values ​​of the predicted cleaning image into three-dimensional coordinates in the 6-base coordinate system of the robotic arm to obtain the predicted cleaning position.

[0089] A5. Based on the predicted cleaning position, calculate the coordinate difference between the current position of the robotic arm 6 and the predicted cleaning position to obtain the deviation value;

[0090] A6, transmit the deviation value to the robotic arm 6, adjust the position of the robotic arm 6 according to the deviation value, control the laser emitter 7 to perform cleaning, and collect the actual result image after cleaning;

[0091] A7. Based on the actual result image after cleaning, determine whether it is clean. If it is clean, transmit the actual result image to the display component 3 for display. Otherwise, use the actual result image as the surface image and continue the cleaning operation.

[0092] The controller 2 embeds a deep learning model, which automatically learns features from images and constructs object detection and image segmentation networks to identify and locate one or more objects in the image. This greatly improves the accuracy and stability of sample classification during laser paint removal, thereby enhancing the system's performance and robustness.

[0093] The deep learning models include improved Yolov5 network models and U-Net network models, and may also include other deep learning models capable of object detection and localization, such as Faster R-CNN and SSD.

[0094] Furthermore, this invention employs an improved YOLOv5 network model for target detection. YOLOv5 is a novel algorithm within the YOLO series. This algorithm consists of four parts: the input layer, the backbone network, the neck layer, and the prediction layer.

[0095] like Figure 3As shown, the input of YOLOv5 consists of three parts: Mosaic data augmentation, image resizing, and adaptive anchor box calculation. Mosaic data augmentation combines four images to enrich the background. Image resizing adjusts the original images with different aspect ratios to adaptively and uniformly scale them to a standard size, minimizing the addition of black borders. Adaptive anchor box calculation uses the original anchor boxes as a basis, compares the predicted output image with the real image, calculates the difference between the two, and updates the anchor box parameters in reverse. This process is repeated until the most suitable anchor box value is obtained.

[0096] Backbone comprises two main modules: BottleneckCSP and Focus. The BottleneckCSP module significantly reduces computational cost while improving the learning performance of convolutional neural networks, thus meeting the demands of complex tasks. The Focus module performs downsampling by slicing the input image into four times the original number of channels and then performing convolution. Furthermore, it can generate more refined feature maps through convolution. This approach achieves downsampling while reducing computation and improving speed. Specifically, the Focus module divides the input image into several blocks and convolves each block's pixels as a channel. This operation expands the number of channels to four times the original, thereby improving feature extraction. After convolution, the resulting downsampled feature map better represents object information in the image.

[0097] The Neck layer combines the Feature Pyramid Network (FPN) with the Path Aggregation Network (PAN) to achieve efficient integration of semantic and positional features. Furthermore, Neck fuses features from the backbone layer and the detection layer, significantly improving the model's accuracy and reliability, and enabling the model to acquire richer feature information.

[0098] The output of the prediction layer is a vector containing information such as the target object's class probability, the object's score, and the location of its bounding box. To better detect targets of different sizes, the detection network consists of three independent layers that use feature maps of different sizes to detect different objects. Each detection layer outputs a corresponding vector, ultimately generating predicted bounding boxes and classes for targets in the original image, and achieving accurate feature recognition through labeling.

[0099] The improved YOLOv5 network model introduces the Focal EIoU Loss function on top of YOLOv5 to improve its accuracy for convolutional neural networks. The loss function plays a crucial role in training convolutional networks. It considers three geometric parameters: overlap area, center-to-center distance, and aspect ratio.

[0100] The traditional Focal Loss formula is shown below:

[0101] Loss F (p t )=-(1-p t ) γ lg(p t );

[0102] Where, p t This represents the probability that a sample belongs to the true bounding box, while γ is a parameter that controls the degree of suppression of outliers, also known as the modulation coefficient. Traditional Focal loss algorithms primarily analyze more difficult samples, their main function being to identify these challenging examples. Currently, existing models suffer from an imbalance between positive and negative values. In an image, the number of frames that match the correct bounding box is small, but the number of unmatched frames can reach tens of thousands. Focal loss not only reduces the weight of easily classified samples but also learns better for difficult-to-classify samples.

[0103] Therefore, this invention employs the Focal EIoU Loss function, which combines the EIoU Loss function with Focal Loss to increase the accuracy of regression and solve the sample imbalance problem in BBox regression. The formula for the Focal EIoU Loss function is as follows:

[0104] Loss F =IOUγLoss E ;

[0105] Where IoU represents the intersection-over-union ratio of the predicted box and the ground truth box, γ represents the modulation coefficient, and Loss... E The EIoU Loss function is used because the definition of the aspect ratio parameter v in the CIoU formula is somewhat vague and differs from the actual width and height. To solve this problem, the length and width losses of the target box and the anchor box are calculated separately.

[0106] The EIoU LoSS loss function includes overlap loss, center distance loss, and width-height loss. The formula for the EIoU LoSS loss function is as follows:

[0107]

[0108] in, ρ represents the center distance loss. 2 (b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes (e.g., ...). Figure 4 (as shown), Indicates width loss. This indicates a significant loss.

[0109] The improved Yolov5 network model, which incorporates the Focal EIoU Loss function, is trained through the following steps:

[0110] The surface image of the substrate after laser cleaning is acquired, and the surface image is preprocessed to obtain a preprocessed image;

[0111] A classification and annotation tool was used to classify and annotate the detected targets in the preprocessed images to obtain a dataset, which was then divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0112] The model is trained and iteratively optimized using the training set to obtain the optimal model;

[0113] Based on the optimal model, target detection is performed on the validation set to obtain the detection results.

[0114] like Figure 5 As shown, the test results can be displayed intuitively on the monitor.

[0115] In another preferred embodiment of this invention, the U-Net network model is used for target segmentation and localization.

[0116] The U-Net network is a classic fully convolutional network (i.e., the network has no fully connected operations). The main idea of ​​a fully convolutional network is to supplement a typical shrinking network with successive layers, where pooling operators are replaced by upsampling operators. For localization, high-resolution features from the shrinking path are combined with the upsampled output. Then, successive convolutional layers learn to combine this information to produce a more accurate output, enabling it to work with fewer training images and produce more accurate segmentation results.

[0117] By stacking multiple convolutional layers, activation functions, and pooling layers, U-Net effectively captures various levels of complexity in image features, from simple to complex. Compared to traditional convolutional neural networks, U-Net has better localization capabilities for smaller objects due to its special network architecture called a U-shaped structure. Through this training method, the U-Net model can accurately predict the location of objects while recognizing them, thereby improving the performance and accuracy of object detection algorithms.

[0118] like Figure 6As shown, the U-Net network model includes a shrinking path (left) and an expanding path (right). The shrinking path downsamples the input feature map, while the expanding path upsamples the feature image. The shrinking path follows the typical architecture of a convolutional network. It involves repeatedly applying two 3x3 convolutions (without padding), each followed by a rectified linear unit (ReLU) and a 2x2 max pooling operation (stride of 2) for downsampling. In each downsampling step, the number of feature channels is doubled. Each step in the expanding path involves upsampling the feature map, followed by a 2x2 convolution to halve the number of feature channels; merging with the corresponding cropped feature map from the shrinking path; and two 3x3 convolutions, each followed by a ReLU. Cropping is necessary because each convolution loses boundary pixels. In the final layer, a 1x1 convolution maps each 64-component feature vector to the desired number of classes. The network has a total of 23 convolutional layers.

[0119] The U-Net network model is trained through the following steps:

[0120] The surface image of the substrate after laser cleaning is acquired, and the surface image is preprocessed to obtain a preprocessed image;

[0121] A classification and annotation tool was used to classify and annotate the detected targets in the preprocessed images to obtain a dataset, which was then divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0122] The model is trained and iteratively optimized using the training set to obtain the optimal model;

[0123] Based on the optimal model, the validation set is segmented to obtain the segmentation results.

[0124] like Figure 7 The image segmentation results of the U-Net network model are shown in the interface.

[0125] The U-Net network model is trained using Caffe's stochastic gradient descent implementation. Due to unpadded convolutions, the output image is smaller than the input image by a constant boundary width. The energy function is calculated from the pixel maximum values ​​on the final feature map and the cross-entropy loss function.

[0126] The maximum value is defined as:

[0127]

[0128] Among them, a k (x) represents the activation level of feature channel k at pixel location, where K is the number of classes, p k (x) is the approximate maximum function.

[0129] By pre-computing the weight map for each true segmentation to compensate for the different frequencies of pixels from a certain class in the training dataset and forcing the network to learn the small separation boundaries introduced between adjacent cells. The calculation formula for the weight map is as follows:

[0130] ...

[0131] where, w c (x) is the weight map for balancing class frequencies, d1(x) represents the distance to the nearest cell boundary, and d2(x) represents the distance to the second nearest cell boundary.

[0132] In the laser cleaning experiment, we set w o = 10, σ ≈ 5 pixels.

[0133] In the upsampling part of the model, there are also a large number of feature channels, enabling the network to propagate context information to higher resolution layers. Therefore, the expansion path is basically symmetric to the contraction path, forming a U-shaped structure. This network has no fully connected layers and only uses the valid part of each convolution, that is, the segmentation map only contains pixels with complete context in the input image. To predict the pixels in the image boundary region, it is necessary to infer the missing context by mirroring the input image. This tiling strategy is very important for applying the network to large images, otherwise the resolution will be limited by the GPU memory.

[0134] In this embodiment, to determine whether the workpiece is cleaned, an existing function model can be used for judgment. For example, let the judgment result of the function model be set to two categories: being cleaned and cleaning completed F2, and corresponding to two discriminant equations F1 and F2 respectively. If the result is F1 < F2, it means that the cleaning has been completed; otherwise, the cleaning is not completed, and it is fed back to the control system for another cleaning until the cleaning is completed.

[0135] In another preferred embodiment of this embodiment, the model of the laser is a nanosecond laser. The robotic arm 6 can realize the lifting of the laser cleaner for the focusing operation of the cleaning sample. The workpiece clamping workbench 5 is made of stainless steel with a length, width and height of 500mm × 300mm × 30mm, mainly used to withstand the laser energy irradiation outside the sample range and play an auxiliary focusing role at the same time. The main light source uses a uniform annular LED lamp, and the side auxiliary light source is a symmetric rectangular structure, which can solve the imaging reflection problem caused by the light change on the metal surface. By adjusting the position of the robotic arm 6, the angle between the lens and the light source with respect to the sample can also be changed to achieve cleaning experiments under different conditions.

[0136] Such as Figure 8The image shown is a side view of the visual acquisition device. Since the laser is at a 90° angle to the sample surface, to ensure accurate capture of detailed changes in the sample during the cleaning process without obstructing it, orthogonal experiments were conducted to repeatedly determine the angles of the acquisition lens and the light field illumination. Ultimately, the distance between the robotic arm 6 and the sample was adjusted to approximately 280mm. The visual camera and the main light source were positioned at an 80° angle to the sample for high-angle imaging. A rectangular side-mounted auxiliary light source was adjusted to a 45° angle to the sample surface for uniform illumination from both sides, resolving the issue of inconsistent color imaging of the paint layer on the metal surface under different light source changes. Because metal surface imaging is highly sensitive to changes in light, a long, symmetrical light source was used for low-angle uniform illumination to improve image quality. To address the issue of inconsistent reflections on the metal surface at different angles, a ring-shaped light source was used for supplementary lighting, making the light field more uniform.

[0137] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0138] (1) The present invention transmits the real-time acquired image information to the controller for target detection, analysis and positioning, dynamically adjusts and controls the laser cleaning process, and displays the cleaning results in real time, which solves the problem that the existing laser cleaning technology does not have real-time monitoring and has low cleaning quality and efficiency.

[0139] (2) This invention adds Focal Loss to the EIoU Loss loss function, which solves the sample imbalance problem in bounding box regression task and reduces the optimization contribution of many anchor boxes to BBox regression. Because these anchor boxes have less overlap with the target box, the regression process focuses more on high-quality anchor boxes, solves the problem of penalty term degradation and difficulty in minimizing simultaneously when the shapes are similar, and improves the modeling accuracy of the frame model.

[0140] (3) The present invention uses a target detection network and an image segmentation network to identify and locate one or more objects in an image, which greatly improves the accuracy and stability of sample classification during laser paint removal, and improves the performance and robustness of the system.

[0141] Example 2

[0142] like Figure 9 As shown, this embodiment of the invention provides a deep learning-based online monitoring method for laser cleaning, applied to an online monitoring system for laser cleaning. The online monitoring system for laser cleaning includes a robotic arm 6, and its steps include:

[0143] S1, acquire the image of the cleaned workpiece surface and the current position of the robotic arm 6, and preprocess the workpiece surface image to obtain the initial cleaned image and the initial coordinates of the image;

[0144] S2, based on the current position of the robotic arm 6 and the initial coordinates of the image, convert the initial coordinates of the image into three-dimensional coordinates in the base coordinate system of the robotic arm 6;

[0145] S3, after inputting the initial cleaned image into the network model for target detection, recognition, segmentation and localization, the predicted cleaned image is obtained;

[0146] S4. Clean the image according to the prediction result, obtain the coordinate values ​​of the cleaned image according to the prediction result, and convert them into three-dimensional coordinates in the 6-base coordinate system of the robotic arm to obtain the predicted cleaning position.

[0147] S5. Based on the predicted cleaning position, calculate the coordinate difference between the current position of the robotic arm 6 and the predicted cleaning position to obtain the deviation value;

[0148] S6, Based on the deviation value, adjust the position of the robotic arm 6 and perform cleaning, and collect the actual result image after cleaning;

[0149] S7. Based on the actual result image after cleaning, determine whether it is clean. If it is clean, the actual result image after cleaning is the final cleaning result image; otherwise, use the actual result image as the surface image and continue the cleaning operation.

[0150] Example 3

[0151] The third embodiment of the present invention also provides a computer-readable storage medium, which includes computer-readable instructions stored on the computer-readable storage medium. When the computer-readable instructions are executed by the processor of the device where the computer-readable storage medium is located, they implement the above-described deep learning-based online monitoring method for laser cleaning.

[0152] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0154] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0156] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0157] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0158] The use of "first" and "second" in the embodiments is merely to distinguish similar objects and does not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0159] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deep learning-based online monitoring system for laser cleaning, characterized in that, Includes an information acquisition mechanism, controller, laser assembly, and display assembly, among which: The information acquisition mechanism includes a data acquisition card and a laser vision sensor, used to acquire images of the workpiece surface and send them to the controller; The laser vision sensor is connected via a vertically movable mechanism; The laser assembly includes a workpiece clamping table, a robotic arm, and a laser emitter; The workpiece clamping stage is used to withstand laser energy irradiation outside the sample range and to assist in focusing; The robotic arm controls the lifting and focusing of the laser emitter to clean the sample. By adjusting the position of the robotic arm, the angle between the lens and the light source relative to the sample can be changed. The robotic arm is mounted at the end of the laser emitter; The display component is connected to the controller, and the controller controls the transmission of display data. The controller includes a processor and a memory, wherein the processor is used to run a computer program stored in the memory to achieve: A1. The current position of the robotic arm and the workpiece surface image collected by the information acquisition mechanism are acquired in real time, and the workpiece surface image is preprocessed to obtain the initial cleaning image and the initial coordinates of the image. A2, Based on the current position of the robotic arm and the initial coordinates of the image, convert the initial coordinates of the image into three-dimensional coordinates in the base coordinate system of the robotic arm; A3, initialize the initial cleaned image, and after target detection and localization, obtain the predicted cleaned image; A4. Convert the coordinate values ​​of the predicted cleaning image into three-dimensional coordinates in the robot arm base coordinate system to obtain the predicted cleaning position; A5. Based on the predicted cleaning position, calculate the coordinate difference between the predicted position and the current position of the robotic arm to obtain the deviation value; A6. The deviation value is transmitted to the robotic arm, the position of the robotic arm is adjusted according to the deviation value, the laser emitter is controlled to perform cleaning, and the actual result image after cleaning is acquired. A7. Based on the actual result image after cleaning, determine whether the cleaning is complete. If it is complete, transmit the actual result image to the display component for display; otherwise, use the actual result image as the surface image and continue the cleaning operation.

2. The deep learning-based online monitoring system for laser cleaning according to claim 1, characterized in that, The target detection employs an improved YOLOv5 network model, which includes an input layer, a backbone network, intermediate layers, and a prediction layer. The input layer performs data augmentation, image resizing, and adaptive anchor box calculation on the input image. The backbone network performs image segmentation, convolution, and feature extraction. The intermediate layers achieve image feature fusion. The prediction layer generates predicted bounding boxes, predicted categories, and labels for targets in the original image.

3. The deep learning-based online monitoring system for laser cleaning according to claim 2, characterized in that, The improved Yolov5 network model is also trained using the following steps: The surface image of the substrate after laser cleaning is acquired, and the surface image is preprocessed to obtain a preprocessed image; A classification and annotation tool was used to classify and annotate the detected targets in the preprocessed images to obtain a dataset, which was then divided into a training set, a validation set, and a test set in a ratio of 7:2:

1. The model is trained and iteratively optimized using the training set to obtain the optimal model; Based on the optimal model, target detection is performed on the validation set to obtain the detection results.

4. The deep learning-based online monitoring system for laser cleaning according to claim 1, characterized in that, Target localization is performed using the U-Net network model, which includes a shrinking path and an expanding path. The shrinking path performs a downsampling operation on the input feature map, and the expanding path performs an upsampling operation on the feature image.

5. The deep learning-based online monitoring system for laser cleaning according to claim 4, characterized in that, The process also includes training the U-Net network model through the following steps: The surface image of the substrate after laser cleaning is acquired, and the surface image is preprocessed to obtain a preprocessed image; A classification and annotation tool was used to classify and annotate the detected targets in the preprocessed images to obtain a dataset, which was then divided into a training set, a validation set, and a test set in a ratio of 7:2:

1. The model is trained and iteratively optimized using the training set to obtain the optimal model; Based on the optimal model, the validation set is segmented to obtain the segmentation results.

6. The deep learning-based online monitoring system for laser cleaning according to claim 3, characterized in that, When training the improved Yolov5 network model, the Focal EIoU Loss function is used for loss calculation and accuracy optimization. The Focal EIoU Loss function combines the EIoU Loss function with Focal Loss to increase the accuracy of regression. The formula for the Focal EIoU Loss function is as follows: ； Where IoU represents the intersection-union ratio between the predicted bounding box and the ground truth bounding box. Represents the modulation coefficient. The EIoU Loss function includes overlap loss, center distance loss, and width / height loss. The formula for the EIoU Loss function is as follows: ； in, Indicates the center distance loss. denoted by , where represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, and c represents the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes. Indicates width loss. This indicates a significant loss.

7. The deep learning-based online monitoring system for laser cleaning according to claim 5, characterized in that, The U-Net network model is trained using Caffe's stochastic gradient descent method, and the loss function is calculated using an energy function. This energy function is derived from the maximum pixel value on the final feature map and the cross-entropy loss function. The formula for calculating the maximum value is as follows: ； in, This represents an approximately maximum function. This represents the activation level of feature channel k at pixel location, where K is the number of categories.

8. A deep learning-based online monitoring method for laser cleaning, applied to the online monitoring system for laser cleaning as described in any one of claims 1 to 7, characterized in that, include: S1, acquire the image of the cleaned workpiece surface and the current position of the robotic arm, and preprocess the workpiece surface image to obtain the initial cleaned image and the initial coordinates of the image; S2, Based on the current position of the robotic arm and the initial coordinates of the image, convert the initial coordinates of the image into three-dimensional coordinates in the base coordinate system of the robotic arm; S3, after inputting the initial cleaned image into the network model for target detection, recognition, segmentation and localization, the predicted cleaned image is obtained; S4. Clean the image according to the prediction result, obtain the coordinate values ​​of the cleaned image according to the prediction result, and convert them into three-dimensional coordinates in the robot arm base coordinate system to obtain the predicted cleaning position. S5. Based on the predicted cleaning position, calculate the coordinate difference between the predicted position and the current position of the robotic arm to obtain the deviation value; S6. Based on the deviation value, adjust the position of the robotic arm and perform cleaning, and collect an image of the actual result after cleaning; S7. Based on the actual result image after cleaning, determine whether it is clean. If it is clean, the actual result image after cleaning is the final cleaning result image; otherwise, use the actual result image as the surface image and continue the cleaning operation.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by the processor of the device on which the computer-readable storage medium resides, implement the deep learning-based online monitoring method for laser cleaning as described in claim 8.

Images