Indoor spraying robot wall spraying defect detection and positioning method

Abstract

The present invention discloses a method for detecting and locating wall spray defects of indoor spray painting robots, comprising: step S1: fusing a reversible column network and OSNet to construct a defect feature extraction network; step S2: using a full-scale feature extraction module and a reversible branch to obtain feature information at different levels; step S3: using a twin network as a spray defect detection end to extract the feature difference between positive and negative samples and obtain the feature vector L2 distance; step S4: adding an Edge‑Seg module to the twin network as a spray defect location end to extract the defect shape and edge information of different scales and obtain spray defect location information; step S5: using a multi-task learning method to construct an integrated model RSNet for spray defect detection and location; step S6: using RSNet to extract the feature difference between positive and negative samples to obtain sample similarity, and performing semantic segmentation and edge extraction on abnormal images to obtain the defect location. This method can effectively realize online detection and location of spray defects, and improve detection accuracy and location accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for detecting and locating wall spraying defects by an indoor spraying robot. Background Art

[0002] With the rapid development of construction robots, indoor spray painting robots are gradually coming into the spotlight. Currently, some indoor spray painting robots are capable of autonomously spraying walls. However, spraying operations inevitably result in defects such as air holes and sagging. Traditional manual inspection is no longer suitable for automated spraying systems, and spray defect detection has become a key technology for achieving automated spraying.

[0003] Wall paint defect detection is a surface defect detection technology. Deep learning-based defect detection methods are divided into three main frameworks: detection network, segmentation network, and twin network. Both detection network and segmentation network require a large number of negative samples and corresponding labels, which is difficult to achieve for indoor spray paint defects. The twin network uses the same weights and architecture, working collaboratively on two different input vectors to generate comparable output vectors, allowing the full role of positive samples in defect detection tasks and reducing the dependence on negative sample data to a certain extent. Miao et al. proposed a cost-sensitive twin network that uses threshold shifting to distinguish the authenticity of PCBs, reducing training time by 33.32% while maintaining accuracy. Chen et al. proposed a defect-sensitive loss function based on the twin network for industrial surface defect detection, achieving 100% accuracy when the positive-to-negative sample ratio is 10:1 and 50:1. Li et al. proposed an end-to-end method that uses the twin network as part of the downstream classifier to capture defect features. Zhang Haigang proposed a twin network-based industrial defect visual inspection algorithm. This algorithm uses a weight-sharing twin network to derive differential features from positive and negative sample pairs. Combined with a weakly supervised localization algorithm, it achieves defect detection and localization. Jiarui Lei et al. proposed PyramidFlow, a high-resolution defect anomaly localization algorithm. This algorithm uses the twin network training process to optimize the model by minimizing frequency differences within the image. During the evaluation phase, the latent template used during training is applied to obtain anomaly localization maps. X Tao et al. proposed a dual twin network for surface defect anomaly detection. The first twin network captures defect-free samples and randomly generated defects and uses a dense feature fusion module to obtain a dense feature representation of the dual inputs. The second twin network reconstructs and maps the dual dense features from the previous stage, improving defect detection accuracy. Dai Gang et al. proposed a PCB defect classification algorithm based on a three-channel twin network. This algorithm uses feature vector similarity to determine whether a PCB is defective and integrates a deep feature residual network to improve model accuracy. Compared to detection and segmentation networks, twin networks are more suitable for defect detection tasks with small sample sizes. However, twin networks require twice as much input during training, significantly increasing computing resources and training time, making them difficult to implement on mobile devices like robots. Furthermore, twin networks cannot accurately locate defects, preventing the spray robot from performing subsequent polishing work.

[0004] In summary, in terms of surface defect detection, twin network-based defect detection algorithms can effectively leverage positive samples, reduce reliance on negative data, and are suitable for wall paint defect detection. However, twin networks still have high computational overhead, making them unsuitable for mobile devices such as robots. Furthermore, twin networks cannot accurately locate defects, resulting in limited adaptability of existing twin network-based surface defect detection algorithms. Detecting and locating wall paint defects remains a challenging research topic.

[0005] Therefore, it is very meaningful to provide a method for detecting and locating wall spraying defects using an indoor spraying robot. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a wall spraying defect detection and positioning method for an indoor spraying robot that can solve the problems of low autonomy of indoor spraying robots, inability to perform online spraying quality detection, and low spraying detection and positioning accuracy.

[0007] The technical solution of the present invention is: a method for detecting and locating wall spray defects by an indoor spray painting robot, comprising:

[0008] S1: Fusing the reversible column network and the lightweight backbone network OSNet to build the defect feature extraction network R-OSNet;

[0009] S2: Use the full-scale feature extraction module and reversible branch in R-OSNet to obtain feature information at different levels;

[0010] S3: Using the twin network structure as the spray defect detection end, the feature difference between positive and negative samples is extracted to obtain the feature vector L2 distance, thereby calculating the similarity between positive and negative samples and making defect judgments based on the similarity;

[0011] S4: Add the Edge-Seg module to the twin network as the spray defect location end to extract the defect shape and edge information to obtain the spray defect location information;

[0012] S5: Use multi-task learning method to build RSNet, an integrated model for spray defect detection and localization;

[0013] S6: Use RSNet to extract the feature differences between positive and negative samples to obtain sample similarity, and perform semantic segmentation and edge extraction on abnormal images to obtain the defect location.

[0014] Specifically, the reversible column network and the lightweight backbone network OSNet are integrated in S1 to construct the defect feature extraction network R-OSNet, which includes:

[0015] S1.1: Four convolutional feature streams are used to construct the main branch to obtain different receptive fields and enrich semantic information. Depthwise separable is used to replace ordinary convolution to reduce the number of model parameters.

[0016] S1.2: Generate channel weights using a unified aggregation gate and dynamically fuse multi-scale spray defect features;

[0017] S1.3: Add a reversible auxiliary branch to the main branch to separate shallow texture information and deep semantic information, providing information for the spray defect detection task and the spray defect localization task respectively.

[0018] Specifically, in S4, the Edge-Seg module is added to the twin network to extract the defect shape and edge information to obtain the spray defect location information, including:

[0019] S4.1: A U-shaped structure is used to process feature information at four scales. The decoder restores the image resolution and concatenates the feature maps between the encoder and decoder in the channel dimension to obtain a spray defect segmentation map at four scales.

[0020] S4.2: Fusion edge detection, adding two side outputs to the four-scale spray defect segmentation map, processing spray defect segmentation information and semantic boundary information respectively;

[0021] S4.3: Create a weight map for each of the shape and segmentation streams and upsample them to a uniform output resolution.

[0022] S4.4: Use 1×1 convolution as the merging head to fuse the outputs of the shape stream and the segmentation stream, and apply the softmax function at different resolution levels to convert them into classification probability maps to obtain the defect type and location.

[0023] This invention provides a method for detecting and locating wall paint defects using an indoor painting robot. This method constructs an integrated model for detecting and locating paint defects, effectively detecting anomalies in wall paint defects, determining their type, and locating them. This model enables real-time detection on indoor painting robots, improving their autonomy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A diagram of the R-OSNet feature extraction network structure provided by the disclosed embodiment of the present invention;

[0027] Figure 2 A structural diagram of a full-scale feature learning module provided in an embodiment disclosed in the present invention;

[0028] Figure 3 A network structure diagram of an integrated model for spray defect detection and positioning provided by an embodiment of the present invention;

[0029] Figure 4 A diagram showing the spray defect location network structure provided by the disclosed embodiment of the present invention;

[0030] Figure 5 A diagram of a multi-task learning architecture provided for an embodiment of the present invention;

[0031] Figure 6 This is a visualization diagram of the last layer features of different backbone networks provided in the disclosed embodiments of the present invention.

[0032] Figure 7 This is a diagram showing the spray defect positioning effect provided in the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0033] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of systems consistent with certain aspects of the present invention, as detailed in the appended claims.

[0034] In order to solve the problems of low autonomy of indoor spray painting robots in the existing technology, inability to perform online spraying quality inspection, and low spraying inspection and positioning accuracy. This embodiment provides a method for detecting and positioning wall spraying defects of indoor spray painting robots. The overall network structure diagram is shown in the figure. Figure 3 Specifically, the method includes the following steps:

[0035] Step 1: Fusion the reversible column network and the lightweight backbone network OSNet to construct the defect feature extraction network R-OSNet.

[0036] Specifically,

[0037] 1) If Figure 1 As shown in . Four convolutional feature flows are used to construct the main branch. Figure 2 As shown in the figure, each convolutional stream uses a scaling factor, T, to determine the size of the feature increment. When T = 4, the corresponding receptive fields are 3×3, 5×5, 7×7, and 9×9. The Lite 3×3 architecture introduces depthwise separable convolution, which reduces computational complexity to a certain extent. is the convolution feature flow output, and its calculation formula is:

[0038]

[0039] 2) If Figure 2As shown in Figure 2. A unified aggregation gate is used to calculate the weights of the four channels, thereby achieving feature fusion at different scales. The four feature streams all use the same aggregation gate, which contains three layers: a global average pooling layer and two fully connected layers. Figure 1 As shown. Adjust the input size to 224×224, remove the global pooling layer and the fully connected layer, change the output layer of conv3 to 512, and the output layer of conv5 to 1024. Then reassemble, conv1 is the STEM layer, conv2 and transition are Layer1, conv3 and transition are Layer2, conv4 is Layer3, and conv5 is Layer4. Figure 1 As shown, the STEM layer is used as the shared layer of all reversible branches, and Layer1, Layer2, Layer3 and Layer4 are stacked to form the main branch;

[0040] 3) Add a reversible branch based on the main branch. The structure of the reversible branch is the same as that of the main branch, consisting of different columns. The columns are connected by fusion blocks, which are composed of upsampling modules in different ways.

[0041] Step 2: Use the full-scale feature extraction module and reversible branch in R-OSNet to obtain feature information at different levels. The feature map is as follows Figure 6 The feature extraction network not only fuses features from different scales, but also fuses features from different levels, taking into account both global and local features, and can express the contour information of shallow features and the semantic information of deep features.

[0042] Step 3: For the paint anomaly detection task, a pair of positive and negative samples are fed into two parameter-sharing twin networks. Visual features are extracted from each, and a feature difference operation is performed to determine the L2 distance between the feature vectors of the normal and abnormal samples. Since the L2 distance between a defect-free sample and a normal sample is small, and the L2 distance between a defective sample and a normal sample is large, paint anomaly detection can be achieved by setting a threshold based on the L2 distance. Euclidean distance is used as the metric distance formula for the feature vector to accommodate contrast loss.

[0043] Step 4: For the defect localization task, we propose the Edge-Seg module. This module shares the feature extraction network of the negative sample stream with the defect anomaly detection task. It uses feature maps from different layers as input and outputs the results of the shape stream and segmentation stream. The shape stream contains defect boundary information, and the segmentation stream contains defect segmentation information. The two are then fused to obtain the defect shape and location results.

[0044] Specifically,

[0045] 1) The Edge-Seg module adopts a U-shaped structure, such as Figure 4 As shown in the figure, the decoder structure on the left uses the second branch of the Siamese network for downsampling, while the decoder structure on the right is used to restore image resolution. Downsampling in the encoder continuously refines the paint defect features, while upsampling in the decoder gradually restores the refined features. The feature maps between the encoder and decoder are then concatenated in the channel dimension, and finally a 1×1 convolution is performed to output the paint defect segmentation map.

[0046] 2) Two side outputs are added to each feature map, namely the shape stream and the segmentation stream. The shape stream processes the shape information of spray defects through the boundaries of the feature map, and the segmentation stream processes the normal segmentation information.

[0047] 3) Create a weight map for each prediction level, upsample the weight map to match the output resolution, and convert it into a classification probability map by applying a softmax function on the connected resolution levels.

[0048] 4) Finally, a merging head is designed to fuse the multi-scale features of the boundary stream and the segmentation stream into a multi-resolution prediction. The formula is as follows:

[0049]

[0050] Step 5: Use multi-task learning method to build the spray defect detection and positioning integrated model RSNet; Figure 5 As shown in the figure, a multi-task learning architecture with hard parameter sharing is used to build an integrated model for spray defect detection and localization, enabling the model to perform simultaneous training for both spray defect detection and localization tasks.

[0051] During the training phase, the model inputs a defect-free normal sample A in channel 1 and a normal sample or an abnormal sample B with defects in channel 2. After feature extraction using a shared agent, the losses for both tasks are calculated. Backpropagation is then used to iteratively update the network parameters, completing both anomaly detection and defect location. During the inference phase, normal sample A undergoes only a single forward propagation, and its features are stored. The input is the sample to be tested. After feature extraction, a direct feature difference operation is performed against the stored normal features, further improving the model's computation speed and the efficiency of the spray robot.

[0052] Step 6: Use RSNet to extract the feature differences between positive and negative samples to obtain sample similarity, and perform semantic segmentation and edge extraction on the abnormal image to obtain the defect location.

[0053] Experiments have demonstrated that the proposed RSNet network model can effectively address the problem of wall paint defect detection and location for indoor spray painting robots. The model achieves 99.7% accuracy in detecting paint defect anomalies and 87.3% defect location accuracy with a detection speed of 0.53 seconds, enabling real-time detection on indoor spray painting robots.

[0054] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. Indoor spray robot wall spray defect detection and positioning method, characterized in that, include: S1: The reversible column network and the lightweight backbone network OSNet are integrated to construct the defect feature extraction network R-OSNet. In the network R-OSNet, conv1 is the STEM layer, conv2 and transition are Layer 1, conv3 and transition are Layer 2, conv4 is Layer 3, and conv5 is Layer 4. The STEM layer is used as the shared layer for all reversible branches, and Layer 1, Layer 2, Layer 3, and Layer 4 are stacked to form the main branch. On the basis of the main branch, a reversible branch is added. The structure of the reversible branch is the same as that of the main branch, forming different columns. The columns are connected by Fusion Blocks, which are composed of upsampling modules in different ways. S2: Use the full-scale feature extraction module and reversible branch in R-OSNet to obtain feature information at different levels; S3: Using the twin network structure as the spray defect detection end, the feature difference between positive and negative samples is extracted to obtain the feature vector L2 distance, thereby calculating the similarity between positive and negative samples and making defect judgments based on the similarity; S4: Add the Edge-Seg module to the twin network as the spray defect location end to extract the defect shape and edge information to obtain the spray defect location information; S5: Use multi-task learning method to build RSNet, an integrated model for spray defect detection and localization; S6: Use RSNet to extract the feature differences between positive and negative samples to obtain sample similarity, and perform semantic segmentation and edge extraction on abnormal images to obtain the defect location.

2. The indoor spray painting robot wall spraying defect detection and positioning method according to claim 1 is characterized in that: In S1, the reversible column network and the lightweight backbone network OSNet are integrated to construct the defect feature extraction network R-OSNet, which includes: S1.1: Four convolutional feature streams are used to construct the main branch to obtain different receptive fields and enrich semantic information. Depthwise separable is used to replace ordinary convolution to reduce the number of model parameters. S1.2: Generate channel weights using a unified aggregation gate and dynamically fuse multi-scale spray defect features; S1.3: Add a reversible auxiliary branch to the main branch to separate shallow texture information and deep semantic information, providing information for the spray defect detection task and the spray defect localization task respectively.

3. The indoor spray painting robot wall spraying defect detection and positioning method according to claim 1 is characterized in that: In S4, the Edge-Seg module is added to the twin network to extract the defect shape and edge information to obtain the spray defect location information, including: S4.1: A U-shaped structure is used to process feature information at four scales. The decoder restores the image resolution and concatenates the feature maps between the encoder and decoder in the channel dimension to obtain a spray defect segmentation map at four scales. S4.2: Fusion edge detection, adding two side outputs to the four-scale spray defect segmentation map, processing spray defect segmentation information and semantic boundary information respectively; S4.3: Create a weight map for each of the shape and segmentation streams and upsample them to a uniform output resolution. S4.4: Use 1×1 convolution as the merging head to fuse the outputs of the shape stream and the segmentation stream, and apply the softmax function at different resolution levels to convert them into classification probability maps to obtain the defect type and location.

Images