Wall surface spraying defect detecting and positioning method of indoor spraying robot

Abstract

The invention discloses an indoor spraying robot wall surface spraying defect detection and positioning method comprising the following steps: S1, fusing a reversible column network and an OSNet, and constructing a defect feature extraction network; s2, acquiring feature information of different levels by using a full-scale feature extraction module and a reversible branch; s3, taking the twin network as a spraying defect detection end, and extracting the feature difference of positive and negative samples to obtain a feature vector L2 distance; s4, adding an Edge-Seg module into the twin network to serve as a spraying defect positioning end, and extracting defect shapes and edge information of different scales to obtain spraying defect position information; s5, constructing a spraying defect detection and positioning integrated model RSNet by using a multi-task learning method; and S6, extracting the feature difference of the positive and negative samples by using RSNet to obtain the sample similarity, and carrying out semantic segmentation and edge extraction on an abnormal image to obtain a defect position. According to the method, online detection and positioning of the spraying defects can be effectively realized, and the detection precision and the positioning precision are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a method for detecting and positioning wall spraying defects by an indoor spraying robot. Background Art

[0002] With the rapid development of construction robots, indoor spraying robots have gradually come into people's view. Nowadays, some indoor spraying robots can spray walls autonomously. However, spraying defects such as pores and sagging are inevitable during spraying operations. Traditional manual inspection is no longer suitable for automated spraying systems, and spraying defect detection has become a key technology to achieve automated spraying.

[0003] Wall spray defect detection belongs to one of the surface defect detection technologies. Defect detection methods based on deep learning are divided into three main frameworks: detection network, segmentation network and twin network. Both the detection network and the segmentation network require a large number of negative samples and corresponding labels, which is difficult to achieve for indoor spray defects. The twin network uses the same weights and architecture, and works together on two different input vectors to generate comparable output vectors, so that the role of positive samples can be fully utilized in defect detection tasks, and the dependence on negative sample data can be reduced to a certain extent. Miao et al. proposed a cost-sensitive twin network that uses threshold movement to distinguish the authenticity of PCBs, while ensuring accuracy and shortening the training time by 33.32%. Chen et al. proposed a defect-sensitive loss function based on a twin network for industrial surface defect detection, with an accuracy of 100% when the positive and negative samples are 10:1 and 50:1. Li et al. proposed an end-to-end method that uses the twin network as part of the downstream classifier to capture defect features. Zhang Haigang proposed an industrial defect visual detection algorithm based on a twin network. The algorithm uses a weight-sharing twin network to obtain the difference features of positive and negative sample pairs, and combines it with a weakly supervised positioning algorithm to achieve defect detection and positioning functions. Jiarui Lei et al. proposed a high-resolution defect anomaly positioning algorithm PyramidFlow. The algorithm uses the training process of the twin network to optimize the model by minimizing the frequency difference within the image, and applies the potential template during training to obtain the anomaly positioning map in the evaluation stage. X Tao et al. proposed a double twin network for surface defect anomaly detection. The first twin network captures defect-free samples and randomly generated defects, and obtains a dense feature representation of dual inputs with a dense feature fusion module. The second twin network improves the defect detection accuracy by reconstructing and drawing the dual dense features of the previous stage. Dai Gang et al. proposed a PCB defect classification algorithm based on a three-channel twin network. The algorithm uses feature vector similarity to determine whether the PCB has defects, and integrates a deep feature residual network to improve the model accuracy. Compared with detection and segmentation networks, twin networks are more suitable for defect detection tasks with small sample data. However, twin networks need to process twice as much input during training, which greatly increases computing resources and training time, making them difficult to be installed on mobile devices on robots. In addition, twin networks cannot accurately locate defects, so spray robots cannot perform subsequent polishing work.

[0004] In summary, in terms of surface defect detection, the defect detection algorithm based on the twin network can effectively play the role of positive samples, reduce the dependence on negative sample data, and is suitable for wall spray defect detection. However, the twin network still has a large amount of calculation and cannot be carried on mobile devices on robots. In addition, the twin network cannot achieve accurate defect positioning, resulting in the existing twin network-based surface defect detection algorithm. The adaptability is not strong, and the wall spray defect detection and positioning task is still a challenging research topic.

[0005] Therefore, it is very meaningful to provide a method for detecting and locating wall spraying defects of an indoor spraying robot. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a wall spraying defect detection and positioning method for an indoor spraying robot that can solve the problems of low autonomy of the indoor spraying robot, inability to perform online spraying quality detection, and low spraying detection and positioning accuracy.

[0007] The technical solution of the present invention is: a method for detecting and locating wall spraying defects by an indoor spraying robot, comprising:

[0008] S1: Integrate the reversible column network and the lightweight backbone network OSNet to build the defect feature extraction network R-OSNet;

[0009] S2: Use the full-scale feature extraction module and reversible branch in R-OSNet to obtain feature information at different levels;

[0010] S3: Use the twin network structure as the spray defect detection end to extract the feature difference between positive and negative samples, obtain the feature vector L2 distance, and then calculate the similarity of positive and negative samples, and make defect judgment based on the similarity;

[0011] S4: Add the Edge-Seg module to the twin network as the spray defect location end to extract the defect shape and edge information to obtain the spray defect location information;

[0012] S5: Use multi-task learning method to build RSNet, an integrated model for spray defect detection and positioning;

[0013] S6: Use RSNet to extract the feature differences between positive and negative samples to obtain sample similarity, and perform semantic segmentation and edge extraction on abnormal images to obtain the defect location.

[0014] Specifically, the reversible column network and the lightweight backbone network OSNet are integrated in S1 to construct a defect feature extraction network R-OSNet, including:

[0015] S1.1: Four convolutional feature streams are used to construct the main branch to obtain different receptive fields and enrich semantic information. At the same time, depthwise separable is used to replace ordinary convolution to reduce the number of model parameters.

[0016] S1.2: Generate channel weights using a unified aggregation gate and dynamically fuse multi-scale spray defect features;

[0017] S1.3: Add a reversible auxiliary branch to the main branch to separate shallow texture information and deep semantic information, and provide information for the spraying defect detection task and the spraying defect localization task respectively.

[0018] Specifically, in S4, the Edge-Seg module is added to the twin network to extract the defect shape and edge information to obtain the spray defect location information, including:

[0019] S4.1: A U-shaped structure is used to process feature information at four scales, the image resolution is restored through the decoder, and the feature maps between the encoder and decoder are spliced ​​in the channel dimension to obtain spray defect segmentation maps at four scales;

[0020] S4.2: Fusion edge detection, adding two side outputs to the 4-scale spray defect segmentation map, processing the spray defect segmentation information and semantic boundary information respectively;

[0021] S4.3: Create a weight map for each of the shape and segmentation streams and upsample to a uniform output resolution.

[0022] S4.4: A 1×1 convolution is used as the merging head to fuse the outputs of the shape stream and the segmentation stream, and a softmax function is applied at different resolution levels to convert them into classification probability maps to obtain the defect type and location.

[0023] The present invention provides a method for detecting and locating wall spraying defects of an indoor spraying robot. The method constructs an integrated model for spraying defect detection and positioning, which can effectively realize abnormal detection of wall spraying defects, determination of defect types, and defect positioning. The model can be detected in real time on the indoor spraying robot, thereby improving the autonomy of the indoor spraying robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 The R-OSNet feature extraction network structure diagram provided for the disclosed embodiment of the present invention;

[0027] Figure 2 A structural diagram of a full-scale feature learning module provided in the disclosed embodiment of the present invention;

[0028] Figure 3 A network structure diagram of an integrated model for spray defect detection and positioning provided in an embodiment of the present invention;

[0029] Figure 4 A diagram of a spray defect location network structure provided by an embodiment of the present invention;

[0030] Figure 5 A multi-task learning architecture diagram provided for the disclosed embodiment of the present invention;

[0031] Figure 6 A visualization diagram of the last layer features of different backbone networks provided in the disclosed embodiments of the present invention.

[0032] Figure 7 This is a spray defect positioning effect diagram provided in the disclosed embodiment of the present invention. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of systems consistent with some aspects of the present invention as detailed in the appended claims.

[0034] In order to solve the problems of low autonomy of indoor spray painting robots in the prior art, inability to perform online spray quality inspection, and low spray inspection and positioning accuracy. This embodiment provides a wall spray defect detection and positioning method for indoor spray painting robots. The overall network structure diagram is as follows Figure 3 Specifically, the method comprises the following steps:

[0035] Step 1: Integrate the reversible column network and the lightweight backbone network OSNet to build the defect feature extraction network R-OSNet.

[0036] Specifically,

[0037] 1) If Figure 1 As shown in Figure 2, four convolutional feature flows are used to construct the main branch. Figure 2 As shown in the figure. Each convolutional flow determines the size of the feature increment by introducing a scale factor T. When T = 4, the corresponding receptive fields are 3×3, 5×5, 7×7, and 9×9. The Lite 3×3 structure introduces a depth-separable convolution, which reduces the computational complexity to a certain extent. is the convolution feature flow output, and its calculation formula is:

[0038]

[0039] 2) If Figure 2As shown in Figure 1. A unified aggregation gate is used to calculate the weights of the four channels, thereby achieving feature fusion at different scales. The four feature streams all use the same aggregation gate, which contains three layers: a global average pooling layer and two fully connected layers. Figure 1 As shown in the figure. The input size is adjusted to 224×224, the global pooling layer and the fully connected layer are removed, the output layer of conv3 is changed to 512, and the output layer of conv5 is changed to 1024. Then reassemble, conv1 is the STEM layer, conv2 and transition are Layer1, conv3 and transition are Layer2, conv4 is Layer3, and conv5 is Layer4. Figure 1 As shown, the STEM layer is used as a shared layer for all reversible branches, and Layer1, Layer2, Layer3, and Layer4 are stacked to form the main branch;

[0040] 3) On the basis of the main branch, a reversible branch is added. The structure of the reversible branch is the same as that of the main branch, and it is composed of different columns. The columns are connected by Fusion Block, which is composed of upsampling modules in different ways.

[0041] Step 2: Use the full-scale feature extraction module and reversible branch in R-OSNet to obtain feature information at different levels. Figure 6 The feature extraction network not only fuses features from different scales, but also fuses features from different levels, taking into account both global and local features, and can express the contour information of shallow features and the semantic information of deep features.

[0042] Step 3: For the spraying anomaly detection task, a pair of positive and negative samples are input into two parameter-sharing twin networks, and visual features are extracted respectively. After the feature difference operation, the L2 distance of the feature vector between the normal sample and the abnormal sample is obtained. Since the L2 distance between the defect-free sample and the normal sample is small, and the L2 distance between the defective sample and the normal sample is large. Therefore, spraying anomaly detection can be achieved by setting a threshold according to the size of the L2 distance. The Euclidean distance is used as the metric distance formula of the feature vector to adapt to the contrast loss.

[0043] Step 4: For the defect location task, an Edge-Seg module is proposed. This module shares the feature extraction network of the negative sample stream with the defect anomaly detection task, takes feature maps of different levels as module input, and outputs the results of the shape stream and segmentation stream. The shape stream is the defect boundary information, and the segmentation stream is the defect segmentation information. The two are then fused to obtain the shape and location results of the defect.

[0044] Specifically,

[0045] 1) The Edge-Seg module adopts a U-shaped structure, such as Figure 4 As shown in the figure. The decoder structure on the left uses the second branch of the twin network for downsampling, and the decoder structure on the right is used to restore the image resolution. The spray defect features are continuously refined by downsampling in the encoder, and the refined features are gradually restored by upsampling in the decoder. Then, the feature maps between the encoder and decoder are spliced ​​in the channel dimension, and finally the spray defect segmentation map is output through 1×1 convolution.

[0046] 2) Two side outputs are added to each feature map, namely the shape stream and the segmentation stream. The shape stream processes the shape information of the spraying defects through the boundary of the feature map, and the segmentation stream processes the normal segmentation information.

[0047] 3) Create a weight map for each prediction level, upsample the weight map to match the output resolution, and convert it into a classification probability map by applying a softmax function on the concatenated resolution levels.

[0048] 4) Finally, a merging head is designed to fuse the multi-scale features of the boundary stream and the segmentation stream into a multi-resolution prediction. Final prediction The formula is as follows:

[0049]

[0050] Step 5: Use multi-task learning method to build the spray defect detection and positioning integrated model RSNet; Figure 5 As shown in the figure, a multi-task learning architecture with hard parameter sharing is used to build an integrated model for spray defect detection and positioning, so that the model can perform the same training for both spray defect detection and positioning tasks.

[0051] In the training phase of the model, channel 1 inputs a normal sample A without defects, and channel 2 inputs a normal sample or an abnormal sample B with defects. After the shared subject extracts features, the losses of the two tasks are calculated respectively, and the network parameters are updated iteratively through back propagation, thereby completing anomaly detection and defect location. In the model inference phase, normal sample A only needs to perform one forward propagation and store its features. The input is the sample to be tested. After feature extraction, the feature difference operation is directly performed with the stored normal features, which can further improve the model calculation speed and improve the work efficiency of the spray robot.

[0052] Step 6: Use RSNet to extract the feature differences between positive and negative samples, obtain sample similarity, and perform semantic segmentation and edge extraction on the abnormal image to obtain the defect location.

[0053] Experiments have shown that the RSNet network model proposed in this paper can effectively solve the problem of wall spraying defect detection and positioning of indoor spraying robots. The model can achieve an accuracy of 99.7% in spraying defect anomaly detection, a defect positioning accuracy of 87.3%, and a detection speed of 0.53s, which can realize real-time detection on indoor spraying robots.

[0054] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These changes and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. Indoor spraying robot wall spraying defect detection and positioning method, characterized in that: include: S1: Integrate the reversible column network and the lightweight backbone network OSNet to build the defect feature extraction network R-OSNet; S2: Use the full-scale feature extraction module and reversible branch in R-OSNet to obtain feature information at different levels; S3: Use the twin network structure as the spray defect detection end to extract the feature difference between positive and negative samples, obtain the feature vector L2 distance, and then calculate the similarity of positive and negative samples, and make defect judgment based on the similarity; S4: Add the Edge-Seg module to the twin network as the spray defect location end to extract the defect shape and edge information to obtain the spray defect location information; S5: Use multi-task learning method to build RSNet, an integrated model for spray defect detection and positioning; S6: Use RSNet to extract the feature differences between positive and negative samples to obtain sample similarity, and perform semantic segmentation and edge extraction on abnormal images to obtain the defect location.

2. The indoor spraying robot wall spraying defect detection and positioning method according to claim 1 is characterized in that: In S1, the reversible column network and the lightweight backbone network OSNet are integrated to construct the defect feature extraction network R-OSNet, including: S1.1: Four convolutional feature streams are used to construct the main branch to obtain different receptive fields and enrich semantic information. At the same time, depthwise separable is used to replace ordinary convolution to reduce the number of model parameters. S1.2: Generate channel weights using a unified aggregation gate and dynamically fuse multi-scale spray defect features; S1.3: Add a reversible auxiliary branch to the main branch to separate shallow texture information and deep semantic information, and provide information for the spraying defect detection task and the spraying defect localization task respectively.

3. The indoor spraying robot wall spraying defect detection and positioning method according to claim 1 is characterized in that: In S4, the Edge-Seg module is added to the twin network to extract the defect shape and edge information to obtain the spray defect location information, including: S4.1: A U-shaped structure is used to process feature information at four scales, the image resolution is restored through the decoder, and the feature maps between the encoder and decoder are spliced ​​in the channel dimension to obtain spray defect segmentation maps at four scales; S4.2: Fusion edge detection, adding two side outputs to the 4-scale spray defect segmentation map, processing the spray defect segmentation information and semantic boundary information respectively; S4.3: Create a weight map for each of the shape and segmentation streams and upsample to a uniform output resolution; S4.4: A 1×1 convolution is used as the merging head to fuse the outputs of the shape stream and the segmentation stream, and a softmax function is applied at different resolution levels to convert them into classification probability maps to obtain the defect type and location.

Images