A method for detecting the construction quality of a wall surface with multi-angle lighting

Abstract

The present invention discloses a method for detecting the construction quality of a wall with multi-angle lighting, belonging to the technical field of building wall construction detection, including: using a camera in front of the equipment pan-tilt, and fill lights in front of and on the front side of the equipment pan-tilt to collect image data; using the method of convolving the original image and then deconvolving it to generate an image corresponding to the original image; performing a shadow adjustment algorithm on the above image; for all the edges obtained above, calculating the total length q occupied by the edges, and q is the judgment criterion for whether there are traces on the wall. For the entire image, calculate its variance δ, and use the variance δ as the judgment criterion for whether the wall is homogeneous as a whole. Through empirical debugging, select appropriate q and δ thresholds, and any wall with a value greater than the threshold is unqualified. The present invention processes images through traditional vision algorithms, and by calculating the number and length of image edges, it can be used as a basis for judging whether there are obvious traces on the wall, and can detect the quality of home wall decoration from multiple angles.

Description

Technical Field

[0001] The present invention relates to the technical field of building wall construction inspection, and in particular to a method for inspecting the quality of wall construction with multi-angle lighting. Background Technique

[0002] During the wall construction process of a construction robot, a suitable scheme is needed to evaluate the final result of the equipment construction. The on-site evaluation process is strongly related to the observer's angle and the light angle. The acceptance standard for factory wall decoration is relatively low. Only front lighting and front observation are required, and if it passes the acceptance, it is considered qualified. For the home decoration standard, a method of side lighting and front observation is also required. Only when both front and side lighting are qualified, the wall construction quality inspection is considered qualified.

[0003] After the wall treatment robot for building walls finishes the construction, it takes pictures of the wall through the camera and fill light in front of the equipment pan-tilt. The construction quality is judged through the images. However, the positions of the camera and fill light are fixed, and only images with front lighting and front shooting can be captured, which only suits the inspection method of the factory standard. For high-standard home decoration, an algorithm is needed to simulate the effect of side lighting and front shooting to check whether the home wall decoration effect meets the standard. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for inspecting the quality of wall construction with multi-angle lighting to solve the technical problem in the prior art that the side lighting and front shooting of the wall effect need to be jointly inspected for home wall decoration.

[0005] The present invention provides a method for inspecting the quality of wall construction with multi-angle lighting. This quality inspection method needs to be used in cooperation with a building wall treatment robot. The building wall treatment robot controls the wall treatment mechanism to scrape and polish the wall putty through the equipment pan-tilt. A camera and a fill light are fixed directly in front of the equipment pan-tilt. It is characterized in that: a fill light is also fixed on the front side of the equipment pan-tilt. The quality inspection of the wall construction is carried out by the building wall treatment robot. This quality inspection method includes:

[0006] Sp1: Data collection: Use the camera in front of the equipment pan-tilt and the fill light in front of the equipment pan-tilt to collect image data. The collected image data is used as the input of the training data of the training set. At the same position, turn off the front fill light and turn on the side fill light as the label corresponding to the training data. At the same position, take an image once with front lighting and once with side lighting through the fill light, and the two correspond one by one, as the input and label during the neural network training;

[0007] Sp2: Deep learning: Use the method of convolving the original image and then deconvolving to generate an image corresponding to the original image. After the new image is generated, it corresponds to the image taken directly under side lighting;

[0008] Sp3: Traditional vision algorithm: Perform a shadow adjustment algorithm on the above image to enhance the darkening effect of the shadow part in the original image, form a more obvious visual contrast, and simulate the effect of side lighting. First, identify the shadow area, then perform color transformation on the shadow area to achieve the darkening effect, and finally smooth the edge between the shadow area and the non-shadow area;

[0009] Sp4: Judge the construction quality: For all the edges obtained above, calculate the total length q occupied by the edges. q is the judgment standard for whether there are marks on the wall. Calculate the variance δ for the entire image, and use the variance δ as the judgment standard for whether the wall is uniform as a whole. According to experience debugging, select appropriate q and δ thresholds. Any wall with a value greater than the threshold is unqualified.

[0010] Furthermore, in the deep learning method, the following steps are included:

[0011] Sp2-1: The image generation network uses a fully convolutional neural network. The left network is used as the feature extraction network, using conv convolution and pooling. On the contraction path, every two 3*3 convolutional layers are followed by a 2*2 max pooling layer with a stride of 2, and a relu activation function is used after each convolutional layer to perform downsampling on the original image. At the same time, the number of channels doubles with each downsampling;

[0012] Sp2-2: The right network is used as the feature generation network. Use the feature map generated by upsampling to connect with the left feature map. In the upsampling of the expansion path, there is a 2*2 convolutional layer in each step, and its activation function is also relu, as well as two 3*3 convolutional layers. At the same time, the feature map from the corresponding contraction path is added in each step of upsampling and cropped to maintain the same shape;

[0013] Sp2-3: Finally, after two more convolutional operations, generate a feature map, and then use two convolutions and a 1*1 convolution for classification to obtain three images, representing the R, G, and B layers of the image, and then calculate the loss function with the images in the database and perform backpropagation calculation.

[0014] Furthermore, in the traditional vision algorithm, the following steps are included:

[0015] Sp3-1: Convert the image to grayscale and normalize it;

[0016] Sp3-2: Determine the shadow area: The grayscale value of each image point is gray. For each point, calculate d = (1 - gray) * (1 - gray), calculate the average value d1 of d for the entire image, and use d1 as the dividing line.

[0017] Sp3-3: In the area where d > d1, that is, the darker area of the original image, reduce the values of the RGB channels of the original image. In the area where d < d1, that is, the brighter area of the original image, increase the values of the RGB channels of the original image. After processing, change the values exceeding 255 to 255.

[0018] Furthermore, based on the image data obtained by the traditional vision algorithm, after the image light source is converted, the construction quality can be evaluated. The visual evaluation standard for wall construction is uniform color without stripe undulations, and the corresponding image is that the color of the image is uniform without obvious edges.

[0019] Furthermore, use the edge detection algorithm to detect the number and length of edges in the image. Use the Canny edge detection method, which includes the following steps:

[0020] Step 1: Remove noise: For the edge noise generated by the pixels of the original image, use Gaussian blurring to remove most of the edge noise, or minimize the unnecessary image details that generate unnecessary edges.

[0021] Step 2: Calculate the brightness gradient value of the image: After the image is smoothed, use the Sobel horizontal and vertical convolution kernels to filter the image, and calculate the amplitude G and direction θ of the brightness gradient using the filtering results. The calculation methods are as follows:

[0022]

[0023]

[0024] Step 3: Remove false edges: After removing the noise and calculating the brightness gradient in the image, use non-maximum suppression to remove the unnecessary pixels. Remove the false edges by comparing the gradient values of each pixel with the surrounding pixels in the horizontal and vertical directions. If the gradient corresponding to a certain pixel is the largest locally, that is, larger than the gradients of its upper, lower, left, and right pixels, it is retained; otherwise, the pixel is set to 0.

[0025] Step 4: Compare the gradient value with two thresholds, one of the two thresholds is smaller than the other, and judge the intensity of the gradient value.

[0026] Further, in the gradient value comparison and judgment, if the image gradient value is larger than the larger threshold, it means that this pixel is a very strong edge and it is retained in the final edge map; if the gradient value is smaller than the smaller threshold, then this pixel is suppressed and removed from the final edge map; if the gradient value falls within the range of the two given thresholds, this pixel is marked as a weak edge. If the weak edge is connected to a strong edge, it is retained in the edge map, otherwise, it is removed.

[0027] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0028] In the prior art, after the construction of the building wall surface by the wall surface treatment robot, the wall surface is photographed by the camera and the fill light in front of the equipment pan-tilt, and the construction quality is judged through the image. However, the positions of the camera and the fill light are fixed, and only the images of the front illumination and the front shooting can be obtained, which only adapts to the inspection method of the factory standard. For the high-standard home decoration, an algorithm needs to be used to simulate the effect of the side illumination and the front shooting to check whether the decoration effect of the home wall surface meets the standard. In view of this situation, the present invention designs a method for detecting the construction quality of the wall surface. By cooperating with the building wall surface treatment robot, the camera is used to collect images, and fill lights are arranged in front of and on the front side of its equipment pan-tilt. The image data taken after lighting in the two directions is sequentially passed through the deep learning and the traditional vision algorithm. The deep learning uses the method of convolutional operation on the original image and then deconvolution to generate an image corresponding to the original image. After the new image is generated, it corresponds to the image of the front shooting under the oblique light illumination. And through the traditional vision algorithm for image processing, by calculating the number and length of the image edges, the basis for judging whether there are obvious traces on the wall surface can be obtained, and the quality of the home wall surface decoration can be detected from multiple angles. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the specific embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0030] Figure 1 It is a schematic diagram of collecting wall surface image data in the present invention;

[0031] Figure 2 It is a flowchart of the wall surface detection method in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The following will clearly and completely describe the technical solutions of the present invention with reference to the drawings. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0033] The components of the embodiments of the present invention that are typically depicted and shown in the accompanying drawings herein can be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention.

[0034] All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention. Specific embodiments:

[0036] Below in conjunction with Figures 1 to 2 As shown, an embodiment of the present invention provides a method for detecting the construction quality of a wall with multi-angle lighting. This quality detection method needs to be used in conjunction with a building wall treatment robot. The building wall treatment robot controls a wall treatment mechanism to scrape and polish wall putty through an equipment pan-tilt. A camera and a fill light are fixed in front of the equipment pan-tilt. It is characterized in that: a fill light is also fixed on the front side of the equipment pan-tilt. The construction quality of the wall is detected by the building wall treatment robot. This quality detection method includes:

[0037] Sp1: Collect data: Use the camera in front of the equipment pan-tilt and the fill light in front of the equipment pan-tilt to collect image data. The collected image data is used as the input of the training data of the training set. At the same position, turn off the front fill light and turn on the side fill light as the label corresponding to the training data. At the same position, collect an image once with front lighting and once with side lighting by the fill light, and the two correspond one by one as the input and label during neural network training;

[0038] Sp2: Deep learning: Use the method of original image convolution and then deconvolution to generate an image corresponding to the original image. After the new image is generated, it corresponds to the image taken directly under side lighting.

[0039] In the deep learning method, the following steps are included:

[0040] Sp2-1: The image generation network adopts a fully convolutional neural network. The left network is used as the feature extraction network. Use conv convolution and pooling. On the contraction path, every two 3*3 convolutional layers are followed by a 2*2 max pooling layer with a stride of 2, and a relu activation function is used after each convolutional layer to perform downsampling operations on the original image. At the same time, the number of channels doubles with each downsampling;

[0041] Sp2-2: Generate a network characterized by the right network. Perform a concatenation operation on the feature map generated by upsampling and the left feature map. In the upsampling of the expansion path, there will be a 2*2 convolutional layer at each step, and its activation function is also relu, as well as two 3*3 convolutional layers. At the same time, the feature map from the corresponding contraction path will be added to the upsampling at each step and cropped to maintain the same shape;

[0042] Sp2-3: Finally, perform two more convolutional operations to generate a feature map, and then use two convolutions and a 1*1 convolution for classification to obtain three maps, representing the R, G, and B layers of the image, and then calculate the loss function with the images in the database and perform backpropagation calculation;

[0043] Sp3: Traditional vision algorithm: Perform a shadow adjustment algorithm on the above image to increase the darkening effect of the shadow part in the original image, form a more obvious visual contrast, and simulate the effect of side lighting. First, identify the shadow area, then perform color transformation on the shadow area to achieve the darkening effect, and finally smooth the edge between the shadow area and the non-shadow area;

[0044] In the traditional vision algorithm, the following steps are included:

[0045] Sp3-1: Convert the image to grayscale and normalize it;

[0046] Sp3-2: Determine the shadow area: The grayscale value of each image point is gray, and for each point, calculate d=(1-gray)*(1-gray). Calculate the average value d1 of d for the entire image, and use d1 as the dividing line;

[0047] Sp3-3: In the area where d>d1, that is, the darker area of the original image, reduce the values of the RGB channels of the original image. In the area where d<d1, that is, the brighter area of the original image, increase the values of the RGB channels of the original image. After processing, change the values exceeding 255 to 255;

[0048] Based on the image data obtained from the traditional vision algorithm, after the image light source is converted, the construction quality can be evaluated. The visual evaluation standard for wall construction is uniform color and no stripe undulations. The corresponding image is that the color of the image is uniform and there are no obvious edges. Use an edge detection algorithm to detect the number and length of edges in the image. Using the Canny edge detection method, the following steps are included:

[0049] Step 1: Remove noise: For the edge noise generated by the pixels of the original image, use Gaussian blurring to remove most of the edge noise, or minimize unnecessary image details that generate unnecessary edges;

[0050] Step 2: Calculate the brightness gradient value of the image: After the image is smoothed, use Sobel horizontal and vertical convolution kernels to convolve and filter the image, and calculate the amplitude G and direction θ of the brightness gradient using the filtering results. The calculation method is as follows:

[0051]

[0052]

[0053] Step 3: Remove false edges: After removing the noise in the image and calculating the brightness gradient, use non-maximum suppression to remove unnecessary pixels. By comparing the gradient values of each pixel with the surrounding pixels in the horizontal and vertical directions, false edges are removed. If the gradient corresponding to a certain pixel is the largest locally, that is, larger than the gradients of its upper, lower, left, and right pixels, it is retained; otherwise, the pixel is set to 0.

[0054] Step 4: Compare the gradient value with two thresholds. One of the two thresholds is smaller than the other, and the intensity of the gradient value is judged.

[0055] In the above gradient value comparison and judgment, if the image gradient value is larger than the larger threshold, it means that this pixel is a very strong edge and it is retained in the final edge map; if the gradient value is smaller than the smaller threshold, the pixel is suppressed and removed from the final edge map; if the gradient value falls within the range of the two given thresholds, the pixel is marked as a weak edge. If the weak edge is connected to a strong edge, it is retained in the edge map; otherwise, it is removed.

[0056] Sp4: Judge the construction quality: For all the edges obtained above, calculate the total length q occupied by the edges. q is the judgment standard for whether there are marks on the wall. Calculate the variance δ of the whole image, and use the variance δ as the judgment standard for whether the wall is uniform as a whole. According to experience debugging, select appropriate q and δ thresholds. Any wall with a value greater than the threshold is unqualified.

[0057] In the prior art, after a construction wall surface treatment robot finishes the construction of a wall surface, it takes pictures of the wall surface through a camera and a supplementary light in front of the equipment pan-tilt head, and judges the construction quality through the images. However, the positions of the camera and the supplementary light are fixed, and only images with frontal illumination and frontal shooting can be captured, which only suit the inspection methods of factory standards. For high-standard home decoration, algorithms need to be used to simulate the effects of side illumination and frontal shooting to check whether the home wall decoration effect meets the standards. In view of this situation, the present invention designs a method for detecting the construction quality of a wall surface. By cooperating with a construction wall surface treatment robot, image acquisition is carried out using a camera, and supplementary lights are arranged in front of and on the front side of its equipment pan-tilt head. The image data taken after lighting from two directions are sequentially passed through deep learning and traditional vision algorithms. Deep learning uses the method of convolving the original image and then deconvolving it to generate an image corresponding to the original image. After the new image is generated, it corresponds to the image taken with frontal shooting under oblique light illumination. And the image is processed through traditional vision algorithms, and by calculating the number and length of the image edges, the basis for judging whether there are obvious traces on the wall surface can be obtained, and the quality of home wall decoration can be detected from multiple angles.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the construction quality of a wall with multi-angle lighting, which is used in conjunction with a building wall treatment robot. The building wall treatment robot controls a wall treatment mechanism to scrape and polish wall putty through an equipment pan-tilt. A camera and a fill light are fixed in the front of the equipment pan-tilt. It is characterized in that: A fill light is also fixed on the front side of the device pan-tilt. The wall construction quality is detected by a building wall treatment robot. The quality detection method includes: Sp1: Data collection: Use the camera directly in front of the device pan-tilt and the fill light directly in front of the device pan-tilt to collect image data. The collected image data is used as the training data of the training set. At the same position, turn off the fill light directly in front and turn on the fill light on the front side as the label corresponding to the training data. At the same position, collect images once with the fill light shining directly and once with the fill light shining from the side respectively, and they correspond one by one as the input and label during neural network training; Sp2: Deep learning: Use the method of original image convolution and then deconvolution to generate an image corresponding to the original image. After the new image is generated, it corresponds to the image taken directly under side lighting; Sp3: Traditional vision algorithm: Perform a shadow adjustment algorithm on the above image to increase the darkening effect of the shadow part in the original image to form a more obvious visual contrast and simulate the effect of side lighting. First, identify the shadow area, then perform color transformation processing on the shadow area to make it achieve the darkening effect, and finally smooth the edge between the shadow area and the non-shadow area; Sp4: Judge the construction quality: For all the obtained edges, calculate the total length q occupied by the edges. q is the judgment standard for whether there are marks on the wall. Calculate the variance δ of the whole image. The variance δ is used as the judgment standard for whether the wall is uniform as a whole. According to experience debugging, select appropriate q and δ thresholds. Any wall with a value greater than the threshold is unqualified; Sp2-1: The image generation network adopts a fully convolutional neural network. The left network is used as the feature extraction network. Use conv convolution and pooling. On the contraction path, there are two 3*3 convolutional layers followed by a 2*2 max pooling layer with a stride of 2. And a relu activation function is used after each convolutional layer to perform downsampling on the original image. At the same time, the number of channels doubles with each downsampling; Sp2-2: The right network is used as the feature generation network. Connect the feature map generated by upsampling with the left feature map. In the upsampling of the expansion path, there is a 2*2 convolutional layer at each step, and its activation function is also relu, as well as two 3*3 convolutional layers. At the same time, the feature map from the corresponding contraction path is added at each step of upsampling and cropped to keep the same shape; Sp2-3: Finally, after two more convolutional operations, generate a feature map, then use two convolutions and a 1*1 convolution to do classification to get three images representing the R, G, and B layers of the image, and then calculate the loss function with the images in the database and perform backpropagation calculation; Sp3-1: Convert the image to a grayscale image and normalize it; Sp3-2: Determine the shadow area: The grayscale value of each image point is gray. Calculate d=(1-gray)*(1-gray) for each point, and calculate the average value d1 of d for the whole image and use d1 as the dividing line; Sp3-3: In the area where d > d1, that is, the darker area of the original image, reduce the values of the RGB channels of the original image. In the area where d < d1, that is, the brighter area of the original image, increase the values of the RGB channels of the original image. After processing, change the values exceeding 255 to 255.

2. The method for detecting the construction quality of a wall with multi-angle lighting according to claim 1, characterized in that: Use the Canny edge detection algorithm to detect the number and length of edges in the image, including the following steps: Step 1: Remove noise: For the edge noise generated by the pixels of the original image, use Gaussian blurring to remove most of the edge noise; Step 2: Calculate the brightness gradient value of the image: After the image is smoothed, use Sobel horizontal and vertical convolution kernels to filter the image, and use the filtering results to calculate the amplitude G and direction θ of the brightness gradient. Among them, the calculation method is as follows: Step 3: Subtract false edges: After removing the noise and calculating the brightness gradient in the image, use non-maximum suppression to remove unnecessary pixels. By comparing the gradient values of each pixel with the surrounding pixels in the horizontal and vertical directions, false edges are removed. If the gradient corresponding to a certain pixel is the largest locally, that is, larger than the gradients of its upper, lower, left, and right pixels, it is retained, otherwise, the pixel is set to 0; Step 4: Compare the gradient value with two thresholds, one of the two thresholds is smaller than the other, and judge the intensity of the gradient value.

3. A method for detecting the construction quality of a wall with multi-angle lighting according to claim 2, characterized in that: In the above gradient value comparison and judgment, if the image gradient value is larger than the larger threshold, it means that this pixel is a very strong edge and it is retained in the final edge map; if the image gradient value is less than the smaller threshold, the pixel is suppressed and removed from the final edge map; if the gradient value falls within the range of the two given thresholds, the pixel is marked as a weak edge. If the weak edge is connected to a strong edge, it is retained in the edge map, otherwise, it is removed.

Images