Building robot brick angle detection method based on artificial intelligence

Abstract

The building robot brick angle detection method based on artificial intelligence comprises the steps that wall brick images in different backgrounds are collected, real picture data are obtained, and the real picture data are classified; collecting a planar graph, an elevation graph and a profile graph of the building; building a virtual environment according to a planar graph, an elevation graph and a profile graph of the building, and generating virtual picture data; performing data enhancement operation on the obtained virtual picture data and the collected real picture data; the real picture data set and the virtual picture data set which are of the same classification are mixed into a semantic segmentation picture data set, and the semantic segmentation picture data set is input into the anchor-frame-free full convolutional network for wall brick recognition; detecting angular points and coordinates thereof in the picture; carrying out semantic segmentation on the obtained accurate position picture by adopting an image semantic segmentation algorithm combining a depth image and an RGB image; and accurately detecting and positioning brick angles of the image subjected to semantic segmentation.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection and relates to a brick corner detection method for a construction robot based on artificial intelligence. Background Art

[0002] In the field of interior decoration in the construction industry, the use of robots for wall tile laying has higher laying efficiency and better laying quality than manual laying methods. However, the current wall tile laying robot is greatly disturbed by complex environments during the construction process, including the difficulty in extracting internal corner points due to the complex texture and special materials of wall tiles; the indoor interference of buildings such as door frames, window frames, and railings leads to misidentification of targets, which in turn interferes with brick corner detection; the indoor and outdoor light of different weather such as light and sky causes overexposure or too dark, resulting in unstable lighting conditions during construction; the construction site, apartment type, floor, different buildings, and even different urban backgrounds cause the target detector to fail. The positioning accuracy of tile space is the key factor for tile laying robots to complete high-precision tile laying. The positioning accuracy of tiles will directly determine whether the tile laying robot is correct or not. At present, the positioning of tile laying robots mostly adopts visual positioning technology. The pixel of the camera, the surrounding lighting conditions, the color texture of the tiles, and the complex background will affect the positioning accuracy of the system, thereby affecting the efficiency and construction cycle of tile laying, and will also cause waste of raw materials. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a brick corner detection method for a construction robot based on artificial intelligence.

[0004] The present invention provides a method for detecting brick corners of a construction robot based on artificial intelligence, comprising:

[0005] Step 1: Collect wall brick images in different backgrounds, obtain real image data, and classify the real image data;

[0006] Step 2: Collect the floor plan, elevation and section drawings of the building;

[0007] Step 3: Build a virtual environment based on the floor plan, elevation and section of the building to generate virtual image data;

[0008] Step 4: Perform data enhancement operations on the obtained virtual image data and the collected real image data;

[0009] Step 5: Mix the real image dataset and the virtual image dataset of the same classification into a semantic segmentation image dataset, and input it into the anchor-free fully convolutional network for wall brick recognition;

[0010] Step 6: Detect the corner points and their coordinates in the image;

[0011] Step 7: Use an image semantic segmentation algorithm that combines the depth image and the RGB image to perform semantic segmentation on the precise location image obtained in step 6;

[0012] Step 8: Accurately detect and locate the brick corners of the semantically segmented image.

[0013] The invention discloses a method for detecting brick corners of a construction robot based on artificial intelligence, which has the following beneficial effects:

[0014] 1. In view of the changes in light at different construction times, seasons, and on different floors, ray tracing is combined with building models to generate multi-angle virtual image data of the specified building model under various lighting environments, making up for the difficulty in collecting complex background samples in actual environments.

[0015] 2. For limited real image data, virtual image data obtained by virtual reality are combined to form a real and virtual mixed data set. Based on adaptive learning and reinforcement learning, search algorithms are used to automatically search and select data enhancement strategies suitable for specific data sets.

[0016] 3. High-precision wall tile recognition model in complex environments based on deep learning

[0017] The real image data collected in the real environment is combined with the virtual image data set generated in the virtual environment after data enhancement to generate a semantic segmentation image data set specifically for this task. The depth camera is combined with the RGB camera to shield the background information far away from the camera through the depth image. Three algorithm models are established based on the deep learning method, namely the single target detection model of wall tiles, the multi-target detection model with added interference items, and the semantic segmentation model of wall tiles area. The most advanced target detection and semantic segmentation algorithm networks are trained and reasoned. After a series of comparative analysis and improvement, the model has better detection performance in complex environments.

[0018] 4. Brick corner detection and positioning

[0019] Under the given brick area conditions, the brick corner positioning accuracy can be guaranteed. The brick corner recognition is accelerated by the CPU platform to meet the time requirements of the project. The detected brick corner points are eliminated to improve the positioning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is a flowchart of a method for detecting brick corners of a construction robot based on artificial intelligence. DETAILED DESCRIPTION

[0021] like Figure 1 As shown, a method for detecting brick corners of a construction robot based on artificial intelligence of the present invention comprises:

[0022] Step 1: Collect wall brick images in different backgrounds to obtain real image data, and classify the real image data, specifically:

[0023] Step 1.1: Collect images of different wall tiles in different working backgrounds.

[0024] Step 1.2: Classify the real image data according to the wall brick background, wall background, building background and ambient light.

[0025] In the specific implementation, the collection of actual construction environment data, that is, the collection of real image data, includes image shooting, image processing, and image annotation. First, the image shooting work of the wall brick robot in different construction scenes is carried out, including but not limited to the texture characteristics, structural materials, engineering properties and other semantic information of the construction site. After screening and processing the collected semantic images, image labeling, scene classification and other work are carried out.

[0026] Step 2: Collect floor plans, elevations, and sections of the building.

[0027] Step 3: Build a virtual environment based on the floor plan, elevation and section of the building to generate virtual image data.

[0028] In view of the lack of existing complex background data sets, high collection costs, and the fact that the data distribution during collection is not necessarily similar to the actual building scene distribution, we conduct research on model generation of virtual reality building construction scenes based on complex backgrounds. We build virtual environments, generate, analyze and optimize virtual data, specifically:

[0029] Step 3.1: Based on the building plan, elevation and section, a virtual environment is built by combining architectural modeling technology with VR technology. The virtual simulation environment of architectural modeling solves the problems of multiple viewing angles, multiple lighting conditions and less complex background data during the construction process.

[0030] Step 3.1.1: Display the effects of various angles of the building in images based on the architectural floor plan, elevation and section drawings, form the architectural space relationship and reference objects, and design the three-dimensional model with the help of architectural modeling technology.

[0031] Step 3.1.2: Associate the geometric information and semantic information of the modeled 3D model with the VR virtual environment.

[0032] Step 3.1.3: Add interference factors such as lighting changes, texture features, and complex backgrounds to the VR virtual environment, create appropriate lighting conditions and building backgrounds, and present the virtual three-dimensional model more realistically.

[0033] In the above steps, the difference between real and virtual light is studied by setting the light close to the visual effect of the real construction site, and then the impact of multi-lighting condition scene data on target recognition and corner detection is studied. In view of the interference of complex background and texture features on target recognition, it is necessary to simulate and reproduce scenes of various complex backgrounds such as the sky, door frames, window bars, etc., so as to make the virtual data diverse.

[0034] Step 3.2: Generate virtual image data similar to the real data training effect based on computer vision rendering technology. The lighting conditions and texture of the virtual image data are consistent with those of the real image data. This avoids collecting data in complex on-site construction environments, reduces the difficulty of data collection, and diversified data is conducive to improving the robustness of the algorithm.

[0035] Step 3.3: Classify the virtual image data according to the wall brick background, wall background, building background and ambient light.

[0036] The main function of step 3 is to combine ray tracing with the building model to generate multi-angle virtual image data of the specified building model under various lighting environments based on the changes in light at different construction times, seasons, and floors, thereby making up for the difficulty in collecting complex background samples in actual environments.

[0037] Step 4: Perform data enhancement operations on the obtained virtual image data and the collected real image data. In this way, the real image data and virtual image data are expanded to improve the model effect in specific complex scenes. In this step, the problem of autonomous enhancement of texture differences between the region of interest and other regions in the image is studied for the situation where the foreground and background are not clearly distinguished in complex environments. An image enhancement algorithm based on morphological component analysis is proposed, specifically:

[0038] Step 4.1: View the image as a linear combination of components representing different texture properties.

[0039] Step 4.2: The image is decomposed into components with different texture characteristics by using wavelet nonlinear transformation that is sensitive to different texture characteristics as a dictionary for morphological component analysis.

[0040] Step 4.3: Use a nonlinear enhancement function that highlights the characteristics of different image components to enhance them, and recombine the enhanced image components to obtain an enhanced image with increased differences in different regions.

[0041] Step 5: Mix the real image dataset and virtual image dataset of the same category into a semantic segmentation image dataset and input it into the anchor-free fully convolutional network for wall brick recognition.

[0042] According to the actual construction needs of the intelligent tiling robot, a target recognition algorithm that can adapt to complex backgrounds is designed to provide accurate ROI areas for robot tiling positioning and provide a basis for accurate positioning of brick corners. Multi-target detection, single-target detection, and semantic segmentation are performed on the semantic segmentation image dataset.

[0043] The anchor-free fully convolutional network includes: a trunk part, a neck part and a head part; the trunk part includes three layers of convolutional networks connected in sequence, and the output of each layer of convolutional network is connected to an attention module. After the image passes through three layers of convolution and the corresponding attention module, multiple levels of feature maps are generated. The detector is applied to the multi-level feature maps to detect objects of different sizes in the image, which also helps to solve the problem of overlapping bounding boxes; an adaptive feature fusion module is used in the neck part to accurately detect small objects; an accelerated computing strategy is used in the head part of the network to further improve efficiency.

[0044] After the anchor-free fully convolutional network is designed, it is trained using semantic segmentation image datasets of different categories, and its target detection accuracy, detection speed, and AP are evaluated overall, with a focus on evaluating its detection performance in complex background environments. The model and optimization method with the best overall performance are selected to obtain the final model. This model can identify the wall tiles in the dataset images.

[0045] Step 6: Detect the corner points and their coordinates in the image, specifically:

[0046] Step 6.1: For the ROI area given by the wall tile recognition result in step 5, use the Harris corner detection algorithm to determine the corner point position and calculate the candidate corner point pixel coordinates, specifically:

[0047] Step 6.1.1: Convert the RGB color image to grayscale;

[0048] Step 6.1.2: Use the Sobel gradient operator to calculate the gradient in the u and v directions of each point in the grayscale image and

[0049] Step 6.1.3: and Perform Gaussian filtering on a local window;

[0050] Step 6.1.4: For each region w in the image, construct the following matrix:

[0051]

[0052] The response value A is calculated according to the following formula:

[0053] A=det(M)-α×trace(M) 2

[0054] Where Det(M) means finding the product of the eigenvalues ​​of the matrix M, and trace(M) means finding the sum of the eigenvalues ​​of the matrix M. The parameter α is 0.004 to 0.006, and the area where the response value A is greater than 0 is the corner point.

[0055] Step 6.2: Use the maximum suppression strategy to remove redundant corner points from the corner points obtained in step 6.1 to obtain the final corner point pixel coordinates, and test and verify the accuracy and positioning precision to select the best one to make the result more robust and accurate.

[0056] Step 6.3: After determining the corner point position, other ranging sensors provide the depth distance from the camera to the plane of the wall brick, so as to use the camera's internal and external parameters and coordinate system conversion method to solve the corner point position and obtain the corner point world coordinates, specifically:

[0057] Given the corner point coordinates u, v and their corresponding depth value depth, the world coordinates x, y, z of the point are obtained based on the camera parameters and the following formula:

[0058]

[0059]

[0060] Where: f x and f y is the focal length of the camera, c x and c y is the principal point of the camera, are the camera internal parameters, and d is the height of the camera from the ground.

[0061] Step 6.4: Accelerate the CPU device used in actual operations on the construction site to meet the real-time requirements of the operations.

[0062] Step 7: Use an image semantic segmentation algorithm that combines the depth image and the RGB image to perform semantic segmentation on the precise location image obtained in step 6. Specifically:

[0063] Step 7.1: Combine the features of the RGB image and depth image of the precise location image to obtain RGB-D data.

[0064] Step 7.2: The RGB-D data is input into the cross-modal guided encoder, the complementary information of the two modalities is recalibrated and fused through the SA-Gate unit, and then the fused multimodal features and specific modality features are propagated together through the bidirectional multi-step propagation module; then, this information is decoded by the segmentation decoding network to generate a segmentation map.

[0065] Step 7.3: The segmentation map is input into the deep convolutional neural network to extract the deep features of the image, which are then fused with the low-level features of the image based on the affine propagation clustering algorithm through a joint probability density model to calculate the image saliency.

[0066] Step 8: Accurately detect and locate the brick corners of the semantically segmented image.

[0067] The convolutional neural network method is used to detect brick corners. Specifically, a convolutional neural network containing 16 modules is used to accurately detect the brick corners of the image after semantic segmentation. Each module contains a convolution layer, a normalization layer and a pooling layer.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the concept of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A brick corner detection method for construction robots based on artificial intelligence, It is characterized in that include: Step 1: Collect wall brick images in different backgrounds, obtain real image data, and classify the real image data; Step 2: Collect the floor plan, elevation and section drawings of the building; Step 3: Build a virtual environment based on the floor plan, elevation and section of the building to generate virtual image data; Step 4: Perform data enhancement operations on the obtained virtual image data and the collected real image data; Step 5: Mix the real image dataset and the virtual image dataset of the same classification into a semantic segmentation image dataset, and input it into the anchor-free fully convolutional network for wall brick recognition; Step 6: Detect the corner points and their coordinates in the image; Step 7: Use an image semantic segmentation algorithm that combines the depth image and the RGB image to perform semantic segmentation on the precise location image obtained in step 6; Step 8: Accurately detect and locate the brick corners of the semantically segmented image.

2. The artificial intelligence-based brick angle detection method for a construction robot as claimed in claim 1, It is characterized in that The step 1 is specifically as follows: Step 1.1: Collect images of different wall tiles in different working backgrounds; Step 1.2: Classify the images according to wall brick background, wall background, building background and ambient light.

3. The artificial intelligence-based brick angle detection method for a construction robot as claimed in claim 1, It is characterized in that The step 3 is specifically as follows: Step 3.1: Build a virtual environment based on the building plan, elevation and section drawings by combining architectural modeling technology with VR technology; Step 3.1.1: Display the effects of the building from various angles in images based on the architectural plan, elevation and section, form the architectural space relationship and reference, and design the three-dimensional model with the help of architectural modeling technology; Step 3.1.2: Associating the geometric information and semantic information of the modeled 3D model with the VR virtual environment; Step 3.1.3: Add interference factors such as lighting changes, texture features, and complex backgrounds to the VR virtual environment, create appropriate lighting conditions and building backgrounds, and present the virtual 3D model more realistically; Step 3.2: Generate virtual image data similar to the real data training effect based on computer vision rendering technology, and the lighting conditions and texture of the virtual image data are consistent with the real image data; Step 3.3: Classify the virtual image data according to the wall brick background, wall background, building background and ambient light.

4. The method for detecting brick corners of a construction robot based on artificial intelligence as claimed in claim 1, It is characterized in that The step 4 is specifically as follows: Step 4.1: Consider the image as a linear combination of components representing different texture properties; Step 4.2: Using wavelet nonlinear transformation that is sensitive to different texture characteristics as a dictionary for morphological component analysis, the image is decomposed into components with different texture characteristics; Step 4.3: Use a nonlinear enhancement function that highlights the characteristics of different image components to enhance them, and recombine the enhanced image components to obtain an enhanced image with increased differences in different regions.

5. The method for detecting brick corners of a construction robot based on artificial intelligence as claimed in claim 1, It is characterized in that The anchor-free fully convolutional network in step 5 includes: a trunk part, a neck part and a head part; the trunk part includes three layers of convolutional networks connected in sequence, and the output of each layer of convolutional network is connected to an attention module. After the image passes through three layers of convolution and the corresponding attention module, multiple levels of feature maps are generated. The detector is applied to the multi-level feature maps to detect objects of different sizes in the image, which also helps to solve the problem of overlapping bounding boxes; an adaptive feature fusion module is used in the neck part to accurately detect small objects; an accelerated computing strategy is used in the head part of the network to further improve efficiency.

6. The artificial intelligence-based brick angle detection method for a construction robot as claimed in claim 1, It is characterized in that The step 6 is specifically as follows: Step 6.1: For the ROI area given by the wall tile recognition result in step 5, use the Harris corner detection algorithm to determine the corner point position and calculate the pixel coordinates of the candidate corner points; Step 6.2: Use the maximum suppression strategy to remove redundant corner points from the corner points obtained in step 6.1 to obtain the final corner point pixel coordinates, and test and verify the accuracy and positioning precision to select the best one to make the result more robust and accurate; Step 6.3: After determining the corner point position, other ranging sensors provide the depth distance from the camera to the plane of the wall brick, so as to use the camera's internal and external parameters and coordinate system conversion method to solve the corner point position and obtain the corner point world coordinates; Step 6.4: Accelerate the CPU device used in actual operations on the construction site to meet the real-time requirements of the operations.

7. The method for detecting brick corners of a construction robot based on artificial intelligence as claimed in claim 1, It is characterized in that The step 6.1 is specifically as follows: Step 6.1.1: Convert the RGB color image to grayscale; Step 6.1.2: Use the Sobel gradient operator to calculate the gradient in the u and v directions of each point in the grayscale image and Step 6.1.3: and Perform Gaussian filtering on a local window; Step 6.1.4: For each region w in the image, construct the following matrix: The response value A is calculated according to the following formula: A=det(M)-α×trace(M) 2 Where Det(M) means finding the product of the eigenvalues ​​of the matrix M, and trace(M) means finding the sum of the eigenvalues ​​of the matrix M. The parameter α is 0.004 to 0.006, and the area where the response value A is greater than 0 is the corner point.

8. The artificial intelligence-based brick angle detection method for a construction robot as claimed in claim 1, It is characterized in that The step 6.3 is specifically as follows: Given the corner point coordinates u, v and their corresponding depth value depth, the world coordinates x, y, z of the point are obtained based on the camera parameters and the following formula: Where: f x and f y is the focal length of the camera, c x and c y is the principal point of the camera, are the camera internal parameters, and d is the height of the camera from the ground.

9. The method for detecting brick corners of a construction robot based on artificial intelligence as claimed in claim 1, It is characterized in that The step 7 is specifically as follows: Step 7.1: Combine the features of the RGB image and the depth image of the precise location image to obtain RGB-D data; Step 7.2: The RGB-D data is input into the cross-modal guided encoder, the complementary information of the two modalities is recalibrated and fused through the SA-Gate unit, and then the fused multi-modal features and specific modality features are propagated together through the bidirectional multi-step propagation module; then, this information is decoded by the segmentation decoding network to generate a segmentation map; Step 7.3: The segmentation map is input into the deep convolutional neural network to extract the deep features of the image, which are then fused with the low-level features of the image based on the affine propagation clustering algorithm through a joint probability density model to calculate the image saliency.

10. The method for detecting brick corners of a construction robot based on artificial intelligence as claimed in claim 1, It is characterized in that In step 8, a convolutional neural network including 16 modules is used to accurately detect the brick corners of the image after semantic segmentation, and each module includes a convolution layer, a normalization layer and a pooling layer.

Images