Curtain wall frame recognition method and system based on multi-scale boundary feature fusion

Abstract

The present application relates to a kind of curtain wall frame identification method and system based on multiscale boundary feature fusion, belong to building curtain wall construction technical field.The method specifically includes: S1: acquisition curtain wall frame color image data;S2: data is preprocessed, and using data labeling software is pixel by pixel labeled to curtain wall frame area;S3: construct curtain wall frame identification network model, including multiscale boundary feature fusion module and channel guide pyramid convolution module;And using cosine annealing algorithm training optimization model;S4: the front end camera of curtain wall installation robot and laser radar are calibrated, and laser point cloud is registered with image;The hardware measurement system of laser radar is combined with the curtain wall frame identification network model trained to measure the three-dimensional space position and attitude of target curtain wall frame.The present application can realize the efficient and accurate identification of curtain wall frame, and real-time monitoring is realized in terminal, also provides fast and accurate installation parameters for curtain wall construction.

Description

Technical Field

[0001] This invention belongs to the field of building curtain wall construction technology, and relates to a curtain wall frame recognition method and system based on multi-scale boundary feature fusion. Background Technology

[0002] In the field of prefabricated buildings, curtain walls, as exterior decorative elements, are widely used. However, curtain wall installation largely relies on manual labor, which is not only labor-intensive but also extremely dangerous. Therefore, mechanized and intelligent curtain wall installation robots are indispensable in curtain wall installation projects, improving both construction efficiency and quality while enhancing worker safety.

[0003] Currently, most building curtain wall installations still rely on manual labor. A single glass curtain wall panel can weigh hundreds of kilograms, requiring multiple people to work together for installation. One person is responsible for moving the curtain wall, while another needs to observe from multiple angles for alignment guidance. The entire installation process is time-consuming, labor-intensive, and poses certain safety hazards. While glass handling vehicles on the market replace manual handling, they lack automatic target recognition and positioning, and are devoid of information perception capabilities. For curtain wall installation tasks, rapid and accurate identification of the curtain wall frame is a key prerequisite for achieving automated curtain wall robots. Simultaneously, the requirement for lightweight algorithms must be considered to facilitate robot hardware deployment. Therefore, this invention enables accurate identification of the curtain wall frame. The network employs a lightweight design for easy deployment on embedded devices, improving engineering efficiency in the curtain wall installation process and significantly contributing to the intelligentization of curtain wall installation. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a curtain wall frame recognition method and system based on multi-scale boundary feature fusion. By using deep learning, the curtain wall frame can be identified quickly and accurately, thereby improving the intelligence of curtain wall panel installation.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Option 1:

[0007] A curtain wall frame recognition method based on multi-scale boundary feature fusion specifically includes the following steps:

[0008] S1: Use a high-definition camera to capture color image data of the curtain wall frame;

[0009] S2: Preprocess the acquired image data and construct a dataset, including a training set, a validation set, and a test set, and use data annotation software to annotate the curtain wall frame area pixel by pixel;

[0010] S3: Construct a curtain wall frame recognition network model based on multi-scale boundary feature fusion. This network model includes a multi-scale boundary feature fusion module (MBA) and a channel-guided pyramid convolution module (CGPC). It achieves efficient curtain wall frame recognition with a lightweight network architecture. The output of the multi-scale boundary feature fusion module is the input of the channel-guided pyramid convolution module. The cosine annealing algorithm is used to train the curtain wall frame recognition network model based on multi-scale boundary feature fusion to obtain the optimal parameters.

[0011] S4: The front-end camera of the curtain wall installation robot is calibrated, and the laser point cloud is registered with the image. A hardware measurement system using LiDAR is combined with a trained curtain wall frame recognition network model to measure the three-dimensional spatial position and orientation of the target curtain wall frame.

[0012] Furthermore, in step S1, the collected color image data of the curtain wall frame includes: multiple curtain wall frames, low-contrast curtain wall frames, and curtain wall frames against a noisy and cluttered background, wherein the interference from the cluttered background comes from distant buildings, railings, columns, etc.

[0013] Furthermore, in step S2, the collected image data is preprocessed, specifically including: filtering out blurry image pairs, and then expanding the data volume through three methods: translation transformation, flip transformation, and random cropping to construct a curtain wall frame image dataset; dividing the dataset into training set, validation set, and test set; then using the data labeling software labelme to manually determine the curtain wall frame region; and finally converting the labels into binary images, where white represents the target curtain wall frame and black represents the background.

[0014] Furthermore, in step S3, the multi-scale boundary feature fusion module (MBA) is used to fuse the boundary features of the curtain wall frame at different scales to improve the segmentation performance of the network. It includes: boundary awareness module (BA), inverted residual module (IRB) and average pooling layer.

[0015] The multi-scale boundary feature fusion module (MBA) specifically includes: combining the feature F0 extracted by convolution with features F1, F2, F3, and F4 respectively, inputting them into the boundary awareness module (BA), concatenating the four results into channels and inputting them into the inverse residual module (IRB), and then dividing them into five paths. The first path multiplies the feature F0 pixel by pixel and then adds it to F0 to output B0; the second path uses an average pooling layer and multiplies the feature F1 pixel by pixel and then adds it to F1 to output B1; the third path uses an average pooling layer and multiplies the feature F2 pixel by pixel and then adds it to F2 to output B2; the fourth path uses an average pooling layer and multiplies the feature F3 pixel by pixel and then adds it to F3 to output B3; the fifth path uses an average pooling layer and multiplies the feature F4 pixel by pixel and then adds it to F4 to output B4. Among them, features F0, F1, F2, F3, and F4 are low-level detail information and high-level semantic information extracted from the data after preprocessing in step S2, respectively.

[0016] Further, in step S3, the boundary awareness module (BA) is used to output a weight map, specifically including: combining the feature F0 extracted by convolution with features F1, F2, F3, and F4 respectively; inputting F0 and Fi into two 1×1×1 three-dimensional convolutions respectively, where i = 1, 2, 3, 4; after the Fi convolution operation, two paths are derived. The first path uses a 3×7×7 three-dimensional convolution operation, maps it with the Sigmoid activation function, then upsamples the feature by a factor of 2, and multiplies it pixel by pixel with the result of the feature F0 convolution operation; the second path upsamples the feature by a factor of 2 and subtracts it pixel by pixel from the result of the first path; the absolute value operation is performed on the result of the pixel-by-pixel subtraction, then a 3×7×7 three-dimensional convolution operation is performed, and then mapped with the Sigmoid activation function, finally outputting the boundary awareness weight map;

[0017] The inverse residual module (IRB) is used to reduce the number of parameters and retain more feature information. The total number of network parameters is 2.6M. Specifically, it includes: first, using 1×1 convolution to expand the channels and using the ReLU activation function; then using 3×3 depth-wise convolution to extract features and using the ReLU activation function; and finally using 1×1 convolution to compress the number of channels back to the original number of channels and using the Linear activation function.

[0018] Furthermore, in step S3, the channel-guided pyramid convolution module (CGPC) is used to more fully fuse the curtain wall frame features, including: inverted residual module (IRB), channel attention module, convolution, channel stitching and pixel-by-pixel addition, etc.; the channel attention module adopts efficient channel attention mechanism (ECA) to locate the curtain wall frame area and suppress background information.

[0019] The channel-guided pyramid convolution module specifically includes: first, performing a 1×1 convolution operation on the input and then splitting it into two paths. The first path uses a channel attention mechanism (ECA), which is divided into four branches. Pixel-by-pixel addition is performed, and the convolution and dilation rates are set. Then, the four branches are concatenated and input into a 1×1 convolution, which is then added pixel-by-pixel to the first path. Subsequently, each layer takes all the previous feature maps as input, using convolution and inverse residual modules to change the feature map size and reduce the number of parameters.

[0020] Furthermore, in step S3, a cosine annealing algorithm is used to train the curtain wall frame recognition network model based on multi-scale boundary feature fusion. Specifically, this includes setting the initial learning rate to 0.002, using a stochastic gradient descent (SGD) optimizer for training, and adjusting the learning rate using the cosine annealing algorithm to make the training process more stable and efficient. Simultaneously, because the curtain wall frame images suffer from sample imbalance (with negative samples (background) far outnumbering positive samples (curtain wall frames), the focal loss function is used as the loss function during training.

[0021] The cosine annealing principle is expressed as follows:

[0022]

[0023] Where, α t Indicates the current learning rate; i represents the i-th index value; and T represents the minimum and maximum values ​​of the learning rate, respectively; t This indicates the current training cycle number; epoch indicates the total number of training cycles; π represents the mathematical constant pi.

[0024] The focus loss function is:

[0025]

[0026] Where Loss is the loss value of the focus loss function, N represents the total number of pixels in a single curtain wall frame image, and y i For binary classification labels 0 or 1, y i 'Is the output belonging to y i The probability of a label; hyperparameter α is used to control class imbalance, and hyperparameter γ is used to control the difficulty of samples.

[0027] Furthermore, in step S4, the calibration of the front-end camera and LiDAR of the curtain wall installation robot specifically includes: calibrating the camera's intrinsic parameters using the Zhang Zhengyou calibration method, and then using the P3P (Perspective-3-Point) algorithm to jointly calibrate the camera and LiDAR, transforming the point cloud data from the LiDAR coordinate system to the camera coordinate system. The images acquired by the camera are processed and locally adaptively thresholded to solve the problem of uneven brightness in different areas of the image, and the Harris corner detection algorithm is used to detect corners to improve the calibration accuracy of the camera's intrinsic parameters.

[0028] The hardware measurement system using LiDAR is combined with a trained curtain wall frame recognition network model. Specifically, this includes: mapping the recognition results to the LiDAR point cloud; using a high-quality corner detection algorithm to detect the corners of the curtain wall frame from the recognition results and LiDAR data; outputting the coordinate information of the four corners; establishing corresponding geometric relationships based on imaging principles and the collinearity between the curtain wall frame's position in the image coordinate system and the world coordinate system and the camera's center point; and solving for their projections, thereby registering the LiDAR point cloud and the camera image, integrating spatial information into the curtain wall frame recognition results. When outputting the curtain wall frame recognition results, the spatial position information of the curtain wall frame is also output, obtaining the distances from the camera and LiDAR to the center point of the curtain wall frame and their angles to the four corners of the curtain wall frame. This allows for pose estimation to measure the three-dimensional spatial position and orientation of the target curtain wall frame, providing rapid and accurate installation parameters for curtain wall construction.

[0029] Option 2:

[0030] A curtain wall frame recognition system based on multi-scale boundary feature fusion, characterized in that the system includes an image acquisition module, a laser ranging module, an AI embedded development board, a wireless communication module, a power supply module, a display module, and a cloud server;

[0031] The image acquisition module uses a camera to acquire images of the curtain wall frame. The acquired images are then input into a cloud server for network training, into an AI embedded development board for real-time segmentation, and finally into a display device for visualization.

[0032] The laser ranging module uses a lidar to acquire the spatial position information of the curtain wall frame, and inputs the acquired spatial position information of the curtain wall frame into the AI ​​embedded development board.

[0033] The AI ​​embedded development board deploys the curtain wall frame recognition network model based on multi-scale boundary feature fusion, which was built and trained in Scheme 1, to the AI ​​embedded development board for real-time curtain wall frame segmentation and matching LiDAR data to estimate the distance and pose of the curtain wall frame.

[0034] The wireless communication module is used to enable communication between the AI ​​embedded development board and the cloud server;

[0035] The power module is used to supply power to the AI ​​embedded development board, image acquisition module, laser ranging module, and display module.

[0036] The display module establishes communication with the AI ​​embedded development board, monitors the front-end curtain wall frame recognition in real time, and combines the recognition results with the laser point cloud to output the spatial information of the curtain wall frame.

[0037] The cloud server is used to receive the collected curtain wall frame images, train a curtain wall frame recognition network model based on multi-scale boundary feature fusion, and deploy the optimal model to the AI ​​embedded development board after obtaining the optimal parameters.

[0038] The beneficial effects of this invention are as follows: This invention designs an efficient curtain wall frame recognition method based on multi-scale boundary feature fusion, which adopts a multi-scale boundary feature fusion module and a channel-guided pyramid convolution module to fully integrate the boundary features of the curtain wall frame, enabling accurate recognition of the curtain wall frame; at the same time, this invention designs an efficient curtain wall frame recognition system based on multi-scale boundary feature fusion, which can deploy the network model on an AI embedded development board to achieve real-time and efficient curtain wall frame recognition, and the curtain wall frame recognition effect can be monitored in real time on the display module.

[0039] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0041] Figure 1 This is a schematic diagram illustrating the implementation process of the efficient curtain wall frame recognition method based on multi-scale boundary feature fusion of the present invention.

[0042] Figure 2 This is a diagram of the efficient curtain wall frame recognition network structure based on multi-scale boundary feature fusion in this invention.

[0043] Figure 3 This is a structural diagram of the multi-scale boundary feature fusion module of the present invention;

[0044] Figure 4 This is a structural diagram of the channel-guided pyramid convolution module of the present invention;

[0045] Figure 5 This is a structural diagram of the inverted residual module of the present invention;

[0046] Figure 6 This is a framework diagram of the curtain wall frame high-efficiency recognition system based on multi-scale boundary feature fusion of the present invention. Detailed Implementation

[0047] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0048] Example 1:

[0049] Please see Figures 1-6 This embodiment provides a curtain wall frame recognition method based on multi-scale boundary feature fusion, such as... Figure 1 As shown, this method, used for precise positioning of the curtain wall frame and achieving real-time and efficient segmentation of the curtain wall frame, specifically includes the following steps:

[0050] Step 1: Use a high-definition camera to capture color image data of the curtain wall frame. Curtain wall frame types include: multiple curtain wall frames under different weather conditions, low-contrast curtain wall frames, and curtain wall frames against noisy and cluttered backgrounds, where the cluttered background interference comes from distant buildings, railings, columns, etc.

[0051] Step 2: Preprocess the acquired curtain wall frame image data. Preprocessing methods include: filtering out blurry image pairs, and then expanding the data volume through translation transformation, flip transformation, and random cropping.

[0052] Step 3: Construct a curtain wall frame image dataset, dividing the dataset into a training set, a validation set, and a test set;

[0053] Step 4: Use data annotation software to annotate the curtain wall frame area pixel by pixel. The data annotation software used is Labelme. Manually determine the curtain wall frame area, then convert the labels into a binary image, where white represents the target curtain wall frame and black represents the background.

[0054] Step 5: Construct a high-efficiency curtain wall frame recognition network architecture based on multi-scale boundary feature fusion, such as... Figure 2As shown, the network comprises a boundary awareness module (BA), a multi-scale boundary feature fusion module (MBA), an inverted residual module (IRB), a channel-guided pyramid convolutional module (CGPC), and average pooling layers. The boundary awareness module uses 3D convolution to obtain a larger receptive field and extract curtain wall frame features. The multi-scale boundary feature fusion module uses the boundary awareness module, inverted residual blocks, and average pooling layers to fuse multi-scale curtain wall frame boundary features, improving recognition accuracy. The channel-guided pyramid convolutional module uses an efficient channel attention module (ECA) to enhance the representational power of curtain wall frame features, suppress background features, improve recognition performance, and reduce the number of network parameters. Different convolutional dilation rates are set to extract features at different levels; a smaller dilation rate captures detailed information of the curtain wall frame, while a larger dilation rate captures global features. Skip connections are also included to enhance the fusion effect of curtain wall frame features.

[0055] like Figure 3 The diagram shows the Multi-Scale Boundary Feature Fusion (MBA) module. It needs to fuse the boundary features of the curtain wall frame at different scales. Specifically, this involves: combining the feature F0 extracted by convolution with F1, F2, F3, and F4 respectively, inputting them into the Boundary Awareness (BA) module; concatenating the four results into channels and inputting them into the Inverse Residual (IRB) module; then splitting the result into five paths. The first path multiplies the feature F0 pixel-by-pixel and adds it back to F0 to output B0; the second path uses an average pooling layer, multiplies the feature F1 pixel-by-pixel, and adds it back to F1 to output B1; the third path uses an average pooling layer, multiplies the feature F2 pixel-by-pixel, and adds it back to F2 to output B2; the fourth path uses an average pooling layer, multiplies the feature F3 pixel-by-pixel, and adds it back to F3 to output B3; and the fifth path uses an average pooling layer, multiplies the feature F4 pixel-by-pixel, and adds it back to F4 to output B4. The boundary-aware module is used to output the weight map. Specifically, it includes: combining the feature F0 extracted from the convolution with F1, F2, F3, and F4 respectively; inputting F0 and Fi into two 1×1×1 convolutions respectively; after the Fi convolution operation, two paths are derived. The first path uses a 3×7×7 convolution operation, then maps it with the Sigmoid activation function, and multiplies it pixel by pixel with the result of the feature F0 convolution operation; the second path subtracts the result of the first path pixel by pixel; the absolute value operation is performed on the result of the pixel-by-pixel subtraction, and then a 3×7×7 convolution operation is performed, followed by mapping with the Sigmoid activation function, finally outputting the boundary-aware weight map.

[0056] like Figure 4The diagram shows a Channel Guided Pyramid Convolutional Module (CGPC). To more fully integrate the curtain wall frame features, operations include convolution, attention mechanisms, channel concatenation, pixel-wise addition, and inverse residual (IRB). First, the input undergoes a 1×1 convolution and is split into two paths. The first path uses a channel attention mechanism, divided into four branches, performing pixel-wise addition and setting the convolution and dilated convolution dilation rates. Then, the four branches are concatenated and fed into a 1×1 convolution, adding pixel-wise to the first path. Subsequent layers use all the previous feature maps as input, employing convolution and inverse residual to change the feature map size and reduce the number of parameters.

[0057] like Figure 5 The diagram shows the inverse residual module (IRB). It reduces the number of parameters and preserves more feature information. The total network parameter count is 2.6M. Specifically, it includes: first, using 1×1 convolutions for channel expansion with ReLU activation; then using 3×3 depth-wise convolutions for feature extraction with ReLU activation; and finally, using 1×1 convolutions to compress the number of channels back to the original number with Linear activation.

[0058] Step 6: Select the focus loss function as the loss function during training, which is expressed as:

[0059]

[0060] Where Loss is the loss value of the focus loss function, N represents the total number of pixels in a single curtain wall frame image, and y i For binary classification labels 0 or 1, y i 'Is the output belonging to y i The probability of a label; hyperparameter α is used to control class imbalance, and hyperparameter γ is used to control the difficulty of samples.

[0061] Step 7: Train a high-efficiency curtain wall frame recognition network model using the cosine annealing algorithm with multi-scale boundary feature fusion. Specifically, this includes setting the initial learning rate to 0.002, using the SGD optimizer for training, and adjusting the learning rate using the cosine annealing algorithm to make the training process more stable and efficient. The principle of cosine annealing is expressed as follows:

[0062]

[0063] Where, α t Indicates the current learning rate; i represents the i-th index value; and T represents the minimum and maximum values ​​of the learning rate, respectively; t This indicates the current training cycle number; epoch indicates the total number of training cycles, set to a fixed value of 200; π represents the mathematical constant pi.

[0064] Step 8: The camera intrinsic parameters are calibrated using Zhang Zhengyou's calibration method. Then, the P3P (Perspective-3-Point) algorithm is used for joint calibration of the camera and LiDAR, transforming the point cloud data from the LiDAR coordinate system to the camera coordinate system. The images acquired by the camera are processed and local adaptive thresholding is performed to address the problem of uneven brightness in different areas of the image. The Harris corner detection algorithm is then used to detect corners, improving the calibration accuracy of the camera intrinsic parameters. The resulting camera intrinsic parameters can be expressed as:

[0065]

[0066] Where f represents the camera's focal length, and (u0, v0) represents the origin of the image, thus enabling the transformation between the pixel coordinate system and the world coordinate system:

[0067]

[0068] Where R represents the camera rotation matrix, T represents the camera translation vector, (u,v) represents the pixel coordinate system, and (X) represents the pixel coordinate system. c ,Y c Z c () represents the camera coordinate system, (X) W ,Y W Z W ) represents the world coordinate system.

[0069] Step 9: Use CloudCompare software to select the laser point cloud area of ​​the curtain wall frame to obtain the point cloud of the specified curtain wall frame area, facilitating the identification of curtain wall frame corner points. Use the Harris 3D corner detection algorithm to extract the curtain wall frame corner points from the laser point cloud data, outputting the XYZ spatial coordinates of the four corner points. Use the Harris 2D corner detection algorithm to extract the curtain wall frame corner points from the curtain wall frame recognition results, outputting the coordinates of the four corner points. Based on the imaging principle and the collinearity between the position of the curtain wall frame in the image coordinate system and the world coordinate system and the position of the camera center point, establish the corresponding geometric relationship and solve the projection between them, mapping the recognition result to the laser point cloud, thus completing the registration of the laser point cloud and the recognition image, and integrating spatial information into the curtain wall frame recognition result. The projection can be expressed by the formula:

[0070]

[0071]

[0072] Where (x,y) are the image coordinates of the curtain wall frame recognition result, (X... FW ,Y FW Z FW(X) represents the world coordinates of the corner point of the curtain wall frame. CW ,Y CW Z CW ) represents the world coordinates of the camera center, r ij (i = 1, 2, 3; j = 1, 2, 3) are the rotation parameters for image coordinates and world coordinates, which can be expressed as:

[0073]

[0074] in, ω and θ are attitude angles in the world coordinate system. The projection relationship between the laser point cloud and the image is established. When outputting the curtain wall frame recognition results, the spatial position information of the curtain wall frame is also output, obtaining the distances of the camera and lidar from the center point of the curtain wall frame and the angles with the four corners of the curtain wall frame. Thus, pose estimation is used to measure the three-dimensional spatial position and attitude of the target curtain wall frame, providing fast and accurate installation parameters for curtain wall construction.

[0075] Example 2:

[0076] Please see Figure 6 The present invention also provides a curtain wall frame recognition system based on multi-scale boundary feature fusion, which is used to accurately locate the curtain wall frame and realize real-time and efficient segmentation of the curtain wall frame. The system includes an image acquisition module, a laser ranging module, an AI embedded development board, a wireless communication module, a power supply module, a display module and a cloud server.

[0077] The image acquisition module uses a camera to acquire images of the curtain wall frame. The acquired images are then input into a cloud server for network training, into an AI embedded development board for real-time segmentation, and into a display device for visualization.

[0078] The laser ranging module uses a lidar to acquire the spatial position information of the curtain wall frame, and inputs the acquired spatial position information of the curtain wall frame into the AI ​​embedded development board.

[0079] The AI ​​embedded development board deploys the curtain wall frame recognition network model based on multi-scale boundary feature fusion constructed in Example 1 to the AI ​​embedded development board for real-time curtain wall frame segmentation and matching LiDAR data to estimate the distance and pose of the curtain wall frame.

[0080] The wireless communication module is used to enable communication between the AI ​​embedded development board and the cloud server;

[0081] The power module is used to supply power to the AI ​​embedded development board, image acquisition module, laser ranging module, and display module.

[0082] The display module establishes communication with the AI ​​embedded development board, monitors the front-end curtain wall frame recognition in real time, and combines the recognition results with the laser point cloud to output the spatial information of the curtain wall frame.

[0083] The cloud server is used to receive the collected curtain wall frame images, train a curtain wall frame recognition network model based on multi-scale boundary feature fusion, and deploy the optimal model to the AI ​​embedded development board after obtaining the optimal parameters.

[0084] The identification method of this system specifically includes the following steps:

[0085] Step 1: Train a high-efficiency curtain wall frame recognition network model based on multi-scale boundary feature fusion on a cloud server. Deploy the trained network model onto an NVIDIA Jetson series development board for real-time curtain wall frame segmentation. Install CUDA and cuDNN on the NVIDIA Jetson series development board and configure the PyTorch deep learning framework to improve the model's inference performance. Run the application by connecting to the NVIDIA Jetson series development board. The power module supplies power to the NVIDIA Jetson series development board, camera, LiDAR, and display.

[0086] Step 2: Add an optical filter to the industrial camera connected to the NVIDIA Jetson series development board. Since the curtain wall frame captured by the camera is easily affected by light, resulting in unclear images and hindering frame identification, placing an optical filter in front of the camera filters out visible light of specific wavelengths and incoherent directions, reducing light interference and improving image quality.

[0087] Step 3: Use an industrial camera to capture images of the curtain wall frame in real time.

[0088] Step 4: Use LiDAR to collect laser point cloud data of the curtain wall frame.

[0089] Step 5: Use a wireless communication module to enable communication between the NVIDIA Jetson series development board and the cloud server. Download the pre-trained optimal model from the cloud server and deploy it to the development board.

[0090] Step 6: Use the trained optimal model on the NVIDIA Jetson series development board to perform real-time and efficient recognition of the curtain wall frame. Register the model with the curtain wall frame's laser point cloud data to integrate spatial information into the recognition results. Output the distances of the camera and LiDAR from the center point of the curtain wall frame, as well as the angles with the four corners of the curtain wall frame. This allows for pose estimation to measure the three-dimensional spatial position and orientation of the target curtain wall frame.

[0091] Step 7: Display the curtain wall frame recognition results on the display module in real time.

[0092] This invention enables efficient and accurate identification of curtain wall frames and allows for real-time monitoring at the terminal, achieving efficient curtain wall frame identification with a lightweight network architecture.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A curtain wall frame recognition method based on multi-scale boundary feature fusion, characterized in that, The method specifically includes the following steps: S1: Use a high-definition camera to capture color image data of the curtain wall frame; S2: Preprocess the acquired image data and construct a dataset, including a training set, a validation set, and a test set, and use data annotation software to annotate the curtain wall frame area pixel by pixel; S3: Construct a curtain wall frame recognition network model based on multi-scale boundary feature fusion. This network model includes a multi-scale boundary feature fusion module and a channel-guided pyramid convolution module. The output of the multi-scale boundary feature fusion module is the input of the channel-guided pyramid convolution module. The cosine annealing algorithm is used to train the curtain wall frame recognition network model based on multi-scale boundary feature fusion to obtain the optimal parameters. The multi-scale boundary feature fusion module is used to fuse the boundary features of curtain wall frames at different scales to improve the segmentation performance of the network. It includes a boundary awareness module, an inverse residual module, and an average pooling layer. Specifically, the multi-scale boundary feature fusion module combines the feature F0 extracted from the convolution with features F1, F2, F3, and F4 respectively, inputs them to the boundary awareness module, concatenates the four results into channels, and inputs them to the inverse residual module. Then, it is divided into five paths. The first path multiplies the feature F0 pixel-by-pixel and then adds it back to F0 to output B0; the second path uses... The average pooling layer is used to multiply the data with feature F1 pixel by pixel, and then the result is added back to F1 to output B1; the third path uses an average pooling layer to multiply the data with feature F2 pixel by pixel, and then the result is added back to F2 to output B2; the fourth path uses an average pooling layer to multiply the data with feature F3 pixel by pixel, and then the result is added back to F3 to output B3; the fifth path uses an average pooling layer to multiply the data with feature F4 pixel by pixel, and then the result is added back to F4 to output B4; where features F0, F1, F2, F3, and F4 are the low-level detail information and high-level semantic information extracted from the data after preprocessing in step S2, respectively; The boundary awareness module is used to output a weight map, specifically including: combining the feature F0 extracted by convolution with features F1, F2, F3, and F4 respectively, and F0 and F... i The inputs are respectively fed into two 1×1×1 3D convolutions, where, i =1,2,3,4; in F i After the convolution operation, two paths are derived. The first path uses a 3×7×7 three-dimensional convolution operation, maps it with the Sigmoid activation function, then upsamples the features by a factor of 2, and multiplies the result of the feature F0 convolution operation pixel by pixel. The second path upsamples the features by a factor of 2 and subtracts the result of the first path pixel by pixel. The absolute value of the result after pixel-by-pixel subtraction is then performed, followed by a 3×7×7 three-dimensional convolution operation, and then mapping it with the Sigmoid activation function, finally outputting a boundary-aware weight map. The inverse residual module is used to reduce the number of parameters. Specifically, it includes: first, using a 1×1 convolution to expand the channels and using the ReLU activation function; then, using a 3×3 depth-wise convolution to extract features and using the ReLU activation function; and finally, using a 1×1 convolution to compress the number of channels back to the original number of channels and using the Linear activation function. The channel-guided pyramid convolution module is used to more fully fuse the curtain wall frame features, including: inverted residual module, channel attention module, convolution, channel stitching and pixel-by-pixel addition; the channel attention module adopts an efficient channel attention mechanism to locate the curtain wall frame area and suppress background information. The channel-guided pyramid convolution module specifically includes: first, performing a 1×1 convolution operation on the input and then splitting it into two paths. The first path uses a channel attention mechanism, which is divided into four branches. Pixel-by-pixel addition is performed, and the convolution and dilation rates are set. Then, the four branches are concatenated and input into a 1×1 convolution, which is added pixel-by-pixel to the first path. Subsequently, each layer takes all the previous feature maps as input, where convolution and inverse residual modules are used to change the feature map size and reduce the number of parameters. S4: The front-end camera and lidar of the curtain wall installation robot are calibrated, and the lidar point cloud is registered with the image; the lidar hardware measurement system is combined with the trained curtain wall frame recognition network model to measure the three-dimensional spatial position and orientation of the target curtain wall frame.

2. The curtain wall frame identification method according to claim 1, characterized in that, In step S1, the collected color image data of the curtain wall frame includes: multiple curtain wall frames, low-contrast curtain wall frames, and curtain wall frames against a noisy and cluttered background.

3. The curtain wall frame identification method according to claim 1, characterized in that, In step S2, the collected image data is preprocessed, specifically including: filtering out blurry images, and then expanding the data volume through three methods: translation transformation, flip transformation, and random cropping to construct a curtain wall frame image dataset; dividing the dataset into training set, validation set, and test set; then using data annotation software to manually determine the curtain wall frame area; and finally converting the labels into binary images, where white represents the target curtain wall frame and black represents the background.

4. The curtain wall frame identification method according to claim 1, characterized in that, In step S3, the cosine annealing algorithm is used to train the curtain wall frame recognition network model based on multi-scale boundary feature fusion. Specifically, this includes: setting the initial learning rate, using the stochastic gradient descent optimizer for training, and adjusting the learning rate by combining the cosine annealing algorithm to make the training process more stable and efficient. At the same time, because the curtain wall frame image has a sample imbalance problem, with negative samples (background) far outnumbering positive samples (curtain wall frames), the focus loss function is used as the loss function in the training process. The cosine annealing principle is expressed as follows: in, Indicates the current learning rate; Indicates the first Secondary index value; and These represent the minimum and maximum values ​​of the learning rate, respectively. Indicates the current training cycle number; This indicates the total number of training cycles; Represents pi; The focus loss function is: ， in, The loss value is the focus loss function. This represents the total number of pixels in a single curtain wall frame image. The labels are binary, 0 or 1. The output belongs to Label probability, hyperparameter Hyperparameters used to control class imbalance Used to control the difficulty level of samples.

5. The curtain wall frame identification method according to claim 1, characterized in that, In step S4, the calibration of the front-end camera and LiDAR of the curtain wall installation robot specifically includes: calibrating the camera intrinsic parameters using the Zhang Zhengyou calibration method, and then using the P3P algorithm to jointly calibrate the camera and LiDAR, transforming the point cloud data from the LiDAR coordinate system to the camera coordinate system; organizing the images acquired by the camera and performing local adaptive thresholding to solve the problem of uneven brightness in different areas of the image, and using the Harris corner detection algorithm to detect corners to improve the calibration accuracy of the camera intrinsic parameters; after the camera and LiDAR are calibrated, images are acquired in real time and the trained curtain wall frame recognition network model is used to recognize the curtain wall frame to obtain the recognition result of the curtain wall frame.

6. The curtain wall frame identification method according to claim 1, characterized in that, In step S4, the hardware measurement system using LiDAR is combined with the trained curtain wall frame recognition network model. Specifically, this includes: mapping the recognition results to the LiDAR point cloud; using a high-quality corner detection algorithm to detect the corners of the curtain wall frame from the recognition results and LiDAR; outputting the coordinate information of the four corners; establishing corresponding geometric relationships and solving the projection between them based on the imaging principle and the collinearity between the position of the curtain wall frame in the image coordinate system and the world coordinate system and the position of the camera center point, thereby registering the LiDAR point cloud and the camera image, so that the curtain wall frame recognition results incorporate spatial information; when outputting the curtain wall frame recognition results, the spatial position information of the curtain wall frame is output, obtaining the distances of the camera and LiDAR from the center point of the curtain wall frame and the angles with the four corners of the curtain wall frame, thereby measuring the three-dimensional spatial position and orientation of the target curtain wall frame through pose estimation, providing fast and accurate installation parameters for curtain wall construction.

7. A curtain wall frame recognition system based on multi-scale boundary feature fusion, characterized in that, The system includes an image acquisition module, a laser ranging module, an AI embedded development board, a wireless communication module, a power supply module, a display module, and a cloud server; The image acquisition module uses a camera to acquire images of the curtain wall frame. The acquired images are then input into a cloud server for network training, into an AI embedded development board for real-time segmentation, and into a display device for visualization. The laser ranging module uses a lidar to acquire the spatial position information of the curtain wall frame, and inputs the acquired spatial position information of the curtain wall frame into the AI ​​embedded development board. The AI ​​embedded development board deploys the curtain wall frame recognition method according to any one of claims 1 to 6 to the AI ​​embedded development board for real-time curtain wall frame segmentation and matching LiDAR data to estimate the distance and attitude of the curtain wall frame. The wireless communication module is used to enable communication between the AI ​​embedded development board and the cloud server; The power module is used to supply power to the AI ​​embedded development board, image acquisition module, laser ranging module, and display module. The display module establishes communication with the AI ​​embedded development board, monitors the front-end curtain wall frame recognition in real time, and combines the recognition results with the laser point cloud to output the spatial information of the curtain wall frame. The cloud server is used to receive the collected curtain wall frame images, train a curtain wall frame recognition network model based on multi-scale boundary feature fusion, and deploy the optimal model to the AI ​​embedded development board after obtaining the optimal parameters.

Images