Curtain wall frame real-time detection method and device based on border-aware cross-modal fusion network

Abstract

The present invention relates to a curtain wall frame real-time detection method and device based on a frame-aware cross-modal fusion network, and belongs to the technical field of building curtain wall construction. The method includes: collecting curtain wall frame RGB images and depth maps; preprocessing the collected curtain wall frame RGB images and depth maps and constructing and dividing curtain wall frame data sets; constructing a curtain wall frame detection model based on a frame-aware cross-modal fusion network, the encoder part uses ConvNeXt as a feature extraction network, and the decoder constructs FFA modules and CFF modules to achieve overall multi-scale multi-modal path aggregation; solving the binary cross entropy loss function, training the curtain wall frame detection model and saving the optimal model; real-time collection of curtain wall frame RGB images and depth maps that need to be detected and using the saved optimal model for real-time detection. The present invention can meet the actual curtain wall frame detection accuracy requirements while realizing intelligent detection, and can improve the efficiency of building construction.

Description

Technical Field

[0001] The present invention belongs to the technical field of building curtain wall construction, and relates to a curtain wall frame real-time detection method and device based on a frame perception cross-modal fusion network. Background Art

[0002] The most important thing for curtain wall installation technology is to efficiently and accurately locate and detect the curtain wall frame.

[0003] At present, there are few curtain wall frame detection technologies based on deep learning, and traditional curtain wall frame detection and positioning mainly rely on manual work. In addition, this method has low efficiency, cannot meet actual needs, and is not conducive to the development of curtain wall installation related technologies, so it has great limitations. Therefore, thinking and researching on curtain wall frame detection methods has certain practical value and progressive significance for improving the mechanization and intelligence level of building curtain wall construction, and improving construction efficiency and construction quality. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a curtain wall frame real-time detection method and device based on a frame-aware cross-modal fusion network, so as to achieve efficient, accurate and intelligent detection of building curtain walls.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] Solution 1: A curtain wall frame real-time detection method based on border-aware cross-modal fusion network, specifically including the following steps:

[0007] S1: Collect curtain wall frame RGB image and depth map;

[0008] S2: perform image preprocessing on the collected curtain wall frame RGB image and depth map, and construct and divide the curtain wall frame dataset;

[0009] S3: Construct a curtain wall frame detection model based on border-aware cross-modal fusion network;

[0010] S4: Solve the binary cross entropy loss function based on the predicted image and target image generated by the border-aware cross-modal fusion network, train and learn the optimal parameters of the curtain wall frame detection model, and save the trained optimal curtain wall frame detection model to the real-time detection platform;

[0011] S5: Collect the RGB image and depth map of the curtain wall frame to be detected in real time, and use the saved optimal curtain wall frame detection model to perform real-time detection.

[0012] Furthermore, in step S1, the collected curtain wall frame RGB image and depth map include: a variety of simple or even complex interference effects, such as single frame, multiple frames and entire frame images collected under different weather, different lighting, different distances, different angles, different obstacles, and with or without dust.

[0013] Further, step S2 specifically includes: using the GaussianBlur function in OpenCV to perform Gaussian filtering and denoising on the RGB image, using the warpPerspective function and the getPerspectiveTransform function to perform perspective transformation and correct the image, using logarithmic transformation to remap the depth information, using Mosaic data enhancement and rotating some curtain wall frame images by 90°, 180°, and 270°, as well as translating and flipping in the horizontal and vertical directions, solving the problem of fewer images in certain scenes in the data set to expand the image data volume, using the labelme labeling tool to annotate the curtain wall borders in the image and divide the curtain wall frame data set into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0014] Furthermore, in step S3, the depth information is remapped using logarithmic transformation, so that the distance between distant scenes in the depth map is reduced, the distance between close scenes is increased, and the influence of the background is reduced. The mathematical expression of the logarithmic transformation is:

[0015] y=λlog2(x+1)

[0016] Among them, x represents the distance from the image acquisition point to each point in the scene, expressed in pixel value, x∈[0,255]; y represents the distance after mapping; λ represents the scale constant, and the value of λ ensures that y∈[0,255].

[0017] Further, step S3 specifically includes: designing an overall "W"-shaped encoder-decoder structure according to the three-dimensional strip characteristics of the curtain wall frame to realize multi-scale multi-modal fusion, the encoder part uses ConvNeXt as the feature extraction network, replaces the traditional 2D convolution with 3D convolution, and then pre-fuse the curtain wall frame RGB image and depth map and inputs them into the corresponding module of the decoder to form features at four scales, and the decoder part constructs an FFA module and a CFF module to realize overall multi-scale multi-modal path aggregation to adapt to the special three-dimensional strip feature requirements of the curtain wall frame; the frame feature perception module (FFA) and the cross-modal feature fusion module (CFF) are connected to realize the merging of upsampled low-resolution features and high-resolution features and the merging of downsampled streams from high-resolution features to low-resolution features from top to bottom and bottom to top, and then connect the prediction head to obtain the prediction map.

[0018] Furthermore, in step S3, the curtain wall frame RGB image and the depth map are pre-fused, specifically including: splicing the curtain wall frame RGB image and the depth map in the data set into a four-channel RGB-D image in a new dimension to realize feature fusion. The form of this new dimension is defined as C×V×H×W, where C, V, H, and W represent the channel, visual, height, and width dimensions of the feature, respectively.

[0019] Further, in step S3, an FFA module is constructed, specifically including: in order to enhance the network's attention to the bar features, a set of RGB images and depth images are input and 6 paths are derived, and the first path is not processed; the second path uses 1×1×1 convolution to adjust the channel features; the third, fourth, and fifth paths perform 1×1×1 convolution and use 3×1×1, 1×3×1, and 1×1×3 asymmetric convolutions respectively to achieve feature interaction in the V dimension, learn features in the H dimension and W dimension, and then add them pixel by pixel with path 2 and use 3×3×3 convolution to form the mainstream features; the sixth path uses a 1×1×1 convolution kernel for convolution, maps it with a Sigmoid function, and multiplies it pixel by pixel with the mainstream features, and finally adds it pixel by pixel with the initial features of the first path and outputs the results.

[0020] Further, in step S3, a CFF module is constructed, specifically including: in order to achieve effective fusion of cross-modal features, two sets of RGB images and depth images are input, and operations of copying and reordering, splicing RGB images and splicing depth images are performed to derive three paths, the first path uses 2×1×1 convolution and LN layer; the second path uses 3×3×3 convolution, LN and GELU activation functions, and adds the original depth map of the third path pixel by pixel, and then uses a 2×3×3 convolution kernel for convolution; the third path uses 3×3×3 convolution, LN and GELU activation functions and adds the original RGB image of the second path pixel by pixel, and then uses a 2×3×3 convolution kernel for convolution; the features of the first path and the second path are fused and 1×1×1 convolution, LN, GELU activation function and 1×1×1 convolution are used in turn, and finally added to the first path to output the result.

[0021] Further, in step S4, the curtain wall frame detection model is trained, specifically including: the initial learning rate is set to 0.001, the Adam optimizer is used for training, and the binary cross entropy function is used as the loss function in the training process. The expression of the binary cross entropy function is as follows:

[0022]

[0023] Among them, BCELoss represents the loss value, y i represents the label value of the i-th pixel, p(y i) represents the predicted probability value of pixel i, and n represents the total pixels of a single image.

[0024] Solution 2: A curtain wall frame real-time detection device based on a border-aware cross-modal fusion network, comprising:

[0025] Data acquisition module: Use the depth camera to collect RGB images and depth maps of the curtain wall frame, and obtain images under the influence of various simple or even complex interferences;

[0026] Pre-processing module: used to pre-process the collected RGB images and depth maps, perform image denoising, perspective transformation, depth information remapping, data enhancement, and rotation, translation and flipping operations to obtain the curtain wall frame data set, and divide the data set;

[0027] Model processing module: used to build and train a curtain wall frame detection model based on a border-aware cross-modal fusion network and save the optimal curtain wall frame detection model;

[0028] Real-time detection module: including mobile robotic arm, depth camera and embedded system, used for real-time shooting and real-time detection of curtain wall frame images. The mobile robotic arm includes a large arm, a small arm, an end effector and a rotatable roller base, which is used for multi-directional movement and rotation to shoot curtain wall frame images in front and back, left and right directions; the embedded system is used to receive the curtain wall frame for parallel real-time detection.

[0029] The beneficial effects of the present invention are:

[0030] 1) Compared with the traditional manual curtain wall frame detection method, the present invention adopts a new curtain wall frame real-time detection framework based on border perception cross-modal fusion network, which can meet the requirements of detection efficiency and detection accuracy, and the operation method is simple and feasible, which is easy to promote and apply;

[0031] 2) The present invention introduces a border feature perception module and a cross-modal feature fusion module in the decoder part of the curtain wall frame detection model, which can well adapt to the special three-dimensional strip feature requirements of the curtain wall frame.

[0032] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0034] Figure 1It is an overall flow chart of the curtain wall frame real-time detection method based on the frame perception cross-modal fusion network of the present invention;

[0035] Figure 2 It is an overall structural diagram of the curtain wall frame detection model based on the frame-aware cross-modal fusion network of the present invention;

[0036] Figure 3 This is a structural diagram of the frame feature perception (FFA) module in the present invention;

[0037] Figure 4 This is a structural diagram of the cross-modal feature fusion (CFF) module in the present invention;

[0038] Figure 5 A comparison diagram of the real-time detection result and the true value of the detection method of the present invention;

[0039] Figure 6 It is a block diagram of a curtain wall frame real-time detection device based on a frame-aware cross-modal fusion network of the present invention. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] Embodiment 1:

[0042] This embodiment provides a curtain wall frame real-time detection method based on a border-aware cross-modal fusion network. Figure 1 As shown, the curtain wall frame real-time detection method comprises the following steps:

[0043] Step 1: Collect the curtain wall frame RGB image and depth map, where the collected curtain wall frame RGB image and depth map data specifically include a variety of simple or even complex interference effects, such as single frame, multiple frames, and entire frame images collected under different weather, different lighting, different distances, different angles, different obstacles, and with or without dust.

[0044] In this embodiment, a depth camera is used to photograph a curtain wall frame on a construction site to collect RGB images and depth map data of the curtain wall frame.

[0045] Step 2: Preprocess the collected curtain wall frame RGB images and depth maps and construct and divide the curtain wall frame dataset

[0046] The GaussianBlur function in OpenCV is used to perform Gaussian filtering and denoising on the RGB image. The warpPerspective function and getPerspectiveTransform function are used to perform perspective transformation to correct the image. The logarithmic transformation is used to remap the depth information, thereby reducing the distance between the distant views in the depth map and increasing the distance between the close views to reduce the impact of the background. Mosaic data enhancement is used to rotate some curtain wall frame images by 90°, 180°, and 270°, as well as translate and flip them in the horizontal and vertical directions to solve the problem of fewer images in some scenes in the dataset and expand the amount of image data. The labelme labeling tool is then used to annotate the curtain wall borders in the image, and the curtain wall frame dataset is divided into training set, validation set, and test set in a ratio of 7:2:1.

[0047] In this embodiment, in order to increase the difference between the foreground and background of the depth map and reduce the influence of the background, the depth information is remapped using logarithmic transformation, so that the distance between distant scenes in the depth map is reduced and the distance between close scenes is increased.

[0048] The mathematical expression of logarithmic transformation is:

[0049] y=λlog2(x+1)

[0050] Among them, x represents the distance from the image acquisition point to each point in the scene, expressed in pixel value, x∈[0,255], y represents the mapped distance; λ represents the scale constant, and the value of λ ensures that y∈[0,255].

[0051] Step 3: Construct a curtain wall frame detection model based on the border-aware cross-modal fusion network. The overall structure diagram is as follows: Figure 2 shown.

[0052] According to the three-dimensional strip characteristics of the curtain wall frame, an overall "W"-shaped encoder-decoder structure is designed to realize multi-scale and multi-modal fusion. The encoder part uses ConvNeXt as the feature extraction network, and replaces the traditional 2D convolution with 3D convolution. The curtain wall frame RGB image and depth map are pre-fused and input into the corresponding module of the decoder to form features at four scales. In order to adapt to the special three-dimensional strip feature requirements of the curtain wall frame, the decoder part constructs a frame feature perception module (FFA) and a cross-modal feature fusion module (CFF) to realize the overall multi-scale and multi-modal path aggregation. The FFA module and the CFF module are connected to realize the merging of the upsampled low-resolution features with the high-resolution features and the downsampled streams from the high-resolution features to the low-resolution features from the bottom to the top, and then connected to the prediction head to obtain the prediction map.

[0053] In this embodiment, the model extracts the features of the object from the two modalities and pre-fuse the curtain wall frame RGB image and the depth map, that is, the curtain wall frame RGB image and the depth map in the data set are spliced ​​in a new dimension to form a four-channel RGB-D image to achieve feature fusion. The form of this new dimension is defined as C×V×H×W, where C, V, H, and W represent the channel, visual, height, and width dimensions of the feature, respectively, and the image input size is 256×256.

[0054] like Figure 3 As shown in the figure, it is the structural diagram of the border feature perception module. In order to achieve effective fusion of cross-modal features, a set of curtain wall frame RGB images and depth images are input and 6 paths are derived. Path 1 is not processed; Path 2 uses 1×1×1 convolution to adjust the channel features; Paths 3, 4, and 5 perform 1×1×1 convolution and use 3×1×1, 1×3×1, and 1×1×3 asymmetric convolutions respectively to achieve feature interaction in the V dimension, learn features in the H and W dimensions, and then add them pixel by pixel with path 2 and use 3×3×3 convolution to form the mainstream features; Path 6 uses a 1×1×1 convolution kernel for convolution, maps it with a Sigmoid function, and multiplies it pixel by pixel with the mainstream features, and finally adds it pixel by pixel with the initial features of path 1 and outputs the results.

[0055] like Figure 4 As shown in the figure, it is the structural diagram of the cross-modal feature fusion module. In order to achieve effective fusion of cross-modal features, two sets of curtain wall frame RGB images and depth images are input, and the operations of copying and reordering, splicing RGB images and splicing depth images are performed to derive three paths. The path uses 2×1×1 convolution and LN layer; Path 2 uses 3×3×3 convolution, LN and GELU activation functions and adds it pixel by pixel with the original depth map of path 3, and then uses 2×3×3 convolution kernel for convolution; Path 3 uses 3×3×3 convolution, LN and GELU activation functions and adds it pixel by pixel with the original RGB image of path 2, and then uses 2×3×3 convolution kernel for convolution; The features of path 1 and path 2 are fused and 1×1×1 convolution, LN, GELU activation function and 1×1×1 convolution are used in turn, and finally added with path 1 to output the result.

[0056] It should be noted that the two sets of input curtain wall frame RGB images and depth images are obtained through dual input of the output of the border feature perception module and the multi-scale features. In particular, part of the first input features of the cross-modal feature fusion module are the output of the border feature perception module, and the other part are the output of ConvNeXt3.

[0057] Step 4: Solve the binary cross entropy loss function according to the predicted image and the target image generated by the frame-aware multimodal fusion network, train and learn the optimal parameters of the curtain wall frame detection model, and save the trained optimal curtain wall frame detection model to the real-time detection platform.

[0058] In this embodiment, the initial learning rate is set to 0.001, the Adam optimizer is used for training, and the binary cross entropy function is used as the loss function in the training process, wherein the expression of the binary cross entropy loss function is:

[0059]

[0060] Among them, BCELoss represents the loss value, y i represents the label value of the i-th pixel, p(y i ) represents the predicted probability value of the i-th pixel, and n represents the total pixels of a single image.

[0061] Step 5: Real-time capture of the RGB image and depth map of the curtain wall frame to be detected and real-time detection using the saved optimal curtain wall frame detection model, such as Figure 5 The figure shows a comparison between the real-time detection results and the true values ​​in this embodiment. The results show that the border-aware cross-modal fusion network model designed in the present invention can achieve a certain detection accuracy, has a strong intelligent effect, and is simple and feasible to operate.

[0062] Embodiment 2:

[0063] This embodiment provides a block diagram of a curtain wall frame real-time detection device based on a frame perception cross-modal fusion network, such as Figure 6 As shown, the curtain wall frame real-time detection device 600 provided in this embodiment specifically includes:

[0064] Data acquisition module 610: uses a depth camera to collect RGB images and depth maps of the curtain wall frame, and obtains images under the influence of various simple or even complex interferences.

[0065] Pre-processing module 620: used to pre-process the collected RGB images and depth maps, perform image denoising, perspective transformation, depth information remapping, data enhancement, and rotation, translation and flipping operations to obtain a curtain wall frame data set, and divide the data set.

[0066] Model processing module 630: used to construct and train a curtain wall frame detection model based on a border-aware cross-modal fusion network and save the optimal curtain wall frame detection model.

[0067] Real-time detection module 640: includes a mobile mechanical arm, a depth camera and an embedded system, and is used to capture and detect curtain wall frame images in real time. The mobile mechanical arm includes a large arm, a small arm, an end effector, and a rotatable roller base. The depth camera is installed on the top of the small arm of the mechanical arm and supported by a bracket to prevent the end effector from blocking the position. The mechanical arm uses its roller mobile platform to move forward and backward, left and right to capture images at different positions, and the rotating base rotates left and right to capture curtain wall frame images at multiple angles.

[0068] In an embodiment of the present invention, by collecting RGB images and depth maps of curtain wall frames, including single-frame, multiple-frame, and entire-frame images collected under different weather, different lighting, different distances, different angles, different obstacles, and with or without dust, a complete data set is constructed, which is divided into a training set, a validation set, and a test set in a ratio of 7:2:1. The model is processed, including building, training, and saving a detection model, which is used for real-time detection. During the detection process, a depth camera installed on a mobile robotic arm captures images in real time and transmits them to an embedded system, wherein the embedded system encapsulates the optimal detection model in advance, and then performs real-time detection to display the final detection result.

[0069] Embodiment 3:

[0070] The present invention provides an electronic device, including a memory and a processor. The memory is used to store a program that supports the processor to execute the curtain wall frame real-time detection method based on the frame-aware cross-modal fusion network described in the above embodiment 1. The processor is configured to execute the program stored in the memory.

[0071] Embodiment 4:

[0072] The present embodiment provides a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the curtain wall frame real-time detection method based on the border-aware cross-modal fusion network in Example 1 of the present invention.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A curtain wall frame real-time detection method based on a frame-aware cross-modal fusion network, characterized in that: The method specifically comprises the following steps: S1: Collect curtain wall frame RGB image and depth map; S2: perform image preprocessing on the collected curtain wall frame RGB image and depth map, and construct and divide the curtain wall frame dataset; S3: Construct a curtain wall frame detection model based on a border-aware cross-modal fusion network, specifically including: designing an overall "W"-shaped encoder-decoder structure according to the three-dimensional strip characteristics of the curtain wall border to achieve multi-scale and multi-modal fusion, the encoder part uses ConvNeXt as a feature extraction network, replaces the traditional 2D convolution with 3D convolution, and then pre-fuse the curtain wall frame RGB image and depth map and input them into the corresponding module of the decoder to form features on four scales, the decoder part constructs FFA module and CFF module to achieve overall multi-scale and multi-modal path aggregation to meet the special three-dimensional strip feature requirements of the curtain wall frame; the FFA module and the CFF module are connected to realize the merging of upsampled low-resolution features and high-resolution features and the merging of downsampled streams from high-resolution features to low-resolution features from top to bottom and bottom to top, and then connect the prediction head to obtain the prediction map; wherein, the FFA module represents the border feature perception module, and the CFF module represents the cross-modal feature fusion module; Constructing the FFA module, specifically including: inputting a set of RGB images and depth images and deriving 6 paths, the first path is not processed; the second path uses 1×1×1 convolution to adjust the channel features; the third, fourth, and fifth paths perform 1×1×1 convolution and use 3×1×1, 1×3×1, and 1×1×3 asymmetric convolutions respectively to achieve feature interaction in the V dimension, learn features in the H and W dimensions, and then add them pixel by pixel with path 2 and use 3×3×3 convolution to form the mainstream features; the sixth path uses a 1×1×1 convolution kernel for convolution, maps it with a Sigmoid function, and multiplies it pixel by pixel with the mainstream features, and finally adds it pixel by pixel with the initial features of the first path and outputs the results; Constructing a CFF module, specifically including: inputting two sets of RGB images and depth images, performing copy and reordering, splicing RGB images and splicing depth images, and deriving three paths, the first path uses 2×1×1 convolution and LN layer; the second path uses 3×3×3 convolution, LN and GELU activation functions, and adds the original depth map of the third path pixel by pixel, and then uses a 2×3×3 convolution kernel for convolution; the third path uses 3×3×3 convolution, LN and GELU activation functions and adds the original RGB image of the second path pixel by pixel, and then uses a 2×3×3 convolution kernel for convolution; the features of the first and second paths are fused and 1×1×1 convolution, LN, GELU activation function and 1×1×1 convolution are used in turn, and finally added to the first path to output the result; S4: Solve the binary cross entropy loss function based on the predicted image and target image generated by the border-aware cross-modal fusion network, train and learn the optimal parameters of the curtain wall frame detection model, and save the trained optimal curtain wall frame detection model to the real-time detection platform; S5: Collect the RGB image and depth map of the curtain wall frame to be detected in real time, and use the saved optimal curtain wall frame detection model to perform real-time detection.

2. The curtain wall frame real-time detection method according to claim 1, characterized in that: In step S1, the collected curtain wall frame RGB images and depth maps include: single frame, multiple frames and whole frame images collected under different weather conditions, different lighting conditions, different distances, different angles, different obstacles, and with or without dust.

3. The curtain wall frame real-time detection method according to claim 1, characterized in that: Step S2 specifically includes: using the GaussianBlur function in OpenCV to perform Gaussian filtering and denoising on the RGB image, using the warpPerspective function and getPerspectiveTransform function to perform perspective transformation to correct the image, using logarithmic transformation to remap the depth information, using Mosaic data enhancement and rotating some curtain wall frame images by 90°, 180°, and 270°, as well as translating and flipping in the horizontal and vertical directions, using the labelme labeling tool to annotate the curtain wall borders in the image and divide the curtain wall frame data set into a training set, a validation set, and a test set in a ratio of 7:2:

1.

4. The curtain wall frame real-time detection method according to claim 3, characterized in that: In step S3, the depth information is remapped using logarithmic transformation, which reduces the distance between distant scenes in the depth map and increases the distance between close scenes, thereby reducing the influence of the background. The mathematical expression of the logarithmic transformation is: y=λlog2(x+1) Among them, x represents the distance from the image acquisition point to each point in the scene, expressed in pixel value, x∈[0,255]; y represents the distance after mapping; λ represents the scale constant, and the value of λ ensures that y∈[0,255].

5. The curtain wall frame real-time detection method according to claim 1, characterized in that: In step S3, the curtain wall frame RGB image and the depth map are pre-fused, specifically including: splicing the curtain wall frame RGB image and the depth map in the data set into a four-channel RGB-D image in a new dimension to achieve feature fusion. The form of this new dimension is defined as C×V×H×W, where C, V, H, and W represent the channel, visual, height, and width dimensions of the feature, respectively.

6. The curtain wall frame real-time detection method according to claim 1, characterized in that: In step S4, the curtain wall frame detection model is trained, specifically including: the initial learning rate is set to 0.001, the Adam optimizer is used for training, and the binary cross entropy function is used as the loss function in the training process. The expression of the binary cross entropy function is as follows: Among them, BCELoss represents the loss value, y i represents the label value of the i-th pixel, p(yi) represents the predicted probability value of the i-th pixel, and n represents the total pixels of a single image.

7. A curtain wall frame real-time detection device based on a frame-aware cross-modal fusion network designed according to the detection method of any one of claims 1 to 6, characterized in that: The device includes: Data acquisition module: Use the depth camera to collect RGB images and depth maps of the curtain wall frame, and obtain images under the influence of various simple or even complex interferences; Pre-processing module: used to pre-process the collected RGB images and depth maps, perform image denoising, perspective transformation, depth information remapping, data enhancement, and rotation, translation and flipping operations to obtain the curtain wall frame data set, and divide the data set; Model processing module: used to build and train a curtain wall frame detection model based on a border-aware cross-modal fusion network and save the optimal curtain wall frame detection model; Real-time detection module: includes a mobile robotic arm, a depth camera and an embedded system, which is used to capture and detect curtain wall frame images in real time.

Images