A visual recognition system and method for window cleaning robots
By using a visual recognition method that combines multi-source data synchronization and 2D/3D fusion, the problem of high computational complexity or lack of depth information in window cleaning robots operating on high-altitude curtain walls has been solved. This method achieves high-precision window frame boundary recognition and real-time performance, thereby improving the safety and intelligence level of window cleaning robots.
Patent Information
- Application Number
- CN202611313969.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
Existing visual recognition technologies for window cleaning robots suffer from high computational complexity or lack of depth information in high-altitude curtain wall operations. They cannot provide accurate three-dimensional physical boundary constraints in complex optical environments, resulting in decreased recognition accuracy or excessive computational burden, making it difficult to meet real-time requirements.
By employing multi-source data acquisition and spatiotemporal synchronization, combined with 2D instance segmentation and 3D point cloud fusion, a coarse mask of the window frame region is extracted through the instance segmentation model. The random sampling consistency algorithm is used to fit the 3D planar model, and morphological processing is combined to refine the boundary features, outputting high-precision 3D coordinates of the window frame.
It significantly reduces the amount of point cloud processing data, improves recognition accuracy and real-time performance, provides reliable safety boundary constraints, and enhances the safety and intelligence level of window cleaning robots operating at heights.
Smart Images

Figure CN123118523A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and robot perception technology, specifically to a visual recognition system and method for a window cleaning robot. Background Technology
[0002] With the continuous advancement of urbanization, the number and height of high-rise buildings are constantly increasing, and glass curtain walls have become the mainstream exterior wall form of modern urban buildings due to their aesthetic appeal and lighting advantages. However, the regular cleaning and maintenance of glass curtain walls has long relied on manual high-altitude operations, which is not only inefficient and costly but also poses serious personal safety hazards. Against this backdrop, automated high-altitude window cleaning robots have attracted widespread attention in the industry as an effective solution to replace manual cleaning. Among them, the visual perception system of the window cleaning robot is the core technology that determines whether it can safely and efficiently complete its work tasks—high-precision window frame boundary recognition and spatial positioning are key prerequisites for ensuring that the robot accurately executes the cleaning trajectory, prevents it from falling out of bounds, and avoids collisions with the window frame structure.
[0003] Currently, the visual recognition technologies used for window cleaning robots mainly include the following implementation schemes.
[0004] (I) Edge Detection and Semantic Segmentation Scheme Based on 2D Images. This scheme adopts a pure 2D image processing approach, acquiring 2D images of the curtain wall scene using a conventional RGB camera, and then employing traditional image processing algorithms such as the Canny edge detection operator and Hough transform, or semantic segmentation networks based on deep learning (such as U-Net, DeepLab series, etc.) to extract the 2D boundary features of the window frame region on the RGB image plane. The significant advantage of this scheme is its relatively low computational complexity, moderate requirements on the computing power of the embedded platform, and ability to output pixel-level position information of the window frame at a high frame rate. However, its inherent defects are also prominent: since it relies solely on monocular RGB images, this scheme inherently faces the problem of missing depth spatial information, and cannot obtain the precise position and orientation of the window frame in 3D physical space. Especially in the actual operation scenario of high-altitude curtain walls, harsh optical conditions such as direct sunlight, large-angle shadows, and high reflectivity of glass surfaces are common, which can easily lead to misjudgment or missed detection of 2D edges. For example, the specular reflection of sunlight by a glass curtain wall may produce false textured edges that closely resemble the actual window frame edges, while the window frame boundaries in large shaded areas may be completely lost due to insufficient contrast. Ultimately, this solution cannot provide accurate three-dimensional physical boundary constraints for the window cleaning robot. If the robot fails to accurately perceive the true three-dimensional position of the window frame boundary when the two-dimensional recognition result is misjudged, it is highly likely to cross the window frame edge, causing adsorption failure, and subsequently fall from a height, resulting in equipment damage or even a safety accident.
[0005] (II) Overall Modeling and Perception Scheme Based on Global 3D Point Cloud. This scheme adopts a 3D perception technology approach, using 3D sensors such as depth cameras (e.g., binocular depth cameras, structured light cameras, ToF cameras) or LiDAR to collect large-scale 3D point cloud data of the panoramic space within the field of view, thereby constructing an overall spatial 3D model including window frames, glass surfaces, wall structures, and the surrounding environment. Based on this, 3D point cloud segmentation algorithms (e.g., 3D segmentation methods based on region growing, Euclidean clustering, or deep learning) are used to extract the spatial location information of the window frames. The advantage of this scheme is that it can provide complete geometric information of the window frames in 3D space, providing direct 3D data support for the robot's spatial localization and path planning. However, this scheme has an inherent drawback of excessively large data volume. The global spatial point cloud inevitably contains a large amount of background data from non-operational areas unrelated to window cleaning operations, such as outdoor distant buildings, sky, and ground, as well as indoor clutter point clouds such as tables, chairs, curtains, and people. This redundant data provides no useful information for the window frame recognition task itself, yet it consumes a large amount of storage space and computing resources, resulting in extremely high computational overhead for the overall algorithm. This makes it difficult to achieve high frame rate real-time perception response on edge computing platforms for window cleaning robots with limited computing power (such as ARM-based embedded devices or lightweight AI acceleration modules). Even when downsampling the point cloud using methods such as voxel filtering, the accuracy of window frame edge details is often sacrificed while reducing the amount of data, making it difficult to balance real-time performance and accuracy.
[0006] Therefore, it can be seen that although Scheme 1 has acceptable computational efficiency, it lacks depth dimension information, resulting in a significant decrease in recognition accuracy under complex optical environments, and cannot meet the three-dimensional physical boundary constraints required for safe operation. Scheme 2, while able to acquire three-dimensional information, suffers from an excessively large global point cloud data scale and computational burden, making it difficult to achieve real-time perception at the edge. Neither of these two schemes can effectively reduce the amount of data computation to meet real-time requirements while ensuring the recognition accuracy of the window cleaning robot in highly reflective and complex curtain wall environments. Summary of the Invention
[0007] To address the shortcomings of existing technologies, this invention provides a visual recognition system and method for window cleaning robots, which significantly reduces the amount of point cloud processing data and meets the real-time requirements of edge computing; effectively suppresses interference from complex curtain wall optical environments, exhibiting strong robustness; and achieves millimeter-level boundary positioning accuracy, providing reliable safety boundary constraints for window cleaning robots and significantly improving the safety and intelligence level of high-altitude operations.
[0008] To achieve the above objectives, the present invention provides the following technical solution:
[0009] A visual recognition method for window cleaning robots includes the following steps:
[0010] Step S1: Perform multi-source data acquisition and spatiotemporal synchronization initialization, control the depth vision sensor of the window cleaning robot to synchronously acquire the original RGB image in the current curtain wall field of view, and read the two-dimensional depth matrix data aligned with its timestamp;
[0011] Step S2: Perform instance segmentation and coarse localization on the two-dimensional pixel plane. Input the original RGB image obtained in step S1 into the pre-trained instance segmentation model for forward inference and extract the coarse mask of the window region on the two-dimensional image plane.
[0012] Step S3: Perform 3D point cloud fusion and extraction of the region of interest. Based on the intrinsic parameters of the depth camera, convert the 2D depth matrix data of step S1 into the overall spatial point cloud. Use the coarse mask generated in step S2 as a spatial filter to extract the depth data corresponding to the mask pixel area, and map the pixels corresponding to the depth data back to the 3D physical space to obtain the local original point cloud data of the window frame.
[0013] Step S4: Perform spatial geometric plane fitting and noise filtering. The local original point cloud extracted in step S3 is fitted with a random sampling consensus algorithm. Sampling points are randomly selected from the candidate point cloud to construct a three-dimensional plane model. The interior points are determined by calculating the distance from each point to the fitted plane. After iteration, the three-dimensional plane model with the most interior points is retained. The plane parameters are refitted using the selected interior points, and outliers that are more than a preset threshold from the fitted plane are removed.
[0014] Step S5: Perform spatial coordinate calculation and boundary feature optimization. Based on the 3D planar model fitted in step S4, project the point cloud onto the camera model and map the projection result onto the 2D image plane. Perform morphological closing operation and dilation processing on the 2D edge candidate region. Use the dilated support region to constrain the edge line candidate pixels to obtain the 2D skeleton boundary of the window frame. Determine the core corner points of the window frame according to the 2D skeleton boundary and calculate the corresponding 3D spatial points. Generate the spatial pose of the window frame in the camera coordinate system. Output the 3D coordinates and spatial pose of each core corner point of the window frame to the downstream path planning and control module.
[0015] Preferably, the timestamp alignment in step S1 specifically involves: caching the original RGB image and the two-dimensional depth matrix data respectively, performing approximate time matching based on the timestamps in their respective message headers, and determining the data pair as timestamp aligned when the time difference between the two does not exceed a preset time tolerance.
[0016] Preferably, in step S2, the instance segmentation model is a deep learning-based instance segmentation network. The model is trained using transfer learning, utilizing pre-trained weights from a publicly available instance segmentation dataset as initialization parameters, and fine-tuned using a self-built window frame scene dataset. The sample images in the self-built window frame scene dataset contain window frame targets under different angles, distances, lighting conditions, and occlusion conditions, and the targets in the sample images are labeled with categories and pixel-level contours.
[0017] The training parameters of the instance segmentation model are set as follows: 100 to 300 training epochs, input image size of 640×640 pixels, batch size of 8 to 32, initial learning rate of 0.001 to 0.01, optimizer of any one of SGD, Adam, or AdamW, and loss function including target classification loss, bounding box regression loss, and mask segmentation loss; during training, data augmentation processing such as brightness perturbation, scaling, translation, flipping, and noise perturbation is performed on the sample images.
[0018] Preferably, in step S3, the intrinsic parameters of the depth camera and the RGB camera include the horizontal focal length fx, the vertical focal length fy, and the principal point coordinates cx and cy. The depth data and the RGB image are aligned using a software reprojection method. Based on the intrinsic parameters of the depth camera and the RGB camera, as well as the rotation and translation parameters between the pre-calibrated depth sensor and the RGB camera, the three-dimensional points in the depth data are transformed to the RGB camera coordinate system and projected onto the RGB image pixel plane to establish the correspondence between the depth data and the RGB image pixels.
[0019] Preferably, in step S4, the specific process of plane fitting using the random sampling consensus algorithm is as follows: No more than 8000 points are randomly selected from the candidate point cloud as a sampling set. Three non-collinear points are randomly selected each time to construct a plane model. The plane model is generally expressed as "A×x+B×y+C×z+D=0", where A, B, and C represent the components of the plane normal vector, D represents the plane offset, and x, y, and z represent the coordinate values in three-dimensional space. Interior points are determined by calculating the distance from each point to the fitting plane, and points with a distance of no more than 0.10 meters are identified as interior points. After multiple random iterations, the plane model with the most interior points is retained. When the number of interior points is no less than 2500 and the proportion of interior points is no less than 12%, the plane model is deemed valid. The selected interior points are then used to refit the plane parameters. Outliers exceeding a preset threshold in distance from the fitting plane are then removed, and spatial connectivity is combined to perform regional filtering of the interior points, obtaining the local plane point cloud where the window frame is located.
[0020] In step S4, the input point cloud is not pre-sampled using voxels; instead, the candidate point cloud is randomly sampled only during the random sampling consistency fitting process.
[0021] Preferably, in step S5, the specific method for performing morphological closing operation and dilation processing on the two-dimensional edge candidate region is as follows: the edge line candidate region is dilated using a rectangular structural element with a side length of 5 pixels, and the point cloud projection support region is dilated using a rectangular structural element with a width of 13 pixels and a height of 13 pixels.
[0022] Preferably, in step S5, the core corner point of the window frame in the two-dimensional skeleton is defined as three feature points, including the upper left corner point of the window frame glass area, a horizontal reference point obtained by offsetting the upper left corner point 200 pixels along the horizontal edge direction, and a vertical reference point obtained by offsetting the upper left corner point 300 pixels along the vertical edge direction; for each two-dimensional feature point, the three-dimensional point in the point cloud is projected onto the image plane according to the camera intrinsic parameters, and the effective three-dimensional point with the smallest projection distance is selected as its corresponding three-dimensional spatial point within a search area with a radius of 25 pixels around the two-dimensional feature point.
[0023] Preferably, a local coordinate system for the window frame is constructed based on the three-dimensional spatial points corresponding to the three-dimensional feature points: the direction from the three-dimensional spatial point corresponding to the upper left corner point to the three-dimensional spatial point corresponding to the horizontal reference point is determined as the horizontal axis, and the direction from the three-dimensional spatial point corresponding to the upper left corner point to the three-dimensional spatial point corresponding to the vertical reference point is determined as the vertical axis. The normal axis of the window frame plane is obtained by the cross product of the two direction vectors. After normalizing and orthogonalizing each coordinate axis, the spatial pose of the window frame in the camera coordinate system is generated.
[0024] The transformation between the camera coordinate system and the window cleaning robot's base coordinate system is achieved using a pre-calibrated and stored fixed four-by-four extrinsic parameter matrix.
[0025] This invention also discloses a visual recognition system and method for a window cleaning robot, comprising:
[0026] The data acquisition module is used to control the external depth camera of the window cleaning robot to synchronously acquire the raw RGB image stream and depth matrix data of the working environment, and to ensure that the data streams of the two are spatiotemporally aligned.
[0027] The 2D segmentation module, connected to the data acquisition module, is used to load the instance segmentation network, receive the original RGB image input from the data acquisition module and perform parallel inference, and output a coarse 2D mask of the window area.
[0028] The local point cloud extraction module is connected to the data acquisition module and the 2D segmentation module respectively. It is used to establish the mapping relationship between 2D pixels and 3D space, use a 2D window frame coarse mask to perform ROI cropping on the global depth data, filter out the background point cloud, and extract the local 3D original point cloud.
[0029] The geometric fitting optimization module, connected to the local point cloud extraction module, is used to perform random sampling consistency iterative calculation on the received local point cloud, remove curtain wall interference noise, and fit the spatial ideal plane equation representing the geometric features of the glass window surface.
[0030] The boundary coordinate calculation module, connected to the geometric fitting optimization module, is used to perform fine pixel dilation and coordinate correction on the window frame edge by combining the fitted plane equation, calculate the three-dimensional coordinate sequence of each boundary corner point of the window frame in the physical space coordinate system of the window cleaning robot, and output the high-precision boundary features.
[0031] Preferably, the instance segmentation network loaded in the two-dimensional segmentation module is a YOLO11-seg model built on the PyTorch framework. The model is pre-trained using transfer learning and outputs the category, bounding box, and instance segmentation mask of the window object.
[0032] This invention provides a visual recognition method for window cleaning robots, which has the following beneficial effects:
[0033] By utilizing a two-dimensional instance segmentation model to extract a coarse mask of the window frame region on the RGB image plane, and then using this mask as a spatial filter to crop the ROI of the synchronously aligned depth matrix, the amount of point cloud data entering the three-dimensional processing stage is reduced by an order of magnitude. This allows the invention to run stably on edge computing platforms with limited computing power for window cleaning robots.
[0034] After extracting the local point cloud of the window frame, a spatial plane geometry fitting mechanism based on the random sampling consensus algorithm is introduced to effectively address the most challenging environmental interference factors in high-altitude glass curtain wall operations. It can accurately distinguish between valid window surface point clouds and outliers, and even when the point cloud contains a large number of outliers, it can still fit a high-precision ideal plane equation representing the real glass window surface. This allows the invention to maintain stable and reliable plane recognition performance under different lighting conditions, weather conditions, and varying degrees of glass surface contamination, significantly improving the adaptability of the window cleaning robot in complex outdoor environments.
[0035] By using a high-precision 3D planar model fitted with the RANSAC algorithm, the local point cloud is reprojected onto the 2D image plane. Then, through a combination of morphological dilation and nearest neighbor search, the 2D segmentation edges are precisely corrected using 3D spatial information. Based on this, a local coordinate system for the window frame is constructed, and its precise spatial pose in the robot's base coordinate system is calculated. Finally, a high-precision 3D coordinate sequence of each boundary corner point of the window frame is output. This effectively eliminates pixel-level positioning errors that may exist in 2D segmentation, providing accurate and reliable safety boundary constraints for downstream path planning and collision avoidance control systems. The window cleaning robot can then accurately locate its spatial position relative to the window frame boundary and promptly perform safety actions such as deceleration, turning, or stopping when approaching the boundary, thus fundamentally avoiding the risk of falling out of bounds. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of this invention or the prior art will be briefly introduced below.
[0037] Figure 1 This is a flowchart of a visual recognition method for a window cleaning robot according to the present invention;
[0038] Figure 2 This is a structural block diagram of a visual recognition system for a window cleaning robot according to the present invention;
[0039] Figure 3 This is a schematic diagram illustrating the RANSAC plane fitting and denoising principle in this invention. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.
[0041] Example 1, as Figure 1 As shown, this embodiment provides a visual recognition method for a window cleaning robot, including the following steps:
[0042] Step S1: Perform multi-source data acquisition and spatiotemporal synchronization initialization;
[0043] The depth vision sensor controlling the window cleaning robot synchronously acquires the original RGB image within the current curtain wall field of view and reads the two-dimensional depth matrix data (e.g., .npy format file) aligned with its timestamp, which serves as the input benchmark for the entire visual recognition system.
[0044] Specifically, the window cleaning robot is equipped with a depth vision sensor consisting of an RGB camera and a depth camera, which work together in a hardware-synchronized or software-synchronized manner. The RGB camera is used to capture color images of the curtain wall scene, while the depth camera is used to capture depth data within the same field of view. In practical deployment, any of the following can be used: a structured light depth camera, a binocular stereo vision camera, or a time-of-flight (ToF) depth camera, as long as it can output depth matrix data that is time-synchronized with the RGB images.
[0045] The specific implementation of timestamp alignment in step S1 is as follows: The original RGB image and the two-dimensional depth matrix data are cached in memory buffers respectively, and approximate time matching is performed based on the timestamps recorded in their respective message headers. Due to unavoidable delays in data acquisition, transmission, and processing, it is difficult for the timestamps of the RGB image and the depth matrix to achieve absolute consistency. Therefore, a preset time tolerance is set (e.g., 30 milliseconds to 50 milliseconds). When the absolute difference between the two timestamps does not exceed this tolerance, they are determined to be valid data pairs for timestamp alignment. Data pairs exceeding the time tolerance will be discarded or wait for matching in the next frame to ensure a strict correspondence between the RGB image and the depth data in the time dimension during subsequent processing.
[0046] In an alternative embodiment, if the depth camera and the RGB camera support a hardware-triggered synchronization mode, the two sensors can be triggered to acquire data simultaneously via an external trigger signal. In this case, timestamp alignment can be directly guaranteed by hardware without the need for approximate matching at the software level.
[0047] Step S2: Perform instance segmentation and coarse localization of the two-dimensional pixel plane.
[0048] The original RGB image obtained in step S1 is input into the pre-trained instance segmentation model for forward inference to extract a coarse mask of the window frame region on the two-dimensional image plane. The mask is used to characterize the region range of the window frame target in the two-dimensional image, that is, to provide a pixel-level spatial index for subsequent three-dimensional point cloud extraction.
[0049] In a preferred embodiment, the instance segmentation model is a deep learning-based instance segmentation network, such as the YOLO11-seg instance segmentation network built on the PyTorch framework. YOLO11-seg is the latest evolution of the YOLO series of object detection algorithms for instance segmentation tasks. It can simultaneously output the target's class label, bounding box, and pixel-level instance segmentation mask in a single forward inference pass, featuring fast inference speed and high accuracy, making it particularly suitable for deployment on edge computing platforms with limited computing power. While the YOLO11-seg model is used as an example, those skilled in the art should understand that other instance segmentation networks (such as Mask R-CNN, SOLO, CondInst, etc.) can also be used to achieve the same or similar functions, all of which fall within the scope of protection of this invention.
[0050] The training process of the instance segmentation model is as follows: First, pre-trained weights on publicly available large instance segmentation datasets (such as the COCO dataset, ADE20K dataset, etc.) are used as the initialization parameters of the model. These pre-trained weights have been fully trained on massive images containing 80 categories to obtain general image feature extraction capabilities. On this basis, fine-tuning is performed in conjunction with a self-built window frame scene dataset to enable the model to adapt to the specific visual features of window frame targets in the window cleaning robot operation scenario.
[0051] The self-built window frame scene dataset is constructed as follows: In actual high-altitude curtain wall operation scenarios, a large number of window frame target sample images are collected using an RGB camera mounted on a window cleaning robot under different angles, distances, lighting conditions (including strong sunlight on sunny days, soft light on cloudy days, and side lighting at dawn and dusk), and different degrees of occlusion (such as the window frame being partially obscured by window screens, decorative strips, or stains). During data collection, the same depth camera as the actual window cleaning robot is used to collect a large number of window frame scene images under different times (such as morning, noon, and evening) and different weather conditions (such as sunny, cloudy, and overcast days). Each window frame target in the collected sample images is precisely manually labeled, including a target category label (e.g., labeled "window_frame") and a pixel-level contour point sequence, forming a high-precision instance segmentation ground truth label. The size of the dataset can be determined according to actual needs, for example, containing more than 5000 labeled samples.
[0052] During training, specific parameters are set as follows: the number of training epochs is 100 to 300, preferably 200; the input image size is uniformly scaled to 640×640 pixels; the batch size is set to 8 to 32, preferably 16; the initial learning rate is set to 0.001 to 0.01, preferably 0.005, and a cosine annealing learning rate decay strategy is used to gradually reduce the learning rate during training; the optimizer is any one of SGD (stochastic gradient descent), Adam, or AdamW, with AdamW being preferred to improve convergence stability; the loss function consists of three components: target classification loss (using cross-entropy loss), bounding box regression loss (using CIoU loss or GIoU loss), and mask segmentation loss (using binary cross-entropy loss or Dice loss).
[0053] To improve the model's generalization ability and robustness in complex outdoor environments, online data augmentation strategies were also adopted during training. The input sample images were subjected to various data augmentation processes in real time, including brightness perturbation, contrast perturbation, random scaling (ranging from 0.5 to 1.5), random translation (ranging from ±20% of the image size), random horizontal flipping, and Gaussian noise perturbation.
[0054] During the inference phase, the trained instance segmentation model takes a single RGB image as input. After forward propagation, it outputs the class confidence of the bounding box target, the bounding box coordinates (including the target's position and size in the image), and an instance segmentation mask (a binary matrix of the same size as the input image; pixels with a mask value of 1 represent the bounding box region, and pixels with a value of 0 represent the background region). This mask characterizes the region extent of the bounding box target in the two-dimensional image, providing pixel-level region guidance for subsequent steps.
[0055] Step S3: Perform 3D point cloud fusion and extraction of the Region of Interest (ROI).
[0056] Global Depth Matrix to Point Cloud Conversion: First, the two-dimensional depth matrix is converted into a global spatial point cloud within the field of view based on the intrinsic parameters of the depth camera. Then, using the coarse mask of the two-dimensional window frame generated in step S2 as a spatial filter, only the depth data corresponding to the mask pixel regions is extracted, and these specific pixels are mapped back to three-dimensional physical space, thereby obtaining the local original point cloud data of the window frame portion. This step achieves effective fusion of two-dimensional segmentation and three-dimensional depth, eliminates invalid background point clouds, and has the technical effect of reducing computational redundancy.
[0057] Specifically, the raw data acquired by the depth camera is a two-dimensional depth matrix. The value of each pixel in the matrix represents the distance (depth value) of the corresponding spatial point from the depth camera's imaging plane. To convert this depth matrix into a three-dimensional point cloud, a back-projection transformation is required using the depth camera's intrinsic parameter matrix. The depth camera's intrinsic parameters include the horizontal focal length fx (in pixels), the vertical focal length fy (in pixels), and the principal point coordinates cx and cy (in pixels). For each pixel (u, v) in the depth matrix, its corresponding depth value is d(u, v). The three-dimensional spatial coordinates (X, Y, Z) of this pixel in the camera coordinate system can be calculated using the following formula:
[0058] Z=d(u, v);
[0059] X = (u - cx) × Z / fx;
[0060] Y = (v - cy) × Z / fy.
[0061] The aforementioned intrinsic parameters can be directly obtained from the camera's factory calibration file, device SDK, or camera information interface. If they cannot be directly obtained, multiple sets of chessboard calibration board images in different poses can be acquired using the classic Zhang's calibration method, and then calculated offline by the camera calibration algorithm.
[0062] Alignment of Depth Data with RGB Images: Pixel-level alignment between depth data and RGB images is achieved through software reprojection. Since the depth camera and RGB camera have a fixed spatial offset in their physical locations (i.e., they have different optical centers and orientations), cross-sensor spatial alignment is required. Specifically, based on the intrinsic parameter matrices of the depth camera and RGB camera, and the rotation (R) and translation (T) parameters from the depth sensor coordinate system to the RGB camera coordinate system obtained through pre-joint calibration, the 3D point corresponding to each valid pixel in the depth data (located in the depth camera coordinate system) is transformed to the RGB camera coordinate system. Then, according to the intrinsic parameter matrix of the RGB camera, it is projected onto the RGB image pixel plane, thereby establishing the correspondence between depth matrix pixels and RGB image pixels (i.e., the "depth-color alignment mapping table"). Specifically, for 3D points in the depth camera coordinate system... Its coordinates in the RGB camera coordinate system are Then The corresponding pixel coordinates are obtained by projecting the RGB camera intrinsic parameters onto the RGB image pixel plane. , The rotation and translation parameters can be obtained directly from the equipment's factory calibration data, or they can be obtained by jointly calibrating the same calibration board using a depth sensor and an RGB camera.
[0063] ROI Cropping and Local Point Cloud Extraction: After depth-color alignment, the coarse mask of the window frame generated in step S2 (a binary image where pixels within the mask area have a value of 1 and those outside have a value of 0) is used as a spatial filter to extract only the depth data corresponding to the mask pixel area. Specifically, the aligned depth matrix is traversed pixel by pixel, retaining only the depth values corresponding to the pixel positions with a mask value of 1, while setting the depth values of all other pixel positions to zero or discarding them directly. Subsequently, these retained effective depth pixels are mapped one by one to three-dimensional physical space (in meters) according to the aforementioned back-projection formula, resulting in a local original point cloud dataset containing only the window frame area. The data volume of this local point cloud is typically only 5% to 15% of the global point cloud, depending on the proportion of the window frame area in the image, and the data compression effect is very significant. After the above filtering, the local original point cloud data of the window frame portion is obtained. This step effectively integrates 2D segmentation and 3D depth, eliminating all background point clouds (such as outdoor scenery, indoor clutter, and adjacent building structures) outside the window frame area, thus reducing the amount of data input to the 3D processing stage by orders of magnitude.
[0064] In a variant embodiment, if there is a large difference in the field of view between the depth camera and the RGB camera (e.g., their field of view angles are inconsistent), the overlapping area between the field of view of the depth camera and the field of view of the RGB camera can be calculated before masking. Depth-color alignment and masking are performed only within the overlapping area to avoid invalid mapping caused by mismatch in field of view boundaries.
[0065] Step S4: Perform spatial geometric plane fitting and noise filtering.
[0066] like Figure 3 As shown, the RANSAC (Random Sample Consensus) algorithm is used to fit the plane of the local original point cloud of the window frame extracted in step S3. A 3D plane model is constructed by randomly selecting sampling points from the candidate point cloud. Interior points are determined by calculating the distance from each point to the fitted plane. After iteration, the 3D plane model with the most interior points is retained. The selected interior points are then used to refit the plane parameters, and outliers exceeding a preset threshold are removed.
[0067] The RANSAC algorithm is a robust parameter estimation method that can obtain optimal model parameter estimates through iterative sampling even with a large proportion of noisy data. In the application scenario of this invention, although most of the background point cloud in the original point cloud of the window frame area has been removed in step S3, a certain proportion of deep outliers inevitably remain in the local point cloud due to the high reflectivity of the glass curtain wall surface, sunlight refraction, and surface dust adhesion. The spatial location of these noise points deviates significantly from the actual glass window surface, and directly fitting them to a plane will seriously interfere with the fitting accuracy. The RANSAC algorithm is designed to solve this kind of problem.
[0068] like Figure 3 As shown, the specific process of RANSAC plane fitting in this step is as follows:
[0069] Step 1: Random Sampling: Randomly select no more than 8000 points from the candidate point cloud as a sampling set. Each time, randomly select three non-collinear points from the sampling set to construct a 3D planar model. The planar model is represented by the general formula:
[0070] A×x+B×y+C×z+D=0;
[0071] Where A, B, and C represent the three components of the plane normal vector, D represents the plane offset, and x, y, and z represent the coordinates of a point in three-dimensional space. The normal vector of this planar model is (A, B, C), which, after normalization, yields a unit normal vector, representing the orientation of the glass window.
[0072] Step 2: Interior Point Determination: For the currently constructed planar model, traverse all points in the candidate point cloud and calculate the perpendicular distance from each point to the plane. The calculation formula is:
[0073]
[0074] Distance Points not exceeding a preset threshold (in this embodiment, the threshold is 0.10 meters, or 10 centimeters) are identified as inliers of the planar model, and the number of inliers and their index set are recorded.
[0075] Step 3: Iterative Optimization: Repeat the above random sampling and interior point determination process. In this embodiment, the number of iterations is set to 160, and the plane model parameter with the most interior points is recorded in each iteration. When the number of interior points of a certain model exceeds the preset early stopping threshold (for example, the number of interior points exceeds 4000), the iteration can be terminated in advance to save computation time.
[0076] Step 4: Model Validation: After iteration, the validity of the retained optimal planar model is verified. In this embodiment, two validity criteria are set: 1) the number of interior points is no less than 2500; 2) the proportion of interior points (the ratio of interior points to the total number of candidate point clouds) is no less than 12%. When both conditions are met simultaneously, the planar model is determined to be valid, indicating that it truly represents the real spatial geometric features of the glass window surface; if either condition is not met, it is considered that no valid window frame plane has been detected in the current frame, and the system will discard the frame data and wait for the next frame input.
[0077] Step 5: Planar Parameter Refinement: After determining the effective planar model, refit the planar parameters using all points identified as interior points. This refinement fitting can employ the least squares method or eigenvalue decomposition method to further optimize the accuracy of the planar parameters while preserving the overall orientation of the optimal model.
[0078] Step 6: Outlier Removal and Region Filtering: Based on the refined plane equations, the distance from each point in the candidate point cloud to the plane is calculated again, and outliers exceeding a preset threshold (0.10 meters in this embodiment) are removed. On this basis, spatial connectivity is used to filter the remaining inliers. Specifically, a clustering algorithm based on Euclidean distance (such as DBSCAN or region growing algorithm) is used to cluster the inliers, selecting the main connected region containing the most points as the local planar point cloud containing the window frame. Isolated, noisy regions are removed, thus obtaining a complete and clean local planar point cloud containing the window frame.
[0079] In the RANSAC fitting process described above, it is important to note that this step does not perform pre-voxel downsampling on the input point cloud. Although voxel downsampling can reduce the amount of point cloud data at the global level, it essentially alters the original spatial distribution of the point cloud, potentially erasing minute geometric features at the window edges, thus affecting the accuracy of subsequent boundary fine localization. This invention employs a strategy of randomly sampling candidate point clouds during the RANSAC fitting process. Specifically, when constructing the planar model, only a subset of sampling points are randomly selected for iteration, rather than traversing all points. This effectively reduces the computational cost per iteration while maintaining the complete point cloud resolution. This strategy balances computational efficiency and accuracy and is one of the key technical features of this invention.
[0080] In an alternative embodiment, depending on the computing power of the actual computing platform, an optional lightweight preprocessing step can be added between steps S3 and S4. For example, a statistical outlier removal (SOR) step can be performed on the local point cloud of the window frame, removing sporadic outliers by calculating the standard deviation of the average distance between each point and its neighboring points. This step serves as preprocessing for RANSAC fitting, further improving fitting efficiency and stability.
[0081] Step S5: Perform spatial coordinate calculation and boundary feature optimization.
[0082] Based on the 3D planar model obtained by fitting in step S4, the point cloud is projected onto the camera model and the projection result is mapped onto the 2D image plane. Morphological closing and dilation operations are performed on the 2D edge candidate regions. The dilated support regions are used to constrain the edge candidate pixels to obtain the 2D skeleton boundary of the window frame. The core corner points of the window frame are determined according to the 2D skeleton boundary and the corresponding 3D spatial points are calculated to generate the spatial pose of the window frame in the camera coordinate system. The 3D coordinates and spatial pose of each core corner point of the window frame are output to the downstream path planning and control module.
[0083] This step aims to combine the high-precision planar information fitted in 3D space with the edge detail information in the 2D image to achieve accurate positioning of the window frame boundary corner points in 3D space. The specific implementation process is as follows:
[0084] Step 1: 3D Point Cloud Projection. The local planar point cloud of the window frame, after outlier removal in step S4, is reprojected onto the 2D image plane according to the intrinsic parameter matrix of the RGB camera. After projection, each 3D point corresponds to a 2D pixel coordinate, thus establishing a mapping relationship between 3D spatial points and 2D image pixels.
[0085] Step 2: Morphological Processing and Edge Support Region Construction. Edge detection (such as Canny edge detection or Sobel operator) is performed on the original image to obtain candidate edge lines of the window frame. Due to factors such as glass surface reflection and noise, the originally detected edge lines often have problems such as breaks and gaps. To overcome this problem, this invention performs morphological closing and dilation operations on the two-dimensional edge candidate regions sequentially. Among them, the closing operation (dilation followed by erosion) is used to connect broken edge segments; on this basis, a separate dilation operation is further performed to expand the edge support region. Specifically, the edge candidate regions are dilated using square rectangular structural elements with a side length of 5 pixels; at the same time, the point cloud projection support region is dilated using rectangular structural elements with a width of 13 pixels and a height of 13 pixels. Through the above dilation operation, the effective support range after the three-dimensional point cloud is projected onto the two-dimensional plane is expanded, providing a wider pixel search space for subsequent edge screening.
[0086] Step 3: Edge Constraint Screening and 2D Skeleton Extraction. The candidate pixels for edge lines are constrained using the projected support region of the dilated point cloud. Only pixels that belong to both the candidate edge line region and fall within the projected support region are retained as candidate edge pixels. This constraint effectively eliminates false edges (such as texture edges inside the window frame or background edges outside the window frame) that are spatially inconsistent with the actual boundary of the window frame. Subsequently, horizontal and vertical edges are screened based on the directional features of the edge lines. Specifically, pixels with edges that are nearly horizontal (i.e., gradient direction is nearly vertical, with an angle less than 15° with the vertical direction) are classified as horizontal edges; pixels with edges that are nearly vertical (i.e., gradient direction is nearly horizontal, with an angle less than 15° with the horizontal direction) are classified as vertical edges. The 2D skeleton boundary of the window frame is obtained by fitting straight lines to the horizontal and vertical edges (e.g., using Hough transform or least squares method). This is the window frame outline formed by four mutually perpendicular straight line segments.
[0087] Step 4: Define the core corner points. Define the core corner points of the window frame in the 2D skeleton as three 2D feature points, including:
[0088] 1) The upper left corner of the window frame glass area (i.e., the inner corner of the upper left corner of the window frame);
[0089] 2) A horizontal reference point obtained by offsetting the top-left corner point 200 pixels along the horizontal edge;
[0090] 3) The vertical reference point is obtained by offsetting the top left corner point by 300 pixels along the vertical edge direction.
[0091] The reason for choosing three 2D feature points instead of four corner points is that in most practical scenarios, the top left corner, horizontal boundary, and vertical boundary of the window frame are the most stable and easiest to image clearly, while the bottom right corner is often difficult to detect accurately due to viewpoint occlusion or blurred imaging at a distance. Using a three-feature-point scheme can reduce the dependence on the completeness of all corner point detection while ensuring the accuracy of pose calculation, thus improving the robustness of the system.
[0092] Step 5: Precise 3D Spatial Point Location. For each of the three 2D feature points mentioned above, in the local planar point cloud of the window frame obtained in step S4, all 3D points are projected onto the 2D image plane according to the RGB camera intrinsic parameters. Within a circular search area with a radius of 25 pixels around the 2D feature point, the effective 3D point with the smallest projection distance (i.e., the smallest Euclidean distance between the 2D projected coordinates and the feature point coordinates) is selected as the corresponding 3D spatial point. Since step S4 has eliminated most outlier noise points, the 3D points in this search area are basically located on the real window frame plane. Therefore, the 3D point selected by the nearest projection distance can accurately reflect the actual 3D position of the window frame corner.
[0093] Step 6: Local Coordinate System and Spatial Pose Calculation for the Window Frame. Construct the local coordinate system of the window frame based on the three 3D spatial points mentioned above (P0 is the top-left corner point, P1 is the horizontal reference point, and P2 is the vertical reference point). Specifically:
[0094] Direction vector The horizontal axis direction of the window frame is determined (i.e., the horizontal direction of the window frame).
[0095] Direction vector The longitudinal axis direction of the window frame (i.e., the vertical direction of the window frame) is determined.
[0096] pass and cross product ( This gives us the direction of the normal axis of the window frame plane, which is the direction of the normal line of the window frame.
[0097] Preferably, when constructing the local coordinate system, the normal axis can be further constrained by the plane normal vector obtained in step S4 to ensure that the local coordinate system of the window frame is strictly consistent with the plane of the glass window.
[0098] right , , Normalization is performed to obtain three orthogonal unit vectors, and the Gram-Schmidt orthogonalization method is used for orthogonalization correction to ensure strict orthogonality between the three axes. This generates a 3×3 rotation matrix (composed of three orthogonal coordinate axes) of the window frame in the camera coordinate system and a 3×1 translation vector (i.e., the three-dimensional coordinates of P0), which constitutes the complete spatial pose of the window frame.
[0099] Step 7: Coordinate System Transformation and Output. The transformation between the camera coordinate system and the window cleaning robot's base coordinate system (or world coordinate system) is achieved using a pre-calibrated and stored fixed 4×4 extrinsic parameter matrix (homogeneous transformation matrix). This extrinsic parameter matrix describes the position offset of the camera coordinate system origin in the robot's base coordinate system and the orientation of the camera coordinate axes relative to the robot's base coordinate system. In this embodiment, this extrinsic parameter matrix is pre-stored in a configuration file (e.g., "bhc_0701.txt"), and is read and loaded from the configuration file when the system starts. This invention does not use an IMU (Inertial Measurement Unit) to estimate the camera pose in real time, but instead uses a fixed extrinsic parameter calibration method. The advantage of this is that it avoids the impact of accumulated IMU drift error on spatial positioning accuracy, and also reduces the computational complexity of the system.
[0100] Finally, the three-dimensional coordinates of each core corner point of the window frame in the robot base coordinate system (including the precise three-dimensional position of the four corner points in the robot base coordinate system) and their spatial pose (including the orientation information of the window frame plane) are output to the downstream path planning module and anti-collision control module in a predefined data format (such as ROS standard message format or custom structure), providing high-precision physical space constraints for the safe and autonomous operation of the window cleaning robot.
[0101] In a preferred embodiment, after step S5 is completed, a result visualization and diagnosis step may be further included: the calculated three-dimensional boundary corner points of the window frame are reprojected onto the original RGB image and a boundary box is drawn, which is displayed in real time on the monitoring interface of the window cleaning robot, so that the operator can remotely view the recognition effect of the system.
[0102] In another alternative embodiment, if the window cleaning robot is equipped with a multi-degree-of-freedom robotic arm or an adjustable-view gimbal, the data acquisition in step S1 can be adjusted to a multi-view acquisition mode according to the actual operation requirements. That is, the window cleaning robot collects data on the same window frame multiple times in different positions or postures, and the window frame pose information identified from each viewpoint is weighted and fused to further improve the accuracy and stability of spatial positioning.
[0103] Example 2, such as Figure 2As shown, the present invention also provides a visual recognition system for a window cleaning robot, used to perform the steps in the above method embodiments. This system is based on a lightweight modular architecture and includes the following functional units:
[0104] The data acquisition module controls the external depth camera of the window cleaning robot to synchronously acquire raw RGB image streams and depth matrix data of the working environment, ensuring spatiotemporal alignment of the two data streams. Specifically, this module includes an image acquisition submodule and a depth acquisition submodule, which drive the RGB camera and depth camera respectively, and bind the two data streams through a timestamp synchronization mechanism. The output of this module is the aligned RGB image and depth matrix data pair, which serves as the input data source for the entire system.
[0105] A two-dimensional segmentation module, connected to the data acquisition module, loads a pre-trained instance segmentation network, receives the raw RGB image input from the data acquisition module, performs efficient parallel inference, and outputs a coarse two-dimensional mask for the window region. This module runs on a GPU (such as NVIDIA Jetson Orin or Jetson Xavier NX) and utilizes CUDA acceleration for efficient tensor operations. In a preferred embodiment, the instance segmentation network is a YOLO11-seg model built on the PyTorch framework. This model is pre-trained on a self-built window scene dataset using transfer learning, and during inference, it can output the class, bounding box, and pixel-level instance segmentation mask of the window target at a rate of over 30 frames per second. This module fully utilizes the NPU (Neural Processing Unit) or GPU acceleration capabilities on edge computing platforms to achieve high frame rate real-time inference.
[0106] The local point cloud extraction module is connected to both the data acquisition module and the 2D segmentation module. It establishes a mapping between 2D pixels and 3D space, uses a coarse 2D window frame mask to crop the region of interest (ROI) of the global depth data, filters out most of the non-window frame background point cloud, and extracts the local 3D original point cloud. The implementation of this module relies on a depth-color alignment mapping table; the specific alignment method has been described in detail in step S3 of Example 1.
[0107] The geometric fitting optimization module, connected to the local point cloud extraction module, performs Random Sample Consensus (RANSAC) iterative calculations on the received local point cloud. This process removes depth interference noise caused by factors such as high reflectivity, refraction, and surface dust on the glass, fitting a spatially ideal plane equation representing the geometric features of the glass window surface. Internally, this module uses a lightweight matrix operation library to implement vector and matrix operations, without relying on heavyweight third-party mathematical libraries, thus reducing memory usage on the embedded platform. During RANSAC iteration, no more than 8000 points are randomly selected from the candidate point cloud as a sampling set. After 160 random iterations, the plane model with the most interior points is retained. A plane model is considered valid when the number of interior points is no less than 2500 and the proportion of interior points is no less than 12%.
[0108] The boundary coordinate calculation module, connected to the geometric fitting optimization module, is used to perform refined pixel dilation and coordinate correction on the window frame edges based on the fitted plane equations, calculating the 3D coordinate sequence of each boundary corner point of the window frame in the physical space coordinate system of the window cleaning robot. This module first projects the 3D planar point cloud onto the 2D image plane, obtains the 2D skeleton boundary of the window frame through morphological dilation and edge constraints, then constructs the local coordinate system of the window frame based on the nearest neighbor 3D mapping of three core feature points, and finally transforms the window frame spatial pose from the camera coordinate system to the robot base coordinate system through a pre-calibrated extrinsic parameter matrix, outputting the high-precision boundary features to the path planning and collision avoidance control system.
[0109] In a preferred embodiment, the five modules are integrated onto the edge computing platform of the window cleaning robot (e.g., an embedded motherboard equipped with an ARM Cortex-A series processor and a lightweight NPU). Data transmission between the modules occurs via inter-process communication (IPC) or shared memory to achieve an end-to-end low-latency visual perception link. The total latency of the entire system is controlled within 100 milliseconds, meeting the real-time operation requirements of the window cleaning robot.
[0110] This invention can be applied to various operational scenarios of high-altitude building glass curtain wall window cleaning robots.
[0111] Scenario 1: Standard rectangular window frame operation:
[0112] In a standard rectangular glass curtain wall scenario, the window cleaning robot adheres to the glass surface and uses an onboard depth camera to acquire RGB images and depth data of the curtain wall in front of it in real time. Following steps 1 to 5, the system calculates the precise 3D coordinates of the window frame boundary within the current field of view in real time on the robot's edge computing platform. The path planning module generates a fully covered, bow-shaped cleaning path based on the 3D coordinates of the four corner points, while the anti-collision module monitors the robot's position in real time using the boundary coordinates to ensure the robot does not overstep the boundary and fall. Actual testing shows that in a standard rectangular window frame scenario, the system's boundary positioning error is less than 5 mm, and the end-to-end processing latency is less than 100 milliseconds.
[0113] Scenario 2: Strong direct sunlight and highly reflective environment:
[0114] Under the intense sunlight of a sunny midday in summer, the glass curtain wall surface experiences strong specular reflection, making traditional 2D image processing methods prone to failure. In this system, under direct sunlight, the 2D instance segmentation model may experience mask positioning deviations due to image reflections. However, this deviation is primarily manifested at the pixel plane. The subsequent RANSAC plane fitting step, based on the geometric consistency of 3D space, removes outliers, effectively compensating for the positioning error of the 2D segmentation and fitting an accurate ideal plane for the glass window surface. Boundary optimization based on this plane model can correct the deviation of the 2D mask to the correct 3D spatial position. Experimental results show that under direct sunlight, the boundary positioning error of this system can still be controlled within 8 mm.
[0115] Scenario 3: Non-standard irregular window frames:
[0116] This system also exhibits excellent adaptability to non-standard irregular window frames such as curved, trapezoidal, or triangular shapes. The two-dimensional instance segmentation model can learn the visual features of irregular window frames and output corresponding masks, while RANSAC plane fitting accurately extracts the plane containing the window frame. The boundary coordinate calculation module can extract contour point sequences of arbitrary shapes based on the actual skeleton boundaries. This feature makes the system suitable not only for standard rectangular curtain walls but also for cleaning irregularly shaped windows on various building facades.
[0117] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A visual recognition method for a window cleaning robot, characterized in that: Includes the following steps: Step S1: Perform multi-source data acquisition and spatiotemporal synchronization initialization, control the depth vision sensor of the window cleaning robot to synchronously acquire the original RGB image in the current curtain wall field of view, and read the two-dimensional depth matrix data aligned with its timestamp; Step S2: Perform instance segmentation and coarse localization on the two-dimensional pixel plane. Input the original RGB image obtained in step S1 into the pre-trained instance segmentation model for forward inference and extract the coarse mask of the window region on the two-dimensional image plane. Step S3: Perform 3D point cloud fusion and extraction of the region of interest. Based on the intrinsic parameters of the depth camera, convert the 2D depth matrix data of step S1 into the overall spatial point cloud. Use the coarse mask generated in step S2 as a spatial filter to extract the depth data corresponding to the mask pixel area, and map the pixels corresponding to the depth data back to the 3D physical space to obtain the local original point cloud data of the window frame. Step S4: Perform spatial geometric plane fitting and noise filtering. The local original point cloud extracted in step S3 is fitted with a random sampling consensus algorithm. Sampling points are randomly selected from the candidate point cloud to construct a three-dimensional plane model. The interior points are determined by calculating the distance from each point to the fitted plane. After iteration, the three-dimensional plane model with the most interior points is retained. The plane parameters are refitted using the selected interior points, and outliers that are more than a preset threshold from the fitted plane are removed. Step S5: Perform spatial coordinate calculation and boundary feature optimization. Based on the 3D planar model fitted in step S4, project the point cloud onto the camera model and map the projection result onto the 2D image plane. Perform morphological closing operation and dilation processing on the 2D edge candidate region. Use the dilated support region to constrain the edge line candidate pixels to obtain the 2D skeleton boundary of the window frame. Determine the core corner points of the window frame according to the 2D skeleton boundary and calculate the corresponding 3D spatial points. Generate the spatial pose of the window frame in the camera coordinate system. Output the 3D coordinates and spatial pose of each core corner point of the window frame to the downstream path planning and control module.
2. The visual recognition method for a window cleaning robot according to claim 1, characterized in that: The timestamp alignment in step S1 specifically involves: caching the original RGB image and the two-dimensional depth matrix data respectively, performing approximate time matching based on the timestamps in their respective message headers, and determining the data pair as timestamp aligned when the time difference between the two does not exceed the preset time tolerance.
3. The visual recognition method for a window cleaning robot according to claim 1, characterized in that: In step S2, the instance segmentation model is a deep learning-based instance segmentation network. The model is trained using transfer learning, using pre-trained weights on a public instance segmentation dataset as initialization parameters, and fine-tuning the training by combining a self-built window frame scene dataset. The sample images in the self-built window frame scene dataset contain window frame targets under different angles, distances, lighting conditions, and occlusion conditions, and the targets in the sample images are labeled with categories and pixel-level contours. The training parameters of the instance segmentation model are set as follows: the number of training rounds is 100 to 300, the input image size is 640×640 pixels, the batch size is 8 to 32, the initial learning rate is 0.001 to 0.01, the optimizer is any one of SGD, Adam or AdamW, and the loss function includes target classification loss, bounding box regression loss and mask segmentation loss. During training, data augmentation processing is performed on the sample images, including brightness perturbation, scaling, translation, flipping, and noise perturbation.
4. The visual recognition method for a window cleaning robot according to claim 1, characterized in that: In step S3, the intrinsic parameters of the depth camera and the RGB camera include the horizontal focal length fx, the vertical focal length fy, and the principal point coordinates cx and cy. The depth data and the RGB image are aligned using a software reprojection method. Based on the intrinsic parameters of the depth camera and the RGB camera, as well as the rotation and translation parameters between the pre-calibrated depth sensor and the RGB camera, the three-dimensional points in the depth data are transformed to the RGB camera coordinate system and projected onto the RGB image pixel plane to establish the correspondence between the depth data and the RGB image pixels.
5. The visual recognition method for a window cleaning robot according to claim 1, characterized in that: In step S4, the specific process of plane fitting using the random sampling consensus algorithm is as follows: No more than 8000 points are randomly selected from the candidate point cloud as a sampling set. Three non-collinear points are randomly selected each time to construct a plane model. The plane model is generally expressed as "A×x+B×y+C×z+D=0", where A, B, and C represent the components of the plane normal vector, D represents the plane offset, and x, y, and z represent the coordinate values in three-dimensional space. Interior points are determined by calculating the distance from each point to the fitting plane, and points with a distance of no more than 0.10 meters are identified as in-plane points. After multiple random iterations, the plane model with the most interior points is retained. When the number of interior points is no less than 2500 and the proportion of interior points is no less than 12%, the plane model is deemed valid. The selected interior points are then used to refit the plane parameters. Outliers exceeding a preset threshold in distance from the fitting plane are then removed, and spatial connectivity is combined to perform regional filtering of in-plane points, obtaining the local plane point cloud where the window frame is located. In step S4, the input point cloud is not pre-sampled using voxels; instead, the candidate point cloud is randomly sampled only during the random sampling consistency fitting process.
6. The visual recognition method for a window cleaning robot according to claim 1, characterized in that: In step S5, the specific method for performing morphological closing operation and dilation processing on the two-dimensional edge candidate region is as follows: the edge line candidate region is dilated using a rectangular structural element with a side length of 5 pixels, and the point cloud projection support region is dilated using a rectangular structural element with a width of 13 pixels and a height of 13 pixels.
7. The visual recognition method for a window cleaning robot according to claim 1, characterized in that: In step S5, the core corner point of the window frame in the two-dimensional skeleton is defined as three feature points, including the upper left corner point of the window frame glass area, a horizontal reference point obtained by offsetting the upper left corner point 200 pixels along the horizontal edge direction, and a vertical reference point obtained by offsetting the upper left corner point 300 pixels along the vertical edge direction. For each two-dimensional feature point, the three-dimensional point in the point cloud is projected onto the image plane according to the camera intrinsic parameters, and the effective three-dimensional point with the smallest projection distance is selected as its corresponding three-dimensional spatial point within a search area with a radius of 25 pixels around the two-dimensional feature point.
8. The visual recognition method for a window cleaning robot according to claim 7, characterized in that: A local coordinate system for the window frame is constructed based on the three-dimensional spatial points corresponding to the three-dimensional feature points: the direction from the three-dimensional spatial point corresponding to the upper left corner point to the three-dimensional spatial point corresponding to the horizontal reference point is determined as the horizontal axis, and the direction from the three-dimensional spatial point corresponding to the upper left corner point to the three-dimensional spatial point corresponding to the vertical reference point is determined as the vertical axis. The normal axis of the window frame plane is obtained by the cross product of the two direction vectors. After normalizing and orthogonalizing each coordinate axis, the spatial pose of the window frame in the camera coordinate system is generated. The transformation between the camera coordinate system and the window cleaning robot's base coordinate system is achieved using a pre-calibrated and stored fixed four-by-four extrinsic parameter matrix.
9. A visual recognition system for a window cleaning robot, characterized in that: include: The data acquisition module is used to control the external depth camera of the window cleaning robot to synchronously acquire the raw RGB image stream and depth matrix data of the working environment, and to ensure that the data streams of the two are spatiotemporally aligned. The 2D segmentation module, connected to the data acquisition module, is used to load the instance segmentation network, receive the original RGB image input from the data acquisition module and perform parallel inference, and output a coarse 2D mask of the window area. The local point cloud extraction module is connected to the data acquisition module and the 2D segmentation module respectively. It is used to establish the mapping relationship between 2D pixels and 3D space, use a 2D window frame coarse mask to perform ROI cropping on the global depth data, filter out the background point cloud, and extract the local 3D original point cloud. The geometric fitting optimization module, connected to the local point cloud extraction module, is used to perform random sampling consistency iterative calculation on the received local point cloud, remove curtain wall interference noise, and fit the spatial ideal plane equation representing the geometric features of the glass window surface. The boundary coordinate calculation module, connected to the geometric fitting optimization module, is used to perform fine pixel dilation and coordinate correction on the window frame edge by combining the fitted plane equation, calculate the three-dimensional coordinate sequence of each boundary corner point of the window frame in the physical space coordinate system of the window cleaning robot, and output the high-precision boundary features.
10. A visual recognition system for a window cleaning robot according to claim 9, characterized in that: The instance segmentation network loaded in the two-dimensional segmentation module is the YOLO11-seg model built on the PyTorch framework. The model is pre-trained using transfer learning and outputs the category, bounding box, and instance segmentation mask of the window object.
