Window switch state detection method based on position coding and self-supervised feature fusion

Abstract

The application relates to the technical field of computer vision and intelligent monitoring, and discloses a window opening and closing state detection method based on position coding and self-supervised feature fusion. The method first extracts image frames from a monitoring video and manually labels four corner points of a target window, and cuts a region containing the target window and a surrounding background according to the labeled coordinates; then the cut image is input into a self-supervised visual model with frozen parameters to extract high-dimensional visual features, and the window corner point coordinates are normalized and a spatial position vector is generated; finally, the visual features and the position features are fused through a cross-attention mechanism, the fused comprehensive features are input into a classification module to obtain a prediction result of the window opening and closing state. The application reduces the labeling cost by using the strong general feature representation capability of the self-supervised model, and introduces position coding to supplement spatial information, effectively improving the precision and generalization capability of the window opening and closing state detection in a complex scene.

Description

Technical Field

[0001] This application relates to the technical field of computer vision and intelligent monitoring, and in particular to a window opening / closing status detection method based on the fusion of location coding and self-supervised features. Background Technology

[0002] With the rapid development of smart cities and intelligent buildings, the demand for automated monitoring of the internal and external environments of buildings is increasing. Windows, as a crucial component of building ventilation and lighting, directly affect indoor temperature, humidity, air quality, as well as building energy consumption and safety. For example, windows left open during heavy rain may cause water to enter the building; windows open while air conditioning is running increase energy consumption; and open windows in unattended areas can pose safety hazards. Therefore, the automatic detection of window open / closed status has significant practical application value.

[0003] Currently, window open / closed status detection methods are mainly divided into two categories: traditional image processing methods and deep learning-based methods. Traditional image processing methods typically rely on techniques such as edge detection, template matching, and color segmentation to extract the window's contour features to determine its open / closed status. However, these methods are highly sensitive to interference factors such as changes in lighting, viewing angle, and occlusion, exhibiting poor robustness and making them difficult to apply in complex real-world scenarios.

[0004] Deep learning-based methods automatically learn feature representations of windows by training neural networks, offering higher detection accuracy and robustness compared to traditional methods. However, most existing deep learning methods rely on supervised learning, requiring large amounts of manually labeled training data, resulting in high labeling costs and long processing times. Furthermore, existing methods typically utilize only content features of the image, ignoring the spatial location information of the window within the image. In real-world surveillance scenarios, multiple similar windows often exist, or windows are partially obscured; relying solely on content features makes it difficult to accurately distinguish different windows and determine their open / closed status, leading to decreased detection accuracy. Simultaneously, existing methods have weak generalization ability across different buildings and window styles. When applied to new scenarios, it is necessary to re-collect data and retrain the model, increasing deployment costs and complexity. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a window opening / closing state detection method based on the fusion of position encoding and self-supervised features, employing the following technical solution: A window opening / closing state detection method based on the fusion of position encoding and self-supervised features includes the following steps: S1: Extract image frames from surveillance video; S2: Initialize the four corner points of the labeled target window in the extracted image frame at once, and record the pixel coordinates of each corner point in the original image; S3: Crop the image area according to the size of the target window, so that the cropped area includes the target window and the surrounding background, and recalculate the relative coordinates of the four corner points in the cropped image; S4: Input the cropped image into a visual model trained with large-scale data and whose parameters are frozen, and extract high-dimensional visual features. The parameters of the self-supervised visual model are kept frozen throughout the subsequent training and inference process. S5: Normalize the relative coordinates of the four corner points, input the normalized coordinates into the position encoding module composed of multilayer perception to generate spatial position vectors, and fuse the visual features as query vectors, spatial position vectors as key vectors and value vectors through cross-attention mechanism to obtain comprehensive features; S6: Input the comprehensive features into the state classification module and output the prediction results of the window opening and closing state; The parameters of the location encoding module and the state classification module are obtained through training with a small number of labeled samples, and the parameters of the self-supervised visual model are kept frozen during the training process.

[0006] Optionally, in step S3, the specific method for cropping the image region is as follows: taking the rectangle determined by the four corner points of the target window as a reference, extend the rectangle in the four directions of up, down, left, and right by a distance equal to the corresponding side length of the target window to form the cropping region.

[0007] Optionally, in step S4, the self-supervised visual model is the DINOv3 model.

[0008] Optionally, in step S5, the formula for normalizing the relative coordinates of the four corner points is: ; in, Let be the relative coordinates of the i-th corner point in the cropped image. To crop the width of the image, To crop the height of the image, The coordinates are normalized. These correspond to the four corner points: top left, top right, bottom left, and bottom right, respectively.

[0009] Optionally, in step S5, the position encoding module is a multilayer perceptron (MLP), whose input is an 8-dimensional normalized coordinate vector and whose output is a spatial position vector with the same dimension as the visual feature.

[0010] Optionally, in step S5, the feature fusion method is a cross-attention mechanism, specifically: using visual features as query vectors, spatial location vectors as key vectors and value vectors, calculating attention weights and performing weighted summation to obtain the fused comprehensive features.

[0011] Optionally, the calculation formula for cross-attention mechanisms is as follows: ; Where Q is the query vector, corresponding to the visual feature; K is the key vector, corresponding to the spatial location vector; and V is the value vector, corresponding to the spatial location vector. is the dimension of the key vector.

[0012] Optionally, in step S6, the state classification module is a fully connected layer, whose input is comprehensive features and output is a two-dimensional vector, which corresponds to the probability of the window being open and closed, respectively.

[0013] Optionally, it also includes a model training step: using the cross-entropy loss function as the training loss, and using a stochastic gradient descent optimizer to train the parameters of the position encoding module and the state classification module, while keeping the parameters of the self-supervised visual model unchanged during the training process.

[0014] Optionally, in step S1, the frame rate for extracting image frames from the surveillance video is 1 frame / second.

[0015] In summary, this application includes at least the following beneficial technical effects: This application employs a visual model trained on large-scale self-supervised data to extract features. Leveraging its powerful general feature representation capabilities, the model requires only a small amount of labeled data for training, significantly reducing labeling costs and improving its generalization ability across different scenarios. By introducing a position encoding module, the spatial location information of windows is transformed into a learnable vector representation and fused with visual features. This allows the model to simultaneously utilize image content and spatial location information for judgment, effectively solving the problems of difficulty in distinguishing multiple similar windows in complex scenes and decreased detection accuracy due to partial occlusion. During training, the self-supervised model parameters are frozen, and only the position encoding and classification modules are trained, greatly accelerating the model's convergence speed, reducing computational resource consumption, and facilitating deployment in practical monitoring systems. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the window switch status detection method in this application. Detailed Implementation

[0017] The embodiments of this application are described in detail below, and examples of the embodiments are shown in the accompanying drawings.

[0018] In the description of this specification, the references to "certain embodiments," "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples" refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0019] This application discloses a window opening / closing status detection method based on the fusion of location coding and self-supervised features. This method can be applied to fields such as intelligent buildings, security monitoring, and energy management, enabling automated and real-time monitoring of the window opening / closing status of buildings. (Refer to...) Figure 1 This includes the following steps: S1: Extract image frames from the surveillance video; acquire real-time surveillance video streams from surveillance cameras deployed outside or inside the building, and extract image frames from the video stream according to a preset frame rate. In this embodiment, the extraction frame rate is set to 1 frame / second. This frame rate ensures timely response to changes in window status while avoiding processing excessive redundant data and reducing computational burden. The extracted image frames have a resolution of 1920×1080 pixels and are in RGB format.

[0020] S2: Initialize the four corner points of the target window in the extracted image frames at once, and record the pixel coordinates of each corner point in the original image; in the extracted original image frames, manually label the four corner points of the target window to be monitored using an image annotation tool, namely the upper left corner, upper right corner, lower left corner, and lower right corner, and record the pixel coordinates of each corner point in the original image. ,in Each of the four corner points corresponds to one of the four corner points. For scenarios that require monitoring multiple windows, the four corner points of each target window are marked sequentially.

[0021] S3: Crop the image region according to the size of the target window, ensuring the cropped region includes the target window and surrounding background, and recalculate the relative coordinates of the four corner points in the cropped image; calculate the width of the target window based on the coordinates of the marked four corner points. and height ,in , Using the rectangle defined by the four corner points of the target window as a reference, extend the rectangle above it. The distance, extending below The distance, extending to the left The distance, extending to the right The distance is used to form the cropping region. The resulting image not only contains the complete target window, but also retains information about other windows and the environment around the target window as a reference background, which helps improve the model's ability to recognize the target window.

[0022] After cropping, recalculate the relative coordinates of the four corner points in the cropped image. The calculation formula is: , ;in, This represents the x-coordinate of the top-left corner of the cropped region in the original image. This represents the ordinate of the top-left corner of the cropped area in the original image.

[0023] S4: The cropped image is input into a self-supervised visual model trained on large-scale data with its parameters frozen to extract high-dimensional visual features. The parameters of the self-supervised visual model remain frozen throughout subsequent training and inference. Specifically, the cropped window region image is resized to a fixed size of 224×224 pixels and then input into the DINOv3 model trained on large-scale data to extract high-dimensional visual features. The DINOv3 model is a self-supervised visual model based on the Transformer architecture. Through comparative learning on massive amounts of unlabeled images, it learns rich and general image feature representations, possessing strong transfer capabilities and scene adaptability, and is less prone to overfitting to task-specific data. During model training, all parameters of the DINOv3 model are kept completely frozen, and only the subsequent position encoding and state classification modules are trained. This fully utilizes the strong feature representation capabilities of the DINOv3 model while significantly reducing the number of parameters that need to be trained, accelerating the convergence speed of the task model, and reducing computational resource consumption. The visual feature dimension output by the DINOv3 model is 768 dimensions.

[0024] S5: Normalize the relative coordinates of the four corner points, input the normalized coordinates into the position encoding module composed of multilayer perception to generate spatial position vectors, and fuse the visual features as query vectors, spatial position vectors as key vectors and value vectors through cross-attention mechanism to obtain comprehensive features; First, the relative coordinates of the four corner points in the cropped image are normalized, mapping the coordinate values ​​to the range [0,1]. The normalization formula is as follows: ; in, To crop the width of the image, To crop the height of the image, These are the normalized coordinates. The normalized coordinates of the four corner points are concatenated in order to form an 8-dimensional coordinate vector. .

[0025] An 8-dimensional coordinate vector is input into the position encoding module to generate a spatial position vector. In this embodiment, the position encoding module uses a three-layer multilayer perceptron (MLP), with the following structure: an input layer of 8 neurons, a hidden layer of 256 neurons using ReLU activation, and an output layer of 768 neurons, matching the dimensionality of the visual features. The position encoding module can transform discrete coordinate values ​​into continuous, learnable vector representations, thereby incorporating the window's spatial position information into the model.

[0026] Then, a cross-attention mechanism is used to fuse visual features with spatial location vectors. Specifically, the 768-dimensional visual features output by the DINOv3 model are used as the query vector Q, and the 768-dimensional spatial location vector output by the location encoding module are used as the key vector K and value vector V. Attention weights are calculated and weighted summation is performed to obtain a comprehensive feature that simultaneously contains image content information and spatial location information. The calculation formula for the cross-attention mechanism is as follows: ; Where is the dimension of the key vector, in this embodiment... =768. Through a cross-attention mechanism, the model can adaptively learn the correlation between visual features and positional features, thereby more accurately determining the open / closed state of a window.

[0027] S6: Input the comprehensive features into the state classification module and output the prediction result of the window opening / closing state. The parameters of the position encoding module and the state classification module are obtained through training with a small number of labeled samples, and the parameters of the self-supervised visual model are kept frozen during training. Specifically, in this embodiment, the fused 768-dimensional comprehensive features are input to the state classification module, which outputs the prediction result of the window's open / closed state. The state classification module uses a fully connected layer with an input dimension of 768 and an output dimension of 2, corresponding to the probabilities of the window's open and closed states, respectively. The output two-dimensional vector is normalized using the Softmax function to obtain the probability values ​​of the two states, and the state with the larger probability value is taken as the final prediction result.

[0028] During the model training phase, the cross-entropy loss function is used as the training loss. The formula for calculating the cross-entropy loss function is as follows: ; in, For true labels, when the window is in state c. =1, otherwise =0; Let be the probability of the c-th state predicted by the model. The parameters of the position encoding module and the state classification module are trained using a stochastic gradient descent (SGD) optimizer with a learning rate of 0.001, momentum of 0.9, batch size of 32, and 50 training epochs. The parameters of the DINOv3 model remain unchanged during training.

[0029] Once training is complete, the model can be deployed to a real-world monitoring system to detect the opening and closing status of windows in real time. For the input monitoring video, the above steps are followed sequentially to perform frame extraction, region cropping, feature extraction, feature fusion, and state classification, ultimately outputting the opening and closing status of each target window.

[0030] Experimental example: Comparative test of this method with existing methods: The test was conducted on a dataset containing 50 different surveillance scenarios (covering different building types, lighting conditions, and occlusion levels). Only 10 labeled images were used for each scenario to train the location encoding and classification modules. The test results are as follows: In particular, on test samples where windows partially obscured more than 40%, the accuracy of DINOv3+ linear classification dropped to 65.3%, while the accuracy of our method remained at 87.2%, demonstrating that spatial location coding provides location priors and effectively compensates for the lack of information in visual features.

[0031] This application employs a visual model trained on large-scale self-supervised data to extract features. Leveraging its powerful general feature representation capabilities, the model requires only a small amount of labeled data for training, significantly reducing labeling costs and improving its generalization ability across different scenarios. By introducing a position encoding module, the spatial location information of windows is transformed into a learnable vector representation and fused with visual features. This allows the model to simultaneously utilize image content and spatial location information for judgment, effectively solving the problems of difficulty in distinguishing multiple similar windows in complex scenes and decreased detection accuracy due to partial occlusion. During training, the self-supervised model parameters are frozen, and only the position encoding and classification modules are trained, greatly accelerating the model's convergence speed, reducing computational resource consumption, and facilitating deployment in practical monitoring systems.

[0032] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A window opening / closing state detection method based on the fusion of position encoding and self-supervised features, characterized in that, Includes the following steps: S1: Extract image frames from surveillance video; S2: Initialize the four corner points of the labeled target window in the extracted image frame at once, and record the pixel coordinates of each corner point in the original image; S3: Crop the image area according to the size of the target window, so that the cropped area includes the target window and the surrounding background, and recalculate the relative coordinates of the four corner points in the cropped image; S4: Input the cropped image into a visual model that has been trained with large-scale data and whose parameters are frozen, and extract high-dimensional visual features. The parameters of the self-supervised visual model are kept frozen throughout the subsequent training and inference process. S5: Normalize the relative coordinates of the four corner points, input the normalized coordinates into the position encoding module composed of multilayer perception to generate spatial position vectors, and fuse the visual features as query vectors, spatial position vectors as key vectors and value vectors through cross-attention mechanism to obtain comprehensive features; S6: Input the comprehensive features into the state classification module and output the prediction results of the window opening and closing state; The parameters of the location encoding module and the state classification module are obtained through training with a small number of labeled samples, and the parameters of the self-supervised visual model are kept frozen during the training process.

2. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S3, the specific method for cropping the image area is as follows: using the rectangle determined by the four corner points of the target window as a reference, extend the rectangle in the four directions of up, down, left, and right by a distance equal to the corresponding side length of the target window to form the cropping area.

3. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S4, the self-supervised visual model is the DINOv3 model.

4. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S5, the formula for normalizing the relative coordinates of the four corner points is as follows: ; in, Let be the relative coordinates of the i-th corner point in the cropped image. To crop the width of the image, To crop the height of the image, The coordinates are normalized. These correspond to the four corner points: top left, top right, bottom left, and bottom right, respectively.

5. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S5, the position encoding module is a multilayer perceptron (MLP), whose input is an 8-dimensional normalized coordinate vector and whose output is a spatial position vector with the same dimension as the visual feature.

6. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S5, the feature fusion method is a cross-attention mechanism, specifically: visual features are used as query vectors, spatial location vectors are used as key vectors and value vectors, attention weights are calculated and weighted summation is performed to obtain the fused comprehensive features.

7. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 6, characterized in that, The calculation formula for cross-attention mechanism is as follows: ; Where Q is the query vector, corresponding to the visual feature; K is the key vector, corresponding to the spatial location vector; and V is the value vector, corresponding to the spatial location vector. is the dimension of the key vector.

8. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S6, the state classification module is a fully connected layer. Its input is comprehensive features, and its output is a two-dimensional vector, which corresponds to the probability of the window being open and closed, respectively.

9. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, It also includes model training steps: using the cross-entropy loss function as the training loss, and using the stochastic gradient descent optimizer to train the parameters of the position encoding module and the state classification module. During the training process, the parameters of the self-supervised visual model remain unchanged.

10. The window opening / closing state detection method based on position encoding and self-supervised feature fusion according to claim 1, characterized in that, In step S1, the frame rate for extracting image frames from the surveillance video is 1 frame / second.

Images