A window state detection method based on a learnable multi-modal fusion gating mechanism

Abstract

This invention discloses a window state detection method based on a learnable multimodal fusion gating mechanism. The invention relates to the field of state detection technology and includes: acquiring time-synchronized and spatially aligned RGB video frames and IR images using a camera device; generating a window region mask in the RGB video frames by user-annotated window corner points; cropping the RGB video frames and IR images to obtain RGB sub-images Img_rgb and IR sub-images Img_ir; and introducing a learnable gating fusion model FGM in the dual-modal input stage to achieve adaptive weighted fusion of RGB modal features Frgb and infrared modal features Fir. The learnable parameter α is automatically optimized during network training and constrained by a Sigmoid function, enabling the model to dynamically adjust modal weights according to lighting conditions and infrared signal quality. This maintains the stability and accuracy of window state recognition results in complex environments such as strong light, reflection, or low illumination, effectively improving the system's adaptability to different environmental scenarios.

Description

Technical Field

[0001] This invention relates to the field of state detection technology, specifically to a window state detection method based on a learnable multimodal fusion gating mechanism. Background Technology

[0002] With the rapid development of intelligent buildings and energy-saving control systems, windows, as important components affecting energy consumption, air circulation, and security monitoring in the building environment, have gradually become an important research direction for indoor environmental intelligent sensing systems. Traditional window status detection mainly relies on mechanical sensors or reed switches, but these methods are costly to install and maintain, have complex wiring, and are easily affected by environmental factors, resulting in false alarms or missed alarms. In recent years, computer vision-based detection technology has begun to be applied to the field of window status recognition. By acquiring window images through cameras and using deep learning models for image classification, non-contact status judgment can be achieved. However, single-modal visible light images are prone to recognition errors in environments with strong reflections, backlighting, and low illumination at night, leading to insufficient detection accuracy. To solve this problem, researchers have gradually introduced infrared imaging technology and multimodal fusion strategies, combining the robust thermal radiation information of infrared modal with the structural texture features of RGB images to improve the model's robustness under complex lighting conditions. A multimodal window status detection method based on a learnable gating mechanism has been developed, which can achieve accurate window opening and closing recognition under multiple lighting and scene conditions, which is of great significance for improving the level of building intelligence and security management.

[0003] Existing window detection boxes tend to include a large number of irrelevant regions (background, parts of other windows) when extracting features. In scenarios with multiple windows nearby, glass reflection, or changes in lighting, feature confusion is severe, affecting classification accuracy. In addition, existing window detection boxes are easily affected by strong environmental background interference when classifying directly based on full-image features, making it difficult to identify the state of a specific window and resulting in poor performance in multi-window scenarios.

[0004] To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a window state detection method based on a learnable multimodal fusion gating mechanism.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a window state detection method based on a learnable multimodal fusion gating mechanism, comprising:

[0007] S1. Use a camera device to acquire time-synchronized and spatially aligned RGB video frames and IR images. By having the user annotate the window corner points, generate a window area mask in the RGB video frame.

[0008] S2. Crop the RGB video frame and the IR image to obtain the RGB sub-image Img_rgb and the IR sub-image Img_ir;

[0009] S3. Input the RGB sub-image Img_rgb into the first encoder network Enc_rgb to extract the RGB modal feature representation Frgb; input the IR sub-image Img_ir into the second encoder network Enc_ir to extract the infrared modal feature representation Fir;

[0010] S4. Introduce a learnable parameter α, constrain it to the range of 0 to 1 by the sigmoid function, and perform weighted fusion of the feature representation Frgb and the infrared modal feature representation Fir to obtain the fused feature Ffused.

[0011] S5. Input the fused features into the classification network and output the window state category.

[0012] The camera device includes an RGB_cam light acquisition unit and an IR_cam infrared acquisition unit. The RGB_cam and the IR_cam achieve frame-level alignment through a time synchronization module (TSM) and a spatial calibration module (SCM). The TSM controls the sampling clock signals of the two modes to be consistent, and the SCM calculates the extrinsic and intrinsic parameters between the two modes, completing the alignment mapping of the image space through a geometric transformation matrix (T_cal).

[0013] The process of aligning and mapping the image space using the geometric transformation matrix T_cal includes:

[0014] S11. For images of the same calibration target, extract the set of feature points from each image.

[0015] S12. Perform matching calculations based on the set of feature points to obtain the spatial mapping relationship from the RGB coordinate system to the IR coordinate system;

[0016] S13. Solve for the extrinsic matrix R_ext and the translation vector T_ext using the least squares optimization algorithm, and construct the geometric transformation matrix T_cal by combining their respective intrinsic parameters K_rgb and K_ir.

[0017] S14. The pixel coordinates of the RGB video frame are homogeneously mapped using the geometric transformation matrix T_cal to obtain the pixel projection position in the infrared coordinate system, thereby achieving spatial alignment between the RGB video frame and the IR image.

[0018] Perform a cropping process on the RGB video frame and the IR image, including:

[0019] S21. Extract the coordinates of four vertices from the user-annotated set of corner coordinates, set C_set={(x1,y1),(x2,y2),(x3,y3),(x4,y4)}.

[0020] S22. Calculate the center position parameters of the window area based on the corner coordinates:

[0021] S23. Calculate the width parameter Wth and height parameter Hth of the window area;

[0022] S24. Verify the effective pixel distribution of the window area based on the window area mask, and perform morphological dilation operation at the edge of the mask;

[0023] S25. Using the center position parameter Ctr as the center point, and defining the cropping area according to the width parameter Wth and height parameter Hth, perform synchronous cropping on the RGB video frame and IR image.

[0024] S3 includes:

[0025] In the RGB video frame feature extraction branch, the RGB sub-image RGB_sub obtained by synchronous cropping is concatenated with the corresponding mask Mask in the channel direction to form a multi-channel input tensor T_rgb=Concat(RGB_sub,Mask), which is then input to the convolutional encoder network CNN_rgb.

[0026] The convolutional encoder CNN_rgb includes a feature extraction stack structure consisting of several levels of convolutional layers (Conv), batch normalization layers (BN), and nonlinear activation function layers (ReLU). It extracts low-level texture edge features and mid-level semantic structure features of the window region and outputs RGB modal feature representation Frgb.

[0027] S3 further includes:

[0028] In the IR image feature extraction branch, the infrared image IR_sub obtained by synchronous cropping is concatenated with the corresponding mask Mask by channel to form the input tensor T_ir=Concat(IR_sub,Mask), which is then input to the lightweight encoder network CNN_ir.

[0029] The lightweight encoder CNN_ir consists of a depthwise separable convolutional layer (DConv) and a pointwise convolutional layer (PConv), which extracts the thermal distribution features and illumination compensation features in the infrared mode and outputs the infrared mode feature representation Fir.

[0030] The fusion model FGM outputs a learnable parameter α, which includes an input layer, multiple hidden layers, and an output layer. The input layer receives a concatenated feature vector from the RGB modal feature Frgb and the IR modal feature Fir. The hidden layers include a first hidden layer, a second hidden layer, and a third hidden layer, containing 128, 256, and 128 neural nodes, respectively. Each hidden layer uses the ReLU activation function for nonlinear mapping, and a BatchNormalization normalization layer is set between the second and third hidden layers to stabilize the training process. A Dropout layer is set after the third hidden layer to prevent overfitting, and the Dropout rate is set to 0.3. The output layer uses the Sigmoid activation function to generate the learnable parameter α. The learnable parameter α performs a weighted fusion of the feature representation Frgb and the infrared modal feature representation Fir to obtain the fused feature Ffused.

[0031] The optimization process of the learnable parameter α:

[0032] Using the window state classification error as the loss function, an adaptive moment estimation optimizer is employed to perform gradient descent iterations on the weight parameters in the learnable gated fusion model (FGM). The learnable parameter α is nonlinearly mapped in each forward propagation by the output layer's linear transformation result z into the Sigmoid function, with the specific calculation formula as follows:

[0033]

[0034] Where z is the linear weighted result of the output layer of the fusion model FGM, e is the natural constant, and the value of α is restricted to between 0 and 1.

[0035] The fused features F_fused are input into the classification network Clsnet, which includes a feature compression module and a multilayer perceptron (MLP) structure. Feature abstraction is achieved through multilayer nonlinear mapping, and a Softmax classification layer is set at the output of the MLP structure. The class probability vector P is calculated according to the following formula: when the probability value corresponding to the open state in the class probability vector P is greater than the preset threshold THR, the output window is in the open label; otherwise, the output window is in the closed label.

[0036] This invention provides a window state detection method based on a learnable multimodal fusion gating mechanism. Compared with existing technologies, it has the following advantages:

[0037] This invention introduces a learnable gated fusion model (FGM) in the dual-modal input stage to achieve adaptive weighted fusion of RGB modal features (Frgb) and infrared modal features (Fir). The learnable parameter α is automatically optimized during network training and constrained by the Sigmoid function, enabling the model to dynamically adjust modal weights according to lighting conditions and infrared signal quality. This maintains the stability and accuracy of window state recognition results in complex environments such as strong light, reflection, or low illumination, effectively improving the system's adaptability to different environmental scenarios.

[0038] This invention constructs a window state classification network Cls_net, which includes a feature compression module Cmp and a multilayer perceptron structure MLP. It performs hierarchical feature abstraction and classification probability inference on the fused features F_fused, and combines the output of the Softmax classification layer to show the open and closed state labels of the window, thereby achieving high-precision window state recognition and discrimination. This recognition mechanism can effectively improve the intelligence level of building energy consumption monitoring and safety management, and reduce energy consumption loss and safety risks caused by misjudgment.

[0039] This invention sets up a Softmax classification layer at the output of the MLP and combines it with a threshold parameter THR (threshold parameter thr) for decision boundary control, making the probability outputs of window opening and closing adjustable. The system can flexibly adjust the threshold THR according to actual environmental needs, further enhancing the reliability and stability of the model in multi-scenario deployment. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the principle framework of the present invention;

[0041] Figure 2 This is a schematic diagram of the principle framework of the internal camera device of the present invention;

[0042] Figure 3 This is a schematic diagram illustrating the principle framework of the internal classification network Clsnet of this invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please see Figure 1 This application provides a window state detection method based on a learnable multimodal fusion gating mechanism, including:

[0045] S1. Use a camera device to acquire time-synchronized and spatially aligned RGB video frames and IR images. By having the user annotate the window corner points, generate a window area mask in the RGB video frame.

[0046] S2. Crop the RGB video frame and IR image to obtain the RGB sub-image Img_rgb and the IR sub-image Img_ir;

[0047] S3. Input the RGB sub-image Img_rgb into the first encoder network Enc_rgb to extract the RGB modal feature representation Frgb; input the IR sub-image Img_ir into the second encoder network Enc_ir to extract the infrared modal feature representation Fir.

[0048] S4. Introduce a learnable parameter α, constrain it to the range of 0 to 1 by the sigmoid function, and perform weighted fusion of the feature representation Frgb and the infrared modal feature representation Fir to obtain the fused feature Ffused.

[0049] S5. Input the fused features into the classification network and output the window state category.

[0050] The camera device includes an RGB_cam light acquisition unit and an IR_cam infrared acquisition unit. The RGB_cam and IR_cam achieve frame-level alignment of sampling through a time synchronization module (TSM) and a spatial calibration module (SCM). The TSM controls the sampling clock signals of the two modes to be consistent, and the SCM calculates the extrinsic and intrinsic parameters between the two modes. The alignment mapping of the image space is completed through the geometric transformation matrix (T_cal).

[0051] In this embodiment, the camera device consists of an integrated dual-mode sensing component comprised of an RGB_cam light acquisition unit and an IR_cam infrared acquisition unit. The RGB_cam is used to acquire visible light video frames, obtaining color, texture, and geometric contour information of the window area; the IR_cam is used to acquire infrared thermal radiation images from the corresponding viewpoint to reflect the temperature distribution and energy transmission differences on the window surface. To ensure the temporal and spatial correspondence between the two modal images, the system is equipped with a time synchronization module (TSM) and a spatial calibration module (SCM). The TSM includes a master clock signal source (CLK_gen) and a synchronization trigger controller (Sync_ctl). The master clock signal source (CLK_gen) generates a unified timing signal and distributes it to the sampling control ports of the RGB_cam and IR_cam via the synchronization trigger controller (Sync_ctl), thereby ensuring that the two acquisition units complete image exposure and data reading at the same time, achieving time alignment at the sampling frame level.

[0052] The Spatial Calibration Module (SCM) comprises a calibration target (Cal_tar), a feature extraction unit (Feat_ext), a parameter solving unit (Param_sol), and a geometric mapping unit (Geo_map). Specifically, the system first acquires multiple sets of corresponding feature point images from both camera channels using the calibration target (Cal_tar). The feature extraction unit (Feat_ext) then uses corner detection and sub-pixel optimization algorithms to extract the coordinate set of the calibration points. Subsequently, the parameter solving unit (Param_sol) performs dual-target calibration calculations based on the feature point coordinates, using least-squares optimization to solve for the intrinsic parameter matrix K_rgb of the RGB channel, the intrinsic parameter matrix K_ir of the infrared channel, and the extrinsic parameter rotation matrix R_ext and translation vector T_ext between them, thereby constructing the geometric transformation matrix T_cal. Finally, the geometric mapping unit (Geo_map) uses the geometric transformation matrix T_cal to project the coordinates of each pixel acquired by the RGB_cam onto the IR_cam coordinate system, achieving pixel-level alignment of the two modal images in space.

[0053] The process of aligning and mapping the image space using the geometric transformation matrix T_cal includes:

[0054] S11. For images of the same calibration target, extract the set of feature points from each image.

[0055] S12. Perform matching calculations based on the feature point set to obtain the spatial mapping relationship from the RGB coordinate system to the IR coordinate system;

[0056] S13. Solve for the extrinsic matrix R_ext and the translation vector T_ext using the least squares optimization algorithm, and construct the geometric transformation matrix T_cal by combining their respective intrinsic parameters K_rgb and K_ir.

[0057] S14. Perform homogeneous coordinate mapping on the pixel coordinates of the RGB video frame through the geometric transformation matrix T_cal to obtain the pixel projection position in the infrared coordinate system, thereby achieving spatial alignment between the RGB video frame and the IR image.

[0058] In this embodiment, a calibration target Cal_tar (e.g., a checkerboard or dot array) with high-contrast feature points is selected. The target is simultaneously captured by the light acquisition unit RGB_cam and the infrared acquisition unit IR_cam to obtain calibration images of RGB and IR respectively. Then, the feature extraction module Feat_ext is called to detect key features such as corner points and edge intersections in the RGB and IR images respectively, forming the RGB feature point set Set_rgb and the IR feature point set Set_ir.

[0059] The process of cropping RGB video frames and IR images includes:

[0060] S21. Extract the coordinates of four vertices from the user-annotated set of corner coordinates, set C_set={(x1,y1),(x2,y2),(x3,y3),(x4,y4)}.

[0061] S22. Calculate the center position parameters of the window area based on the corner coordinates:

[0062] S23. Calculate the width parameter Wth and height parameter Hth of the window area;

[0063] S24. Verify the effective pixel distribution of the window region based on the window region mask, and perform morphological dilation operation at the edge of the mask;

[0064] S25. Using the center position parameter Ctr as the center point, define the cropping area according to the width parameter Wth and the height parameter Hth, and perform synchronous cropping on the RGB video frame and IR image.

[0065] In this embodiment, the center position parameter Ctr is used as the cropping center. The cropping rectangular region ROI = (Ctr_x, Ctr_y, Wth, Hth) is determined based on the width parameter Wth and the height parameter Hth, and synchronous cropping is performed on the RGB video frame and the IR image. During the cropping process, the region coordinates are transformed by the geometric mapping matrix T_cal of the spatial calibration module SCM to ensure that the corresponding regions in the RGB and IR images are completely aligned in space. After cropping, the RGB sub-image Img_rgb and the IR sub-image Img_ir are output respectively, providing a spatially consistent input basis for subsequent modal feature extraction and gating fusion.

[0066] S3 includes:

[0067] In the RGB video frame feature extraction branch, the RGB sub-image RGB_sub obtained by synchronous cropping is concatenated with the corresponding binarized mask Mask in the channel direction to form a multi-channel input tensor T_rgb=Concat(RGB_sub,Mask), which is then input to the convolutional encoder network CNN_rgb.

[0068] The convolutional encoder CNN_rgb consists of a feature extraction stack structure composed of several levels of convolutional layers (Conv), batch normalization layers (BN), and nonlinear activation function layers (ReLU). It extracts low-level texture edge features and mid-level semantic structure features of the window region and outputs the RGB modal feature representation Frgb.

[0069] In this embodiment, the structure of the convolutional encoder CNN_rgb is designed as follows:

[0070] The first convolutional layer, Conv1, uses a 3×3 convolutional kernel with a stride of 1 to extract window edge and texture detail features.

[0071] The second convolutional layer Conv2 and the third convolutional layer Conv3 use 5×5 convolutional kernels, and after each convolutional layer, a batch normalization layer (BN) and an activation function layer (ReLU) are added to stabilize the gradient distribution and accelerate network convergence.

[0072] After every two convolutional layers, a downsampling operation is performed through a max pooling layer (MaxPool) to reduce the spatial size of the feature map while preserving the main structural information.

[0073] The last two convolutional layers, Conv4 and Conv5, are used to extract semantic structural features from the middle layers, including semantic information such as window frame boundaries, glass reflection areas, and local texture changes.

[0074] Throughout the feature extraction process, each convolutional module employs the non-linear activation function ReLU to achieve non-linear mapping of the feature space; the BN layer is used to reduce internal covariate shifts and improve training stability and generalization performance.

[0075] After multiple convolutions, normalization, and activation operations, CNN_rgb outputs an RGB modal feature tensor Frgb with dimensions (H / 8, W / 8, C), where C is the number of output channels. This feature tensor Frgb represents the multi-level semantic structure features of the window region in the visible light modality, including information such as texture, edges, and color gradients.

[0076] S3 also includes:

[0077] In the IR image feature extraction branch, the infrared image IR_sub obtained by synchronous cropping is concatenated with the corresponding mask Mask by channel to form the input tensor T_ir=Concat(IR_sub,Mask), which is then input to the lightweight encoder network CNN_ir.

[0078] The lightweight encoder CNN_ir consists of depthwise separable convolutional layers (DConv) and pointwise convolutional layers (PConv), which extract thermal distribution features and illumination compensation features in the infrared mode and output the infrared mode feature representation Fir.

[0079] The specific implementation steps are as follows:

[0080] The infrared sub-image IR_sub obtained through synchronous cropping is stitched together with the corresponding image mask Mask in the channel dimension to form the input tensor T_ir=Concat(IR_sub,Mask), where the infrared image IR_sub is a grayscale single-channel image and the mask Mask is a binarized image corresponding to the window area. The input tensor T_ir formed after stitching has a dual-channel structure to simultaneously preserve the target thermal signal distribution and window area constraint information.

[0081] Furthermore, the input tensor T_ir is fed into the lightweight convolutional encoder network CNN_ir for feature extraction. The lightweight encoder CNN_ir is constructed using a depthwise separable convolutional architecture, which includes multiple convolutional units consisting of alternating deep convolutional layers (DConv) and pointwise convolutional layers (PConv). The deep convolutional layers (DConv) are used to perform independent convolution operations on each input channel to extract the spatial local features of infrared thermal distribution. The pointwise convolutional layers (PConv) are used to perform linear combination in the channel direction to achieve feature fusion and illumination difference compensation between different channels, thereby effectively reducing local artifacts caused by different thermal radiation intensities.

[0082] Furthermore, a batch normalization (BN) layer is set after each convolutional operation to stabilize the network training process, and a ReLU activation layer is used to enhance the network's responsiveness to changes in infrared thermal texture. Finally, after processing by several levels of depthwise separable convolutional modules, the infrared modal feature representation Fir is output.

[0083] Beneficial effects:

[0084] This invention introduces a learnable gated fusion model (FGM) in the dual-modal input stage to achieve adaptive weighted fusion of RGB modal features (Frgb) and infrared modal features (Fir). The learnable parameter α is automatically optimized during network training and constrained by the Sigmoid function, enabling the model to dynamically adjust modal weights according to lighting conditions and infrared signal quality. This maintains the stability and accuracy of window state recognition results in complex environments such as strong light, reflection, or low illumination, effectively improving the system's adaptability to different environmental scenarios.

[0085] The fusion model FGM outputs a learnable parameter α. The FGM includes an input layer, multiple hidden layers, and an output layer. The input layer receives a concatenated feature vector from the RGB modal feature Frgb and the IR modal feature Fir. The hidden layers include a first hidden layer, a second hidden layer, and a third hidden layer, containing 128, 256, and 128 neural nodes, respectively. Each hidden layer uses the ReLU activation function for non-linear mapping. A BatchNormalization layer is placed between the second and third hidden layers to stabilize the training process. A Dropout layer is placed after the third hidden layer to prevent overfitting, with a Dropout rate set to 0.3. The output layer uses the Sigmoid activation function to generate the learnable parameter α. This learnable parameter α is used to weight and fuse the feature representation Frgb and the infrared modal feature representation Fir to obtain the fused feature Ffused.

[0086] The optimization process of the learnable parameter α:

[0087] Using the window state classification error as the loss function, an adaptive moment estimation optimizer is employed to perform gradient descent iterations on the weight parameters in the learnable gated fusion model (FGM). The learnable parameter α is nonlinearly mapped in each forward propagation by the output layer's linear transformation result z into the Sigmoid function, with the specific calculation formula as follows:

[0088]

[0089] Where z is the linear weighted result of the output layer of the fusion model FGM, e is the natural constant, and the value of α is restricted to between 0 and 1.

[0090] The fused feature F_fused input classification network Clsnet includes a feature compression module and a multilayer perceptron (MLP) structure. Feature abstraction is achieved through multilayer nonlinear mapping, and a Softmax classification layer is set at the output of the MLP structure. The class probability vector P is calculated according to the following formula: when the probability value corresponding to the open state in the class probability vector P is greater than the preset threshold THR, the output window is in the open label; otherwise, the output window is in the closed label.

[0091] Its structural design and classification decision-making mechanism are as follows:

[0092] The fused feature F_fused, which is the output of the feature fusion module from the RGB modal feature representation Frgb and the infrared modal feature representation Fir, is input into the classification network Clsnet. The classification network Clsnet consists of two parts: a feature compression module and a multilayer perceptron structure MLP.

[0093] The feature compression module is used to perform dimensionality compression and feature reconstruction on the input high-dimensional fusion feature F_fused. Specifically, it is achieved by connecting a set of linear transformation layers (Linear Layer Lin) and batch normalization layers (BN) in series, which compresses the channel dimension of the input feature from N1 to N2 to reduce the computational complexity of the subsequent classification network, while preserving the significant texture and thermal feature distribution information of the window area.

[0094] This invention sets up a Softmax classification layer at the output of the MLP and combines it with a threshold parameter THR (threshold parameter thr) for decision boundary control, making the probability outputs of window opening and closing adjustable. The system can flexibly adjust the threshold THR according to actual environmental needs, further enhancing the reliability and stability of the model in multi-scenario deployment.

[0095] The compressed feature vectors are sequentially input into a multilayer perceptron (MLP) structure. The MLP consists of several layers of fully connected (FC) and nonlinear activation function (ReLU) layers stacked alternately. Each layer realizes the nonlinear mapping and abstract expression of the input features, which is used to further extract high-level semantic information and cross-modal discriminative features from the fused features.

[0096] When the probability value P corresponding to the open state of the window in the category probability vector P is greater than the preset threshold THR, that is, P>THR, the output window of the classification network Clsnet is labeled as open; if P≤THR, the output window is labeled as closed.

[0097] Beneficial effects:

[0098] This invention constructs a window state classification network Cls_net, which includes a feature compression module Cmp and a multilayer perceptron structure MLP. It performs hierarchical feature abstraction and classification probability inference on the fused features F_fused, and combines the output of the Softmax classification layer to show the open and closed state labels of the window, thereby achieving high-precision window state recognition and discrimination. This recognition mechanism can effectively improve the intelligence level of building energy consumption monitoring and safety management, and reduce energy consumption loss and safety risks caused by misjudgment.

[0099] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. A computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0100] Some of the data in the above formulas are numerical calculations with dimensions removed, and the contents not described in detail in this specification are all prior art known to those skilled in the art.

[0101] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A window state detection method based on a learnable multimodal fusion gating mechanism, characterized in that, include: S1. Use a camera device to acquire time-synchronized and spatially aligned RGB video frames and IR images. By having the user annotate the window corner points, generate a window area mask in the RGB video frame. S2. Crop the RGB video frame and the IR image to obtain the RGB sub-image Img_rgb and the IR sub-image Img_ir; S3. Input the RGB sub-image Img_rgb into the first encoder network Enc_rgb to extract the RGB modal feature representation Frgb; input the IR sub-image Img_ir into the second encoder network Enc_ir to extract the infrared modal feature representation Fir; S4. Introduce a learnable parameter α, constrain it to the range of 0 to 1 by the sigmoid function, and perform weighted fusion of the feature representation Frgb and the infrared modal feature representation Fir to obtain the fused feature Ffused. S5. Input the fused features into the classification network and output the window state category.

2. The window state detection method based on a learnable multimodal fusion gating mechanism according to claim 1, characterized in that, The camera device includes an RGB_cam light acquisition unit and an IR_cam infrared acquisition unit. The RGB_cam and the IR_cam achieve frame-level alignment through a time synchronization module (TSM) and a spatial calibration module (SCM). The TSM controls the sampling clock signals of the two modes to be consistent, and the SCM calculates the extrinsic and intrinsic parameters between the two modes, completing the alignment mapping of the image space through a geometric transformation matrix (T_cal).

3. The window state detection method based on a learnable multimodal fusion gating mechanism according to claim 2, characterized in that, The process of aligning and mapping the image space using the geometric transformation matrix T_cal includes: S11. For images of the same calibration target, extract the set of feature points from each image; S12. Perform matching calculations based on the set of feature points to obtain the spatial mapping relationship from the RGB coordinate system to the IR coordinate system; S13. Solve for the extrinsic matrix R_ext and the translation vector T_ext using the least squares optimization algorithm, and construct the geometric transformation matrix T_cal by combining their respective intrinsic parameters K_rgb and K_ir. S14. The pixel coordinates of the RGB video frame are homogeneously mapped using the geometric transformation matrix T_cal to obtain the pixel projection position in the infrared coordinate system, thereby achieving spatial alignment between the RGB video frame and the IR image.

4. The window state detection method based on a learnable multimodal fusion gating mechanism according to claim 1, characterized in that, Perform a cropping process on the RGB video frame and the IR image, including: S21. Extract the coordinates of four vertices from the user-annotated set of corner coordinates, set C_set={(x1,y1),(x2,y2),(x3,y3),(x4,y4)}. S22. Calculate the center position parameters of the window area based on the corner coordinates: S23. Calculate the width parameter Wth and height parameter Hth of the window area; S24. Verify the effective pixel distribution of the window area based on the window area mask, and perform morphological dilation operation at the edge of the mask; S25. Using the center position parameter Ctr as the center point, and defining the cropping area according to the width parameter Wth and height parameter Hth, perform synchronous cropping on the RGB video frame and IR image.

5. The window state detection method based on a learnable multimodal fusion gating mechanism according to claim 1, characterized in that, S3 includes: In the RGB video frame feature extraction branch, the RGB sub-image RGB_sub obtained by synchronous cropping is concatenated with the corresponding binarized mask Mask in the channel direction to form a multi-channel input tensor T_rgb=Concat(RGB_sub,Mask), which is then input to the convolutional encoder network CNN_rgb. The convolutional encoder CNN_rgb includes a feature extraction stack structure consisting of several levels of convolutional layers (Conv), batch normalization layers (BN), and nonlinear activation function layers (ReLU). It extracts low-level texture edge features and mid-level semantic structure features of the window region and outputs RGB modal feature representation Frgb.

6. The window state detection method based on a learnable multimodal fusion gating mechanism according to claim 1, characterized in that, S3 further includes: In the IR image feature extraction branch, the infrared image IR_sub obtained by synchronous cropping is concatenated with the corresponding mask Mask by channel to form the input tensor T_ir=Concat(IR_sub,Mask), which is then input to the lightweight encoder network CNN_ir. The lightweight encoder CNN_ir consists of a depthwise separable convolutional layer (DConv) and a pointwise convolutional layer (PConv), which extracts the thermal distribution features and illumination compensation features in the infrared mode and outputs the infrared mode feature representation Fir.

7. The window state detection method based on a learnable multimodal fusion gating mechanism according to claim 1, characterized in that, The fusion model FGM outputs a learnable parameter α, which includes an input layer, multiple hidden layers, and an output layer. The input layer receives a concatenated feature vector from the RGB modal feature Frgb and the IR modal feature Fir. The hidden layers include a first hidden layer, a second hidden layer, and a third hidden layer, containing 128, 256, and 128 neural nodes, respectively. Each hidden layer uses the ReLU activation function for nonlinear mapping, and a BatchNormalization normalization layer is set between the second and third hidden layers to stabilize the training process. A Dropout layer is set after the third hidden layer to prevent overfitting, and the Dropout rate is set to 0.

3. The output layer uses the Sigmoid activation function to generate the learnable parameter α. The learnable parameter α performs a weighted fusion of the feature representation Frgb and the infrared modal feature representation Fir to obtain the fused feature Ffused.

8. A window state detection method based on a learnable multimodal fusion gating mechanism according to claim 7, characterized in that, The optimization process of the learnable parameter α: Using the window state classification error as the loss function, an adaptive moment estimation optimizer is employed to perform gradient descent iterations on the weight parameters in the fusion model FGM. The learnable parameter α is nonlinearly mapped in each forward propagation by the output layer's linear transformation result z into the Sigmoid function, with the specific calculation formula as follows: ； Where z is the linear weighted result of the output layer of the fusion model FGM, e is the natural constant, and the value of α is restricted to between 0 and 1.

9. A window state detection method based on a learnable multimodal fusion gating mechanism according to claim 1, characterized in that, The fused features F_fused are input into the classification network Clsnet, which includes a feature compression module and a multilayer perceptron (MLP) structure. Feature abstraction is achieved through multilayer nonlinear mapping, and a Softmax classification layer is set at the output of the MLP structure. The class probability vector P is calculated according to the following formula: when the probability value corresponding to the open state in the class probability vector P is greater than the preset threshold THR, the output window is in the open label; otherwise, the output window is in the closed label.

Images