Intelligent door and window state detection system based on deep learning and video analysis

Abstract

The invention discloses an intelligent door and window state detection system based on deep learning and video analysis, which comprises a client, a data acquisition end and a server, and is characterized in that the output end of the client is in communication connection with the data acquisition end, the output end of the data acquisition end is in communication connection with the server, and the output end of the server is in communication connection with the client. The server side comprises a model construction module, a model training module, a video learning module, a model determination module and a detection result feedback module. The intelligent door and window state detection system has the beneficial effects that a target detection deep learning algorithm is used for the first time in door and window state detection, so that intelligent detection of door and window states is realized; the improved YOLOv5 network structure ensures that fine-grained information is captured, a CBAM module is added to a detection head, the detection performance under the dark light condition is improved, and understanding and judgment of the model on door and window state changes are enhanced through multi-frame fusion and time sequence information analysis, and the detection robustness is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent doors and windows, and specifically to an intelligent door and window status detection system based on deep learning and video analysis. Background Art

[0002] In the development of intelligent security systems, door and window status detection technology has gradually become a key link. Doors and windows are the main entrances and exits of homes, commercial buildings and industrial parks, and their open and closed states directly affect the safety of the premises. Effective door and window status detection can not only detect and prevent potential intrusion risks in a timely manner, but also improve the overall security level and ensure the safety of property and personnel. In addition, the application of door and window status detection technology is not limited to preventing intrusion and theft, but can also play an important role in emergency situations such as fires and earthquakes, by monitoring the status of doors and windows in real time, and assisting emergency escape and rescue. With the continuous advancement of technology, door and window status detection technology will play its unique advantages in more scenarios and provide solid guarantees for building an intelligent and safe environment.

[0003] In the early days, door and window status detection mainly relied on manual patrol or static monitoring. Manual patrol relies on manpower, and the patrol range and frequency are limited, making it difficult to achieve 24-hour uninterrupted monitoring. This not only requires a lot of human resources, but also increases costs and management expenses. Static monitoring has a limited monitoring range, is prone to blind spots, and is difficult to fully cover all corners. In addition, static equipment cannot adjust the monitoring range as the environment changes, and lacks flexibility.

[0004] With the development of technology, the method of installing sensors on doors and windows to determine their status has gradually emerged. Although this method has improved the monitoring efficiency to a certain extent, it also has some problems. First, this method requires the doors and windows to be modified to carry sensors, which may affect the normal use of doors and windows. Secondly, the sensors need to be accurately installed and regularly maintained to ensure their normal operation. This increases the complexity and maintenance cost of the system. In addition, the accuracy of the sensor is easily affected by environmental factors, resulting in false alarms or missed alarms. For example, the Chinese patent (CN110794394A) published on February 14, 2020 describes a method for determining the status of doors and windows by measuring the empty area of ​​the room with a millimeter wave sensor. However, the application scenarios of this technology are limited, it can only be detected indoors, and the sensor signal is easily interfered by other electronic devices or environmental noise. Therefore, it is crucial to develop more accurate and efficient door and window status detection technology. Summary of the invention

[0005] The purpose of the present invention is to provide an intelligent door and window status detection system based on deep learning and video analysis to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] An intelligent door and window status detection system based on deep learning and video analysis, characterized in that it includes a client, a data acquisition end and a server, the client output end is communicatively connected to the data acquisition end, the data acquisition end output end is communicatively connected to the server end, the server end output end is communicatively connected to the client end, and the server end includes a model construction module, a model training module, a video learning module, a model determination module and a detection result feedback module;

[0008] The client is used for the user to send a test request and receive a test result;

[0009] The data collection end is used to collect door and window data on the spot and divide the collected door and window data set into a training set, a verification set and a validation set according to a preset ratio;

[0010] The server is used to process and detect the collected data;

[0011] The model building module is used to build a deep learning model based on the YOLOv5 convolutional neural network framework;

[0012] The model training module is used to train and optimize the deep learning network model through the collected data set;

[0013] The video learning module is used to design reasoning detection code based on video analysis, and enhances the model's understanding and judgment of door and window state changes through multi-frame fusion and timing information analysis, thereby enhancing the accuracy and robustness of the algorithm;

[0014] The model determination module is used to determine the trained and optimized model and generate the algorithm calling interface code of the client;

[0015] The detection result feedback module is used to feed back the detection result to the client.

[0016] As a preferred solution of the present invention: the client, data acquisition end and server end and the specific workflow are as follows: S1, the client sends a detection request to the data acquisition end;

[0017] S2, the data collection end obtains the door and window data set, and divides the data set into a training set, a validation set, and a verification set according to a preset ratio;

[0018] S3, the server builds an improved model based on YOLOv5, trains it with the training data set obtained in S2, and then iteratively optimizes the model based on the loss function to obtain the trained YOLOv5 model;

[0019] S4. The server designs reasoning detection code based on video analysis. Through multi-frame fusion and time series information analysis, the model's understanding and judgment of door and window state changes are enhanced, thereby enhancing the accuracy and robustness of the algorithm.

[0020] S5. The server is designed to use the interface code for calling the algorithm, enabling multiple users to call the algorithm simultaneously, and returning the detection results to the client.

[0021] As a preferred solution of the present invention: the door and window dataset in S2 includes multiple different types of doors, such as glass doors, wooden doors, revolving doors, elevator doors, glass opening and closing windows, sliding windows and other doors and windows of different shapes, colors and sizes. A total of 20,000 images are obtained as the initial dataset, and expansion is performed based on this dataset.

[0022] As a preferred solution of the present invention: the deep learning network model includes an input end, a feature extraction Backbone network, a feature processing Neck network and a Head layer, the input end is composed of three structures: a Mosaic data enhancement module, an adaptive anchor frame calculation and an adaptive image scaling, the feature extraction Backbone network is composed of three structures: a Focus module, a CSP module and a convolution Conv module, the feature processing Neck network is used to enhance the feature fusion and detection capabilities of different scales, and the Head layer is composed of a Detect module, which is used to convert the feature maps extracted and fused from the Backbone and Neck networks into the final detection results.

[0023] As a preferred solution of the present invention: the deep learning network model performs forward propagation through the input end, feature extraction Backbone network, feature processing Neck network and Head layer, and uses a loss function to iteratively optimize the model. The function iterative optimization model uses a composite loss function, and the composite loss function includes bounding box regression loss, classification loss and confidence loss. The goal of the composite loss function is to minimize the difference between the predicted result and the true label.

[0024] As a preferred solution of the present invention: the bounding box regression loss is used to measure the difference between the predicted bounding box and the true bounding box, and the formula used by the bounding box regression loss is:

[0025] L bbox =1-CIOU

[0026]

[0027] Among them, IOU is the intersection of the predicted box and the real box, ρ 2 (b,b gt) is the Euclidean distance between the center point of the predicted box and the center point of the real box, c is the diagonal length of the minimum closure area between the predicted box and the real box, w and h are the width and height of the predicted box, w gt and h gt are the width and height of the real box.

[0028] As a preferred solution of the present invention: the classification loss is used to measure the difference between the predicted category probability and the true category, and the classification loss uses the formula:

[0029]

[0030] Where N is the number of prediction boxes, C is the number of categories, and y ic represents the true category of the i-th prediction box, p ic is the probability that the i-th predicted box belongs to category c.

[0031] As a preferred solution of the present invention: the confidence loss is used to measure the difference between the confidence score of whether the predicted box contains the target and the actual situation, and the formula used by the confidence loss is:

[0032]

[0033] Where N is the number of prediction boxes, y i Indicates whether the i-th prediction box contains the target, p i Represents the confidence score of the i-th predicted box.

[0034] As a preferred solution of the present invention: the total loss function of the deep learning network model is a weighted sum of bounding box regression loss, classification loss and confidence loss, and the formula is as follows:

[0035] L total =λ bbox L bbox +λ conf L conf +λ cls L cls .

[0036] As a preferred solution of the present invention: in each iteration of the deep learning network model, the model parameters are updated to reduce the total loss, and this process continues until the performance of the model on the training set and the validation set reaches the expected effect.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention proposes an intelligent door and window status detection system based on deep learning and video analysis. It is the first to use a target detection deep learning algorithm to solve the door and window status detection problem. This method uses data-driven deep learning technology to automatically learn and identify the characteristics of doors and windows and opening and closing, thereby realizing intelligent detection of door and window status. Compared with the sensor hardware deployment method, this method does not need to install complex sensor hardware, and only requires a camera to achieve detection, making the deployment process simpler and faster. It reduces the purchase and maintenance costs of sensors and related hardware, and the overall cost is lower. The system has strong adaptability to different environments and is not limited by physical sensors. It also has obvious advantages in improving detection accuracy, enhancing system functions, and data storage and analysis. This makes the system have broad application prospects and practical value in the fields of smart cities, smart homes, and security monitoring.

[0039] 2. This invention improves the network structure of YOLOv5, uses the SPD-Conv module to replace the stride convolution and pooling layer in the backbone architecture to capture more fine-grained information of the image, and adds a CBAM module to the detection head to improve the detection performance under dark light conditions. The above improvements can effectively improve the detection efficiency and detection accuracy. Unlike traditional single image detection, this method uses video as input to solve the problem that single-frame detection is easily affected by occlusion and visual blind spots. By fusing multi-frame image information and time series analysis, the model can better understand and predict changes in door and window status and improve the robustness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Design a training flow chart for the algorithm of the present invention;

[0041] Figure 2 Deploy an inference flow chart for the algorithm of the present invention;

[0042] Figure 3 This is a structural diagram of the detection network model based on the improvement of YOLOv5 of the present invention;

[0043] Figure 4 This is a pseudo code diagram of the video analysis algorithm of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1-Figure 4 , the present invention provides a technical solution:

[0046] An intelligent door and window status detection system based on deep learning and video analysis, including a client, a data acquisition terminal and a server, wherein the client output terminal is connected to the data acquisition terminal in communication, the data acquisition terminal output terminal is connected to the server in communication, the server output terminal is connected to the client in communication, and the server terminal includes a model building module, a model training module, a video learning module, a model determination module and a detection result feedback module;

[0047] The client is used by users to send test requests and receive test results;

[0048] The data collection end is used to collect the door and window data in the field and divide the collected door and window data set into a training set, a validation set and a verification set according to a preset ratio;

[0049] The server is used to process and detect the collected data;

[0050] The model building module is used to build a deep learning model based on the YOLOv5 convolutional neural network framework;

[0051] The model training module is used to train and optimize the deep learning network model through the collected data set;

[0052] The video learning module is used to design reasoning detection code based on video analysis. Through multi-frame fusion and time series information analysis, the model's understanding and judgment of door and window state changes are enhanced, thereby enhancing the accuracy and robustness of the algorithm.

[0053] The model determination module is used to determine the trained and optimized model and generate the algorithm call interface code for the client;

[0054] The detection result feedback module is used to feed back the detection result to the client.

[0055] Furthermore, the specific workflow of the client, the data collection end and the server is as follows: S1, the client sends a detection request to the data collection end;

[0056] S2, the data collection end obtains the door and window data set, and divides the data set into a training set, a validation set, and a verification set according to a preset ratio;

[0057] S3, the server builds an improved model based on YOLOv5, trains it with the training data set obtained in S2, and then iteratively optimizes the model based on the loss function to obtain the trained YOLOv5 model;

[0058] S4. The server designs reasoning detection code based on video analysis. Through multi-frame fusion and time series information analysis, the model's understanding and judgment of door and window state changes are enhanced, thereby enhancing the accuracy and robustness of the algorithm.

[0059] S5. The server is designed to use the interface code for calling the algorithm, enabling multiple users to call the algorithm simultaneously, and returning the detection results to the client.

[0060] Furthermore, the door and window dataset in S2 contains many different types of doors, such as glass doors, wooden doors, revolving doors, elevator doors, glass opening and closing windows, sliding windows, and other doors and windows of different shapes, colors, and sizes. A total of 20,000 images are obtained as the initial dataset, and expansion is performed based on this dataset.

[0061] Furthermore, the deep learning network model includes an input end, a feature extraction Backbone network, a feature processing Neck network and a Head layer. The input end consists of three structures: a Mosaic data enhancement module, an adaptive anchor box calculation and an adaptive image scaling module. The feature extraction Backbone network consists of three structures: a Focus module, a CSP module and a convolution Conv module. The feature processing Neck network is used to enhance the feature fusion and detection capabilities of different scales. The Head layer consists of a Detect module, which is used to convert the feature maps extracted and fused from the Backbone and Neck networks into the final detection results.

[0062] Furthermore, the deep learning network model performs forward propagation through the input end, feature extraction Backbone network, feature processing Neck network and Head layer, and uses the loss function to iteratively optimize the model. The function iterative optimization model uses a composite loss function. The composite loss function includes bounding box regression loss, classification loss and confidence loss. The goal of the composite loss function is to minimize the difference between the predicted result and the true label.

[0063] Furthermore, the bounding box regression loss is used to measure the difference between the predicted bounding box and the true bounding box. The formula used for the bounding box regression loss is:

[0064] L bbox =1-CIOU

[0065]

[0066] Among them, IOU is the intersection of the predicted box and the real box, ρ 2 (b,b gt ) is the Euclidean distance between the center point of the predicted box and the center point of the real box, c is the diagonal length of the minimum closure area between the predicted box and the real box, w and h are the width and height of the predicted box, w gt and h gt are the width and height of the real box.

[0067] Furthermore, the classification loss is used to measure the difference between the predicted category probability and the true category. The formula used for the classification loss is:

[0068]

[0069] Where N is the number of prediction boxes, C is the number of categories, and y ic represents the true category of the i-th prediction box, p ic is the probability that the i-th predicted box belongs to category c.

[0070] Furthermore, the confidence loss is used to measure the difference between the confidence score of whether the predicted box contains the target and the actual situation. The formula used for the confidence loss is:

[0071]

[0072] Where N is the number of prediction boxes, y i Indicates whether the i-th prediction box contains the target, p i Represents the confidence score of the i-th predicted box.

[0073] Furthermore, the total loss function of the deep learning network model is the weighted sum of the bounding box regression loss, classification loss, and confidence loss, as follows:

[0074] L total =λ bbox L bbox +λ conf L conf +λ cls L cls .

[0075] Furthermore, in each iteration of the deep learning network model, the model parameters are updated to reduce the total loss, and this process continues until the performance of the model on the training set and the validation set reaches the expected effect.

[0076] In order to highlight the practical advantages of the present invention, the improved network part based on YOLOv5 and the reasoning process based on video analysis are further described in detail below in combination with specific implementation methods.

[0077] The present invention is mainly divided into an algorithm design training phase and an algorithm deployment reasoning phase.

[0078] Example 1: Algorithm Design Training Phase

[0079] The algorithm design and training phase specifically includes the following steps:

[0080] S1.1. Video acquisition: The patrol robot car shoots videos of doors and windows during fixed-point patrols. During the process, the robot car moves, and the camera rotates and swings to capture images from all angles;

[0081] S1.2, data labeling: extract frames from the collected video, and use the image labeling tool LabelImg to label the door and window switch status in the image, build a door and window status dataset, and divide it into a training dataset D and a test dataset T;

[0082] S1.3, data expansion: Since doors and windows have different sizes and shapes in real life, in order to obtain enough samples for training the deep learning network model, avoid overfitting, and enhance the robustness of the network, the constructed training data set is further expanded to obtain the expanded training data set Dk;

[0083] S1.4, Model construction: Based on the improved YOLOv5 convolutional neural network architecture, a deep learning network model for door and window detection is constructed;

[0084] S1.5, model training: use the expanded training data set Dk to train the constructed deep learning network model;

[0085] S1.6, Model optimization and testing: Deploy the trained model to actual scenarios for testing, evaluate its performance in the real environment, and continuously optimize the model based on problems encountered during the test.

[0086] The deep learning network model training part in S1.5 is as follows:

[0087] 1. In the traditional CNN network architecture, downsampling and pooling layer operations will lead to the loss of fine-grained information, so the model's performance degrades when detecting small objects or low-resolution images. For door and window detection, doors and windows are of different sizes, and they contain small features such as door and window handles and frames. These features play an important role in distinguishing doors and windows from other interferences. Therefore, in order to improve this problem, the SPD-Conv module is introduced to replace the downsampling and pooling layers in the backbone architecture, and can efficiently extract image features;

[0088] SPD-Conv consists of a space-to-depth (SPD) layer and a non-strided convolution layer. The SPD layer downsamples the feature map but retains all information in the channel dimension, so there is no information loss. A non-strided convolution operation is added after each SPD to reduce the number of (increased) channels using learnable parameters in the added convolution layer. For an intermediate feature map with size, the feature map is divided as follows:

[0089] f 0,0 =X[0:S:scale,0:S:scale],

[0090] f 1,0=X[1:S:scale,0:S:scale],..,

[0091] f scale-1,0 =X[scale-1:S:scale,0:S:scale];

[0092] f 0,1 =X[0:S:scale,1:S:scale],

[0093] f 1,1 ,...,

[0094] f scale-1,1 =X[scale-1:S:scale,1:S:scale]; ...

[0096] f 0,scale-1 =X[0:S:scale,scale-1:S:scale],

[0097] f 1,scale-1 ,...,

[0098] f scale-1,scale-1 =X[scale-1:S:scale,scale-1:S:scale];

[0099] Where scale is the division step size, each sub-feature map is obtained by downsampling the feature map X, and then these sub-feature maps are spliced ​​along the channel direction to obtain a new feature map X'. X' is reduced in the spatial dimension by scale, but increased in the channel dimension. It can be expressed as:

[0100]

[0101] After the SPD conversion layer, a non-strided convolution layer with a C2 convolution kernel is added, where C2 < scale 2 C1, the reason for doing this is to optimize the number of channels while retaining all the discriminant feature information as much as possible.

[0102] 2. b) When patrol robots conduct night patrols, the collected video images are often dim due to insufficient light. This problem of insufficient light will make it difficult to clearly display the targets in the image, which is manifested in the blurred edges of the targets, loss of color information and other problems. This not only affects the visual effect of the image, but also brings great challenges to the target detection task. Therefore, when extracting features of such images, the network needs to pay more attention to important feature information and ignore unnecessary parts. In order to address this problem, the present invention introduces a convolutional block attention module (CBAM) in the detection head. CBAM is a simple and effective attention mechanism that can significantly improve the detection capability of the model in dark light scenes by weighted processing of channel attention and spatial attention on the feature map.

[0103] CBAM consists of two independent submodules: channel attention module CAM and spatial attention module SAM. These two modules perform attention operations on feature maps in the channel and spatial dimensions respectively. The specific steps are as follows:

[0104] Channel Attention Module:

[0105] The channel attention map is generated by utilizing the channel relationship of the features. The channel attention module mainly explores the importance of the semantic level. First, the feature map is average pooled and max pooled to obtain two different spatial descriptors. and These two descriptors are fed into a shared network consisting of one layer to obtain the channel attention map M c , after applying the shared network to each descriptor, we merge the output feature vectors using element-wise summation. In short, the channel attention is calculated as:

[0106] M c (F)=sig(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0107] Spatial Attention Module:

[0108] Unlike channel attention, spatial attention focuses on the “where” part of the information, which is complementary to channel attention. To calculate spatial attention, we first perform average pooling and maximum pooling operations on each channel of the feature map and concatenate them to generate a valid feature descriptor. We apply a convolutional layer on the concatenated feature descriptors to generate a spatial attention map M. s (F) The channel information of the feature map is aggregated through two pooling operations to generate and Each represents the average pooled features and the maximum pooled features across channels. They are then concatenated and convolved through standard convolutional layers to generate a 2D spatial attention map. The spatial attention is calculated as:

[0109] M s (F)=sig(f([AvgPool(F);MaxPool(F)]))

[0110] By making the above two improvements to the network, the model performance has been greatly improved, and it can achieve good detection capabilities in complex scenes and environments with poor lighting conditions.

[0111] Example 2: Algorithm deployment inference phase

[0112] The algorithm deployment inference phase specifically includes the following steps:

[0113] S2.1, Video shooting and uploading: The client sends a detection request to the patrol robot car. When the robot car patrols at a fixed point, it shoots a 3 to 5 second video of a single target door or window and uploads it to the server.

[0114] S2.2, Model detection: After receiving the video, the server uses it as model input for detection;

[0115] S2.3, Result analysis and feedback: The detection result of each frame is further processed by the video analysis algorithm to obtain the final detection result and return it to the client.

[0116] Specifically, the video analysis algorithm in S2.3 is as follows:

[0117] The output of the YOLOv5 model is to predict the detection targets that may exist in the input image. When the amount of training data is small and the environmental differences are large, the model is often unstable and prone to false detection and missed detection. Therefore, the previous single image input detection cannot cope with complex environmental changes and detection accuracy requirements. In order to make the model have higher accuracy and robustness, this method uses video as input. When shooting doors and windows, the patrol car takes a single target as the standard, shoots the target doors and windows from all angles, and designs the corresponding video analysis algorithm by fusing multi-frame image information and time series changes to obtain the detection results. The detection results of the target are returned to the client.

[0118] Although a single target is taken as the standard when shooting, considering the actual deployment environment of doors and windows and the limited shooting conditions of the algorithm, the video analysis algorithm will return four types of results to the client. The first case is that the detection results of multiple frames in the video are a single target, then the detection results of the target are returned. The second case is that the model does not detect the target, then 'no_det' is returned to the client. However, considering that some doors and windows are close in real life, in order to obtain a more comprehensive monitoring picture, doors or windows other than the target may be captured during shooting. In this case, if the multi-frame detection results in the video show two targets, the status of the two doors and windows are returned. If the multi-frame detection results in the video show more than two targets, the patrol car's shooting distance is too far from the target, and the model cannot accurately determine the target to be detected, then 'reshoot' is returned to remind you to readjust the shooting. The pseudo code of the algorithm can be found in Figure 4 .

[0119] In summary, the present invention uses the target detection deep learning algorithm for the first time in the door and window status detection to realize the intelligent detection of the door and window status. Unlike the sensor detection that requires hardware deployment, the data collection method of this system only needs to be completed by the non-contact patrol robot car. The patrol robot car will dynamically and omnidirectionally scan the target with the camera to avoid the blind spot of the viewing angle or the false detection caused by the angle problem. Then the above process is recorded into a video and sent to the deep learning model based on YOLOv5 target detection to automatically learn the features of the doors and windows in the video, so as to realize the intelligent and automatic detection of the switch status of the doors and windows. For the YOLOv5 model, its network structure is improved to adapt to the current task. First, in order to reduce the loss of fine-grained information of the original video and let the algorithm learn more effective feature representation, the SPD-Conv module is introduced in the backbone, and it replaces the downsampling and pooling operations to reduce information loss. Secondly, in order to increase the detection accuracy of dark light scenes and the applicability of the model scene, the present invention adds a CBAM attention module to the detection head of the algorithm to improve the detection performance. In addition, during the inference detection phase of the model, a video analysis algorithm is designed to consider the detection results of each frame in the video, and a comprehensive analysis and judgment is made to obtain the final detection result.

[0120] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent door and window status detection system based on deep learning and video analysis, characterized in that: It includes a client, a data acquisition terminal and a server, wherein the client output terminal is connected to the data acquisition terminal in communication, the data acquisition terminal output terminal is connected to the server in communication, the server output terminal is connected to the client in communication, and the server terminal includes a model construction module, a model training module, a video learning module, a model determination module and a detection result feedback module; The client is used for the user to send a test request and receive a test result; The data collection end is used to collect door and window data on the spot and divide the collected door and window data set into a training set, a verification set and a validation set according to a preset ratio; The server is used to process and detect the collected data; The model building module is used to build a deep learning model based on the YOLOv5 convolutional neural network framework; The model training module is used to train and optimize the deep learning network model through the collected data set; The video learning module is used to design reasoning detection code based on video analysis, and enhances the model's understanding and judgment of door and window state changes through multi-frame fusion and timing information analysis, thereby enhancing the accuracy and robustness of the algorithm; The model determination module is used to determine the trained and optimized model and generate the algorithm calling interface code of the client; The detection result feedback module is used to feed back the detection result to the client.

2. According to claim 1, the intelligent door and window status detection system based on deep learning and video analysis is characterized in that: The client, data collection end and server end and the specific working process are as follows: S1, the client sends a detection request to the data collection end; S2, the data collection end obtains the door and window data set, and divides the data set into a training set, a validation set, and a verification set according to a preset ratio; S3, the server builds an improved model based on YOLOv5, trains it with the training data set obtained in S2, and then iteratively optimizes the model based on the loss function to obtain the trained YOLOv5 model; S4. The server designs reasoning detection code based on video analysis. Through multi-frame fusion and time series information analysis, the model's understanding and judgment of door and window state changes are enhanced, thereby enhancing the accuracy and robustness of the algorithm. S5. The server is designed to use the interface code for calling the algorithm, enabling multiple users to call the algorithm simultaneously, and returning the detection results to the client.

3. According to claim 2, the intelligent door and window status detection system based on deep learning and video analysis is characterized in that: The door and window dataset in S2 contains many different types of doors, such as glass doors, wooden doors, revolving doors, elevator doors, glass opening and closing windows, sliding windows and other doors and windows of different shapes, colors and sizes. A total of 20,000 images are obtained as the initial dataset, and expansion is performed based on this dataset.

4. The intelligent door and window status detection system based on deep learning and video analysis according to claim 1 is characterized in that: The deep learning network model includes an input end, a feature extraction Backbone network, a feature processing Neck network and a Head layer. The input end consists of three structures: a Mosaic data enhancement module, an adaptive anchor frame calculation and an adaptive image scaling module. The feature extraction Backbone network consists of three structures: a Focus module, a CSP module and a convolution Conv module. The feature processing Neck network is used to enhance the feature fusion and detection capabilities of different scales. The Head layer consists of a Detect module, which is used to convert the feature maps extracted and fused from the Backbone and Neck networks into the final detection results.

5. The intelligent door and window status detection system based on deep learning and video analysis according to claim 4 is characterized in that: The deep learning network model performs forward propagation through the input end, feature extraction Backbone network, feature processing Neck network and Head layer, and uses a loss function to iteratively optimize the model. The function iterative optimization model uses a composite loss function, which includes a bounding box regression loss, a classification loss and a confidence loss. The goal of the composite loss function is to minimize the difference between the predicted result and the true label.

6. The intelligent door and window status detection system based on deep learning and video analysis according to claim 5 is characterized in that: The bounding box regression loss is used to measure the difference between the predicted bounding box and the true bounding box. The formula used for the bounding box regression loss is: L bbox =1-CIOU Among them, IOU is the intersection of the predicted box and the real box, ρ 2 (b,b gt ) is the Euclidean distance between the center point of the predicted box and the center point of the real box, c is the diagonal length of the minimum closure area between the predicted box and the real box, w and h are the width and height of the predicted box, w gt and h gt are the width and height of the real box.

7. The intelligent door and window status detection system based on deep learning and video analysis according to claim 5 is characterized in that: The classification loss is used to measure the difference between the predicted category probability and the true category. The formula used for the classification loss is: Where N is the number of prediction boxes, C is the number of categories, and y ic represents the true category of the i-th prediction box, p ic is the probability that the i-th predicted box belongs to category c.

8. The intelligent door and window status detection system based on deep learning and video analysis according to claim 5 is characterized in that: The confidence loss is used to measure the difference between the confidence score of whether the predicted box contains the target and the actual situation. The formula used for the confidence loss is: Where N is the number of prediction boxes, y i Indicates whether the i-th prediction box contains the target, p i Represents the confidence score of the i-th predicted box.

9. The intelligent door and window status detection system based on deep learning and video analysis according to claim 4 is characterized in that: The total loss function of the deep learning network model is a weighted sum of bounding box regression loss, classification loss, and confidence loss, and the formula is as follows: L total =λ bbox L bbox +λ conf L conf +λ cls L cls 。 10. The intelligent door and window status detection system based on deep learning and video analysis according to claim 9, characterized in that: In each iteration of the deep learning network model, the model parameters are updated to reduce the total loss, and this process continues until the performance of the model on the training set and the validation set reaches the expected effect.

Images