Building safety intelligent monitoring system and method based on Internet of Things

Abstract

The present application relates to the field of intelligent monitoring, and discloses an intelligent monitoring system and method for building safety based on the Internet of Things. The system obtains a collection of image data of fire-fighting facilities in buildings from a cloud platform, and uses artificial intelligence-based image analysis and extraction technology to extract image semantic features of the image data of fire-fighting facilities in each building. In this way, the system automatically judges whether each fire-fighting facility has an abnormality based on the layered spatial domain enhanced representation of the semantic coding features of the images of fire-fighting facilities in each building, and generates abnormal values ​​of fire-fighting facilities. In this way, the high-level semantic information in the image can be better understood, so as to more accurately judge whether the fire-fighting facilities have an abnormality, and reduce misjudgments caused by differences in shape or model. At the same time, it can distinguish between changes in non-target objects in the background and the fire-fighting facilities themselves, avoid false alarms caused by background interference, and thus improve the intelligence of abnormal judgment of fire-fighting facilities.

Description

Technical Field

[0001] The present application relates to the field of intelligent monitoring, and more specifically, to an intelligent building safety monitoring system and method based on the Internet of Things. Background Art

[0002] Patent CN117975375A proposes an intelligent building safety monitoring system and method based on the Internet of Things. It monitors the status of fire-fighting facilities through image acquisition and sensor data, calculates the abnormal coefficients of fire-fighting equipment and sign lamps using an abnormality recognition network, and combines these coefficients to evaluate the overall safety status of the building. If the safety factor is lower than the set threshold, a rectification alarm is triggered. The system includes modules such as data acquisition, abnormality recognition, comprehensive evaluation and early warning, and uses image processing technology and algorithm models to effectively improve the efficiency of fire safety management and reduce fire risks.

[0003] The fire-fighting facility anomaly value in the above patent is obtained by calculating the cumulative pixel-by-pixel difference between the actually collected image data of the fire-fighting facilities in the building and the image data of the standard or finished fire-fighting facilities. However, methods based solely on pixel differences may not be able to capture high-level semantic information in the image. For example, two fire hydrants are slightly different in shape but function intact. In this case, pixel-level comparison may be judged as abnormal, when in fact they are normal. This method lacks a deep understanding of the function and state of the object, and it is easy to misjudge normal changes as abnormal. In addition, if there are other objects or decorations in the background, changes in these non-target objects may also be misjudged as abnormalities of the fire-fighting facilities themselves, thereby increasing the false alarm rate and reducing the accuracy and reliability of the system.

[0004] Therefore, an optimized IoT-based intelligent building safety monitoring solution is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides an intelligent building safety monitoring system and method based on the Internet of Things, which obtains a collection of image data of fire-fighting facilities in the building from a cloud platform, and uses artificial intelligence-based image analysis and extraction technology to extract image semantic features of the image data of fire-fighting facilities in each building, thereby automatically judging whether each fire-fighting facility has an abnormality according to the layered spatial domain enhanced representation of the semantic coding features of the images of fire-fighting facilities in each building, and generating fire-fighting facility abnormality values. In this way, the high-level semantic information in the image can be better understood, so as to more accurately judge whether the fire-fighting facilities have an abnormality, and reduce misjudgments caused by differences in shape or model. At the same time, it can distinguish between changes in non-target objects in the background and the fire-fighting facilities themselves, avoid false alarms caused by background interference, and thus improve the intelligence of abnormal judgment of fire-fighting facilities.

[0006] According to one aspect of the present application, a method for intelligent monitoring of building safety based on the Internet of Things is provided, which includes:

[0007] Obtain image data of fire-fighting facilities, fire-fighting equipment information and images of fire-fighting passage sign lights in buildings;

[0008] Performing fire equipment abnormality identification based on the fire equipment image data in the building and the fire equipment information to obtain a fire equipment abnormality coefficient, including: obtaining a fire equipment abnormality value based on the fire equipment image data in the building; obtaining the fire equipment abnormality coefficient based on the fire equipment abnormality value and the fire equipment information;

[0009] Based on the fire passage sign light image, obtaining a fire passage sign light abnormality coefficient;

[0010] Based on the abnormality coefficient of the fire-fighting equipment and the abnormality coefficient of the fire-fighting passage sign light, the building safety factor is obtained, and based on the comparison between the building safety factor and a preset building safety factor threshold, it is determined whether building safety rectification is needed.

[0011] According to another aspect of the present application, there is provided a building safety intelligent monitoring system based on the Internet of Things, which includes:

[0012] An information acquisition module is used to acquire image data of fire-fighting facilities in a building, information on fire-fighting equipment, and images of fire-fighting passage sign lights;

[0013] A fire equipment abnormality identification module, used for performing fire equipment abnormality identification based on the fire equipment image data in the building and the fire equipment information to obtain a fire equipment abnormality coefficient, comprising: a fire equipment abnormality value generation unit, used for obtaining a fire equipment abnormality value based on the fire equipment image data in the building; a fire equipment abnormality coefficient generation unit, used for obtaining the fire equipment abnormality coefficient based on the fire facility abnormality value and the fire equipment information;

[0014] A sign light abnormality coefficient generating module, used for obtaining a fire passage sign light abnormality coefficient based on the fire passage sign light image;

[0015] The building safety rectification determination module is used to obtain the building safety factor based on the fire equipment abnormality coefficient and the fire passage marker light abnormality coefficient, and determine whether the building safety rectification is needed based on the comparison between the building safety factor and a preset building safety factor threshold.

[0016] Compared with the prior art, the present application provides an intelligent building safety monitoring system and method based on the Internet of Things. It obtains a collection of image data of fire-fighting facilities in buildings from a cloud platform, and uses artificial intelligence-based image analysis and extraction technology to extract image semantic features of the image data of fire-fighting facilities in each building. It automatically judges whether each fire-fighting facility has an abnormality based on the layered spatial domain enhancement of the semantic coding features of the images of fire-fighting facilities in each building, and generates anomaly values ​​of fire-fighting facilities. In this way, the high-level semantic information in the image can be better understood, so as to more accurately judge whether the fire-fighting facilities have an abnormality, and reduce misjudgments caused by differences in shape or model. At the same time, it can distinguish between changes in non-target objects in the background and the fire-fighting facilities themselves, avoid false alarms caused by background interference, and thus improve the intelligence of abnormal judgment of fire-fighting facilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A flowchart of a building safety intelligent monitoring method based on the Internet of Things according to an embodiment of the present application;

[0019] Figure 2 A data flow diagram of a building safety intelligent monitoring method based on the Internet of Things according to an embodiment of the present application;

[0020] Figure 3 Flow chart of sub-step S2 of the method for intelligent monitoring of building safety based on the Internet of Things according to an embodiment of the present application;

[0021] Figure 4 It is a block diagram of an IoT-based building safety intelligent monitoring system according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0023] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0024] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0025] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0026] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0027] In the technical solution of the present application, a method for intelligent monitoring of building safety based on the Internet of Things is proposed. Figure 1 The present invention is a flowchart of an intelligent building safety monitoring method based on the Internet of Things according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the building safety intelligent monitoring method based on the Internet of Things according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the intelligent building safety monitoring method based on the Internet of Things includes the following steps: S1, acquiring image data of fire-fighting facilities, fire-fighting equipment information and images of fire-fighting passageway marker lights in the building; S2, identifying fire-fighting equipment abnormalities based on the image data of fire-fighting facilities in the building and the fire-fighting equipment information to obtain a fire-fighting equipment abnormality coefficient, including: obtaining fire-fighting facility abnormality values ​​based on the image data of fire-fighting facilities in the building; obtaining fire-fighting equipment abnormality coefficients based on the fire-fighting facility abnormality values ​​and the fire-fighting equipment information; S3, obtaining fire-fighting passageway marker light abnormality coefficients based on the fire-fighting passageway marker light images; S4, obtaining a building safety factor based on the fire-fighting equipment abnormality coefficient and the fire-fighting passageway marker light abnormality coefficient, and determining whether building safety rectification is needed based on a comparison between the building safety factor and a preset building safety factor threshold.

[0028] In particular, the S1 obtains the image data of fire-fighting facilities, fire-fighting equipment information and the image of fire-fighting passage sign lights in the building. In a specific example of the present application, the image data of the fire-fighting facilities inside the building can be collected by an image acquisition terminal (such as a camera). These image data will then be imported into the image processing software to obtain the pixel value of each pixel point. In addition, the fire-fighting equipment information inside the fire-fighting facilities can be collected by sensors, including but not limited to the fire-fighting water pressure and the use time of the fire-fighting equipment. This process ensures that the system can obtain comprehensive fire-fighting facility status data. More specifically, the image information of the fire-fighting sign lamps on the fire-fighting passage can be obtained by the image acquisition terminal. These image information will be imported into the image processing software to obtain the distance, brightness and normal use number of the fire-fighting sign lamps relative to the specified position, so as to evaluate whether the fire-fighting sign lamps are in the correct position and whether their working state is normal. Obtaining the image data of fire-fighting facilities and the information of internal fire-fighting equipment, as well as the image information of fire-fighting sign lamps in the fire-fighting passage, helps to establish a comprehensive database of building fire safety conditions. This is very important for subsequent anomaly detection and risk assessment.

[0029] In particular, S2 performs fire equipment anomaly identification based on the fire facility image data in the building and the fire equipment information to obtain the fire equipment anomaly coefficient. It should be understood that the status of the fire equipment directly affects the fire safety of the building. The anomaly coefficient can be used as an early warning indicator. By calculating the fire equipment anomaly coefficient, the current status of each fire equipment can be comprehensively evaluated and potential problems can be discovered in time. This helps managers take measures to prevent accidents before problems become serious. In a specific example of the present application, Figure 3 As shown, the S2 includes: S21, obtaining the abnormal value of the fire-fighting facilities based on the image data of the fire-fighting facilities in the building; S22, obtaining the abnormal coefficient of the fire-fighting equipment based on the abnormal value of the fire-fighting facilities and the fire-fighting equipment information.

[0030] Specifically, the S21 obtains the abnormal value of the fire-fighting facilities based on the image data of the fire-fighting facilities in the building. In an embodiment of the present application, first, a collection of image data of the fire-fighting facilities in the building is obtained from the cloud platform; in the building safety intelligent monitoring system based on the Internet of Things, the image data of the fire-fighting facilities in the building is collected by an Internet of Things device (such as a camera) and transmitted to the cloud platform through a wireless network for storage and processing. That is, after the cloud platform receives the image data transmitted from the image acquisition terminal and stores it in the distributed storage system, the system sends a data request to the cloud platform through the API interface, requesting to obtain the image data of the fire-fighting facilities within a specific time period or a specific area. The request parameters may include the acquisition time range, location information, equipment number, etc. After receiving the request, the cloud platform queries the database according to the request parameters to find the image data records that meet the conditions.

[0031] Next, image features are extracted from each image data of fire-fighting facilities in the set of image data of fire-fighting facilities in the building to obtain a set of semantic coding features of images of fire-fighting facilities in the building. Considering that each semantic coding feature map of fire-fighting facilities in the building contains rich visual information and semantic content about fire-fighting facilities, such as local detail information such as appearance, color, shape, position, and global semantic information. Therefore, it is possible to capture local details and global information of fire-fighting facilities in the building at the same time. In the technical solution of the present application, each image data of fire-fighting facilities in the set of image data of fire-fighting facilities in the building is respectively input into a fire-fighting facility image feature extractor based on a Mobile-Former network to obtain a set of semantic coding feature maps of images of fire-fighting facilities in the building. It should be understood that the Mobile-Former network is a parallel design of MobileNet and Transformer. Specifically, MobileNet decomposes the traditional convolution into two steps through depth-separable convolution: first, depth-wise convolution, which independently applies a single filter to each input channel, focusing on capturing local details; second, point-by-point convolution (1x1 convolution), which is responsible for cross-channel information fusion. This design significantly reduces the amount of computation and model parameters while maintaining good performance. It is particularly suitable for identifying fine-grained features such as edges and textures in images, and can more accurately and meticulously capture the local details of firefighting facilities in buildings. In contrast, the Transformer model uses a self-attention mechanism to efficiently learn long-distance dependencies when processing data. That is, the Transformer can better understand the complex associations between different regions in the semantic encoding features of images of firefighting facilities in buildings, such as the relationship between firefighting facilities and the surrounding environment, avoiding the limitations of traditional convolutional networks in the receptive field, thereby enhancing the model's ability to understand the global context.

[0032] Then, the semantic coding features of the fire-fighting facilities images in each building in the set of semantic coding features of the fire-fighting facilities images in the building are respectively subjected to hierarchical spatial domain semantic enhancement to obtain a set of semantically enhanced coding features of the fire-fighting facilities images in the building. Considering that in each semantic coding feature of the fire-fighting facilities images in the building, different local areas have different degrees of fine-grained information, which reflects the specific status and details of the equipment. In addition, the importance and influence of different local detail features in the entire spatial range are also different. For example, some areas such as the pressure gauge of a fire extinguisher or the valve of a fire hydrant may be crucial to the overall safety assessment, while other areas such as the background wall are relatively minor. Therefore, in order to dynamically enhance the key features of different local areas in the semantic coding features of the fire-fighting facilities images in each building, in the technical solution of the present application, the semantic coding features of the fire-fighting facilities images in each building in the set of semantic coding features of the fire-fighting facilities images in the building are respectively subjected to hierarchical spatial domain semantic enhancement to obtain a set of semantically enhanced coding features of the fire-fighting facilities images in the building, so as to more accurately judge the status of the fire-fighting facilities, thereby improving the accuracy of anomaly detection.

[0033] Specifically, the specific steps of performing hierarchical spatial domain semantic enhancement on each semantic coding feature of the image of fire-fighting facilities in the set of semantic coding features of the image of fire-fighting facilities in the building include: first, performing pixel granularity feature decoupling on the semantic coding feature map of the image of fire-fighting facilities in the building to obtain a set of pixel granularity decoupled feature vectors of the image of fire-fighting facilities in the building; by performing pixel granularity feature decoupling on the semantic coding feature map of the image of fire-fighting facilities in the building, the semantic coding feature map of the image of fire-fighting facilities in the building is decomposed into a series of independent pixel granularity decoupled feature vectors of the image of fire-fighting facilities in the building, so as to eliminate unnecessary coupling relationships between pixels and enable the features of each pixel to be processed separately. In particular, through feature decoupling, not only the redundancy of the feature space is reduced, but also the model's ability to understand local features is improved, making it easier to discover and enhance local patterns that are critical to classification or recognition tasks.

[0034] More specifically, pixel-level feature decoupling is performed on the semantic coding feature map of the fire-fighting facilities image in the building using the following feature decoupling formula to obtain a set of pixel-level decoupled feature vectors of the fire-fighting facilities in the building; wherein the feature decoupling formula is:

[0035]

[0036] in, for each semantic coding feature map of an image of fire-fighting facilities in a building in the set of semantic coding feature maps of images of fire-fighting facilities in a building, For Decouple pixel-level features. , , and They are the first, second, and third pixel-size decoupling feature vectors of fire-fighting facilities in buildings. and The pixel-granularity decoupling feature vector of fire-fighting facilities in a building.

[0037] Next, the significant features of the set of pixel granularity decoupling feature vectors of the fire-fighting facilities in the building are calculated to obtain the semantic saliency distribution matrix of the structure of the fire-fighting facilities in the building. Specifically, each pixel granularity decoupling feature vector of the fire-fighting facilities in the building is input into the hyperbolic space mapper to obtain the set of hyperbolic space modulation pixel granularity decoupling feature vectors of the fire-fighting facilities in the building; and then the fine-grained structural semantic saliency measurement coefficient of each hyperbolic space modulation pixel granularity decoupling feature vector of the fire-fighting facilities in the building is calculated to obtain the semantic saliency distribution matrix of the structure of the fire-fighting facilities in the building, which is formed by arranging the fine-grained structural semantic saliency measurement coefficients of the fire-fighting facilities in the building. That is, the set of pixel granularity decoupling feature vectors of the fire-fighting facilities in the building is sent to the hyperbolic space mapper to transform the features, that is, map from the Euclidean geometric space to the non-Euclidean geometric space, so as to better preserve the hierarchical relationship and topological structure between high-dimensional data. Hyperbolic geometry provides a unique perspective to deal with complex data structures, especially when dealing with hierarchical data sets, where distance metrics in hyperbolic space are more suitable for reflecting hierarchical data structures. The operation of mapping to hyperbolic space can effectively maintain the relative distance between input features and may reveal hidden hierarchical structures, thereby improving feature expression capabilities. Then, the fine-grained structural semantic saliency metric coefficient of each vector in the set of hyperbolic space modulated pixel granularity decoupling feature vectors of the fire-fighting facilities in the building is calculated to obtain the structural semantic saliency distribution matrix of the fire-fighting facilities in the building. The structural semantic saliency distribution matrix actually quantifies the importance of each fine-grained decoupling feature, and evaluates the contribution of each feature in a specific context by defining a metric function. The distribution matrix composed of these metric coefficients reflects the semantic saliency of each part in the semantic encoding feature map of the fire-fighting facilities image in the building.

[0038] More specifically, the fine-grained structure semantic saliency measurement coefficient of each building fire-fighting facility hyperbolic space modulation pixel granularity decoupling feature vector in the set of the building fire-fighting facility hyperbolic space modulation pixel granularity decoupling feature vector is calculated by the following fine-grained structure semantic saliency measurement coefficient calculation formula to obtain the building fire-fighting facility structure semantic saliency distribution matrix formed by arranging the fine-grained structure semantic saliency measurement coefficients of multiple building fire-fighting facilities; wherein the formula for the fine-grained structure semantic saliency measurement coefficient is:

[0039]

[0040]

[0041] in, and are the first weight matrix and the second weight matrix respectively, for The corresponding hyperbolic spatial modulation pixel granularity decoupling feature vector of the fire protection facilities in the building, To calculate the square of the vector-norm, represents the logarithmic function value with base 2, is the semantic saliency distribution matrix of the fire protection facilities structure in the building The corresponding semantic saliency coefficient of the fine-grained structure of fire-fighting facilities in the building.

[0042] Then, each eigenvalue in the semantic significance distribution matrix of the fire-fighting facilities structure in the building is weighted to obtain a semantic significance distribution weight matrix of the fire-fighting facilities structure in the building composed of multiple semantic weights of the fire-fighting facilities structure in the building. That is, in the technical solution of the present application, the semantic significance distribution matrix of the fire-fighting facilities structure in the building is input into a feature explicit hierarchical precipitation module based on a multi-layer mask function to obtain the semantic significance distribution weight matrix of the fire-fighting facilities structure in the building. That is, after obtaining the semantic significance distribution matrix of the fire-fighting facilities structure in the building, it is passed as input to a feature explicit hierarchical precipitation module based on a multi-layer mask function to achieve selective amplification of different parts in the semantic encoding feature map of the fire-fighting facilities image in the building by applying different masks to features at different levels, thereby strengthening those feature parts that are considered important and ignoring minor features. Using the semantic significance distribution matrix of the fire-fighting facilities structure in the building as a guide, the feature explicit hierarchical precipitation module highlights important features through a multi-layer mask mechanism while suppressing noise or irrelevant features.

[0043] More specifically, the following weight conversion formula is used to perform weight conversion on each eigenvalue in the semantic significance distribution matrix of the fire protection facility structure in the building to obtain a semantic significance distribution weight matrix of the fire protection facility structure in the building composed of multiple semantic weights of the fire protection facility structure in the building; wherein the formula is:

[0044]

[0045] in, is the preset threshold, For masking operation, is the semantic saliency weight matrix of the fire protection facilities structure in the building The corresponding semantic significance distribution weight value of the fire protection facility structure in the building.

[0046] Subsequently, the semantic significance distribution weight matrix of the fire-fighting facilities structure in the building and each feature matrix of the semantic coding feature map of the fire-fighting facilities image in the building are multiplied by the position points along the channel dimension to obtain the semantic enhancement coding feature map of the fire-fighting facilities image in the building as the semantic enhancement coding feature of the fire-fighting facilities image in the building. That is, the semantic significance distribution weight matrix of the fire-fighting facilities structure in the building is multiplied element by element with the semantic coding feature map of the fire-fighting facilities image in the building, that is, the feature matrix is ​​multiplied bitwise along the channel dimension to generate an enhanced feature map. This step realizes the weighted adjustment of the original feature map, ensuring that the feature map outputted in the end can highlight the feature information that is identified as the most representative and discriminative.

[0047] More specifically, the semantic significance distribution weight matrix of the fire-fighting facilities structure in the building is calculated by multiplying the position points of each feature matrix of the semantic coding feature map of the fire-fighting facilities image in the building along the channel dimension by the following weighted formula to obtain the semantic enhancement coding feature map of the fire-fighting facilities image in the building in the spatial domain; wherein the weighted formula is:

[0048]

[0049] in, is the semantic saliency distribution weight matrix of the fire protection facilities structure in the building, In order to use the matrix to weight each feature matrix along the channel dimension of the feature map, It is a spatial domain semantic enhancement coding feature map of the fire-fighting facilities image in the building.

[0050] Finally, based on the set of spatial domain semantic enhancement coding features of the fire-fighting facilities images in the building, the fire-fighting facilities abnormal value is obtained. Specifically, each of the spatial domain semantic enhancement coding feature maps of the fire-fighting facilities images in the building in the set of spatial domain semantic enhancement coding feature maps is input into the fire-fighting facilities state identifier based on the classifier to obtain a set of state recognition results, and the state recognition results are used to indicate whether the fire-fighting facilities are abnormal; that is, each of the spatial domain semantic enhancement coding feature maps of the fire-fighting facilities images in the building in the set of spatial domain semantic enhancement coding feature maps is input into the fire-fighting facilities state identifier based on the classifier to obtain a set of state recognition results, and the state recognition results are used to indicate whether the fire-fighting facilities are abnormal, and the number of fire-fighting facilities with abnormal state recognition results in the set of state recognition results is counted, and the number is divided by the total number of the set of state recognition results as the fire-fighting facilities abnormal value. That is, the set of spatial domain semantic enhancement coding features of the fire-fighting facilities images in the building obtained by layered spatial domain semantic enhancement using the set of semantic coding feature maps of the fire-fighting facilities images in the building is classified, so as to automatically determine whether each fire-fighting facility is abnormal and generate a fire-fighting facilities abnormal value. Specifically, the specific process of inputting each of the spatial domain semantic enhancement coded feature maps of the fire-fighting facility images in the building in the set of spatial domain semantic enhancement coded feature maps into a fire-fighting facility state identifier based on a classifier to obtain a set of state recognition results includes: expanding each of the spatial domain semantic enhancement coded feature maps of the fire-fighting facility images in the building in the set of spatial domain semantic enhancement coded feature maps into a set of classification feature vectors based on row vectors or column vectors; using multiple fully connected layers of the classifier to fully connect encode each of the classification feature vectors in the set of classification feature vectors to obtain a set of encoded classification feature vectors; and passing the set of encoded classification feature vectors through the Softmax classification function of the classifier to obtain the set of state recognition results.

[0051] Preferably, the set of state recognition results obtained by inputting the spatial domain semantic enhancement coding feature maps of the fire-fighting facility images in the building in the set of spatial domain semantic enhancement coding feature maps of the fire-fighting facility images in the building into a fire-fighting facility state recognizer based on a classifier comprises:

[0052] The absolute value sum and the square root of the square sum of all eigenvalues ​​of the spatial domain semantic enhancement coding feature graphs of the fire-fighting facilities images in each building are calculated to obtain the spatial structure value of the spatial domain semantic enhancement coding of the fire-fighting facilities images in the first and second buildings, that is:

[0053]

[0054]

[0055] in, represents each feature value in the spatial domain semantic enhancement coding feature map of the fire-fighting facilities image in the building, represents the spatial structure value of the spatial domain semantic enhancement encoding of the fire-fighting facilities image in the first building, Represents the spatial structure value of the spatial domain semantic enhancement encoding of the fire-fighting facilities image in the second building;

[0056] Determine the total number of eigenvalues ​​of all eigenvalues ​​of the set of spatial domain semantic enhancement coding feature maps of the fire-fighting facilities image in the building ;

[0057] For each eigenvalue of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building, the long-range dependency value of the spatial domain semantic enhancement coding of the fire-fighting facilities images in the first building is obtained by subtracting the product of the eigenvalue and the total number of eigenvalues ​​from the spatial structure value of the spatial domain semantic enhancement coding of the fire-fighting facilities images in the first building. , and calculate the square root of the total number of eigenvalues ​​multiplied by the product of the eigenvalues ​​minus the spatial structure value of the second fire-fighting facility image spatial domain semantic enhancement coding to obtain the long-range dependency value of the second fire-fighting facility image spatial domain semantic enhancement coding ;

[0058] The exponential value calculated by taking the long-range dependency value of the semantic enhancement coding of the space domain of the fire-fighting facilities image in the first building as the exponent of the natural constant and the inverse of the long-range dependency value of the semantic enhancement coding of the space domain of the fire-fighting facilities image in the second building are weighted summed to obtain the optimized feature value ;as well as

[0059] Each of the spatial domain semantic enhancement coding feature maps of the fire-fighting facilities images in the building in the set of optimized spatial domain semantic enhancement coding feature maps of the fire-fighting facilities images in the building composed of the optimized feature values ​​is input into the fire-fighting facilities state identifier based on the classifier to obtain a set of state recognition results.

[0060] Here, in the case where each semantic coding feature map of in-building fire protection facilities images in the set of semantic coding feature maps of in-building fire protection facilities images represents the image semantic features of the corresponding in-building fire protection facilities image data, after performing feature enhancement based on feature explicit hierarchical precipitation, each spatial domain semantic enhancement coding feature map of in-building fire protection facilities images in the set of spatial domain semantic enhancement coding feature maps of in-building fire protection facilities images will also have significant feature explicit spatial structure differences due to the differences in the hierarchical precipitation of image semantic features, affecting the convergence consistency of the classifier and thereby affecting the consistency of the set of state recognition results.

[0061] Based on this, in view of the possible lack of spatial structure consistency in the feature set of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building in the high-dimensional space, which causes the weight matrix of the classifier to implicitly infer the spatial structure information based on the features, resulting in inconsistent convergence. A long-distance feature dependency relationship is established based on the overall feature scale of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building relative to the spatial structure representation of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building, so as to establish the feature local connectivity of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building, and the spatial ambiguous information of the object feature value is captured by predicting the unstructured feature value points of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building, thereby improving the spatial inductive bias perception ability of the feature set of the spatial domain semantic enhancement coding feature map of the fire-fighting facilities images in each building, improving the convergence consistency of the classifier, and improving the consistency of the set of state recognition results obtained by the fire-fighting facility state identifier based on the classifier.

[0062] Specifically, the S22 obtains the abnormal coefficient of the fire-fighting equipment based on the abnormal value of the fire-fighting facility and the fire-fighting equipment information. That is, first, the current water pressure value of the fire-fighting equipment is obtained through the sensor. The fire-fighting water pressure is one of the key indicators to measure whether the fire-fighting equipment is working properly. Too low or too high water pressure may cause the equipment to fail; and the accumulated usage time of the fire-fighting equipment is obtained through the sensor. The usage time can help evaluate the degree of wear and life of the equipment. Equipment that has been used for a long time may require more maintenance and inspection. In an example, the fire-fighting water pressure coefficient can be obtained by calculating the deviation between the fire-fighting water pressure and the safety range; and the usage time coefficient is obtained by calculating the ratio of the usage time of the fire-fighting equipment to the maximum usage time. The above three coefficients (fire-fighting water pressure coefficient, usage time coefficient, and abnormal value coefficient) are calculated through a comprehensive formula to obtain the abnormal coefficient of the fire-fighting equipment.

[0063]

[0064] in, , , is the weight coefficient used to balance the influence of different factors, exp() represents the exponential function of the base of natural logarithm e, represents the fire water pressure coefficient, represents the usage time coefficient, represents the outlier coefficient.

[0065] In particular, the S3 obtains the abnormality coefficient of the fire lane sign light based on the fire lane sign light image. In one example, the distance of the fire sign light relative to the specified position can be obtained through image processing technology (such as edge detection, contour extraction, etc.) to determine whether the sign light has deviated from the predetermined position, thereby affecting its normal use; and the brightness value of the fire sign light is obtained through image processing technology. The brightness value can reflect whether the sign light is in normal working condition. Too low or too high brightness may indicate a fault; in addition, the number of normal uses of the fire sign light is counted through image processing technology. This step helps to evaluate the integrity rate of the overall sign light and ensure sufficient safety identification. Based on the extracted feature information (position, brightness, number of normal uses), the abnormality coefficient of the fire sign light is obtained.

[0066] Specifically, based on the extracted characteristic information (position, brightness, and normal usage quantity), the abnormality coefficient of the fire sign lamp is obtained, including: the distance of the position of the fire sign lamp relative to the specified position, the brightness of the fire sign lamp, and the normal usage quantity are introduced into the fire sign lamp abnormality coefficient calculation formula to calculate the fire sign lamp abnormality coefficient, wherein the fire sign lamp abnormality coefficient calculation formula is:

[0067]

[0068] in, Indicates the number of fire sign lamps. Indicates the number of fire sign lamps in normal use. Indicates the distance of the position of the qth fire sign lamp relative to the specified position, Indicates the maximum safe distance of the set fire sign lamp relative to the specified position. Indicates the brightness of the qth fire sign lamp, Indicates the median value of the safe range of brightness of fire sign lamps. Indicates the maximum value of the safe range of brightness of fire sign lamps. Indicates the minimum value of the safe range of brightness of fire sign lamps. represents the distance ratio, represents the brightness ratio, and + =1, Indicates the abnormal coefficient of fire sign lighting.

[0069] In particular, the S4 obtains the building safety factor based on the abnormal coefficient of the fire equipment and the abnormal coefficient of the fire passage sign light, and determines whether building safety rectification is required based on the comparison between the building safety factor and the preset building safety factor threshold. That is, in a specific example, first, the calculated abnormal coefficient of the fire equipment and the abnormal coefficient of the fire sign light are obtained, and the calculated abnormal coefficient of the fire equipment and the abnormal coefficient of the fire sign light are introduced into the safety factor calculation formula to calculate the building safety factor; wherein, the building safety factor calculation formula is:

[0070]

[0071] in, Indicates the abnormal proportion coefficient of fire equipment, Indicates the abnormal proportion coefficient of fire sign lamps, and + =1.

[0072] Furthermore, based on the comparison between the building safety factor and a preset building safety factor threshold, it is determined whether building safety rectification is required. In one example, in response to the building safety factor being greater than the preset building safety factor threshold, it is output that building safety rectification is required; otherwise, building safety rectification is not required.

[0073] In summary, the method for intelligent building safety monitoring based on the Internet of Things according to the embodiment of the present application is explained, which obtains a collection of image data of fire-fighting facilities in the building from the cloud platform, and uses artificial intelligence-based image analysis and extraction technology to extract image semantic features of the image data of fire-fighting facilities in each building, so as to automatically judge whether each fire-fighting facility has an abnormality according to the layered spatial domain enhanced representation of the semantic coding features of the images of fire-fighting facilities in each building, and generate fire-fighting facility abnormality values. In this way, the high-level semantic information in the image can be better understood, so as to more accurately judge whether the fire-fighting facilities have an abnormality, and reduce misjudgments caused by differences in shape or model. At the same time, it can distinguish between changes in non-target objects in the background and the fire-fighting facilities themselves, avoid false alarms caused by background interference, and thus improve the intelligence of abnormal judgment of fire-fighting facilities.

[0074] Furthermore, a building safety intelligent monitoring system based on the Internet of Things is also provided.

[0075] Figure 4 FIG. 1 is a block diagram of an intelligent building safety monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 4As shown, according to the embodiment of the present application, the building safety intelligent monitoring system 300 based on the Internet of Things includes: an information acquisition module 310, which is used to acquire the image data of fire-fighting facilities, fire-fighting equipment information and the image of fire-fighting passageway marker lights in the building; a fire-fighting equipment abnormality identification module 320, which is used to identify the abnormality of fire-fighting equipment based on the image data of fire-fighting facilities in the building and the fire-fighting equipment information to obtain the abnormality coefficient of fire-fighting equipment, including: a fire-fighting facility abnormality value generation unit, which is used to obtain the abnormal value of fire-fighting facilities based on the image data of fire-fighting facilities in the building; a fire-fighting equipment abnormality coefficient generation unit, which is used to obtain the abnormal coefficient of fire-fighting equipment based on the abnormal value of fire-fighting facilities and the fire-fighting equipment information; a marker light abnormality coefficient generation module 330, which is used to obtain the abnormal coefficient of fire-fighting passageway marker lights based on the image of the fire-fighting passageway marker lights; a building safety rectification determination module 340, which is used to obtain the building safety factor based on the fire-fighting equipment abnormality coefficient and the fire-fighting passageway marker light abnormality coefficient, and determine whether building safety rectification is required based on the comparison between the building safety factor and the preset building safety factor threshold.

[0076] As described above, the building security intelligent monitoring system 300 based on the Internet of Things according to the embodiment of the present application can be implemented in various wireless terminals, such as a server with a building security intelligent monitoring algorithm based on the Internet of Things. In a possible implementation, the building security intelligent monitoring system 300 based on the Internet of Things according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or a hardware module. For example, the building security intelligent monitoring system 300 based on the Internet of Things can be a software module in the operating system of the wireless terminal, or can be an application developed for the wireless terminal; of course, the building security intelligent monitoring system 300 based on the Internet of Things can also be one of the many hardware modules of the wireless terminal.

[0077] Alternatively, in another example, the IoT-based building security intelligent monitoring system 300 and the wireless terminal may also be separate devices, and the IoT-based building security intelligent monitoring system 300 may be connected to the wireless terminal via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0078] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A building safety intelligent monitoring method based on the Internet of Things, characterized in that: include: Obtain image data of fire-fighting facilities, fire-fighting equipment information and images of fire-fighting passage sign lights in buildings; Performing fire equipment abnormality identification based on the fire equipment image data in the building and the fire equipment information to obtain a fire equipment abnormality coefficient, including: obtaining a fire equipment abnormality value based on the fire equipment image data in the building; obtaining the fire equipment abnormality coefficient based on the fire equipment abnormality value and the fire equipment information; Based on the fire passage sign light image, obtaining a fire passage sign light abnormality coefficient; Based on the abnormal coefficient of the fire-fighting equipment and the abnormal coefficient of the fire-fighting passage sign light, a building safety factor is obtained, and based on a comparison between the building safety factor and a preset building safety factor threshold, whether building safety rectification is required; Wherein, based on the image data of the fire-fighting facilities in the building, the abnormal value of the fire-fighting facilities is obtained, including: Obtain a collection of image data of firefighting facilities in buildings from the cloud platform; Extracting image features of each in-building fire-fighting facility image data in the set of in-building fire-fighting facility image data to obtain a set of semantic coding features of the in-building fire-fighting facility images; Performing hierarchical spatial domain semantic enhancement on each of the semantic coding features of the fire-fighting facility images in the set of semantic coding features of the fire-fighting facility images in the building to obtain a set of spatial domain semantic enhancement coding features of the fire-fighting facility images in the building; Obtaining the fire-fighting facility outlier value based on a set of spatial domain semantic enhancement coding features of the fire-fighting facility image in the building; The method of performing hierarchical spatial domain semantic enhancement on each of the semantic coding features of the fire-fighting facilities images in the building to obtain a set of spatial domain semantic enhancement coding features of the fire-fighting facilities images in the building includes: Perform pixel-level feature decoupling on the semantic coding feature map of the fire-fighting facilities image in the building to obtain a set of pixel-level decoupling feature vectors of the fire-fighting facilities in the building; Calculating the saliency features of the set of pixel granularity decoupling feature vectors of the fire-fighting facilities in the building to obtain a semantic saliency distribution matrix of the fire-fighting facilities structure in the building; Performing weight conversion on each eigenvalue in the semantic significance distribution matrix of the fire protection facility structure in the building to obtain a semantic significance distribution weight matrix of the fire protection facility structure in the building composed of multiple semantic weights of the fire protection facility structure in the building; The semantic significance distribution weight matrix of the structure of the fire-fighting facilities in the building is calculated and multiplied by each feature matrix along the channel dimension of the semantic coding feature map of the image of the fire-fighting facilities in the building at each position point to obtain the spatial domain semantic enhancement coding feature map of the image of the fire-fighting facilities in the building as the spatial domain semantic enhancement coding feature of the image of the fire-fighting facilities in the building.

2. The method for intelligent building safety monitoring based on the Internet of Things according to claim 1 is characterized in that: Image feature extraction is performed on each in-building firefighting facility image data in the set of in-building firefighting facility image data to obtain a set of in-building firefighting facility image semantic coding features, including: inputting each in-building firefighting facility image data in the set of in-building firefighting facility image data into a firefighting facility image feature extractor based on a Mobile-Former network to obtain a set of in-building firefighting facility image semantic coding feature maps as the set of in-building firefighting facility image semantic coding features.

3. The method for intelligent building safety monitoring based on the Internet of Things according to claim 2 is characterized in that: Calculating the significant features of the set of pixel granularity decoupling feature vectors of the fire-fighting facilities in the building to obtain the semantic significance distribution matrix of the fire-fighting facilities structure in the building, including: Inputting each of the pixel granularity decoupling feature vectors of the fire-fighting facilities in the building into a hyperbolic space mapper to obtain a set of hyperbolic space modulation pixel granularity decoupling feature vectors of the fire-fighting facilities in the building; The fine-grained structural semantic significance measurement coefficient of each in-building fire-fighting facility hyperbolic spatial modulation pixel granularity decoupling feature vector in the set of in-building fire-fighting facility hyperbolic spatial modulation pixel granularity decoupling feature vectors is calculated to obtain the in-building fire-fighting facility structural semantic significance distribution matrix formed by arranging multiple in-building fire-fighting facility fine-grained structural semantic significance measurement coefficients.

4. The method for intelligent building safety monitoring based on the Internet of Things according to claim 3 is characterized in that: The method comprises the following steps: performing weight transformation on each eigenvalue in the semantic significance distribution matrix of the fire-fighting facility structure in the building to obtain a semantic significance distribution weight matrix of the fire-fighting facility structure in the building composed of multiple semantic weights of the fire-fighting facility structure in the building, including: inputting the semantic significance distribution matrix of the fire-fighting facility structure in the building into a feature explicit hierarchical precipitation module based on a multi-layer mask function to obtain the semantic significance distribution weight matrix of the fire-fighting facility structure in the building.

5. The method for intelligent building safety monitoring based on the Internet of Things according to claim 4 is characterized in that: Based on the set of spatial domain semantic enhancement coding features of the fire-fighting facility image in the building, the fire-fighting facility abnormal value is obtained, including: Inputting the spatial domain semantic enhancement coding feature maps of the fire-fighting facility images in the building into a fire-fighting facility state identifier based on a classifier to obtain a set of state recognition results, wherein the state recognition results are used to indicate whether there is an abnormality in the fire-fighting facility; Based on the set of status identification results, the fire-fighting facility abnormal value is obtained.

6. The method for intelligent building safety monitoring based on the Internet of Things according to claim 5 is characterized in that: Inputting each of the spatial domain semantic enhancement coding feature maps of the fire-fighting facility images in the building in the set of spatial domain semantic enhancement coding feature maps into the fire-fighting facility state identifier based on the classifier to obtain a set of state recognition results, including: Expanding each of the spatial domain semantic enhancement coding feature maps of the fire-fighting facility images in the building into a set of classification feature vectors based on row vectors or column vectors in the set of spatial domain semantic enhancement coding feature maps of the fire-fighting facility images in the building; Using a plurality of fully connected layers of the classifier, fully connected encoding is performed on each classification feature vector in the set of classification feature vectors to obtain a set of encoded classification feature vectors; and The set of the encoded classification feature vectors is passed through the Softmax classification function of the classifier to obtain the set of state recognition results.

7. The method for intelligent building safety monitoring based on the Internet of Things according to claim 6 is characterized in that: Based on the set of status identification results, the abnormal value of the fire-fighting facility is obtained, including: counting the number of fire-fighting facilities with abnormal status identification results in the set of status identification results, and dividing the number by the total number of the set of status identification results as the abnormal value of the fire-fighting facility.

8. An intelligent building safety monitoring system based on the Internet of Things, using the intelligent building safety monitoring method based on the Internet of Things according to any one of claims 1 to 7, characterized in that: include: An information acquisition module is used to acquire image data of fire-fighting facilities in a building, information on fire-fighting equipment, and images of fire-fighting passage sign lights; A fire equipment abnormality identification module, used for performing fire equipment abnormality identification based on the fire equipment image data in the building and the fire equipment information to obtain a fire equipment abnormality coefficient, comprising: a fire equipment abnormality value generation unit, used for obtaining a fire equipment abnormality value based on the fire equipment image data in the building; a fire equipment abnormality coefficient generation unit, used for obtaining the fire equipment abnormality coefficient based on the fire equipment abnormality value and the fire equipment information; A sign light abnormality coefficient generating module, used for obtaining a fire passage sign light abnormality coefficient based on the fire passage sign light image; The building safety rectification determination module is used to obtain the building safety factor based on the fire equipment abnormality coefficient and the fire passage marker light abnormality coefficient, and determine whether the building safety rectification is needed based on the comparison between the building safety factor and a preset building safety factor threshold.

Images