Smart park security management system and method based on Internet of Things

Through the Internet of Things combined with deep learning, the multimodal fusion of visual and text data is used to solve the problem of insufficient real-time and intelligence of traditional campus risk assessment methods, and more efficient security incident identification and response are achieved.

CN118673390BActive Publication Date: 2024-11-22HANGZHOU ANZHIWEI IOT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411155782.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-22
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

Traditional park risk level assessment methods rely on a single type of data, which is difficult to fully reflect the current security situation in real time, and the degree of intelligence of manual monitoring is low, making it impossible to identify small-scale environmental changes and minor security violations.

Method used

The smart park security management system based on the Internet of Things is adopted to obtain visual state images and user-input status text descriptions through the camera, and features are extracted using deep learning algorithms and multimodal fusion to automatically evaluate risk levels.

Benefits of technology

Real-time and comprehensive security information analysis, timely discover security incidents, and independently identify small-scale environmental changes and minor security violations, improving the intelligence level of park safety management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673390B_ABST
    Figure CN118673390B_ABST
Patent Text Reader

Abstract

The present application provides a smart park security management system and method based on the Internet of Things, which relates to the field of intelligent security management. It uses a data analysis algorithm based on deep learning to extract and enhance the features of the visual state image of the park area to be evaluated, and at the same time, semantically encodes the state text description of the park area to be evaluated, so as to automatically evaluate the risk level of the park area to be evaluated based on the multimodal fusion features between the visual state image features and the state text semantic features of the park area to be evaluated. In this way, by analyzing more comprehensive security information in the park area, it is helpful to timely discover and respond to security incidents, and at the same time, it can autonomously identify small-scale environmental changes and minor security violations in the park area, thereby improving the intelligent level of park security management.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent security management, and more specifically, to a smart park security management system and method based on the Internet of Things. Background Art

[0002] With the acceleration of urbanization, various industrial parks, science and technology parks, business parks, etc. have developed rapidly, the scale of the parks has continued to expand, and the functions have become increasingly complex. The activities in the parks are diverse, including production, logistics, warehousing, office, etc. These activities may bring various safety risks, such as fire, explosion, leakage, traffic accidents, etc. Therefore, by evaluating the risk level of the park, potential safety hazards can be discovered in time, preventive measures can be taken, and the risk of accidents can be reduced.

[0003] However, traditional park risk level assessment methods usually rely on a single type of data, such as historical accident records or on-site inspection reports, which can only provide limited information and are usually reviewed after the fact, making it difficult to fully reflect the current safety status of the park in real time, limiting the effectiveness and response speed of park safety management. In addition, some existing methods of monitoring park safety through cameras usually require manual monitoring, but this method has a low degree of intelligence and cannot notice details in the image, such as small-scale environmental changes, minor safety violations and hidden dangers, etc. These details are not easy to detect, but may indicate potential safety risks.

[0004] Therefore, an optimized campus security management solution is needed. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiments of this application provide a smart park security management system and method based on the Internet of Things.

[0006] According to one aspect of the present application, there is provided a smart park security management system based on the Internet of Things, which includes: a visual state image acquisition module of the park area to be evaluated, used to acquire the visual state image of the park area to be evaluated collected by a camera; a state text description acquisition module of the park area to be evaluated, used to acquire the state text description of the park area to be evaluated input by a user; a park visual state feature extraction module, used to input the visual state image into a park visual feature extractor to obtain a park visual state feature map; a park visual state feature enhancement module, used to input the park visual state feature map into a feature space structure maintenance self-attention enhancement module based on cross-channel intersection to obtain a park visual state enhancement feature map; the park area to be evaluated A domain status text description encoding module is used to semantically encode the status text description of the park area to be evaluated to obtain a semantic encoding feature vector of the park area status to be evaluated; a multimodal joint encoding module of the park area to be evaluated is used to input the semantic encoding feature vector of the park area status to be evaluated and the park visual status enhancement feature map into a heterogeneous data-driven multimodal Transformer joint encoder to obtain a multimodal joint encoding feature map of the park area status to be evaluated as a multimodal joint encoding feature of the park area status to be evaluated; an evaluation result generation module is used to obtain an evaluation result based on the multimodal joint encoding feature of the park area status to be evaluated, and the evaluation result is used to represent the risk level label of the park area to be evaluated.

[0007] According to another aspect of the present application, a smart park security management method based on the Internet of Things is provided, which includes: obtaining a visual status image of a park area to be evaluated captured by a camera; obtaining a text description of the status of the park area to be evaluated input by a user; inputting the visual status image into a park visual feature extractor to obtain a park visual status feature map; inputting the park visual status feature map into a feature space structure-maintaining self-attention enhancement module based on cross-channel intersection to obtain a park visual status enhancement feature map; semantically encoding the status text description of the park area to be evaluated to obtain a semantically encoded feature vector of the park area to be evaluated; inputting the semantically encoded feature vector of the park area to be evaluated and the park visual status enhancement feature map into a heterogeneous data-driven multimodal Transformer joint encoder to obtain a multimodal joint encoding feature map of the park area to be evaluated as a multimodal joint encoding feature of the park area to be evaluated; based on the multimodal joint encoding feature of the park area to be evaluated, obtaining an evaluation result, and the evaluation result is used to represent a risk level label of the park area to be evaluated.

[0008] Compared with the prior art, the IoT-based smart park security management system and method provided by this application uses a deep learning-based data analysis algorithm to extract and enhance features of the visual state image of the park area to be evaluated, and semantically encodes the state text description of the park area to be evaluated, so as to automatically evaluate the risk level of the park area to be evaluated based on the multimodal fusion features between the visual state image features and the state text semantic features of the park area to be evaluated. In this way, by analyzing more comprehensive security information in the park area, it is helpful to discover and respond to security incidents in a timely manner, and at the same time, it can autonomously identify small-scale environmental changes and minor security violations in the park area, thereby improving the intelligent level of park security management. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] In the attached picture: Figure 1 This is a system block diagram of an IoT-based smart park security management system according to an embodiment of the present application.

[0011] Figure 2 This is a data flow diagram of the IoT-based smart park security management system according to an embodiment of the present application.

[0012] Figure 3 It is a block diagram of the park visual status feature enhancement module in the smart park security management system based on the Internet of Things according to the embodiment of the present application.

[0013] Figure 4 It is a block diagram of a multimodal joint encoding module of a park area to be evaluated in an IoT-based smart park security management system according to an embodiment of the present application.

[0014] Figure 5 It is a flow chart of a smart park security management method based on the Internet of Things according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0016] With the acceleration of urbanization, various parks such as industrial parks, science and technology parks and business parks have expanded rapidly, and their functions have become increasingly diversified, involving production, logistics, warehousing and office activities. These activities may cause a variety of safety risks, such as fire, explosion, leakage and traffic accidents. Traditional risk assessment methods rely on limited data sources, such as historical accident records or on-site inspection reports. These data are often analyzed after the fact, and it is difficult to fully reflect the safety status of the park in real time, affecting the efficiency and response speed of safety management. In addition, the camera system that relies on manual monitoring has limited intelligence and is difficult to capture subtle environmental changes and potential safety violations. These details may indicate major safety risks.

[0017] Based on this, the present application proposes a technical solution for smart park security management based on the Internet of Things, which uses a camera to collect visual status images of the park area to be evaluated, and a text description of the status of the park area to be evaluated input by the user, and uses an image data analysis and processing algorithm based on the Internet of Things and deep learning to extract and enhance features of the visual status image, and at the same time, semantically encodes the status text description, so as to automatically evaluate the risk level of the park area to be evaluated based on the multimodal fusion features between the visual status image features and the status text semantic features of the park area to be evaluated. In this way, more comprehensive security information is provided through the semantics of the park area image collected by the camera and the text semantics of the state of the area, which helps to timely discover and respond to security incidents. And through deep learning technology, it can autonomously identify details such as small-scale environmental changes and minor security violations in the image, thereby improving the level of intelligence of park security management.

[0018] Figure 1 This is a system block diagram of an IoT-based smart park security management system according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of the smart park security management system based on the Internet of Things according to an embodiment of the present application. Figure 1 and Figure 2As shown, in the smart park security management system 100 based on the Internet of Things, it includes: a visual state image acquisition module 110 of the park area to be evaluated, which is used to obtain the visual state image of the park area to be evaluated collected by a camera; a state text description acquisition module 120 of the park area to be evaluated, which is used to obtain the state text description of the park area to be evaluated input by the user; a park visual state feature extraction module 130, which is used to input the visual state image into the park visual feature extractor to obtain a park visual state feature map; a park visual state feature enhancement module 140, which is used to input the park visual state feature map into a feature space structure maintenance self-attention enhancement module based on cross-channel intersection to obtain a park visual state enhancement feature map; the park area state to be evaluated The text description encoding module 150 is used to semantically encode the text description of the status of the park area to be evaluated to obtain a semantic encoding feature vector of the status of the park area to be evaluated; the multimodal joint encoding module 160 of the park area to be evaluated is used to input the semantic encoding feature vector of the status of the park area to be evaluated and the park visual status enhancement feature map into a heterogeneous data-driven multimodal Transformer joint encoder to obtain a multimodal joint encoding feature map of the status of the park area to be evaluated as a multimodal joint encoding feature of the status of the park area to be evaluated; the evaluation result generation module 170 is used to obtain an evaluation result based on the multimodal joint encoding feature of the status of the park area to be evaluated, and the evaluation result is used to represent the risk level label of the park area to be evaluated.

[0019] In the embodiment of the present application, the visual state image acquisition module 110 of the park area to be evaluated and the state text description acquisition module 120 of the park area to be evaluated are respectively used to obtain the visual state image of the park area to be evaluated collected by the camera and obtain the state text description of the park area to be evaluated input by the user. It should be understood that the visual state image of the park area to be evaluated provides the real-time state of the park area to be evaluated, and displays the overall environment, buildings, equipment and personnel activity information of the park. This information provides basic visual information for subsequent park risk assessment and helps identify potential safety hazards. The state text description of the park area to be evaluated input by the user provides a record of the results of the on-site inspection in the park area, which mainly records the problems found in the park safety inspection. The text description provides rich contextual information for risk assessment and supplements the possible missing parts in the park visual data. In general, by combining the visual state image information of the park area to be evaluated and the state text description information of the park area to be evaluated input by the user, a comprehensive park safety state view can be formed. By analyzing the acquired data, it is conducive to realizing a real-time and comprehensive risk assessment of the park, thereby timely discovering potential safety problems.

[0020] In an embodiment of the present application, the park visual state feature extraction module 130 is used to input the visual state image into the park visual feature extractor to obtain a park visual state feature map. Specifically, in an embodiment of the present application, the park visual state feature extraction module is used to: input the visual state image into the park visual feature extractor based on the hole convolutional neural network model to obtain the park visual state feature map. It should be understood that the visual state image contains a wide range of spatial range features, such as vehicle movement, crowd gathering, etc. The hole convolutional neural network model can expand the receptive field and help capture patterns in a larger spatial range. Based on this, in the technical solution of the present application, the visual state image is input into the park visual feature extractor based on the hole convolutional neural network model to obtain a park visual state feature map. In particular, the essence of the park visual feature extractor based on the hole convolutional neural network model here is the hole convolutional neural network model.

[0021] In an embodiment of the present application, the park visual state feature enhancement module 140 is used to input the park visual state feature map into a feature space structure preservation self-attention enhancement module based on cross-channel intersection to obtain a park visual state enhancement feature map. Accordingly, it is considered that there is a mutual correlation and interaction relationship between the park visual state features in different channels in the park visual state feature map, and there is corresponding spatial structure feature information in each channel. Therefore, in order to make the visual state information context information in each channel in the park visual state feature map better associated and interactive, while maintaining the spatial feature structure information in each channel, in the technical solution of the present application, the channel features of the park visual state feature map are enhanced by using a feature space structure preservation self-attention enhancement module based on cross-channel intersection to obtain a park visual state enhancement feature map.

[0022] Specifically, first, the park visual state feature map is subjected to layer normalization to keep the output distribution between each layer relatively stable. Then, the layer-normalized features are sequentially subjected to point convolution processing and convolution encoding to promote the correlation interaction between the park visual state features in each channel and capture and extract the spatial features of the park visual state. After the channel and spatial context information of the park visual state features are fully extracted, a feature backup step is introduced to retain the spatial context information of the original park visual state features. After that, the channel space and backup information are reshaped to be converted into a matrix form, and the Transformer structure is used to perform global correlation interaction across channels of the park visual state features, so as to capture the feature correlation between each channel to obtain the park visual state channel-spatial context correlation representation feature matrix, and then the correlation representation matrix is ​​activated by the activation function to extract the key information of the visual state and obtain the global interaction attention matrix of the park visual state features. Finally, the global interaction attention matrix is ​​multiplied with the backup features and reshaped to obtain the enhanced park visual state enhanced feature map.

[0023] Specifically, Figure 3 FIG. 1 is a block diagram of a park visual status feature enhancement module in a smart park security management system based on the Internet of Things according to an embodiment of the present application. Figure 3As shown, the park visual state feature enhancement module 140 includes: a park visual state feature map normalization processing unit 141, which is used to perform layer normalization on the park visual state feature map to obtain a normalized park visual state feature map; a normalized park visual state feature point convolution processing unit 142, which is used to perform point convolution processing on the normalized park visual state feature map to obtain a park visual state channel context association representation feature map; a park visual state channel context association representation feature point convolution encoding unit 143, which is used to perform convolution encoding on the park visual state channel context association representation feature map to obtain a park visual state space context association representation feature map; a park visual state space context association representation feature backup unit 144, which is used to copy the park visual state space context association representation feature map to obtain a park visual state backup space context association representation feature map; a park visual state context association representation feature shape reshaping unit 145, which is used to reshape the park visual state channel context association representation feature map, the park visual state space context association representation feature map and the park visual state backup space context association table respectively. The characteristic graph is reshaped to obtain the park visual state channel context association representation feature matrix, the park visual state space context association representation feature matrix and the park visual state backup space context association representation feature matrix; the park visual state cross-channel covariance calculation unit 146 is used to calculate the park visual state cross-channel cross covariance matrix between the park visual state channel context association representation feature matrix and the park visual state space context association representation feature matrix; the park visual state feature global interaction unit 147 is used to activate the park visual state cross-channel cross covariance matrix to obtain the park visual state feature global interaction attention matrix; the park visual state attention enhancement feature representation generation unit 148 is used to calculate the product between the park visual state backup space context association representation feature matrix and the park visual state feature global interaction attention matrix to obtain the park visual state attention enhancement feature representation matrix; the park visual state enhancement feature generation unit 149 is used to reshape the park visual state attention enhancement feature representation matrix to obtain the park visual state enhancement feature graph.

[0024] More specifically, in an embodiment of the present application, the park visual state cross-channel covariance calculation unit is used to: calculate the matrix product between the park visual state channel context association representation feature matrix and the park visual state spatial context association representation feature matrix to obtain the park visual state channel-spatial context association representation feature matrix; calculate the positional division of the park visual state channel-spatial context association representation feature matrix and a predetermined hyperparameter to obtain the park visual state cross-channel cross-covariance matrix.

[0025] More specifically, in an embodiment of the present application, the park visual state feature global interaction unit is used to: use a Softmax function to activate the park visual state cross-channel cross-covariance matrix to obtain the park visual state feature global interaction attention matrix.

[0026] In the embodiment of the present application, specifically, the park visual state feature enhancement module is used to: input the park visual state feature map into the feature space structure maintenance self-attention enhancement module based on cross-channel intersection, and process it with the following self-attention enhancement formula to obtain the park visual state enhancement feature map; wherein the self-attention enhancement formula is: ;in, is the visual state feature map of the park, Indicates the layer normalization operation on the feature map. is the normalized park visual state feature map, is the point convolution operation, is a context-related representation feature map of the park visual state channel, The convolution kernel is The convolution operation, is a context-related representation feature map of the park visual state space, For copy operation, A spatial context association representation feature map is provided for the park visual state backup, For the reshape operation, , and are respectively the park visual state channel context association representation feature matrix, the park visual state space context association representation feature matrix and the park visual state backup space context association representation feature matrix, To predetermine the hyperparameters, for function, is the global interactive attention matrix of the park visual state features, is matrix multiplication, It is a feature map that enhances the visual status of the park.

[0027] In an embodiment of the present application, the text description encoding module 150 of the status of the park area to be evaluated is used to semantically encode the text description of the status of the park area to be evaluated to obtain a semantically encoded feature vector of the status of the park area to be evaluated. It should be understood that the text description of the status of the park area to be evaluated contains semantic context information about the status of the park area. In order to better understand the context information in the status text description and to improve the accuracy of the park risk level classification, therefore, in the technical solution of the present application, the text description of the status of the park area to be evaluated is semantically encoded to capture and extract complex semantic relationships, and obtain a semantically encoded feature vector of the status of the park area to be evaluated.

[0028] In an embodiment of the present application, the multimodal joint encoding module 160 of the park area to be evaluated is used to input the semantic encoding feature vector of the park area state to be evaluated and the park visual state enhancement feature map into a heterogeneous data-driven multimodal Transformer joint encoder to obtain the multimodal joint encoding feature map of the park area state to be evaluated as the multimodal joint encoding feature of the park area state to be evaluated. It should be understood that, considering that the semantic encoding feature vector of the park area state to be evaluated expresses the semantic information of the textual description of the state in the park, a textual expression is provided. The park visual state enhancement feature map expresses the visual feature representation information within a certain area of ​​the park, providing intuitive visual information. In order to integrate the feature data of the two different modalities together to form a unified representation so that the model can understand the situation of the park from multiple perspectives, in the technical solution of the present application, the semantic encoding feature vector of the park area state to be evaluated and the park visual state enhancement feature map are input into a heterogeneous data-driven multimodal Transformer joint encoder to obtain the multimodal joint encoding feature map of the park area state to be evaluated. That is, the heterogeneous data-driven multimodal Transformer joint encoder can utilize the key and important information between the two and realize the fusion between different modalities based on the self-attention mechanism of the Transformer structure. In detail, the semantic coding feature vector of the state of the park area to be evaluated is first linearly transformed and reshaped to generate the semantic coding modulation matrix of the state of the park area to be evaluated. The park visual state enhancement feature map is linearly transformed and reshaped twice to obtain the first linear transformation park visual state enhancement feature matrix and the second linear transformation park visual state enhancement feature matrix. Then, the three matrices are dynamically associated and interacted between channels across modalities through the self-attention mechanism of the Transformer. Subsequently, the features obtained after the interaction are reshaped and point convolutionally encoded to capture and utilize cross-modal information to obtain the park area semantic coding attention query park visual state feature map to be evaluated. Finally, the feature is multiplied with the original park visual state enhancement feature to integrate the interactive information and visual information between the two.

[0029] Specifically, Figure 4 FIG. 1 is a block diagram of a multi-modal joint coding module for a park area to be evaluated in a smart park security management system based on the Internet of Things according to an embodiment of the present application. Figure 4 As shown, the multimodal joint encoding module 160 of the park area to be evaluated includes: a linear transformation shape reshaping unit 161 of the park area state feature to be evaluated, which is used to perform linear transformation and feature shape reshaping on the semantic coding feature vector of the park area state to be evaluated to obtain the semantic coding modulation matrix of the park area state to be evaluated; a park visual state enhancement feature linear transformation unit 162, which is used to perform a first linear transformation and a second linear transformation on the park visual state enhancement feature map to obtain a first linear transformation park visual state enhancement feature map and a second linear transformation park visual state enhancement feature map; a linear transformation park visual state enhancement feature shape reshaping unit 163, which is used to perform feature shape reshaping on the first linear transformation park visual state enhancement feature map and the second linear transformation park visual state enhancement feature map to obtain a first linear transformation park visual state enhancement feature matrix and a second linear transformation park visual state enhancement feature matrix; a park area state attention calculation unit 164 to be evaluated, which is used to use the first linear transformation park visual state enhancement feature matrix and the second linear transformation park visual state enhancement feature matrix as key matrices and value matrices and the park area state semantic coding modulation matrix to be evaluated as a query matrix, and input the query matrix, the key matrix and the value matrix into a Transformer The structure is used to obtain the park area semantic coding attention query park visual state matrix to be evaluated; the park area semantic coding attention query park visual state feature generation unit 165 is used to reshape and point convolution code the park area semantic coding attention query park visual state matrix to be evaluated to obtain the park area semantic coding attention query park visual state feature map to be evaluated; the park area state multimodal joint coding feature generation unit 166 is used to multiply the park area semantic coding attention query park visual state feature map by the position point and the park visual state enhancement feature map to obtain the park area state to be evaluated. Multimodal joint coding feature map; wherein, the park area state attention calculation unit to be evaluated is used to: calculate the matrix product between the transposed matrix of the park area state semantic coding modulation matrix to be evaluated and the first linear transformation park visual state enhancement feature matrix, and then input it into the Softmax function to obtain the park area state semantic-visual feature association matrix to be evaluated; use the park area state semantic-visual feature association matrix to be evaluated as the weight matrix, calculate the matrix product between the weight matrix and the second linear transformation park visual state enhancement feature matrix to obtain the park area semantic coding attention query park visual state matrix to be evaluated.

[0030] More specifically, in an embodiment of the present application, the linear transformation reshape reshaping unit of the park area state feature to be evaluated is used to: matrix multiply the semantic coding feature vector of the park area state to be evaluated with the modulation matrix of the park area state to be evaluated, and then add the resultant vector by position with the bias vector of the park area state to be evaluated to obtain the semantic coding modulation bias vector of the park area state to be evaluated; input the semantic coding modulation bias vector of the park area state to be evaluated into a Sigmoid function for activation processing to obtain the semantic coding modulation activation vector of the park area state to be evaluated; and reshape the semantic coding modulation activation vector of the park area state to be evaluated to obtain the semantic coding modulation matrix of the park area state to be evaluated.

[0031] More specifically, in an embodiment of the present application, the park visual state enhancement feature linear transformation unit is used to: multiply each feature matrix along the channel dimension in the park visual state enhancement feature map by position with the first park visual state modulation matrix, and then add them by position with the first park visual state bias matrix to obtain the first park visual state bias feature map; input the first park visual state bias feature map into the Sigmoid function for activation processing to obtain the first linear transformation park visual state enhancement feature map; multiply each feature matrix along the channel dimension in the park visual state enhancement feature map by position with the second park visual state modulation matrix, and then add them by position with the second park visual state bias matrix to obtain the second park visual state bias feature map; input the second park visual state bias feature map into the Sigmoid function for activation processing to obtain the second linear transformation park visual state enhancement feature map.

[0032] In the embodiment of the present application, specifically, the multimodal joint encoding module of the park area to be evaluated is used to: input the semantic encoding feature vector of the park area state to be evaluated and the enhanced feature map of the park visual state into the heterogeneous data-driven multimodal Transformer joint encoder, and process them with the following joint encoding formula to obtain the multimodal joint encoding feature map of the park area state to be evaluated; wherein the joint encoding formula is:

[0033] ;in, is the semantic encoding feature vector of the state of the park area to be evaluated, is the regional state modulation matrix of the park to be evaluated, is the state bias vector of the park area to be evaluated, is the Sigmoid function, The semantic coding modulation activation vector of the state of the park area to be evaluated is For the reshape operation, is the semantic coding modulation matrix of the park area state to be evaluated, is the visual state enhancement feature map of the park, and are respectively the first park visual state modulation matrix and the second park visual state modulation matrix, and are respectively the first park visual state bias matrix and the second park visual state bias matrix, and are respectively the first linear transformation park visual state enhancement feature map and the second linear transformation park visual state enhancement feature map, and are respectively the first linear transformation park visual state enhancement feature matrix and the second linear transformation park visual state enhancement feature matrix, is the transposed matrix of the semantic coding modulation matrix of the park area state to be evaluated, for function, is the matrix product, Semantically encode the attention query park visual state matrix for the park area to be evaluated, is the point convolution operation, Semantically encode the park area to be evaluated, and query the park visual state feature map. For point-by-position multiplication, It is a multi-modal joint coding feature map of the area status of the park to be evaluated.

[0034] In the embodiment of the present application, the evaluation result generation module 170 is used to obtain the evaluation result based on the multimodal joint coding feature of the state of the park area to be evaluated, and the evaluation result is used to represent the risk level label of the park area to be evaluated. Specifically, in the embodiment of the present application, the evaluation result generation module is used to: input the multimodal joint coding feature map of the state of the park area to be evaluated into the risk assessor based on the classifier to obtain the evaluation result, and the evaluation result is used to represent the risk level label of the park area to be evaluated. That is, the multimodal joint coding feature of the state of the park area to be evaluated obtained by cross-modal fusion of the semantic coding feature vector of the state of the park area to be evaluated and the visual state enhancement feature map of the park is used for classification processing, so as to automatically evaluate the risk level of the park area to be evaluated. In this way, through cameras and other Internet of Things devices, the system can collect visual state images and environmental data in the park in real time, provide more comprehensive security information, so as to timely discover and respond to security incidents. And through deep learning technology, details such as small-scale environmental changes and minor security violations in the image can be identified, thereby improving the intelligence of park security management.

[0035] In particular, in a preferred example, when the semantic coding feature vector of the state of the park area to be evaluated and the enhanced feature map of the park visual state respectively express the textual semantic features of the textual description of the state of the park area to be evaluated and the channel dimension cross-distributed self-enhanced image semantic features of the visual state image of the park area to be evaluated, when heterogeneous data-driven multimodal Transformer joint encoding is performed on them, the obtained multimodal joint coding feature map of the state of the park area to be evaluated will also cause the overall distribution regression mode to shift due to the difference in heterogeneous data feature constraints on the channel dimension, that is, it will cause the overflow of the evaluation result obtained by inputting the multimodal joint coding feature map of the state of the park area to be evaluated into the classifier-based risk assessor, affecting its accuracy.

[0036] Based on this, when the multimodal joint coding feature map of the park area state to be evaluated is input into the classifier-based risk assessor to obtain an evaluation result, the multimodal joint coding feature map of the park area state to be evaluated is optimized, and the optimization process includes the steps of: determining the multimodal joint coding feature map of the park area state to be evaluated input into the classifier-based risk assessor to obtain the same The risk level labels of the park areas to be assessed correspond to risk level label probability values, and calculate the The square root of the sum of the squares of the probability values ​​of the risk level labels is taken to obtain the probability value of the level distribution; the multimodal joint coding feature map of the state of the park area to be evaluated is passed through a probabilistic activation function to obtain a probabilistic multimodal joint coding feature map of the state of the park area to be evaluated; the probabilistic multimodal joint coding feature map of the state of the park area to be evaluated is subtracted from the unit feature map, and the power function with the level distribution probability value as the exponent is calculated using the feature map of the point subtraction result as the base; the power function feature map is point-multiplied with the probabilistic multimodal joint coding feature map of the state of the park area to be evaluated to obtain the park area to be evaluated. domain state multimodal joint coding intermediate feature map; taking the absolute value of each eigenvalue of the probabilistic feature map of the multimodal joint coding of the park area state to be evaluated as the base, calculating the power function with the inverse of the level distribution probability value as the exponent, and summing all the eigenvalues ​​to obtain the bias eigenvalue of the multimodal joint coding of the park area state to be evaluated; multiplying the bias eigenvalue of the multimodal joint coding of the park area state to be evaluated by the weight serving as a hyperparameter, and performing point addition with the intermediate feature map of the multimodal joint coding of the park area state to be evaluated to obtain the optimized multimodal joint coding feature map of the park area state to be evaluated.

[0037] Wherein, the probabilistic multi-modal joint coding feature map of the park area state to be evaluated is recorded as , then the optimized multi-modal joint coding feature map of the park area state to be evaluated is It is expressed as: ;in, is the probabilistic multi-modal joint coding feature map of the park area state to be evaluated, is a unit eigenmap with all eigenvalues ​​being one, and , is the probabilistic multi-modal joint coding feature map of the park area state to be evaluated The eigenvalues ​​at each position of for Risk level label probability value, It is the point multiplication by position. It is subtracted by position point. is the grade distribution probability value, is a power function that calculates the feature values ​​of each position in the feature map with the level distribution probability value as an exponent, It is added by location point. is the weight as a hyperparameter, It is the optimized multi-modal joint coding feature map of the park area state to be evaluated.

[0038] That is, for the multimodal joint coding feature map of the state of the park area to be evaluated, based on the convergence limit of the probability distribution of the power series in the form of Cauchy-Hadamard, the feature set of the multimodal joint coding feature map of the state of the park area to be evaluated is represented by a power series based on the convergence probability, and a set convergence bounded bias based on each eigenvalue of the multimodal joint coding feature map of the state of the park area to be evaluated as a convergence radius is added, so as to avoid the invalid overflow of the classification result caused by the deviation of the distribution mode of the multimodal joint coding feature map of the state of the park area to be evaluated due to the gradient propagation of the model parameters based on the convergence probability in the regression process of the risk assessor based on the classifier under the unified probability convergence limit standard, thereby improving the accuracy of the assessment result obtained by the risk assessor based on the classifier. In this way, through cameras and other IoT devices, the system can collect visual state images and environmental data in the park in real time, provide more comprehensive security information, so as to timely discover and respond to security incidents. In addition, through deep learning technology, it is possible to identify details such as small-scale environmental changes and minor safety violations in images, thereby improving the intelligence of park security management.

[0039] In summary, the IoT-based smart park security management system 100 according to the embodiment of the present application is explained, which uses a data analysis algorithm based on deep learning to extract and enhance features of the visual state image of the park area to be evaluated, and semantically encodes the state text description of the park area to be evaluated, so as to automatically evaluate the risk level of the park area to be evaluated based on the multimodal fusion features between the visual state image features and the state text semantic features of the park area to be evaluated. In this way, by analyzing more comprehensive security information of the park area, it is helpful to timely discover and respond to security incidents, and at the same time, it can autonomously identify small-scale environmental changes and minor security violations in the park area, thereby improving the intelligent level of park security management.

[0040] As described above, the smart park security management system 100 based on the Internet of Things according to the embodiment of the present application can be implemented in various terminal devices, such as a server for smart park security management based on the Internet of Things. In one example, the smart park security management system 100 based on the Internet of Things according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the smart park security management system 100 based on the Internet of Things can be a software module in the operating system of the terminal device, or it can be an application developed for the terminal device; of course, the smart park security management system 100 based on the Internet of Things can also be one of the many hardware modules of the terminal device.

[0041] Alternatively, in another example, the IoT-based smart campus security management system 100 and the terminal device may also be separate devices, and the IoT-based smart campus security management system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0042] According to other embodiments of the present application, a campus safety management system is also provided, which is provided with three directions: a department supervision end, a campus end, and a lessee enterprise end. On the department supervision side, there are risk control modules, hidden danger management modules, enterprise ledger modules and system setting modules. The main functions of these modules include understanding the regional risk situation and the implementation of risk control, understanding the park-related information including the basic information of the park and the basic information of the enterprises in the park, entering the supervision and inspection data, reviewing and confirming the rectification of the park and the enterprise after completion, and querying the ledger-related information uploaded by the park and the enterprise; in the park section, there are responsibility implementation management modules and hidden danger management management modules. The main functions of these modules include the management of production safety management agreements, the management of safety management agency data, the management of safety coordination meeting data, the management of enterprise access and withdrawal system data, the management of lessee unit information data, the management of training and education content data, the management of plant structure acceptance material data, the management of plant area risk four-color map data, and the management of park inspection record data; on the lessee enterprise side, there are responsibility implementation modules and hidden danger management modules. The functions of these modules include the management of lessee enterprise training and education data, the management of lessee enterprise dangerous operation application data, the management of lessee enterprise fire-fighting equipment data, the management of lessee enterprise emergency fire drill data, the management of lessee enterprise hidden danger inspection data, and the management of lessee enterprise hidden danger rectification related data. The establishment of this park management system will help accelerate the construction of digital supervision modules for small and micro parks, and continuously improve the accuracy, effectiveness and timeliness of supervision; at the same time, it will help encourage and guide enterprises to continuously promote industrial digitalization and transformation of development momentum; and this park management system can also be connected with the department’s safety supervision platform to achieve data sharing, complementary advantages, and form a good interactive situation for departmental and enterprise safety management.

[0043] Figure 5 FIG. 1 is a flow chart of a smart park security management method based on the Internet of Things according to an embodiment of the present application. Figure 5As shown, in the smart park security management method based on the Internet of Things, it includes: S110, obtaining a visual state image of the park area to be evaluated collected by a camera; S120, obtaining a text description of the state of the park area to be evaluated input by a user; S130, inputting the visual state image into the park visual feature extractor to obtain a park visual state feature map; S140, inputting the park visual state feature map into a feature space structure-maintaining self-attention enhancement module based on cross-channel intersection to obtain a park visual state enhancement feature map; S150, semantically encoding the state text description of the park area to be evaluated to obtain a semantic encoding feature vector of the state of the park area to be evaluated; S160, inputting the semantic encoding feature vector of the state of the park area to be evaluated and the park visual state enhancement feature map into a heterogeneous data-driven multimodal Transformer joint encoder to obtain a multimodal joint encoding feature map of the state of the park area to be evaluated as a multimodal joint encoding feature of the state of the park area to be evaluated; S170, based on the multimodal joint encoding feature of the state of the park area to be evaluated, obtaining an evaluation result, and the evaluation result is used to represent the risk level label of the park area to be evaluated.

[0044] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned smart park security management method based on the Internet of Things have been referred to above. Figures 1 to 4 The invention has been introduced in detail in the description of the IoT-based smart park security management system, and therefore, its repeated description will be omitted.

[0045] In summary, the IoT-based smart park security management method based on the embodiment of the present application is explained, which uses a data analysis algorithm based on deep learning to extract and enhance features of the visual state image of the park area to be evaluated, and semantically encodes the state text description of the park area to be evaluated, so as to automatically evaluate the risk level of the park area to be evaluated based on the multimodal fusion features between the visual state image features and the state text semantic features of the park area to be evaluated. In this way, by analyzing more comprehensive security information of the park area, it is helpful to discover and respond to security incidents in a timely manner, and at the same time, it can autonomously identify small-scale environmental changes and minor security violations in the park area, thereby improving the level of intelligence of park security management.

[0046] The above is only a preferred embodiment of the present application and does not constitute any form of limitation to the present application. Although the present application has been disclosed as a preferred embodiment as above, it is not intended to limit the present application. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of the technical contents disclosed above without departing from the scope of the technical solution of the present application. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application still falls within the scope of the technical solution of the present application.

Claims

1. A smart park security management system based on the Internet of Things, It is characterized in that include: A visual state image acquisition module for the park area to be evaluated, used to acquire a visual state image of the park area to be evaluated collected by a camera; A module for acquiring a text description of the state of the park area to be evaluated, used to obtain a text description of the state of the park area to be evaluated input by a user; a module for extracting a visual state feature of the park, used to input the visual state image into a park visual feature extractor to obtain a park visual state feature map; a module for enhancing the visual state feature of the park, used to input the park visual state feature map into a self-attention enhancement module based on a feature space structure maintained across channels to obtain a park visual state enhanced feature map; a module for encoding a text description of the state of the park area to be evaluated, used to semantically encode the text description of the state of the park area to be evaluated to obtain a semantically encoded feature vector of the state of the park area to be evaluated; a module for multimodal joint encoding of the park area to be evaluated, used to input the semantically encoded feature vector of the state of the park area to be evaluated and the enhanced feature map of the park visual state into a heterogeneous data-driven multimodal Transformer joint encoder to obtain a multimodal joint encoding feature map of the state of the park area to be evaluated as a multimodal joint encoding feature of the state of the park area to be evaluated; and a module for generating an evaluation result, used to obtain an evaluation result based on the multimodal joint encoding feature of the state of the park area to be evaluated, wherein the evaluation result is used to represent a risk level label of the park area to be evaluated; Among them, the park visual state feature enhancement module includes: a park visual state feature map normalization processing unit, which is used to perform layer normalization on the park visual state feature map to obtain a normalized park visual state feature map; a normalized park visual state feature point convolution processing unit, which is used to perform point convolution processing on the normalized park visual state feature map to obtain a park visual state channel context association representation feature map; a park visual state channel context association representation feature point convolution encoding unit, which is used to perform convolution encoding on the park visual state channel context association representation feature map to obtain a park visual state space context association representation feature map; a park visual state space context association representation feature backup unit, which is used to copy the park visual state space context association representation feature map to obtain a park visual state backup space context association representation feature map; a park visual state context association representation feature shape reshaping unit, which is used to reshape the park visual state channel context association representation feature map, the park visual state space context association representation feature map and the park visual state backup space context association representation feature map respectively. The park visual state cross-channel covariance calculation unit is used to calculate the park visual state cross-channel cross-covariance matrix between the park visual state channel context association representation feature matrix and the park visual state space context association representation feature matrix; the park visual state feature global interaction unit is used to activate the park visual state cross-channel cross-covariance matrix to obtain the park visual state feature global interaction attention matrix; the park visual state attention enhancement feature representation generation unit is used to calculate the product between the park visual state backup space context association representation feature matrix and the park visual state feature global interaction attention matrix to obtain the park visual state attention enhancement feature representation matrix; the park visual state enhancement feature generation unit is used to reshape the park visual state attention enhancement feature representation matrix to obtain the park visual state enhancement feature map; Wherein, the park visual state cross-channel covariance calculation unit is used to: calculate the matrix product between the park visual state channel context association representation feature matrix and the park visual state space context association representation feature matrix to obtain the park visual state channel-space context association representation feature matrix; calculate the positional division of the park visual state channel-space context association representation feature matrix and a predetermined hyperparameter to obtain the park visual state cross-channel cross covariance matrix; The park visual state feature global interaction unit is used to: use a Softmax function to activate the park visual state cross-channel cross covariance matrix to obtain the park visual state feature global interaction attention matrix; The park visual state feature enhancement module is used to: input the park visual state feature map into the feature space structure maintenance self-attention enhancement module based on cross-channel intersection, and process it with the following self-attention enhancement formula to obtain the park visual state enhancement feature map; wherein the self-attention enhancement formula is: ;in, is the visual state feature map of the park, Indicates the layer normalization operation on the feature map. is the normalized park visual state feature map, is the point convolution operation, is a context-related representation feature map of the park visual state channel, The convolution kernel is The convolution operation, is a context-related representation feature map of the park visual state space, For copy operation, A spatial context association representation feature map is provided for the park visual state backup, For the reshape operation, , and are respectively the park visual state channel context association representation feature matrix, the park visual state space context association representation feature matrix and the park visual state backup space context association representation feature matrix, To predetermine the hyperparameters, for function, is the global interactive attention matrix of the park visual state features, is matrix multiplication, Strengthening feature maps for the visual status of the park; Among them, the multimodal joint encoding module of the park area to be evaluated includes: a linear transformation shape reshaping unit of the park area state feature to be evaluated, which is used to perform linear transformation and feature shape reshaping on the semantic coding feature vector of the park area state to be evaluated to obtain the semantic coding modulation matrix of the park area state to be evaluated; a park visual state enhancement feature linear transformation unit, which is used to perform a first linear transformation and a second linear transformation on the park visual state enhancement feature map to obtain a first linear transformation park visual state enhancement feature map and a second linear transformation park visual state enhancement feature map; a linear transformation park visual state enhancement feature shape reshaping unit, which is used to perform feature shape reshaping on the first linear transformation park visual state enhancement feature map and the second linear transformation park visual state enhancement feature map to obtain a first linear transformation park visual state enhancement feature matrix and a second linear transformation park visual state enhancement feature matrix; a park area state attention calculation unit to be evaluated, which is used to use the first linear transformation park visual state enhancement feature matrix and the second linear transformation park visual state enhancement feature matrix as key matrices and value matrices and the park area state semantic coding modulation matrix to be evaluated as a query matrix, and input the query matrix, the key matrix and the value matrix into the Transformer structure to obtain The park area semantic coding attention query park visual state matrix to be evaluated; the park area semantic coding attention query park visual state feature generation unit to be evaluated, used for reshaping and point convolution encoding the park area semantic coding attention query park visual state matrix to be evaluated to obtain the park area semantic coding attention query park visual state feature map to be evaluated; the park area state multimodal joint coding feature generation unit to be evaluated, used for multiplying the park area semantic coding attention query park visual state feature map to be evaluated with the park visual state enhancement feature map by position point to obtain the park area state multimodal joint coding feature map to be evaluated; wherein the park area state attention calculation unit to be evaluated is used to: calculate the matrix product between the transposed matrix of the park area state semantic coding modulation matrix to be evaluated and the first linear transformation park visual state enhancement feature matrix, and then input it into the Softmax function to obtain the park area state semantic-visual feature association matrix to be evaluated; use the park area state semantic-visual feature association matrix to be evaluated as a weight matrix, calculate the matrix product between the weight matrix and the second linear transformation park visual state enhancement feature matrix to obtain the park area semantic coding attention query park visual state matrix to be evaluated; Among them, the linear transformation shape reshaping unit of the state feature of the park area to be evaluated is used to: perform matrix multiplication of the semantic coding feature vector of the state of the park area to be evaluated and the modulation matrix of the state of the park area to be evaluated, and then add them by position with the bias vector of the state of the park area to be evaluated to obtain the semantic coding modulation bias vector of the state of the park area to be evaluated; input the semantic coding modulation bias vector of the state of the park area to be evaluated into the Sigmoid function for activation processing to obtain the semantic coding modulation activation vector of the state of the park area to be evaluated; reshape the semantic coding modulation activation vector of the state of the park area to be evaluated to obtain the semantic coding modulation matrix of the state of the park area to be evaluated; Among them, the park visual state enhancement feature linear transformation unit is used to: multiply each feature matrix along the channel dimension in the park visual state enhancement feature map by position with the first park visual state modulation matrix, and then add them by position with the first park visual state bias matrix to obtain the first park visual state bias feature map; input the first park visual state bias feature map into the Sigmoid function for activation processing to obtain the first linear transformation park visual state enhancement feature map; multiply each feature matrix along the channel dimension in the park visual state enhancement feature map by position with the second park visual state modulation matrix, and then add them by position with the second park visual state bias matrix to obtain the second park visual state bias feature map; input the second park visual state bias feature map into the Sigmoid function for activation processing to obtain the second linear transformation park visual state enhancement feature map; The multimodal joint encoding module of the park area to be evaluated is used to: input the semantic encoding feature vector of the park area state to be evaluated and the enhanced feature map of the park visual state into the multimodal Transformer joint encoder driven by heterogeneous data, and process them with the following joint encoding formula to obtain the multimodal joint encoding feature map of the park area state to be evaluated; wherein the joint encoding formula is: ;in, is the semantic encoding feature vector of the state of the park area to be evaluated, is the regional state modulation matrix of the park to be evaluated, is the state bias vector of the park area to be evaluated, is the Sigmoid function, The semantic coding modulation activation vector of the state of the park area to be evaluated is For the reshape operation, is the semantic coding modulation matrix of the park area state to be evaluated, is the visual state enhancement feature map of the park, and are respectively the first park visual state modulation matrix and the second park visual state modulation matrix, and are respectively the first park visual state bias matrix and the second park visual state bias matrix, and are respectively the first linear transformation park visual state enhancement feature map and the second linear transformation park visual state enhancement feature map, and are respectively the first linear transformation park visual state enhancement feature matrix and the second linear transformation park visual state enhancement feature matrix, is the transposed matrix of the semantic coding modulation matrix of the park area state to be evaluated, for function, is the matrix product, Semantically encode the attention query park visual state matrix for the park area to be evaluated, is the point convolution operation, Semantically encode the park area to be evaluated, and query the park visual state feature map. For point-by-position multiplication, It is a multi-modal joint coding feature map of the area status of the park to be evaluated.

2. According to the IoT-based smart park security management system of claim 1, It is characterized in that The park visual state feature extraction module is used to: input the visual state image into a park visual feature extractor based on a hole convolutional neural network model to obtain the park visual state feature map.

3. According to the IoT-based smart park security management system of claim 2, It is characterized in that The assessment result generation module is used to: input the multimodal joint coding feature map of the park area state to be assessed into a classifier-based risk assessor to obtain the assessment result, and the assessment result is used to represent the risk level label of the park area to be assessed.

4. A smart park security management method based on the Internet of Things, using the smart park security management system based on the Internet of Things according to claim 1, It is characterized in that include: Obtaining a visual status image of the park area to be evaluated collected by a camera; Obtaining a text description of the status of the park area to be evaluated input by a user; The visual state image is input into the park visual feature extractor to obtain a park visual state feature map; the park visual state feature map is input into the feature space structure-maintaining self-attention enhancement module based on cross-channel crossing to obtain a park visual state enhancement feature map; the state text description of the park area to be evaluated is semantically encoded to obtain a semantic encoding feature vector of the park area state to be evaluated; the semantic encoding feature vector of the park area state to be evaluated and the park visual state enhancement feature map are input into a heterogeneous data-driven multimodal Transformer joint encoder to obtain a multimodal joint encoding feature map of the park area state to be evaluated as a multimodal joint encoding feature of the park area state to be evaluated; based on the multimodal joint encoding feature of the park area state to be evaluated, an evaluation result is obtained, and the evaluation result is used to represent the risk level label of the park area to be evaluated.